This post originated from an RSS feed registered with Java Buzz
by Marc Logemann.
Original Post: IBM JDK thread analysis on Linux PPC
Feed Title: Logemann Blog
Feed URL: http://www.logemann.org/blojsom/blog/default/?flavor=rss2
Feed Description: Marc's thoughts on Java and more
Imagine you are in problems. To be specific in java runtime problems. Just imagine that your customer is slowly getting angry because your application is unresponsive and tends to crash here and there. Normally one can solve such issues by asking the customer on what actions things get worse. You are in serious trouble when it is kind of random, so that looking at source code is not enough.
On real out-of-memory crashes, you should definitely first check the heap dump and analyse what objects are in the heap on crash time. I blogged about that before. But what if your program hangs for some seconds and is just unresponsive? First there could be gazillion things be wrong on the server so your main task is to make sure that its really your application that hangs and not the VM or the whole server. So in such a situation you should definitely save a "top" output. This way you can see the server load and the load of all processes on the system.
After that you should signal the VM that it should generate a threaddump called javacore on IBM java platforms. Simply do a "kill -3 PID" on your linux system and replace PID with the current process ID of the VM. You can get it easily by issueing a "ps xa | grep tomcat" (or catalina, depending on your installation of tomcat and the path you chosen...).
To analyze the javacore there is a good free tool from IBM called IBM Thread and Monitor Dump Analyzer. When feeding this tool with your first javacore, you should see something like this:
This is of course the second screen after you selected the opened javacore and clicking Analysis->Thread Details. It shows all active threads of the VM. Some of them are used by the VM itself. On this screenshot you see a bunch of tomcat threads waiting on requests and various VM threads. You see that its a total of 123 Threads. If you click on a thread you see its current stacktrace on the right. You also have different colors for the states of the threads. Its important to know when a thread is suspected to hang or not. The stack mostly helps to identify what the thread is doing.
More interessing is comparing javacore against each other. So you should obtain more than one dump. It would be perfect to create one before the system is in trouble state and one after the system is responsive again. No worry, you can do this in "production" without really killing the process. With the tool you can compare two threads against each other and you will see something like this:
The challenging part is again to interpret this stuff correctly. Even there is a lot of red stuff here, it doesnt necesarily mean that you are in danger. The program basically checks if a thread has still the same status and the same stack and marks it red. But you cant blame a waiting tomcat request handler thread for hanging when it still waits for requests. So some cuaution in this area. But it should be quite easy to track the various stack traces and see when there are non-expected thread lags.
Javacore aka threaddumps wont help you in any situation but its crucial to know what tool you can use and what files are there to have a look at. Besides this, you see a lot of interesting things about your application which opens a hole new view on your application and possible performance bottlenecks. Like i said, its one of a lot of things you can do when apps behave weird somehow.
Previous Topic
