This post originated from an RSS feed registered with Java Buzz
by Mathias Bogaert.
Original Post: CodeCache is full. Compiler has been disabled.
Feed Title: Scuttlebutt
Feed URL: http://feeds.feedburner.com/AtlassianDeveloperBlog
Feed Description: tech gossip by mathias
When I was timing how long it takes JIRA 5.0.1 to reach a steady state for GC & code compilation with JDK 1.6.0_26 for a GC tuning guide; I noticed a log message that I’d never seen before: 12Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled. Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize= This cache is a memory area separate from the JVM heap that contains all the JVM bytecode for a method compiled down to native code, each called an nmethod1. This is where the JIT compiled methods are kept. On server classed 64 bit VMs, the reserved cache size is 48 MB (Used to be 1 GB, see Bug 6245770). Once the compiler is switched off due to the Code Cache being full, it does not switch back on. Existing nmethods will continue to be used until they are flushed from the cache. As this was the first time I’d seen this interesting message, I decided to do some digging as I hadn’t encountered the VM code cache with previous research in to HotSpot garbage collection. Impact of the Code Cache being full is that JIT compilation is now off for the remainder of the JVM’s life. We don’t have any statistics as to the impact of this on long running performance; the impression that I have is that CPU utilisation & request service time would increase over long running VMs for code that could have been optimised. For a system that is over-committed on CPU resources the application throughput could decrease noticeably. The suspicion that I have here is that if I give the Code Cache some fatpants by increasing -XX:ReservedCodeCacheSize=, I’d just be delaying the problem. So I dug in to the HotSpot code to see if there was a way that the JVM can manage this for me. Turns out, there is; in the HotSpot source for jdk6 the following flag is available for product releases, -XX:+UseCodeCacheFlushing: 1product(bool, UseCodeCacheFlushing, false, "Attempt to clean the code cache before shutting off compiler") It is not on by default. So I turned this on, and I no longer observed the log message about the code cache being exhausted with the particular environment being used. I wanted to confirm that this was working as advertised, so I read through the HotSpot code to determine how the Code Cache flushes nmethods during default operation and when -XX:+UseCodeCacheFlushing is used. There is a way to infer when nmethods are flushed by interpreting the log messages printed with -XX:+PrintCompilation, the lines that are important are the ones where an nmethod has been made a zombie: 1552287 4320 made zombie org.ofbiz.core.entity.EntityExpr::makeWhereString (557 bytes) With -XX:+UseCodeCacheFlushing, what you would be looking for is a large group of these messages in quick succession, as it generally indicates the processing of older nmethods to be flushed from the code cache. As opposed to normal flushing due to class unloading or deoptimisation. This doesn’t give [...]
Previous Topic
