Today's "oops, I discovered a self inflicted scaling problem" lesson goes to me, again. In the startup script for the image running this, I set the soft and hard memory limits to a range - which is fine, except for one small problem - I had completely forgotten about an earlier directive to grow memory by a specific amount. Well, it turns out that growing memory by as much as I set it to pushed the image right up to the soft memory limit - which placed the image into a low memory situation once it ran for awhile. That's why the server has been slow or inaccessible the last couple of days - I addressed that problem just before lunch.

Update: The final fix was to modify the settings for this:

ObjectMemory sizesAtStartup

All the values there were set to the defaults (which, for the version of the server I'm running, were 1.0). That was way too low - in particular, new space (where new objects are born) was too small. The way things were working, objects were tenuring too quickly, and ending up in old space (where they never die until a GC is done - which is much more rare until the image switches from "grow memory" to "reclaim memory").

In effect, I was filling memory up with trash until I hit a level that made things slow, and only then did the image start to respond. By setting new space larger, objects have a chance to die off without being tenured - and the image doesn't push CPU as often collecting garbage (and thus spends more time answering actual user requests).