The Artima Developer Community
Sponsored Link

News & Ideas Forum (Closed for new topic posts)
Tomcat and Apache Living Together, Mass Hysteria

16 replies on 2 pages. Most recent reply: Nov 13, 2002 4:41 PM by Bill Venners

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 16 replies on 2 pages [ 1 2 | » ]
Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 5, 2002 4:39 PM
Reply to this message Reply
Advertisement
Artima.com runs on Linux, PostgreSQL, Tomcat, and Apache. Tomcat seems to crash about once a month, which is frankly too unreliable for this site. I am curious if others who use this combination of open source products experience this level of reliability, and what people recommend for a reliable servlet engine.

It is possible that the problem is the JVM , the Linux version, or Tomcat version I'm using. Here are all the version numbers:

java version "1.3.1_02"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_02-b02)
Java HotSpot(TM) Client VM (build 1.3.1_02-b02, mixed mode)

Tomcat Version 3.3a
Apache Version 1.3.19 (Will be upgrading soon)
Linux Kernel Version 2.2

I certainly don't mind paying for a commercial app server, though all I use are servlets so all I need is a servlet engine. I used Tomcat because my impression was that it is very robust. I used 3.3 because at the time the 4.x line was new and, I was told, not very stable. At this point 4.x may be the way to go. What is your experience?

Also, what strategies do people use to automatically detect a problem and fix it, perhaps by restarting the servers.

Thanks.


Gary Lawrence Murphy

Posts: 4
Nickname: teledyn
Registered: Nov, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 5, 2002 6:27 PM
Reply to this message Reply
You don't give enough diagnostics to go on here: Any of 10,000 things could be wrong, everything from whether you have a memory (heap or stack) leak to whether your JDBC bridge is working correctly.

I have run sites using this combination of technology under extreme loads (millions of page-reads a day) over many months without problems, but I have also had this same combination blow in a day, usually because of database issues. Check that your transactions are not the problem (do you vacuum?) Since I don't know the volatility of your data, I just can't say more.

Similarly your Linux 2.2 is pretty old; there were memory issues with that kernel, and compatibility issues with certain hardware. 2.4 might do better, maybe not. Some of my early JSP/Tomcat apps, running on one of the first releases of tomcat, were very stable under 2.2. Without knowing your code, it's impossible to say, but I can say that 2.2 had some VM issues, and it had some SMP issues. Ditto for your Tomcat 3.3a ... if you had better diagnostics of what is actually causing your app to fail, you might find some corresponding issues in the bug reports for the components.

Also you can have problems in allocating and reclaiming objects: Do you pool objects or create them? What does the JVM profiling say? Could you be keeling over from null pointers or software proceeding along even when it did not actually allocate some object? Do you check result codes?
I had one high-traffic portal app where peak-hour activity was such the gc never got a chance to run; we eventually had to force gc calls (which slowed the system, but slowed slightly is far better than fail completely)

I hope you begin to grasp the complexity of your question: Those of us who do this work make a pretty decent hourly wage for doing exactly what you troll for free ;)

But to answer your basic question: Yes, Tomcat+Apache+PostgreSQL are used in production systems all over the world and they are, themselves, really very reliable and trustworthy. /But/ do keep in mind that Tomcat is a *reference* implementation, so it is not the fastest nor the most memory efficient.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 5, 2002 10:15 PM
Reply to this message Reply
> You don't give enough diagnostics to go on here: Any of
> 10,000 things could be wrong, everything from whether you
> have a memory (heap or stack) leak to whether your JDBC
> bridge is working correctly.
>
At least we've narrowed it down to 10,000 possibilities.

> I have run sites using this combination of technology
> under extreme loads (millions of page-reads a day) over
> many months without problems, but I have also had this
> same combination blow in a day, usually because of
> database issues. Check that your transactions are not the
> problem (do you vacuum?) Since I don't know the volatility
> of your data, I just can't say more.
>
OK. That's the kind of real-world experience I was interested in hearing about. I suspected I'm doing something wrong somewhere, because my problems usually turn out to be my fault. But since I didn't pay for any of these products I can't call up a vendor and ask for advice.

I have been vacuuming nightly (my database, not my house), unless cron wasn't able to find vacuumdb. I just looked at my crontab entry and realized I had forgotten to give the full pathname of vacuumdb. So maybe I wasn't vacuuming. I didn't see any email message complaints from cron all this time, though. My friend Peter Kellner recommended I use Postgres, and told me to vacuum. He'd fall off his bicycle if it turned out I wasn't.

> Similarly your Linux 2.2 is pretty old; there were memory
> issues with that kernel, and compatibility issues with
> certain hardware. 2.4 might do better, maybe not. Some of
> my early JSP/Tomcat apps, running on one of the first
> releases of tomcat, were very stable under 2.2. Without
> knowing your code, it's impossible to say, but I can say
> that 2.2 had some VM issues, and it had some SMP issues.
> Ditto for your Tomcat 3.3a ... if you had better
> diagnostics of what is actually causing your app to fail,
> you might find some corresponding issues in the bug
> reports for the components.
>
I have reason to suspect the VM, because one of the other dumb things I do is repeatedly run a java app to build each page on the website from a makefile. It has been on my list forever to switch that to ANT, but I haven't done it yet. Thus, I create a new java process to pump out each page on the site when I do a full build, and every couple thousand of those java processes crashes. I don't remember the output, but I'll pay attention next time. I'll start a build and within a couple hours I should have an example of the crash, which output I will post.

I chose 3.3a, because several people told me it was very stable and that the then new 4.x line was not. If not 3.3a, then what?

> if you had better
> diagnostics of what is actually causing your app to fail,
> you might find some corresponding issues in the bug
> reports for the components.
>
I'm not sure how to get diagnostics of my problem, exactly.

> Also you can have problems in allocating and reclaiming
> objects: Do you pool objects or create them? What does
> the JVM profiling say? Could you be keeling over from
> null pointers or software proceeding along even when it
> did not actually allocate some object? Do you check
> result codes?
>
I use Jive forums, which does database connection pooling. I heavily hacked the Jive JSPs and made a few minor enhancements and bug fixes to the Jive API, but most of it I did not write and am not familiar with.

> I had one high-traffic portal app where peak-hour activity
> was such the gc never got a chance to run; we eventually
> had to force gc calls (which slowed the system, but slowed
> slightly is far better than fail completely)
>
I don't have that much traffic. I get fewer than 2 million hits a month, and most of that is going to Apache, not Tomcat.

> I hope you begin to grasp the complexity of your question:
> Those of us who do this work make a pretty decent hourly
> wage for doing exactly what you troll for free ;)
>
My problem is I don't know how to analyze the problem. It doesn't happen consistently, and when it does happen I don't know how to tell what happened. What I'm really trolling for is if there's something obvious about the VM I'm using or version of something that has a reputation of being unrealiable. Short of that, I guess I need some advice as to how to analyze the problem.

> But to answer your basic question: Yes,
> Tomcat+Apache+PostgreSQL are used in production systems
> all over the world and they are, themselves, really very
> reliable and trustworthy. /But/ do keep in mind that
> Tomcat is a *reference* implementation, so it is not the
> fastest nor the most memory efficient.
>
I currently no complaints about performance. As I said, most of the load is being taken up by Apache, which has only started acting strange twice in the past year, both times right after Tomcat went away. That I can live with. Linux and Postgres have never crashed once. It's quite amazing, really. But it shows that at least some open source apps are incredibly robust and stable. Apache seems fine except very occasionally it doesn't gracefully handle when Tomcat crashes. My problem is Tomcat.

Gary Lawrence Murphy

Posts: 4
Nickname: teledyn
Registered: Nov, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 5, 2002 10:46 PM
Reply to this message Reply
I share your pain --- the Tomcat/Apache link is one I'm not terribly enamoured with as a diagnostic tool. The best I can offer is to fold log4j into your application and use it extensively to map everything that might be going on in your program.

The direct method of diagnostic is to hook the java debugger to the runtime JVM, but I find in practice this gives too much information at too low of a detail; you want to know semantically where you've gone wrong, not concretely at the JIT level.

If you use Log4j, you can emit diagnostics on the heap size and log when you enter and exit important parts of your project; that it happens sporadically is also bad news unless you can correlate this to specific apache access logs. I hope you also keep your tomcat stderr logs as those may also have some value, but really the best you can do to narrow it down is to trap all exceptions and dump what you can of them to a log file through log4j. You can also use unix profiling tools to try and correlate activity to the time of a crash and from there deduce possible causes.

when its sporadic, it's hard to run alternative scenarious, but if you don't mind the bother, you might try to run a pure-Tomcat service. More or less, we're always left with Conan-Doyle's advice, "Eliminate what cannot possibly be true and whatever is left, like it or not, is the answer"

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 6, 2002 12:02 AM
Reply to this message Reply
> I share your pain --- the Tomcat/Apache link is one I'm
> not terribly enamoured with as a diagnostic tool. The
> best I can offer is to fold log4j into your application
> and use it extensively to map everything that might be
> going on in your program.
>
Ouch. That will be touch because I didn't really write the app, I bought it, then hacked at it for a couple months.

> The direct method of diagnostic is to hook the java
> debugger to the runtime JVM, but I find in practice this
> gives too much information at too low of a detail; you
> want to know semantically where you've gone wrong, not
> concretely at the JIT level.
>
> If you use Log4j, you can emit diagnostics on the heap
> size and log when you enter and exit important parts of
> your project; that it happens sporadically is also bad
> news unless you can correlate this to specific apache
> access logs. I hope you also keep your tomcat stderr logs
> as those may also have some value, but really the best you
> can do to narrow it down is to trap all exceptions and
> dump what you can of them to a log file through log4j.
> You can also use unix profiling tools to try and
> correlate activity to the time of a crash and from there
> deduce possible causes.
>
I am not keeping tomcat log files. You're right that's a good thing do to. I'll start.

> when its sporadic, it's hard to run alternative
> scenarious, but if you don't mind the bother, you might
> try to run a pure-Tomcat service. More or less, we're
> always left with Conan-Doyle's advice, "Eliminate what
> cannot possibly be true and whatever is left, like it or
> not, is the answer"

You know, Jason Hunter told me he just uses Tomcat to serve up even static .html pages at servlets.com. But of course, it's servlets.com. I like Apache for static content, and I use server-side includes in most of the static pages, so I expect I'll keep on using Apache and some external servlet container in combination.

My prime suspect right now is my JVM. I should figure out why the VM crashes when I do a site build. If I can solve that it might solve the Tomcat problem. If not, it at least eliminates one possibility. And my builds will finish to boot.

Thanks for your comments.

Gary Lawrence Murphy

Posts: 4
Nickname: teledyn
Registered: Nov, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 6, 2002 6:47 AM
Reply to this message Reply
>> I share your pain --- the Tomcat/Apache link is one I'm
>> not terribly enamoured with as a diagnostic tool. The
>> best I can offer is to fold log4j into your application
>> and use it extensively to map everything that might be
>> going on in your program.

> Ouch. That will be touch because I didn't really write the
> app, I bought it, then hacked at it for a couple months.

Well ... let that be a lesson to you: If it's not at least open source, it's unwise to bet your business on it :)

At least, not _unless_ the owners are shareholders in your company, or they and you are owned by the same people. When you bet critical systems on closed software, you are effectively abdicating the controlling interest in that business process, handing the responsibility for it to total strangers who could care less about your business problems.

Put another way, would you hand all your accounting over to someone who wouldn't let you look at your own books?

Carfield Yim

Posts: 32
Nickname: carfield
Registered: Sep, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 6, 2002 8:09 AM
Reply to this message Reply
I never get this problem with tomcat for 3.3... However, I am using IBM JDK 1.3 when I running tomcat in Linux, you mean consider this: http://www-106.ibm.com/developerworks/java/jdk/

Now I am using Sun JDK 1.4 with JBoss + Jetty, never have any problem since early october, so you can consider Jetty.

Gary Lawrence Murphy

Posts: 4
Nickname: teledyn
Registered: Nov, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 6, 2002 8:33 AM
Reply to this message Reply
I've also been impressed with Jetty --- for most of my recent applications, Tomcat is just too much noise and while the integration with Apache is better with Tomcat, mod_rewrite can use used to do much the same thing with any servlet container.

I actually get my best results using Apache, Jetty and the Linux kHTTP kernel-level server for static files, but I have not (yet) had the opportunity to test this combination in a real production environment.

Joe Cheng

Posts: 65
Nickname: jcheng
Registered: Oct, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 6, 2002 9:23 AM
Reply to this message Reply
Hi Bill,

Do you not have any log output from the time of the crashes?

Could the JVM be running out of heap?

And I'm interested to hear what kind of Java app you're using to generate your static pages...?

Thanks-
jmc

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 6, 2002 10:22 AM
Reply to this message Reply
> I never get this problem with tomcat for 3.3... However, I
> am using IBM JDK 1.3 when I running tomcat in Linux, you
> mean consider this:
> http://www-106.ibm.com/developerworks/java/jdk/
>
That's a good idea. Since my current JVM is crashing when running my build, I should try using a different VM. I've heard good things about the IBM JVM.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 6, 2002 10:39 AM
Reply to this message Reply
> Hi Bill,
>
> Do you not have any log output from the time of the
> crashes?
>
The log files in tomcat's logs directory don't say much at all, and they have no hints as to any problem. What I need to do is capture what is currently being printed to the standard output to a file. Exception stack traces, for example, are currently going to the standard output, which is going nowhere.

> Could the JVM be running out of heap?
>
Good question.

> And I'm interested to hear what kind of Java app you're
> using to generate your static pages...?
>
I use home grown tools. Each page is stored in CVS as a raw HTML file and a small XML file that defines things like title, subtitle, etc.. The tools take the info in the XML file and the raw HTML file and produce a pretty HTML (or JSP) file that goes up on the site.

Since a great deal of the material on this site are articles and other more static content, I wanted to have this make process rather than have each page assembled again and again as it goes out the door. I have python scripts that generate the makefiles, and I need to change those to ANT to speed up the build. I just haven't switched to ANT yet for the site build, which is why I notice the JVM crashing when I do the build. Make runs a new JVM to build each page, and every few thousand runs the JVM crashes.



Posts: 20
Nickname: johnnyo
Registered: Oct, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 9, 2002 5:09 AM
Reply to this message Reply
My ISP has set up a "monitor": every ten minutes, a GET of
a simple page is performed. If there is no response after 1:00, then a restart script is invoked. If I didn't have this, my blood pressure would be substantially higher...

An extra benefit of this simple tool is that it forms a simple but effective performance monitor as well. The logged time of the monitor's GET requests is usually at 04:10:00, 04:20:01, 04:30:01, etc., so you can grep the access log for monitor requests having usually long times (I use 4 seconds or above). Voila - simple tool to monitor your machine's overall performance.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 9, 2002 11:47 PM
Reply to this message Reply
> My ISP has set up a "monitor": every ten minutes, a GET of
>
> a simple page is performed. If there is no response after
> 1:00, then a restart script is invoked. If I didn't have
> this, my blood pressure would be substantially higher...
>
Thanks John. That's another great suggestion. I believe my ISP actually does the same thing, but only for Apache, not Tomcat. I'll talk to them about this.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 9, 2002 11:59 PM
Reply to this message Reply
Frank Sommers sent me an email last night containing a paragraph he found somewhere in the Tomcat documentation:

------------------------------- Linux and Sun JDK 1.2.x - 1.3.x:
Virtual machine crashes can be experienced when using certain combinations of kernel / glibc under Linux with Sun Hotspot 1.2 to 1.3. The crashes were reported to occur mostly on startup. Sun JDK 1.4 does not exhibit the problems, and neither does IBM JDK for Linux. The problems can be fixed by reducing the default stack size. At bash shell, do "ulimit -s 2048"; use "limit stacksize 2048" for tcsh. GLIBC 2.2 / Linux 2.4 users should also define an environment variable: export LD_ASSUME_KERNEL=2.2.5


That sounds promising. I'm off to Japan tomorrow, but will try swapping JVMs as soon as possible.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Tomcat and Apache Living Together, Mass Hysteria Posted: Nov 11, 2002 8:59 PM
Reply to this message Reply
Tomcat crashed again last night, as I slept in Tokyo. I didn't want to replace the VM before my trip, because I wanted to be around the net for awhile after the operation, but I'm in a hotel that has a free LAN connection. So I just replaced Sun's 1.3 VM that has the known crashing problem on Linux with IBM's 1.3 VM that allegedly does not have the problem.

This is my first post being handled by the IBM JVM. It seemed like it took longer for Tomcat to get going by 5 to 10 seconds on the IBM VM, but now that it's going it seems to be working. Hopefully this will solve the problem of Tomcat crashing. Time will tell.

Flat View: This topic has 16 replies on 2 pages [ 1  2 | » ]
Topic: Teaching Java the Extreme Way Previous Topic   Next Topic Topic: Centralized Security Server for Apache/Tomcat

Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2019 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use