Article Discussion
Database Throughput
Summary: In this interview with Artima, Gemstone chief architect Jags Ramnarayan discusses the limits of database throughput, and what developers can do when their applications reach those limits.
5 posts.
The ability to add new comments in this discussion is temporarily disabled.
Most recent reply: July 13, 2007 8:41 AM by Bill
    Frank
     
    Posts: 135 / Nickname: fsommers / Registered: January 19, 2002 7:24 AM
    Database Throughput
    July 3, 2007 1:30 PM      
    In this interview with Artima, Gemstone chief architect Jags Ramnarayan discusses the limits of database throughput, and what developers can do when their applications reach those limits:

    http://www.artima.com/lejava/articles/javaone_2007_jags_ramnarayan.html

    Based on your experience with the databases you've worked with, to what degree would you say typical databases really scale in terms of transaction (query and update) throughput? At what level of transaction volume would you think of clustering as a way to increase the system's throughput?
    • A P
       
      Posts: 1 / Nickname: apc / Registered: June 25, 2004 3:11 AM
      Re: Database Throughput
      July 6, 2007 6:22 AM      
      I was attempting to read the Jags Ramnarayan interview, but the given url just links to a blank page (well, apart from the ads and all the usual folderol around the edges). This seems to be true for the other articles too. Is there a problem with the Artima site?

      Cheers, APC
      • Bill
         
        Posts: 409 / Nickname: bv / Registered: January 17, 2002 4:28 PM
        Re: Database Throughput
        July 6, 2007 3:07 PM      
        > I was attempting to read the Jags Ramnarayan interview,
        > but the given url just links to a blank page (well, apart
        > from the ads and all the usual folderol around the edges).
        > This seems to be true for the other articles too. Is
        > s there a problem with the Artima site?
        >
        > Cheers, APC

        Sorry, I accidentally configured Apache with an overzealous redirect. It has been fixed. I don't like broken URLs, so I was trying to put in a redirect for a URL that people would rarely go to, and ended up temporarily breaking URLs that people are trying to get to.
    • Bill
       
      Posts: 28 / Nickname: billpyne / Registered: January 29, 2007 4:12 AM
      Re: Database Throughput
      July 5, 2007 8:20 AM      
      > Based on your experience with the databases you've worked
      > with, to what degree would you say typical databases
      > really scale in terms of transaction (query and update)
      > throughput? At what level of transaction volume would you
      > think of clustering as a way to increase the system's
      > throughput?

      In addition to Frank's questions, I'd like to know how many applications require this kind of throughput? My belief is that the number of applications requiring this throughput is at the far right of the bell curve.
      • Cameron
         
        Posts: 26 / Nickname: cpurdy / Registered: December 23, 2004 0:16 AM
        Re: Database Throughput
        July 10, 2007 4:52 AM      
        > In addition to Frank's questions, I'd like to know how
        > many applications require this kind of throughput? My
        > belief is that the number of applications requiring this
        > throughput is at the far right of the bell curve.

        We have quite a few customers with applications that have the "5000 events per second" requirement, and a small number over the 100,000 mark. Some of these are "peak load" numbers (e.g. markets that see bursts of activity at open and close) and some of these are sustained (non-stop load of thousands of transactions / operations / events / etc. per second).

        The reasons why a database may or may not be suitable for a task isn't just raw speed. Sometimes it has to do with the isolation requirements, or the serial versus parallel nature of the workload, or the ability to partition the load, or the durability requirements, etc. A database, for example, may impose too high a cost for serial transactions against the same small set of data, due to the isolation+durability requirement. (Serial + isolated + disk durable transactions create a maximum throughput limiter tied directly to the transaction latency, and if the transactions are driven out-of-process, that latency becomes even more significant.)

        Peace,

        Cameron Purdy
        http://www.oracle.com/technology/products/coherence/index.html
        • Bill
           
          Posts: 28 / Nickname: billpyne / Registered: January 29, 2007 4:12 AM
          Re: Database Throughput
          July 13, 2007 8:41 AM      
          > We have quite a few customers with applications that have
          > the "5000 events per second" requirement, and a small
          > number over the 100,000 mark. Some of these are "peak
          > load" numbers (e.g. markets that see bursts of activity at
          > open and close) and some of these are sustained (non-stop
          > load of thousands of transactions / operations / events /
          > etc. per second).

          Thank you for the response. I haven't run into volume requirements that steep before, so it sometimes seems unreal. Still, I would like to know the percentage of data persisted applications requiring that throughput. I believe it's a small percentage but have no figures to back it up.

          > The reasons why a database may or may not be suitable for
          > a task isn't just raw speed. Sometimes it has to do with
          > the isolation requirements, or the serial versus parallel
          > nature of the workload, or the ability to partition the
          > load, or the durability requirements, etc. A database, for
          > example, may impose too high a cost for serial
          > transactions against the same small set of data, due to
          > the isolation+durability requirement. (Serial + isolated +
          > disk durable transactions create a maximum throughput
          > limiter tied directly to the transaction latency, and if
          > the transactions are driven out-of-process, that latency
          > becomes even more significant.)
          >
          > Peace,
          >
          > Cameron Purdy
          > http://www.oracle.com/technology/products/coherence/index.h
          > tml

          Great point.