In this interview with Artima, Gemstone chief architect Jags Ramnarayan discusses the limits of database throughput, and what developers can do when their applications reach those limits:

http://www.artima.com/lejava/articles/javaone_2007_jags_ramnarayan.html

Based on your experience with the databases you've worked with, to what degree would you say typical databases really scale in terms of transaction (query and update) throughput? At what level of transaction volume would you think of clustering as a way to increase the system's throughput?

> Based on your experience with the databases you've worked
> with, to what degree would you say typical databases
> really scale in terms of transaction (query and update)
> throughput? At what level of transaction volume would you
> think of clustering as a way to increase the system's
> throughput?

In addition to Frank's questions, I'd like to know how many applications require this kind of throughput? My belief is that the number of applications requiring this throughput is at the far right of the bell curve.

I was attempting to read the Jags Ramnarayan interview, but the given url just links to a blank page (well, apart from the ads and all the usual folderol around the edges). This seems to be true for the other articles too. Is there a problem with the Artima site?

Cheers, APC

> I was attempting to read the Jags Ramnarayan interview,
> but the given url just links to a blank page (well, apart
> from the ads and all the usual folderol around the edges).
> This seems to be true for the other articles too. Is
> s there a problem with the Artima site?
>
> Cheers, APC

Sorry, I accidentally configured Apache with an overzealous redirect. It has been fixed. I don't like broken URLs, so I was trying to put in a redirect for a URL that people would rarely go to, and ended up temporarily breaking URLs that people are trying to get to.

> In addition to Frank's questions, I'd like to know how
> many applications require this kind of throughput? My
> belief is that the number of applications requiring this
> throughput is at the far right of the bell curve.

We have quite a few customers with applications that have the "5000 events per second" requirement, and a small number over the 100,000 mark. Some of these are "peak load" numbers (e.g. markets that see bursts of activity at open and close) and some of these are sustained (non-stop load of thousands of transactions / operations / events / etc. per second).

The reasons why a database may or may not be suitable for a task isn't just raw speed. Sometimes it has to do with the isolation requirements, or the serial versus parallel nature of the workload, or the ability to partition the load, or the durability requirements, etc. A database, for example, may impose too high a cost for serial transactions against the same small set of data, due to the isolation+durability requirement. (Serial + isolated + disk durable transactions create a maximum throughput limiter tied directly to the transaction latency, and if the transactions are driven out-of-process, that latency becomes even more significant.)

Peace,

Cameron Purdy
http://www.oracle.com/technology/products/coherence/index.html

> We have quite a few customers with applications that have
> the "5000 events per second" requirement, and a small
> number over the 100,000 mark. Some of these are "peak
> load" numbers (e.g. markets that see bursts of activity at
> open and close) and some of these are sustained (non-stop
> load of thousands of transactions / operations / events /
> etc. per second).

Thank you for the response. I haven't run into volume requirements that steep before, so it sometimes seems unreal. Still, I would like to know the percentage of data persisted applications requiring that throughput. I believe it's a small percentage but have no figures to back it up.

> The reasons why a database may or may not be suitable for
> a task isn't just raw speed. Sometimes it has to do with
> the isolation requirements, or the serial versus parallel
> nature of the workload, or the ability to partition the
> load, or the durability requirements, etc. A database, for
> example, may impose too high a cost for serial
> transactions against the same small set of data, due to
> the isolation+durability requirement. (Serial + isolated +
> disk durable transactions create a maximum throughput
> limiter tied directly to the transaction latency, and if
> the transactions are driven out-of-process, that latency
> becomes even more significant.)
>
> Peace,
>
> Cameron Purdy
> http://www.oracle.com/technology/products/coherence/index.h
> tml

Great point.