ocamlmq is a STOMP message broker with features that make it especially suitable for implementing task queues and communication between subsystems:
persistent queues, scaling over arbitrarily large numbers of queues with constant memory usage (i.e. supports millions of queues)
strong durability guarantees: messages are guaranteed to have been saved to disk by the time the sender gets a message receipt
message priorities
per-subscription prefetch limit for queue messages
error handling and ACK timeout: if a subscriber doesn't ACK a message after a (message-specific) timeout, the message will be sent to another subscriber. Messages are also resent automatically if a subscriber dies and its connection expires.
topic subscriptions (messages broadcast to all subscribers) with prefix matching
support for high numbers of subscriptions: millions of subscriptions pose no problem
simple extensions within the STOMP protocol to report the number of messages in a queue and the number of subscribers to a queue or topic
excessive memory footprint (ActiveMQ and RabbitMQ), both at startup (e.g. >120MB for ActiveMQ vs. <3 MB for ocamlmq) and as new queues are created or new topic subscriptions are added
bad performance with ActiveMQ's scalable (to thousands of queues) storage backends (KahaDB, JDBC)
bad performance in RabbitMQ's topic message dispatch: RabbitMQ was doing a linear scan of the subscription table per dispatch
RabbitMQ did not guarantee that persistent messages had been saved to disk before sending the message receipt, which could lead to data loss
ocamlmq is written in OCaml. I originally estimated that a simple STOMP message broker with the feature set I wanted could be done in around 2000 lines. It took less than 1000, which later grew to ~1200 as I added new features (prefix topic subscriptions and control messages). Because ocamlmq is relatively small (the core of the broker takes around 500 lines of code), it is easy to extend and there's little space for bugs. The server is fairly efficient and abstracted over a storage backend; currently only PostgreSQL's is implemented (< 150 lines of code).
Limitations
ocamlmq works well in the intended use case (persistent queues and transient topic destinations, with possibly many queues and subscriptions), but it has some limitations which preclude its use in other domains:
ocamlmq is not designed to scale beyond several hundred / a few thousand simultaneous connections (it will work, but performance will be affected)
there is no flow control for topic messages (in the intended use case, topic messages are assumed to be relatively small and processed fast)
messages are limited to 16 MB on 32-bit platforms
the PostgreSQL storage backend can only persist a few thousand messages per second (note that ocamlmq allows >50K/s persistent message bursts in async mode)
ocamlmq does not support very high message rates (ocamlmq delivers only ~20000 messages/second on a 3GHz AMD64 box)
If you need complex routing rules, scalability to many thousand simultaneous connections or other enterprise messaging features, you'll be better served by AMPQ or JMS brokers. ActiveMQ, in particular, is very configurable, so it'll most likely fit the bill if memory consumption or scaling to many subscriptions are not a concern.
Scalability
ocamlmq has been designed to support millions of queues and topic subscriptions without undue memory usage. This table summarizes the time complexity of some STOMP operations:
SEND to queue O(log subscribers)
SEND to topic O(subscribers + log (total subs))
SUBSCRIBE to queue O(log (total subs))
SUBSCRIBE to topic O(log subscribers)
ACK O(1)
ocamlmq needs typically around 150 bytes per subscription, so 1 million subscriptions will not take much more than 150 MB of memory (compare to >120MB required by ActiveMQ on startup). No extra memory is needed for queues, so you can use lots of them with no concerns for memory usage.
Previous Topic
