Weblogs Forum - Messaging is the Right Way to Build a Distributed System

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Weblogs Forum
Messaging is the Right Way to Build a Distributed System

50 replies on 4 pages. Most recent reply: Aug 22, 2006 4:12 AM by D. A.

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 50 replies on 4 pages [ « | 1 2 3 4 | » ]

Jonathan Dodds

Posts: 464
Nickname: jrdodds
Registered: Mar, 2004

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 3, 2005 6:54 AM

I have worked on a system that used XML messaging. Not Soap, just XML over HTTP. We checked for well-formedness but didn't bother with validating against either a schema or a DTD. We pulled 'records' and 'fields' out of the XML document by querying with XPath. The decision not to validate was originally driven primarily by performance issues. However it proved very flexible because the client and server didn't need to agree on the whole document. As long as its XPath queries continued to be satisfied an older client would not be troubled by newer revisions of a message.

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 6, 2005 12:00 PM

> I have worked on a system that used XML messaging. Not
> Soap, just XML over HTTP. We checked for well-formedness
> but didn't bother with validating against either a schema
> or a DTD. We pulled 'records' and 'fields' out of the XML
> document by querying with XPath. The decision not to
> validate was originally driven primarily by performance
> issues. However it proved very flexible because the client
> and server didn't need to agree on the whole document. As
> long as its XPath queries continued to be satisfied an
> older client would not be troubled by newer revisions of a
> message.
>
Yup. You surely do get flexibile evolution that way.

Come to think of it, validation can be performed on subtrees of the DOM, instead of on the entire document. That makes it possible to achieve "safe flexibility", or something to that effect.

Roger Voss

Posts: 27
Nickname: rogerv
Registered: Aug, 2005

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 14, 2005 10:55 PM

> I guess I should have missed something, but using web
> services (SOAP and etc) as a front-end to high-granularity
> services implemented by my Session EJB facades provides
> all the exposed benefits of messaging systems? Maybe using
> JMS as a buffer between the web services front-end and the
> EJB back-end?
>
> I don't think messaging is better or RPC-like systems are
> better. I just think people build too many inflexible
> systems or inefficient ones by choosing among the two.
> Tighy-coupled components should use RPC (and so EJB seems
> better than plain RPC, RMI or CORBA) but subsystems
> boundaries and integration points (to external systems)
> should use messaging.

When designing distributed apps a successful approach is to conceptualize a discreet node in the distributed system as a state machine. Messaging is thus the means for conveying events bidirectionally between state machine nodes. Messaging is more adept for this approach because of its intrinsically queued semantics. Also often times it is desirable to multi-cast events to multiple nodes. RPC is a highly coupled point-to-point communication mechanism that has none of these highly desirable characteristics.

Indeed RPC's lack of queued behavior is higly problematic in terms of attempting to use it in an asynchronous fashion (RPC neophytes like to think they're mimicing messaging when they use RPC asynchronously but they're not all - they've merely pushed RPC into an entirely different quagmire).

Though, I've cited some of the core fundamentals I've barely scratched the surface as to why messaging is several orders of magnitude superior to RPC. Here's a little snippet from one of my presentations on my real-world distributed applications that are messaging based:

Messages are discreet objects that can be arbitrarily manipulated at practically any stage of their existence.

This makes messaging a completely different paradigm relative to RPC.

Messages can be persisted, transacted, filtered, routed, duplicated, logged, bridged, archived, transformed, uni-cast, mutli-cast, etc., etc.

RPC data payloads, in comparison, are ephemeral

The loose coupling aspect is another highly beneficial characteristic of messaging. A given processing node doesn't care what is on the other side of a messaging queue - it could be a single node or it could be a high availability cluster doing round robin processing. It could be a middle-ware J2EE server or it could be a simple 2 tier topology that is serviced directly by a back-end client-server application. The administrative flexibilities due to messaging loose coupling are endless and in the real world of deployment issues - a vital tool in the arsenal.

Roger Voss

Posts: 27
Nickname: rogerv
Registered: Aug, 2005

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 14, 2005 11:14 PM

Hey Eric, on this messaging thing we're definitely kindred spirits. I've been building and deploying distributed applications using Tibco EMS for over the last two years. My applications have been quite successful for my customers and are used in very demanding situations (marine terminal operations).

I got sold on messaging back in the mid 90s where I worked for Microsoft building middleware software. At that time I was using DCOM RPC and got completely fed up with RPC crap. I did a bit of stuff with MSMQ and saw the potential of things could be much better.

These days am using J2EE and JMS and loving JMS just fine. I completely skipped the RPC session bean nonsense and just use MDBs and special JMX mbeans that also do some messaging. Consequently my J2EE experience has been great as I've steered clear of all the old-school J2EE nonsense that everyone else wallowed through.

On Aug. 15, Monday night I'm doing a presentation on my distributed software systems to the Seattle Java User Group (SeaJUG). I've referenced your article in the notice on the SeaJUG site as recommended reading prior to attending my presentation:

http://www.seajug.org/vqwiki/jsp/Wiki?FutureMeetingTopics

Maybe we ought to get together and do a new book on messaging? I can't recall O'Reilly having any significant titles on this subject. And I think messaging is getting hot as more and more developers are beginning to see the light.

Cheers,
--RogerV

Roger Voss

Posts: 27
Nickname: rogerv
Registered: Aug, 2005

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 14, 2005 11:41 PM

> But using SOAP provides all benefits you talked about on
> the original post, like debugability. I simply can't see
> why RPC per se is a bad thing. Maybe just implementing it
> using binary wire protocols like RMI s a bad thing. What's
> the point in making everything assyncronous?

JMS also supports doing a synchronous request. I've used that in certain situations where strict order of processing had to be observed. For instance, answering a call and hanging up a call occur in a specific sequence. It would never be appropriate for a hang-up-call msg to get processed ahead of a answer-call msg (that is, when both msgs pertain to the same call). Yet with MDBs in a bean pool in a J2EE app server, that could actually happen. Or if you made asynchronous RPC calls you'd have the same potential problem.

So yeah, there are situations when is warranted to observe strict synchronous order.

But a lot of things can be dealt with in an asynchronous fashion. And queuing can preserve correct sequencing (just have to be careful about the server-side in terms of MDB pools). The queuing characteristic of messaging is a most significant distinction of messaging relative to RPC. By and large one is better off to begin to think in terms of asynchronous state machines communicating through event transfers. You'll wind up with more robust designs. And in actuality, IMO, the error handling is actually more cleanly dealt with.

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 15, 2005 9:34 AM

> Messages are discreet objects that can be arbitrarily
> manipulated at practically any stage of their existence.
>
> This makes messaging a completely different paradigm
> relative to RPC.
>
> Messages can be persisted, transacted, filtered, routed,
> duplicated, logged, bridged, archived, transformed,
> uni-cast, mutli-cast, etc., etc.
>
Brilliant. It gets the highest compliment I can give:
I wish *I* had written it.

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 15, 2005 9:43 AM

Roger wrote:
> These days am using J2EE and JMS and loving JMS just fine.
> I completely skipped the RPC session bean nonsense and
> just use MDBs and special JMX mbeans that also do some
> messaging. Consequently my J2EE experience has been great
>
Aha! I was impressed with Sun's app server. As of version 8,
it began becoming friendly enough to use. It's a J2EE platform, but I figured I'd ignore that and use it as a web service platform. It looks pretty good for that. (Especially the free version that's good for something like 50 users.)

But your message tells me that J2EE minus RPC has some value, as well. But surely it's possible to get a messaging service without J2EE. Isn't it? Or was J2EE just convenient? Or does it have other benefits, after ignoring the RPC elements?

> On Aug. 15, Monday night I'm doing a presentation on my
> distributed software systems to the Seattle Java User
> Group (SeaJUG). I've referenced your article in the notice
> on the SeaJUG site as recommended reading prior to
> attending my presentation:
>
> http://www.seajug.org/vqwiki/jsp/Wiki?FutureMeetingTopics
>
Excellent! Let me know how the presentation goes.

> Maybe we ought to get together and do a new book on
> messaging? I can't recall O'Reilly having any significant
> titles on this subject. And I think messaging is getting
> hot as more and more developers are beginning to see the
> light.
>
That sounds like a seriously good idea. You've had the experience to suggest the use cases, anticipate the problems, and supply remedies. It could be a great partnership--and judging from the response to this topic, there is more than enough interest in it! I'll have to see if I can dig up that business card for the O'Reilly guy I met at JavaOne. :_)

Mike Petry

Posts: 34
Nickname: mikepetry
Registered: Apr, 2005

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 15, 2005 3:55 PM

Why not both messaging and document-centric web services for distributed systems?

There are two concepts that keep merging together and I think we should seperate them. Firstly, there is event notifications and synchronization. Secondly, there is the distribution of information.

Simple "document-centric" web services could handle information distribution and a messaging system could serve to provide notifications in support of the venerable publish-subscribe pattern.

This approach would serve to decouple the data intensive part of the system, while the message system is implicitly decoupled.

The trade-off is that the interface to a distribute element of the system provides no value other than a portal for information. I guess that is what WSDL and XML Schema is for - yuck. Why not make each element of the system figure out what the XML document means? It seems like this is where the work should take place.

Anyway, thanks for letting me put in my two cents.

Roger Voss

Posts: 27
Nickname: rogerv
Registered: Aug, 2005

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 17, 2005 7:13 AM

> But your message tells me that J2EE minus RPC has some
> value, as well. But surely it's possible to get a
> messaging service without J2EE. Isn't it? Or was J2EE just
> convenient? Or does it have other benefits, after ignoring
> the RPC elements?

All the services in the J2EE app server - JNDI, XA transactions, JDBC connection pooling, logging and error notification to external monitoring systems, container-based thread pool management - are all valuable to message-driven-bean EJB implementations as well.

Also I heavily use JMX mbeans to write all manner of custom services - such as device control or special purpose items such as the UDP-based auto-discovery service for JMS clients or an Oracle AQ-to-JMS bridge. (The previous paragraph likewise applies to JMX mbean implementation.)

I still have a sprinkling of servlets and JSPs too. More of these will be written in the future as we do more web-based admin and dashboard stuff to control or interact with the apps that are running. A person recently joining staff brings in web expertise - though in the meantime he's learning to write the kind of stuff I create. (On admin stuff - because I can apply AOP to quickly create a JMX mbean out of a POJO, its very convenient. The JMX admin interface approach is okay for my purposes and the support staff. However, for some of the managerial end-users we have in mind, we want them to have well polished web app interfaces - and for some of the dashboard ideas are even considering Java Swing apps that are deployed by WebStart.)

Frequently multiple beans are deployed in a single .ear file (constituting an application) where they are designed to interact - say an MDB that invokes an API directly and locally on a service mbean.

Too, over time I create and deploy various apps for different purposes into the application server so its beneficial to have a host environment to contain these in one place - as opposed to writing a standalone app each time where the various J2EE services have to be reinvented or reintroduced per each app.

> > On Aug. 15, Monday night I'm doing a presentation on my
> > distributed software systems to the Seattle Java User
> > Group (SeaJUG). I've referenced your article in the
> notice
> > on the SeaJUG site as recommended reading prior to
> > attending my presentation:
> >
> >
> http://www.seajug.org/vqwiki/jsp/Wiki?FutureMeetingTopics
> >
> Excellent! Let me know how the presentation goes.

The presentation was very well received. It had the rapt attention of all attendees and everyone had plenty of questions that were asked per practically every slide (50 slides total). I've been to several of their meetings before and by comparison I could tell that this one was definitely connecting to the audience.

Or perhaps folks relished the opportunity to talk about "old-fashioned" rich client GUI applications and J2EE being used for cool things like device control - as opposed to HTML browser stuff and yet another templating solution for dynamic page composition. (I started out telling them they were going to enjoy the evening because we weren't going to be talking about any web development.)

People definitely tuned in to the use of JMS messaging. I think my distributed applications opened some eyes. The things I described doing with messaging that couldn't be done with RPC (or would be very difficult) connected with folks.

For instance, one cool example is where I used a message bridge to duplicate and route the flow of messages from a production site to my private development JMS/J2EE cluster. I then further tested, debugged, and refined this particular distributed application using real live messaging data. A completely parallel system running in a ghost fashion with respect to the production operation yet not having any adverse effect on the production system.

The motivation for doing that is that my simulator program was proving less and less adequate for doing the final phase of fine tuning. I didn't want to keep putting more and more effort into trying to make the simulator realistic with respect to the real world. Hence it was most effective to just use the real world message data and test against that. Thankfully messaging made that very simple to do.

I continue to think a new book on messaging and its pragmatic application is a worthy idea. The experience of my presentation has bolstered my belief of such.

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 17, 2005 10:53 AM

> Why not both messaging and document-centric web services
> for distributed systems?
>
I'm not sure what you mean by document-centric web services, exactly, but the phrase did remind me another point I wanted to make:

It's seductive to think that the server should remember state. After all, one reasons, if I've started a session on the server, then the server should remember where I am in the transaction process so I can pick up where I left off when I supply the next bit of information.

On the whole, though, systems are more fully decoupled when all of the state information resides in the message. This may be what you had in mind when you said "document-centric" processing.

On the one hand, the message gets larger. The time required to process the message gets longer, as well. So there would seem to be a performance cost. On the other hand, the systems are now *fully* decoupled, and the systems become drastically less complex. The savings in complexity translates into a performance gain, in addition to a reliability gain. For one thing, no server resources are *ever* tied up waiting for a client-side action, so the server can conceivably handle many more resources.

Putting all of the state in the message.document makes the subsystems fully asynchronous, so there is no need for sessions to time out to free up resources. That eliminates some of the performance overhead of threading and timer task(s).

Basically, keeping state in the message/document keeps the system simple.

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 17, 2005 10:57 AM

Roger wrote:
> I continue to think a new book on messaging and its
> pragmatic application is a worthy idea. The experience of
> my presentation has bolstered my belief of such.
>
Presentations are definitely a writer's friend. They tell you when a subject is hot, they give you the questions you need to answer in the book, and they give you faces--real people you're writing for, that you want to communicate to. More than anything, knowing that you're talking to a person keeps a writer's prose simple and direct.

Note:
I haven't surfaced my contact yet. Keep prodding me (off list, preferably).

Mike Petry

Posts: 34
Nickname: mikepetry
Registered: Apr, 2005

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 17, 2005 2:58 PM

Documentric-centric is a buzz word that refers to web service protocols that pass XML Documents (as opposed to RPC-like web services). I think the current vision has WSDL defining the end points, and then using XML Schema to validate the XML Document "payload". This is the proverbial web service that appears as

justDoIt(String XMLDocument)

In some ways it almost eliminates the need for WSDL. It seems like they are trying to figure out a way to keep around WSDL by adding WS extensions, but that is a debate for another day.
Anyway, XML Document vs RPC web services is a huge hotbutton and your opion largely reflects how you see SOA and distributed systems evolving.
I guess my previous post had me laying down my cards and saying that I think all web services should do is pass information and request services. This actually means more work at each end point but allows severe decoupling between services.
Another point is an observation that a call to a function, method or an RPC call serves two purposes. Firstly it invokes some type of functionality or distributes information, secondly it triggers an event. I think XML Doc-based web services are too amorphous to denote events. Therefore I think it would be optimal to have XML Doc services going-on, while some type of messaging service is handling event notifications and basically synchronizing activities.
As far as state goes, I haven't fully firmed up my opinion on this and your ideas sound plausible. I am interested in "viral" approaches to state management which is interesting in theory but I don't completely see how it would be done. This basically fosters a totally decentralized approach to distribution systems.
I feel like I don't grok Jini. Is this an application for Jini? Decentralized, fault-tolerant systems?

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 18, 2005 2:12 PM

> Documentric-centric is a buzz word that refers to web
> service protocols that pass XML Documents (as opposed to
> RPC-like web services).
>
Got it. Thanks.

> I think the current vision has
> WSDL defining the end points, and then using XML Schema to
> validate the XML Document "payload".
>
Hmmm. As previously surfaced in this thread, it's a better idea to validate pertinent segments of the message, rather than the whole thing. That allows the message structure to evolve.

> I guess my previous post had me laying down my cards and
> saying that I think all web services should do is pass
> information and request services. This actually means
> more work at each end point but allows severe decoupling
> between services.
>
Yeah.

> Another point is an observation that a call to a function,
> method or an RPC call serves two purposes. Firstly it
> invokes some type of functionality or distributes
> information, secondly it triggers an event.
>
That's an interesting aspect of RPC mechanisms. It's something that *could* be duplicated in a message based system, but it would require extra work. (I like to say that
just because you *can* do something doesn't necessarily make it simple, effective, or even feasible. After all, you *can* implement a compiler in a Turing machine--theoretically.)

> I think XML
> Doc-based web services are too amorphous to denote events.
> Therefore I think it would be optimal to have XML Doc
> c services going-on, while some type of messaging service
> is handling event notifications and basically
> synchronizing activities.
>
Interesting concept.

> As far as state goes, I haven't fully firmed up my opinion
> on this and your ideas sound plausible.
>
It would seem to reasonate with your "pass information and
request services" concept.

> I am interested
> in "viral" approaches to state management which is
> interesting in theory but I don't completely see how it
> would be done.
>
Haven't heard of that before. What is it (in theory)?

> This basically fosters a totally
> decentralized approach to distribution systems.
> I feel like I don't grok Jini. Is this an application for
> Jini? Decentralized, fault-tolerant systems?
>
Maybe we can get Bill or Frank to weigh in, here. They've got a fair amount of experience with Jini, I believe.

Mike Petry

Posts: 34
Nickname: mikepetry
Registered: Apr, 2005

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 18, 2005 6:29 PM

This is Adam Bosworth at the MySQL convention talking about Googles approach to distributed computing and he discusses some of his interests, one of which is this viral state propagation.

http://www.itconversations.com/shows/detail571.html

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 19, 2005 12:52 PM

> This is Adam Bosworth at the MySQL convention talking
> about Googles approach to distributed computing and he
> discusses some of his interests, one of which is this
> viral state propagation.
>
> http://www.itconversations.com/shows/detail571.html
>
First, let me say that was a brilliant discussion. Thanks for clueing me in to it. (The fact that it was an audio recording made it an easy listen, too, instead of a hard read.)

He never actually uses the word "viral" anywhere in that presentation, but since his idea is to use RSS 2.0 or ATOM to create an "open protocol" for data delivery and access, I can see where additional servers could use the same the same-change replication mechanisms that web systems use.

In general, I have to say I agree with his approach. His focus on simplicity and his recognition that the Web grew because "anyone who can think" could author HTML content are right up my alley. I am particularly intrigued by his observation that centeralized databases don't scale--especially if they include processing logic--and that a simple data-exposure mechanism (the equivalent of HTML for data tables) is the likely solution (as opposed to the semantic web, complex query languages, or "standards" that took 4 years to produce, and will rarely be implemented, due to their complexity).

The presentation does seem to be somewhat tangental to the thrust of this thread, but it was a good listen nonetheless.
It does seem to have a place in how one thinks about designing the data-side of a message-based system, but I confess that at the moment, I don't quite see how to apply it. (Bosworth mentions a need for a small extenstion to the RSS 2.0 or ATOM spec to support updates, and his talk was cut short due to time, so perhaps it is simply a matter of
filling in some missing information?...)

Flat View: This topic has 50 replies on 4 pages [ « | 1 2 3 4 | » ]

Previous Topic

Next Topic