Weblogs Forum - Messaging is the Right Way to Build a Distributed System

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Weblogs Forum
Messaging is the Right Way to Build a Distributed System

50 replies on 4 pages. Most recent reply: Aug 22, 2006 4:12 AM by D. A.

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 50 replies on 4 pages [ « | 1 2 3 4 | » ]

Isaac Gouy

Posts: 527
Nickname: igouy
Registered: Jul, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Jul 27, 2005 11:55 AM

A message-based design is fundamentally the right way to think about building a distributed system, as opposed to code sharing, remote procedure calls, and the like.

Seems like you'd be interested in Erlang.
http://www.erlang.org/white_paper.html

Making reliable distributed systems in the presence of software errors
http://www.sics.se/~joe/thesis/armstrong_thesis_2003.pdf

Todd Blanchard

Posts: 316
Nickname: tblanchard
Registered: May, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Jul 27, 2005 2:53 PM

Messaging is the right way to build most systems, distributed or not. All of the benefits you cite for distributed systems continue to hold for local programs.

This philosophy is embodied in a few programming languages as well. Smalltalk is a message oriented language. You send messages to objects, they respond if they can, forward the messages if they know someone who can, or call a default handler if they can't.

Objects might be to local, or not, but messaging is the fundamental interaction for everything from objects representing values like numbers and characters all the way up to machines elsewhere on the network. Each smalltalk object is itself a server with a set of message handlers.

Java and C++ got it wrong with their function-calling static linking semantics. Now you have to switch paradigms, simulating a messaging system in a language that doesn't support it. Complexity increases and you get something like J2EE because you are trying to make the language do something for which it is not suited.

Fernando Lozano

Posts: 2
Nickname: flozano
Registered: Jul, 2005

Re: Messaging is the Right Way to Build a Distributed System

Posted: Jul 28, 2005 10:09 AM

I guess I should have missed something, but using web serivces (SOAP and etc) as a front-end to high-granularity services implemented by my Session EJB facades provides all the exposed benefits of messaging systems? Maybe using JMS as a buffer beteen the web services front-end and the EJB back-end?

I don't think messaging is better or RPC-like systems are better. I just think people build too many unflexible systems or inefficient ones by chooing among the two. Tighy-coupled components should use RPC (and so EJB seems better than plain RPC, RMI or CORBA) but subsystems boundaries and integration points (to external systems) should use messaging.

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Jul 28, 2005 1:23 PM

> I don't think messaging is better or RPC-like systems are
> better. I just think people build too many unflexible
> systems or inefficient ones by chooing among the two.
> Tighy-coupled components should use RPC (and so EJB seems
> better than plain RPC, RMI or CORBA) but subsystems
> boundaries and integration points (to external systems)
> should use messaging.
>
Hmmm. I agree about the importance of a good design. But it's hard to see where RPC has a real place. If components are truly "tightly coupled", why is RPC needed for an invocation? If they're distributed enough to be "loosely coupled", then messaging is proving superior. It's possible there is a middle ground I've missed, where RPC really does make sense. But at the moment I don't see where it is.

Peter Booth

Posts: 62
Nickname: alohashirt
Registered: Aug, 2004

Re: Messaging is the Right Way to Build a Distributed System

Posted: Jul 28, 2005 9:13 PM

Perhaps we can unpick the threads by using an example as a
thought-experiment? I'd be delighted to learn that RPCs don't (or do) make sense.

Imagine a distributed application whose job is to display
historical financial data in a graphical form - an interactive
superset of what yahoo finance shows with a price history for a stock:

One approach includes RPCs would be to have a pure Java
client that is responsible for presnenting a set of possible datasets allowing a user to select a dataset and then visualizing that dataset. Imagine that the data being displayed is stored in a number of back end systems and that for capacity reasons a direct connection between each of these clients and the back end systems isn't feasible. So we have a:

[Java app] <----> [facade/server] <----> [back end systems]

If some of the operations/services exposed by the "server" are:

authenticateUser()
listPossibleDatasets()
retrieveDataset()

then RPC semantics would seem "natural" for the communication between app and server.

What do you think? Am I mistaking "familiar" for "natural"?

These RPCs could just as easily be abstracted as command objects sent as messages. To what end? I'm intereste din understanding why there's a difference in viewpoint.

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Jul 28, 2005 11:13 PM

> Perhaps we can unpick the threads by using an example as a
> thought-experiment? I'd be delighted to learn that RPCs
> don't (or do) make sense.
>
Good idea. A "delineation of cases" is generally good practice, so you know when to use the hammer, and when to use the saw.

> (Example:)
> [Java app] <----> [facade/server] <----> [back end
>
> If some of the operations/services exposed by the "server"
> are:
>
> authenticateUser()
> listPossibleDatasets()
> retrieveDataset()
>
> then RPC semantics would seem "natural" for the
> communication between app and server.
>
> What do you think? Am I mistaking "familiar" for
> "natural"?
>
I believe so. As you indicate, messaging could be used to make the connections. The major benefits of doing so are described in the original post. The most important is that subsystems can be upgraded indivdually.

Calum Shaw-Mackay

Posts: 58
Nickname: calum
Registered: Mar, 2004

Re: Messaging is the Right Way to Build a Distributed System

Posted: Jul 29, 2005 7:05 AM

I agree that messaging is as good a way, if not better, than RPC, however I was wondering what your thoughts are on objects being used in the transmission, rather than XML.

Although I agree that you can add additional content to XML at will, you can't if those additions invalidate a schema, without handling the version differences in the schema. With Java, and a bit of caution, you can add fields to a serialized object instance, and if need be add conditional, version oriented logic to the readExternal(), writeExternal methods. Also, reordering and mutation of XML, that is defined with a schema, can cause problems if that schema does not necessarily address evolution. Evolution is, in general, not an implicit function of any system.

Any system must be able to be tested 'clean-room' i.e. in isolation, whether it is OO or XML-based, just because the complexity of a test framework, can (apparently) be made easier by using messages, does not, in itself,negate the need for a test framework.

On your point regarding wrappers, interfaces and adapters have been doing this kind of thing for years.

Why is using a messaging system based on XML so radically different from having a messaging system developed using a distributed object paradigm? By distributed I mean objects moved around the network? Where a class file describes the contract of a stream of bytes meant to be an instance of that class, a schema describes the contract of a stream of text meant to by a document of that schema definiton

Peter Booth

Posts: 62
Nickname: alohashirt
Registered: Aug, 2004

Re: Messaging is the Right Way to Build a Distributed System

Posted: Jul 29, 2005 9:05 AM

Lets look at pros and cons. There's definitely
something won and lost with both approaches

If we adopt a messaging approach instead of RPC our
command logic becomes asynchronous. This means that
our message handling code has no context to know whether
a particular message is expected, and we don't know
whether message disappear. Some partial failure conditions
become a little harder to handle

Imagine that a user has selected a new portfolio and this
translates to two remote service invokations. (Two to illustrate a point) :

RPC Approach:

/** try to open the requested portfolio */
try
{
   newPortfolio = callOpenPortfolioRPC(p)
   newMembers = callGetMembersRPC( newPortfolio )
}
catch( ManyThingsCanBreakException e)
{}
setPortfolioOnGui( newPortfolio, newMembers);

Naive messaging approach

/** request a new portfolio */
try
{
   fireOpenPortfolioMessage(p);
}
catch( MessageSenderException e)
{}
// ...
//
handleNewPortfolioMessage(msg)
{
   newPortfolio=msg.extract
   updateGuiCleanUpStateMess(newPortfolio, newMembers);
}
handleNewMemberMessage(msg)
{
   newPortfolio=msg.extract
   updateGuiCleanUpStateMess(newPortfolio, newMembers);
}
updateGuiCleanUpStateMess()
{
   if newPortfolio.correspondsTo(newMembers) 
   { 
      setPortfolioOnGui( newPortfolio, newMembers);
      newPortfolio = null;
      newMembers = null;
   }
}

This state logic is ugly. But beyond that - we now need
to consider "when does it make sense to handle new Portfolio message?" What happens if a client receives a message intended for another client? How do we handle that? What if a client receives a newPortfolioMessage without a corresponding newMemberMessage? Who would know and how?

What if we fire the openPortfolioMessage and no response is
generated? How does the client know? Do we have to start creating timers now? Or two responses? Suddenly we're writing new code so we can say we're using messaging.
Smells fishy to me.

If the context for this is an app deployed to a captive userbase (20 traders on same floor) and you own the desktop then upgrading the client and server separately is much less of a benefit than the "foreign userbase" context.

Of course this is a manufactured example to make a point.
Help me understand what I'm not getting and why net-net
we can be better off with messaging semantics in this
kind of example?

Tom Biuso

Posts: 2
Nickname: technotom
Registered: Jul, 2005

Re: Messaging is the Right Way to Build a Distributed System

Posted: Jul 29, 2005 11:49 AM

> ...If components
> are truly "tightly coupled", why is RPC needed for an
> invocation?

I believe it is the case where components are permitted to be tightly coupled (due to design specs), and only when scalability is of little concern, that RPC actually provides the ideal platform for a solution. In my judgment, the primary advantage to RPC is ease of use. The middleware inherent to an RPC implementation will synchronize the remote method calls, eliminating the need for a transactionalized protocol. This also greatly simplifies error handling. The ease of use comes at a great cost, though, as the original article points out.

I think that basically narrows down the ideal case for RPC to a relatively small distributed system where, frankly, the cost of coordinated client-server deployment is offset by infrequent application evolution.

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Jul 29, 2005 12:24 PM

> I agree that messaging is as good a way, if not better,
> than RPC, however I was wondering what your thoughts are
> on objects being used in the transmission, rather than
> XML.
>
The problems with passing objects are versioning and transparency.

Versioning: When you modify an object, it won't work
on a system that expects the old object. So both sides of the
transmission have to be simultaneously upgraded. For many systems, that can be hard.

The lack of transparency makes things harder to debug. You can't see what's going back and forth unless you're in an interactive debugger, or write a special logging tool.

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Jul 29, 2005 12:29 PM

> Of course this is a manufactured example to make a point.
> Help me understand what I'm not getting and why net-net
> we can be better off with messaging semantics in this
> kind of example?
>
I believe the answer lay in an earlier reply, which said to leave this kind of error handling to a messaging transaction system.

However, I do think you hit on the nail on the head when you mentioned synchronicity--if synchronous communications among distributed components, then RPC makes some sense. Although stateless logic and asynchronous communications tend to produce better designs, there are undoubtedly some use cases that require synchronous traffic. (What remains is to characterize them.)

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Jul 29, 2005 12:31 PM

> In my
> judgment, the primary advantage to RPC is ease of use.
> The middleware inherent to an RPC implementation will
> ll synchronize the remote method calls, eliminating the
> need for a transactionalized protocol. This also greatly
> simplifies error handling. The ease of use comes at a
> great cost, though, as the original article points out.
>
There was a reply in which API wrappers around messaging protocols was suggested. That couples ease of development with the advantages of a message-based implementation.

Calum Shaw-Mackay

Posts: 58
Nickname: calum
Registered: Mar, 2004

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 1, 2005 3:49 AM

> The problems with passing objects are versioning and
> transparency.
>
> Versioning: When you modify an object, it won't work
> on a system that expects the old object. So both sides of
> the
> transmission have to be simultaneously upgraded. For many
> systems, that can be hard.

When using Jini, you can use PreferredClassLoaders so code that is downloaded to the machine is the most current version, even though it can also handle the old version. So technically the old side can be upgraded......

> The lack of transparency makes things harder to debug. You
> can't see what's going back and forth unless you're in an
> interactive debugger, or write a special logging tool.

Surely this is just some form of Wire Tap or Detour, that can be implemented as a Message Endpoint, before forwarding on to the intended recipient, unless you are referring to actually peeking at the message on the message queues.

I have no particular issue with the idea that messaging is as good a way (although as a previous poster pointed out, they both have their pros and cons) as RPC for doing distributed systems, however, I feel that evolvability and immunity using XML has more to do with not verifying/validating the XML than it has to do with XML having this implicit feature that wins over object formats.

For instance, with serialization, your marshalled object is compared against the class definition, before you can get to it, in other words the byte-stream is implicitly verified, and of course you can have multiple versions, either by careful changes via non-serialization-breaking changes to your class format, through readExternal, or other mechanisms.

However, with XML, if you have an XML Schema (S1) that describes the types and formats that your parser is expecting from a particular set of data purporting to be applicable to that schema( an XML file.X1), and you have another XML file(X2) that is applicable to an 'evolved' version of that schema(S1.1), for a system using the first Schema format(S1) to read the new format file(X2), the existing schema must be updated, to be able to recognize new element definitions (even though processing further on may ignore them), so you do have to change something, and it does have some impact....

The only advantage in this is that with XML you can turn off implicit validation and verification of the XML stream, and do selective data validation and string-object transform, in your SAX/DOM handling, as part of your message recipient code.

The thing about XML is that you have multi-platform interoperability....as long as you have an XML parser on that platform of course!

--Calum

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 1, 2005 12:50 PM

Calum wrote:
> When using Jini, you can use PreferredClassLoaders so code
> that is downloaded to the machine is the most current
> version, even though it can also handle the old version.
> So technically the old side can be upgraded......
>
I wasn't aware of that. Thanks for the update.

>> The lack of transparency makes things harder to debug.
>> You can't see what's going back and forth unless
>> you're in an interactive debugger, or write a special
>> logging tool.
>
> Surely this is just some form of Wire Tap or Detour, that
> can be implemented as a Message Endpoint, before
> forwarding on to the intended recipient, unless you are
> referring to actually peeking at the message on the
> message queues.
>
Right. Actually seeing the message in a debugger, or with println debugging. Composing test messages. Logging them. Patching a bad message to continue testing. All possible with ASCII messaging, and all but the last of which are possible without additional implementation steps.

> I have no particular issue with the idea that messaging is
> as good a way (although as a previous poster pointed out,
> they both have their pros and cons) as RPC for doing
> distributed systems.
>
It's tempting to agree with that statement in principle. But it has been difficult to find a good "pro" argument for RPC. If co-located, RPC isn't needed. If distributed, messaging seems superior. We assume there must be some sort of middle-ground use case where RPC is ideal, but have yet to identify it.

That use case may involve breaking an existing application into distributed modules. In such circumstances, an RPC wrapper might make the migration go more quickly--until such time as the app can be retrofitted to use messaging instead.

> For instance, with serialization, your marshalled object
> is compared against the class definition, before you can
> get to it, in other words the byte-stream is implicitly
> verified, and of course you can have multiple versions,
> either by careful changes via non-serialization-breaking
> changes to your class format, through readExternal, or
> other mechanisms.
>
Interesting point. It's true, in theory at least. In practice, getting serialization to work properly has tended to be a bit of a hurdle. When you add versioning to that...

> However, with XML, if you have an XML Schema (S1)
> ...and you have 'evolved' version of that schema (S1.1),
> ...the > existing schema must be updated
>
Damn good point, actually. Wish to heck I'd thought of it earlier. I've been assuming non-verified XML, which is silly. Still, it's pretty easy to dispatch to a wrapper that parses the old version and delegates to the new code with appropriate defaults. Unless you're using Jini, that's hard to do with an RPC implementation. (Thanks to you, I knew to add that last sentence. :_)

> The only advantage in this is that with XML you can turn
> off implicit validation and verification of the XML
> stream, and do selective data validation and string-object
> transform, in your SAX/DOM handling, as part of your
> message recipient code.
>
Another design option. A good one to have in your bag of tricks.

> The thing about XML is that you have multi-platform
> interoperability....as long as you have an XML parser on
> that platform of course!
>
That's another great point. That was the "interoperability" feature mentioned at http://www.artima.com/forums/flat.jsp?forum=106&thread=120669#Interoperability

Fernando Lozano

Posts: 2
Nickname: flozano
Registered: Jul, 2005

Re: Messaging is the Right Way to Build a Distributed System

Posted: Aug 2, 2005 7:19 AM

Everyone in this thread is taking the "all or nothing" approach. And everyone is thinking about high-granularity services. It doesn't look like real systems to me. But it could be jut my lack of experience on the subject.

Most systems end up using the facade pattern. This recognizes you have some low level components which are, because of their business logic, tighly coupled. They interact frequently and so the network overhead has a strong influence on performance. But I want to have for them managed transactions, resources (like database connections) and security features J2EE gives me. So I implement them as EJBs. And when I change one of them it's likely there will be some changes to the other ones, because the business rules have changed.

But there are also subsystems boundaries and integration points for external systems. That's were I need flexibility to connect different technologies, and where I should minimize the impact of any changes. Web Services technology makes a lot of sense here.

I've seem some systems where all business objects were implemented as web services, and whenever one business object had to talk to another there were SOAP calls. It became very hard to escalate those systems, and they lost many benefits of EJBs (like declarative transactions and clustering).

Everything that has been talked about here looks to me that I should use a messaging system like JMS to buffer request received from the web services, before passing them to the real business objects (EJBs). That makes sense to me: interoperability by using SOAP, and assyncronous messaging by JMS to provide higher decoupling.

Is my understanding correct? Because I've seen far more use cases where I need a syncronous response to my requests for remote business objects than cases where I could send the request and not bother (at least for now) about the response. So using assyncronous messages would be unatural and complicate the code, and that's allways a bad thing.

But using SOAP provides all benefits you talked about on the original post, like debugability. I simply can't see why RPC per se is a bad thing. Maybe just implementing it using binary wire protocols like RMI s a bad thing. What's the point in making everything assyncronous?

As pointed out by http://www.artima.com/weblogs/viewpost.jsp?thread=121358 ( Messaging may not be the way to Build a Distrbuted System) messaging evolved to be hidden by an RPC-like API. That's the way I see SOAP web services. Is my understading wrong?

Flat View: This topic has 50 replies on 4 pages [ « | 1 2 3 4 | » ]

Previous Topic

Next Topic