The Artima Developer Community
Sponsored Link

Weblogs Forum
Grokking Twisted

8 replies on 1 page. Most recent reply: Jun 16, 2009 8:56 AM by Nanjundi T

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 8 replies on 1 page
Bruce Eckel

Posts: 875
Nickname: beckel
Registered: Jun, 2003

Grokking Twisted (View in Weblogs)
Posted: Apr 15, 2006 8:17 PM
Reply to this message Reply
Summary
For some time now, my intuition has been telling me that the Twisted Matrix is something that would be good to understand. Maybe it's just from seeing the kind of energy that the Twisted guys exhibit at Python conferences.
Advertisement

The problem is that the terminology they use has never made much sense to me. It seems like they make up words that seem sensible to them at the moment, but that I haven't been able to make heads or tails of.

I've been studying distributed systems, things like XML-RPC. XML-RPC is nice when you are making calls across the network and you don't want to worry about how the server is implemented. As long as the server speaks XML-RPC (or SOAP, if you must -- but from what I've heard, web services have mostly been providing XML-RPC as an alternative when they offer SOAP, and that the vast majority of people are using XML-RPC), you don't care what language that server was written in.

But what if you have control over both ends? XML-RPC still works fine, but if you're programming in Java, maybe it makes more sense to use RMI. OK, RMI might end up being more complicated than XML-RPC, so consider a distributed system where all the clients and servers are using Python.

Python has a perfectly fine XML-RPC system, and that makes sense as long as you're doing synchronous calls -- that is, you make a call and expect the result to be returned right away (this is also what Java RMI expects). But in a distributed system it's not uncommon to make asynchronous calls, where you make the call and it takes awhile to complete the result, so you don't want the caller to be stopped while waiting for the result. Instead, what you'd like is for the call to return, so the caller can go on and do something else, and then sometime later, when the server finishes the request, it can somehow transport the result back to the client. And this is where things get tricky, and where simple synchronous systems like XML-RPC or Java RMI don't serve so well.

If you've studied the concurrency library in Java 5 (and this amounts to the longest chapter in Thinking in Java 4th edition), you'll know that a new construct has been introduced -- new to Java, not to the programming community at large -- the Future. A Future is an interface that is returned from an asynchronous call. You can call isDone() to discover whether the underlying call is completed, and get() to get the result. This is fairly primitive, because it requires polling to see whether the call is completed, but it does allow the calling process to continue doing other things rather than being stopped with a synchronous call.

This is where Twisted comes in handy -- it will apparently do a lot of other things, but this is the first feature that I've really understood. And I understood it not from the O'Reilly Twisted book, but rather from the Python Cookbook, 2nd edition, where they have a small, succinct example and short, clear explanation (I find the Python Cookbook one of the best references once you've mastered the basics -- for which I still like Learning Python -- and I also get a lot of use from Python in a Nutshell. And although it's a bit dated now, the Python Standard Library is also full of great examples).

In recipe 15.7, Using Twisted Perspective Broker, they create a server in 11 lines, by using the Twisted Perspective Broker. And here's where I have terminology problems again -- "Perspective Broker" doesn't say to me "thing that you make a network server from," in fact, I don't know what it's supposed to mean. But what you can do with it is indeed very cool. I'm going to try to make an even simpler example here than they showed, because I'm thinking about doing distributed computing in order to distribute long-running processes among machines, just to get the work done faster. So my example will simply be a server that provides a remote method call that takes awhile to complete:

"Server.py: Provides a calculation service across the network"
from twisted.spread import pb
from twisted.internet import reactor
import time

PORT = 8992
COUNT = 3

class Calculator(pb.Root):
    def __init__(self, id):
        self.id = id
        print "Calculator", self.id, "running"
    def remote_calculate(self, a, b):
        print "Calculator", self.id, "calculating ..."
        time.sleep(1)
        return a + b, self.id

for i, port in enumerate(range(PORT, PORT + COUNT)):
    print "port:", port
    reactor.listenTCP(port, pb.PBServerFactory(Calculator(i)))

reactor.run()

The Calculator class provides a remote method called calculate(), which adds its arguments. The call to sleep() simulates a complex, time-consuming task (and we distribute these tasks across machines to get the job done faster).

To create a server that provides this service, you simply inherit your service class from pb.Root and then pass an instance of that class to pb.PBServerFactory(), the result of which is bound to and listens at a particular TCP port, using reactor.listenTCP(). Because other parts of Twisted are more low-level, I think that the goal of the perspective-broker framework was to make this process of creating servers and clients very easy.

The reactor is another one of those words that they use, although I think this one makes sense (it's apparently based on the Reactor Design Pattern, which is not one of the Gang Of Four patterns). It's the object that implements the event loop, and it "reacts" to events (in this case, method calls across the network). This is one of the core elements of Twisted, and they build a lot of stuff on it. But in the above code, we can wave our hands over the code a little bit and say "if you inherit a class from pb.Root and give that class methods that begin with remote_, we can tell a reactor to listen at a TCP port and hand our class to the pb.PBServerFactory constructor, and this will provide our methods as remote network calls. That is, we can create a server with almost no effort.

Twisted itself is mindlessly easy to install; after you have Python installed (from www.Python.org) you go to the Twisted page and, for Windows, download and execute a Windows installer. That's it.

In order for the client to make an asynchronous call, the call must return right away using a mechanism like Java 5's Future -- which is really just a proxy to a result that will appear sometime in the future. Twisted, however, returns an object that is more clever than a Future, called a Deferred. But instead of polling that Deferred as you do in Java, you add callbacks -- one to call back if the call succeeds, and on to call back if the call fails (of course, failure might not be detectable by the server in some cases, so it's easy to add things like watchdog timers in Twisted).

Because of the possibility of failure when making calls across the network, the client tends to be more complex (and in this example I haven't covered all the possible failure points, in the interest of keeping things simple):

"Client.py: Uses the calculation service across the network"
from twisted.spread import pb
from twisted.internet import reactor

PORT = 8992
COUNT = 3

class ModelCalculator:
    def __init__(self, host):
        self.a = 42
        self.b = 47
        self.count = 0
        self.total = 0
        print 'Connecting to Servers...'
        for port in range(PORT, PORT + COUNT):
            factory = pb.PBClientFactory()
            reactor.connectTCP(host, port, factory)
            factory.getRootObject()\
                .addCallbacks(self.connected, self.failure)
    def connected(self, perspective):
        perspective.callRemote('calculate', self.a, self.b)\
            .addCallbacks(self.success, self.failure)
        print "connected"
        self.a += 10
        self.b += 11
    def success(self, result):
        print result
        self.total += result[0]
        self.count += 1 # Alternative: use a DeferredList
        if(self.count == COUNT):
            print "total:", self.total
            reactor.stop()
    def failure(self, _):
        print "remote failure"
        reactor.stop()

ModelCalculator("127.0.0.1")
reactor.run()

This class distributes calculations to three different servers, then collects the results. To connect to a server, you create a pb.PBClientFactory(), pass it in to reactor.connectTCP(), then get the root object of that factory, which is a defferred representing the connection process. To this, we add "success" and "failure" callbacks, for when the connection process either finishes or fails. Assuming it's sucessful, the connected() method will be called, handing it the perspective object. Now that you have a live connection, you can make a remote call using callRemote(). Again, this returns a deferred object (because it's an asynchronous call), and you can add callbacks indicating success and failure for that call. In the example above, the success() method keeps track of the callbacks it receives so that it can print the final result and stop the reactor when the calculation is complete.

Twisted has lots of other features that I haven't fathomed yet; the book "Twisted Network Programming Essentials" by Abe Fettig (O'Reilly, 2006) is the definitive introduction to Twisted and takes you through those other features. For example, it's quite simple to create an XML-RPC server using Twisted. In the above example, you could factor the calculation (business logic) into a separate class, then expose the functionality of that class using both a perspective broker and as an XML-RPC service -- although the client for the XML-RPC service would then have to deal with the fact that the synchronous calls would not return right away.


Mike Coyle

Posts: 1
Nickname: mcoyle
Registered: Apr, 2006

Re: Grokking Twisted Posted: Apr 16, 2006 12:12 AM
Reply to this message Reply
While not specifically related to the topic that Bruce was covering (RPC in Twisted), I give my $.02 on why Twisted is well worth grokking in this blog entry:

http://www.botonomy.com/blogs/mcoyle/WebFrameworkShootouts.html

--Mike

Glenn Vanderburg

Posts: 114
Nickname: glv
Registered: Jun, 2003

Re: Grokking Twisted Posted: Apr 17, 2006 7:26 AM
Reply to this message Reply
Bruce, you don't have to keep polling a Future, unless you want to avoid blocking. The typical use is to submit a task for something you know is going to take a little while, hold onto the Future for the result, and just go off and do some preliminary work that can be done without the Future result. Then, when you reach the point where you need that result, you just call get() on the Future. If it's ready, you get it right away; otherwise you block until it's available.

Where the Future is getting something from the network, you'd want to have timeouts established, but generally you wouldn't want to poll.

Gregg Wonderly

Posts: 317
Nickname: greggwon
Registered: Apr, 2003

Re: Grokking Twisted Posted: Apr 20, 2006 12:07 AM
Reply to this message Reply
This is a great example of where a Tuple space pays off. With Javaspaces from the Jini tools, you can simply submit all the tasks to the javaspace. All your workers will listen for the work to come to the javaspace, and execute the work and then write the results back into the space.

The work creator would so something like:

long leaseTime = getLeaseTimeout();
JavaSpace sp = findSpace();
for( WorkItem item : work ) {
Transaction txn = createTransaction();
Lease l = sp.write( item, txn, leaseTime );
manageLease( l );
txn.commit();
}

while( true ) {
// Get next result
Result r = (Result)sp.take(
resultTemplate, t, leaseTime );
}

The workers would just do

while( true ) {
WorkItem wi = (WorkItem)sp.take( workTemplate,
txn, Lease.FOREVER );
Result res = doWork( wi );
Lease l = sp.write( res, txn, resultLeaseTime );
manageLease( l, wi, res );
txn.commit(); // Use transaction to swap work for
// result.
}

This is the classic distributed worker pattern that javaspaces really excels at. There are a lot of variations of this pattern in JavaSpaces applications.

Bruce Eckel

Posts: 875
Nickname: beckel
Registered: Jun, 2003

Re: Grokking Twisted Posted: Apr 24, 2006 12:45 PM
Reply to this message Reply
Is there a Jini/Javaspaces implementation that doesn't take a lot to get set up and going out of the box? The thing that has kept me away before has been what appears to be a lot of configuration and programming involved to get something going.

Gregg Wonderly

Posts: 317
Nickname: greggwon
Registered: Apr, 2003

Re: Grokking Twisted Posted: Apr 25, 2006 12:24 PM
Reply to this message Reply
> Is there a Jini/Javaspaces implementation that doesn't
> take a lot to get set up and going out of the box? The
> thing that has kept me away before has been what appears
> to be a lot of configuration and programming involved to
> get something going.

There is some newer IDE support from the incaX group in the form of IDE plugins. There is a community edition and a commercial edition. This is probably the easiest place for people to start with Jini, in general.

The Jini 2.1 starter kit at www.jini.org (for now, changes are underway in the community), has an installer, and provides startup of all the services in a fairly automated fashion. This is better for people who are more technical, and whose IDE doesn't have a Jini plugin or some Jini supporting infrastructure.

There are several different containers that have been around for a while, which reduce the barriers as well. The http://harvester.jini.org project and the http://cherion.jini.org project are a couple. My startnow.jini.org project includes a simple class for creating services called PersistentJiniService. You can use it with

/**
 *  The interface for clients to use
 */
public interface MyInterface extends Remote {
     public int myMethod1( String val )
         throws IOException;
     public String myMethod2( int v1, int v2 )
         throws IOException;
}

I use throws IOException instead of throws RemoteException that is typically visible in RMI applications to reduce my dependencies on the RMI package for J2ME and other non-J2SE environments. Either is acceptable to the proxy generation mechanisms.
/**
 *  The service implemenation of the interface
 */
public class MyService extends PersistentJiniService
     implements MyInterface {
 
     public int myMethod1( String val ) {
         return Integer.parseInt(val);
     }
 
     public String myMethod2( int v1, int v2 ) {
         return v1+"+"+v2+"="+(v1+v2);
     }
 
     public MyService( String args[] ) {
         super(args);
         startService();
     }
}

The parent class does all of the dirty work in startService(). It swallows a configuration provided on the command line. But, if you don't provide one, you get reasonable defaults with no security enforced.

There are lots of ways to simplify things, and many people have in house tools for some type of "container" that works great for their application. One of the standing issues with Jini is that it is not a "web service thing". It's a toolkit for a lot of different distributed computing applications.

Thus, one persons tools might not have anything to do with your application domain.

If you need any more specifics, please send me an email directly Bruce, and I'd be glad to help you find the right tools.

Nanjundi T

Posts: 4
Nickname: nanjundi
Registered: Jun, 2009

Re: Grokking Twisted Posted: Jun 15, 2009 10:17 AM
Reply to this message Reply
> class Calculator(pb.Root):
.../snip/...
> for i, port in enumerate(range(PORT, PORT + COUNT)):
> print "port:", port
> reactor.listenTCP(port,
> ort, pb.PBServerFactory(Calculator(i)))
>


Well, this doesn't work as expected but behaves very strangely. Adding more logging and timestamps to the client, the o/p is:

2009-06-15T09:58:27 Connecting to Servers...
2009-06-15T09:58:27 connected: Calculating 42 + 47:
2009-06-15T09:58:27 connected: Calculating 52 + 58:
2009-06-15T09:58:27 connected: Calculating 62 + 69:
2009-06-15T09:58:30 (89, 0)
2009-06-15T09:58:30 (110, 1)
2009-06-15T09:58:30 (131, 2)
2009-06-15T09:58:30 total: 330


Since the additions run in parallel, the approximate time for client execution should be one second. (ignoring the execution time for other computations in client).

Strangeness1: Since these are 3 different servers on different ports, it should take one second to return. Well it took ~3 seconds.

Strangeness2: Accepting strangeness1, and assuming the method calls run in sequence, I expected to see results returned at one second interval, but all three returned at the same time, after 3 seconds.

If we burst the above loop and start 3 servers in 3 different instances of python, the client returns the result in (approximately) one second.

Nanjundi T

Posts: 4
Nickname: nanjundi
Registered: Jun, 2009

Re: Grokking Twisted Posted: Jun 16, 2009 8:50 AM
Reply to this message Reply
never mind, the culprit is time.sleep. Use reactor.callLater instead.


(Server.py)
def remote_calculate(self, a, b):
print "Calculator", self.id, "calculating ..."
+ d = defer.Deferred()
+ reactor.callLater(1, d.callback, (a + b, self.id))
+ return d
- time.sleep(1)
- return a + b, self.id

Nanjundi T

Posts: 4
Nickname: nanjundi
Registered: Jun, 2009

Re: Grokking Twisted Posted: Jun 16, 2009 8:56 AM
Reply to this message Reply
(Server.py)

def remote_calculate(self, a, b):
print "Calculator", self.id, "calculating ..."
+ d = defer.Deferred()
+ reactor.callLater(1, d.callback, (a + b, self.id))
+ return d
- time.sleep(1)
- return a + b, self.id

Flat View: This topic has 8 replies on 1 page
Topic: The Adventures of a Pythonista in Schemeland/27 Previous Topic   Next Topic Topic: The Adventures of a Pythonista in Schemeland/26


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us