This post originated from an RSS feed registered with Python Buzz
by Ian Bicking.
Original Post: Twisted and Threads
Feed Title: Ian Bicking
Feed URL: http://www.ianbicking.org/feeds/atom.xml
Feed Description: Thoughts on Python and Programming.
There seems to be some confusion about Twisted's asynchronous
programming, and the use of threads and blocking code.
As a quick primer, everything in Twisted runs in a single process.
Which means when one piece of code is running the entire server
process is dedicated to that code, and no other request can be
handled. Twisted handles this with a sort of event-driven
programming. When you have long-running code, you chop it up into
pieces where each piece is short, then you let the event processor
call those pieces in turn (the "reactor"
in Twisted terminology). These use something called a "Deferred"
and look something like:
def blocking_code():
val = init_val() # <-- not blocking
val = inner_blocking_code(val)
return val + 10
def non_blocking_code():
val = init_val()
d = inner_non_blocking_code(val)
d.addCallback(non_blocking_code_adder)
return d
def non_blocking_code_adder(val):
return val + 10
inner_non_block_code is a modified version of
inner_blocking_code, which uses events and returns a Twisted
Deferred object. Someday that deferred will produce a
value; when it does non_blocking_code_adder will be called.
If someone wants to use the "return" value of
non_blocking_code they will add another callback, and so on.
The call stack we've gotten used to is gone -- instead a series of
callbacks replaces it. (I'm only a Twisted tourist, so feel free to
correct me in comments if I've got this wrong)
This is a little weird, but there are advantages that I won't talk
about here (mostly performance and some people find concurrent
programming easier/safer with this style).
A big part of this example was the inner_blocking_code to
inner_non_blocking_code rewrite. That rewrite may very well
be hard to do. Or maybe that blocking code exists in a library you
don't control, and most Python libraries are blocking. Once this
occurs to people, a lot of people think Twisted is unusable in a lot
of situations -- and if you don't handle blocking code, your
Twisted server will be broken. At this point people sometimes
dismiss Twisted as being too limited. I was most recently reminded of
this by this
post, but the confusion is very common.
Let's say you can't refactor inner_blocking_code; then you
can do:
from twisted.internet import threads
def inner_non_blocking_code(val):
return threads.deferToThread(inner_blocking_code, val)
Twisted will then run inner_blocking_code in a different
thread, and when the function returns it will trigger the Deferred
that threads.deferToThread returns.
Threads in Twisted aren't great -- you've gone to all that trouble to
factor your program into an asynchronous style, but you still have to
run threads (which you may have been trying to avoid) -- but at least
it's possible. Twisted doesn't put up walls, even if it might
put up some hurdles.
Previous Topic
