My previous post led me to this library, which appears to solve the coarse-grained parallelism problem quite elegantly.
You can find the library here, which was written by Vitalii Vanovschi, a Russian chemist now doing graduate work at USC. It appears he created the library to serve his own computational needs, and designed it to be simple so his colleagues could use it.
Parallel Python is based on a functional model; you submit a function to a "job server" and then later fetch the result. It uses processes (just like I requested in my previous post) and IPC (InterProcess Communication) to execute the function, so there is no shared memory and thus no side effects.
The pp module will automatically figure out the number of processors available and by default create as many worker processes as there are processors.
You provide the function, a tuple of that function's arguments, and tuples with the dependent functions and modules that your function uses. You can also provide a callback function called when your function completes. Here's what the syntax looks like:
job_server = pp.Server() # Uses number of processors in system
f1 = job_server.submit(func1, args1, depfuncs1, modules1)
f2 = job_server.submit(func1, args2, depfuncs1, modules1)
f3 = job_server.submit(func2, args3, depfuncs2, modules2)
# Retrieve results:
r1 = f1()
r2 = f2()
r3 = f3()
What's even more interesting is that Vitalii has already solved the scaling problem. If you want to use a network of machines to solve your problem, the change is relatively minor. You start an instance of the Parallel Python server on each node machine:
Submitting jobs and getting results is the same as before, thus switching from multicores to a cluster of computers is virtually effortless. Notice that it transparently handles the problem of distributing code to the remote machines. It was not clear, however, whether ppserver.py automatically makes use of multiple cores on the node machines, but you would think so.
This library allows you to stay within Python for everything you're doing, although you can easily do further optimizations by writing time-critical sections in C, C++, Fortran, etc. and effortlessly and efficiently linking to them using Python 2.5's ctypes.
This is an exciting development for anyone doing parallel processing work, and something I want to explore further once my dual-core machine comes online.
Hi, I'm a little new to this whole concept... when I mentioned Erlang to a friend, he informed me of Stackless Python, which seems to have a fair history in this sort of thing, and is used for projects such as the EVE online game.
finally, there are many people that use this module so to use multiple cores effectively. I think it performs well, is stable enough and it is up to the person who is in charge to make it part of Python's distribution.
Just imagine this: With this solution in the STANDARD library, who will continue to fight for the GIL anymore ??
Now THAT is an impressive piece of work. Thank you very much for the link. I really love the approach of mirroring the existing threading module. Makes it very easy to use. Has all the nice features of the threading module like locks and such too. wow.
I think this kind of architecture - using a "space" to integrate eterogenous services written in different languages - is at the core of the GigaSpaces infrastracture (and what is called "Space based computing). Talking about Python, maybe NetWorkSpaces for Python http://nws-py.sourceforge.net/ is of some interest; it is developed by Scientific Computing Associates (http://www.lindaspaces.com), the pristine Linda spin-off by Carriero and Gelernter.
Sadly I now realize that for a rounded library you would also need to cater for those cases where you have very little data to exchange between jobs and good job control is paramount. You might as well embed the Sun Grid Engine as a library!