The feedback from my previous article has been very helpful, and it has raised more questions about RPC technology. I've also included notes about converting Python libraries from Python 2 to Python 3.
I've used XML-RPC in the past. In the article, I used JSON-RPC because that's the only thing that Go implements in the standard library (someone on the Go team clearly thought JSON-RPC was valuable). But in my researches, there seems to be a "lost civilization" feel to JSON-RPC. Lots of activity in the 2006-2008 timeframe, then a plague seems to have wiped everyone out. For example, the last entry by Roland Koebler, the proposer for JSON-RPC 2.0 reads: "But the (in my opinion) much improved JSON-RPC 2.0 is stable and will probably be officially released soon." The last update on that page was 2008, so "soon" seems not to have happend. I emailed Koebler asking him about this, and after a protracted delay, he replied:
"My Python JSON-RPC implementation currently supports something "like"
JSON-RPC 2.0. I created it according to a JSON-RPC 2.0 draft (which I
wrote, by the way ;)) before JSON-RPC 2.0 was officially released.
It works flawlessly on several servers out there.
"Unfortunately I did not have the time since then to check if it really
meets the JSON-RPC 2.0 specification or add the missing functionality.
But I'm planning to release an updated version of my library soon."
The various other JSON-RPC libraries I've come across don't seem to have any updates or corrections after around this time, either.
I recently learned about Thrift. There is Go support (which I haven't yet tested). The bug fixes seem to be very active, and the appearance of Thrift seems to coincide with the apparent loss of interest in JSON-RPC. Is that in fact what happened? Did Thrift come along and, promoted by Facebook, effectively wipe out the other RPC strategies like XML-RPC and JSON-RPC? (Or is there something else I should be looking at?).
There are a number of other RPC technologies; Google's "protocol buffers" has been suggested, however that is only a data format and doesn't include a way to make remote calls.
Then there were the heady days of CORBA (Common Object Request Broker Architecture), which began when the original C++ standards committee was meeting. There may have been earlier attempts to "connect everything to everything else" but this was my first exposure and it seemed promising indeed: We'd be able to save legacy code by connecting it to modern code, and we'd be able to get the best of all worlds by combining languages. CORBA was shaping up to be the ultimate answer.
A committee was formed. The Object Management Group (OMG, and the irony is not lost). The OMG then wanted to be an organization on its own, so it needed funding. It was decided that to be on the committee you needed to be (A) a corporation and (B) pay an annual fee that only a fairly successful corporation could afford (the C++ committee, in contrast, was accessible enough that independents like me could participate). The rest you can imagine: it took a very long time and all the obscure problems of the corporations on the committee had to be solved. The result was huge and complex, and didn't get much use (but surprisingly, some aspects of CORBA are still in use if you search in enough dark corners). The Tower of Babel was a project that got waylaid when everyone started speaking different languages (management issues were not so different back then, it seems) and the CORBA project lost its way because no one could agree on what the "common language" was, so they just threw in everything they could think of.
It would have been interesting if they had started with small prototypes and tried them out to see what worked, what was too complicated, etc. But in those days you designed something and then waited for awhile (often years, in the case of C++ features) to see how it came out.
Thanks for any insights. I'd like to figure out the best (current) approach to RPC.
I don't know if this is always possible, but so far I've been able to take several pieces of Python 2 code (I'm running 2.6) and modify it so that it will also work with Python 3 (I've got 3.2).
As an example, we can convert Roland Koebler's JSON-RPC library by going to his page and downloading from his link. Make a copy of this file in a local directory to experiment on.
To see whether it works as-is, you can try "running library module as a script" like this:
python -m jsonrpc
With Python 3, this will produce errors. Fortunately, Python 3 includes the 2to3 utility. If you run this on the file without using flags, you'll get the diffs that it wants to apply. But if you use the -w flag, it will apply the diffs:
2to3 -w jsonrpc.py
Now python -m jsonrpc will discover that the simplejson module is missing. This was required because Koebler wrote his module for an earlier version of Python that did not contain a json module, but now (since 2.6) there's one in the standard library. So we can edit the file and change the line:
import json as simplejson
Now python -m jsonrpc completes successfully -- but we're not done yet, because for some reason 2to3 misses the buffer translations (perhaps it is unable to parse for them -- a place where static type checking wins). When we try this program:
rpc = jsonrpc.ServerProxy(jsonrpc.JsonRpc10(),
for i in range(10):
print(rpc.RPCFunc.Echo("hello " + str(i)))
(With the server from the previous post running) we get the complaint:
'str' does not support the buffer interface
This means we have to follow the line numbers and change line 404 from:
data = self.loads(string)
data = self.loads(string.decode())
and line 780 from:
self.s.sendall( string )
self.s.sendall( string.encode() )
And now it works, for both versions of Python. Note that Koebler's code is posted with an open-source license, so you can republish the modified code if you want.
One thing that was different between jsonrpc.py and the jsonclient.py code from the previous article:: jsonrpc.py opens and closes a connection for each call, while jsonclient.py keeps the connection open for all the calls. You'll notice a speed difference from the overhead of opening and closing all those connections (Perhaps jsonrpc.py has a "keep open" option; I didn't look).
"I'd like to figure out the best (current) approach to RPC."
IMHO, the best approach to general-purpose RPC is to avoid it. The whole idea that there is one grand solution that connects all clients to all services, involves no understanding of the underlying service semantics, and solves all modeling needs is just bankrupt.
Instead, use bespoke APIs which have a well-defined call-and-response document definition, whether that be in XML or JSON. See, e.g., Netflix, GData, etc. Such APIs are tailored to a particular problem space, address the authentication and authorization challenges in a method appropriate to their needs, and have REST URIs which map well to the objects they represent.
In contrast, the workings of "RPC" protocols are often opaque, with complex multiplexed URIs, use of some "platform-independent" datatypes which must in reality be mapped to language-specific representations, etc.