An incident on python-dev today made me appreciate (again) that there's more to language design than puzzle-solving. A ramble on the nature of Pythonicity, culminating in a comparison of language design to user interface design.
Some people seem to think that language design is just like solving a puzzle. Given a set of requirements they systematically search the solution space for a match, and when they find one, they claim to have the perfect language feature, as if they've solved a Sudoku puzzle. For example, today someone claimed to have solved the problem of the multi-statement lambda.
But such solutions often lack "Pythonicity" -- that elusive trait of a good Python feature. It's impossible to express Pythonicity as a hard constraint. Even the Zen of Python doesn't translate into a simple test of Pythonicity.
In the example above, it's easy to find the Achilles heel of the proposed solution: the double colon, while indeed syntactically unambiguous (one of the "puzzle constraints"), is completely arbitrary and doesn't resemble anything else in Python. A double colon occurs in one other place, but there it's part of the slice syntax, where a[::] is simply a degenerate case of the extended slice notation a[start:stop:step] with start, stop and step all omitted. But that's not analogous at all to the proposal's lambda <args>::<suite>. There's also no analogy to the use of :: in other languages -- in C++ (and Perl) it's a scoping operator.
And still that's not why I rejected this proposal. If the double colon is unpythonic, perhaps a solution could be found that uses a single colon and is still backwards compatible (the other big constraint looming big for Pythonic Puzzle solvers). I actually have one in mind: if there's text after the colon, it's a backwards-compatible expression lambda; if there's a newline, it's a multi-line lambda; the rest of the proposal can remain unchanged. Presto, QED, voila, etcetera.
But I'm rejecting that too, because in the end (and this is where I admit to unintentionally misleading the submitter) I find any solution unacceptable that embeds an indentation-based block in the middle of an expression. Since I find alternative syntax for statement grouping (e.g. braces or begin/end keywords) equally unacceptable, this pretty much makes a multi-line lambda an unsolvable puzzle.
And I like it that way! In a sense, the reason I went to considerable length describing the problems of embedding an indented block in an expression (thereby accidentally laying the bait) was that I wanted to convey the sense that the problem was unsolvable. I should have known my geek audience better and expected someone to solve it. :-)
The unspoken, right brain constraint here is that the complexity introduced by a solution to a design problem must be somehow proportional to the problem's importance. In my mind, the inability of lambda to contain a print statement or a while-loop etc. is only a minor flaw; after all instead of a lambda you can just use a named function nested in the current scope.
But the complexity of any proposed solution for this puzzle is immense, to me: it requires the parser (or more precisely, the lexer) to be able to switch back and forth between indent-sensitive and indent-insensitive modes, keeping a stack of previous modes and indentation level. Technically that can all be solved (there's already a stack of indentation levels that could be generalized). But none of that takes away my gut feeling that it is all an elaborate Rube Goldberg contraption.
Mathematicians don't mind these -- a proof is a proof is a proof, no matter whether it contains 2 or 2000 steps, or requires an infinite-dimensional space to prove something about integers. Sometimes, the software equivalent is acceptable as well, based on the theory that the end justifies the means. Some of Google's amazing accomplishments have this nature inside, even though we do our very best to make it appear simple.
And there's the rub: there's no way to make a Rube Goldberg language feature appear simple. Features of a programming language, whether syntactic or semantic, are all part of the language's user interface. And a user interface can handle only so much complexity or it becomes unusable. This is also the reason why Python will never have continuations, and even why I'm uninterested in optimizing tail recursion. But that's for another installment.
I think the flaw in these lambda replacements/augmentations is that they lack sufficient creativity, and they lack proper consideration of the real problems.
Why do you need statements in lambdas? There's nothing that requires exactly that. There are use cases that can be improved using it. But it is by no means the only way to do that.
The most compelling use cases I've seen -- Twisted-style programming among them -- do not require these kind of extended lambdas. The status quo is awkward, but it's for lack of some solution, not any particular solution.
In this case I've seen some proposals for a way to use a function, then define the function later. The problem I see is not that you have to give the function a name (names aren't that hard), but that you have to name the function and define it before it is clear what you are using it for. In the case of single-use functions this makes the code seem entirely backwards. Like you are setting up this magic and mysterious environment full of functions that do small things, and then BOOM! you put them all together to implement your algorithm. It's like a mathematical proof -- it seems magic and strange and deep, but that's only because it is a contortion of the author's original thinking process. The programmer wrote from the bottom up, and then when you read from the top down it doesn't make any sense.
I'm convinced there exist several good solutions for that problem that are still quite Pythonic.
I seem to recall you making noises at certain points about not being much of a designer or a UI person or something like that. I am always skeptical to hear that; the thinking underlying your post here is yet more evidence to me that you have a strong sense of the user perspective and that it is a key reason for the success of Python.
Yes, lambdas is a sugary construct, but so are list comprehensions (replace with a for-loop), generators(class + next func), etc. Of course the difference is you write more lines of code to implement the alternatives, which is the big difference with lambda. But I cannot underestimate the win in readability - lambdas move the callback much closer to the context. Plus, consider the fact that every Python guy who tried Ruby is whining about missing the lambdas (actually Ruby blocks are more than lambdas, but it's the anonymous functions they are whining about) ... I did not start it! I am not going to say that C# has them. Oops, I said it. I think it would be sad to see Python as the last kid on the block without a cool bike. And may be you are perfectly content having these souls gravitate to Ruby... I am just saying the problem should considered for it's non-intrinsic aspects, too.
I think it would be sad to see Python as the last kid on the block without a cool bike. And may be you are perfectly content having these souls gravitate to Ruby...
I don't like the thought of adding stuff to Python just because "language X has it." Surely somebody isn't going to switch to Ruby solely because of blocks, but even if they were, I'd rather not have Python cluttered with new, non-Pythonic-feeling features just to attract a few users.
> I seem to recall you making noises at certain points about > not being much of a designer or a UI person or something > like that. I am always skeptical to hear that; the > thinking underlying your post here is yet more evidence to > me that you have a strong sense of the user perspective > and that it is a key reason for the success of Python.
I have little sense of *graphical* design. I *do* have a strong sense for *usability* issues. "Designer" is a word that means too many things, hence the confusion.
> Plus, consider the fact that every Python guy who tried > Ruby is whining about missing the lambdas (actually Ruby > blocks are more than lambdas, but it's the anonymous > functions they are whining about) ... I did not start it! > Issue here is that some of the very reason why Ruby's block "work" (e.g. look sexy) are just impossible to use in Python, unless you want to reimplement Ruby on CPython that is (and if that's what you want, there is no point in using Python in the first place).
There are only two possible syntaxes to create something as powerful as blocks in python, one has been rejected already (and is extremely ugly and confusing) and the other one would require heaps of confusing ruby-style magic (especially to keep some kind of backward-compatibility with the existing constructs) and is not that readable (more than the first construct, but it's still very far from perfect, and even farther from pythonic)
def foo(func): return func(3)
foo(def x: return x*x )
def bar(func, value): return func(value)
sth = 3 bar(def x: out = x**sth return out , 5 # How the hell do I specify the other arguments with that kind of syntax? )
The second one would require a magic argument (let's say that it's an argument prefixed by "~" in the function definition), behold the fuglyness:
def foo(~func): return func(3)
def myfunction(x): return x**5 print foo(myfunction) # current syntax
# Magic, if an anonymous function is defined behind a function call and the function has a ~ argument, then the ~ argument is populated with the anonymous function. print foo() def (x): return x**5
def bar(~func, value): return func(value)
sth = 3 bar(5) def (x): return x*sth
Second syntax is extremely close to the way Ruby blocks work (post-defined anonymous functions), but it has two gigantic horrors:
First, it requires a "magical argument" (this allows the edition of existing function-using functions such as "map" in backward-compatible ways ), then the number of arguments you call your function with actually varies, and in a way that is much worse than the current self/cls arguments count variability.
Using the "blocks-like" syntax requires you to use one less argument than the regular python syntax for the very same function called almost the same way. Because of the magic.
Of course a placeholder argument could be put in place such as
bar(~, 5) def (x): return x
But that's even uglier and probably even more confusing (what's that ~ doing here?) and error prone.
Ruby's blocks work because it's ruby, no one has found any pythonic way (syntax) to use blocks-like anonymous functions in python, and until that is done I doubt Guido will even consider putting them in.
And as far as Guido is concerned, it seems that this quest is now branded foolish and uninterresting, fit only for the community's Don Quijotes (no offense meant)(and these are of course not Guido's words but merely how I see his position after the last thread on the subject)
Having toyed around with Ruby and blocks, I know how nicely they work, how cool they look, how gracefully they fit in the language. But Python's not ruby, something that works for ruby doesn't necessarily work for Python, and blocks don't. Everyone's time would probably be much better invested in creating useful, new, original features (== ripping Lisp or Smalltalk ;) )
 One could argue that the existing functions could not have the possibility to use the new feature, and use an argument at the end of the arguments list, as Ruby does with the final "&block", but consider the price: any function using this postfix block would be forbidden from using *args and probably **kwargs as well... that's a high price to pay for such a feature.
Thank you, this was an excellent summary (I do not wade onto python-dev often enough). I certainly understand the problem. I guess what I am hoping for is that smart people keep thinking about it. Perhaps this is unsolvable, as Guido sais, but I sure hope not ;)
As far as I can tell, PEP 343 is entirely unrelated to the Twisted-being-backwards problem that Ian is talking about. It could have been a solution, but the semantics of 'with' would have to have been radically different. For example, if:
with x as something: foo()
would have meant something like:
def anon(x): foo() something.__block__(anon)
then perhaps this could have been abused to work with Deferreds. As it is, though, the "with" block is still executed synchronously and therefore useless as a callback - unless I have missed something totally fundamental, and I would be overjoyed if I had. I'm patiently waiting for "when-blocks" that just explicitly support Deferreds :).
when deferredThing() -> foo: when otherDeferredThing() -> bar: doActualThing(foo, bar) except SomeError, se: handleErrorFromOther(se)
(arrows-as-asynchronous-operator idea shamelessly stolen from E)
> As far as I can tell, PEP 343 is entirely unrelated to the > Twisted-being-backwards problem that Ian is talking about.
Point taken. Try PEP 342 instead -- yield expressions and generator.send(value) make it possible to write a handler that does things in the normal order but is suspended while waiting for data. (I believe you can already do this with defer but Python 2.5 makes the syntax much nicer.)
> I think it would be sad to see Python as the last kid > on the block without a cool bike. And may be you are > perfectly content having these souls gravitate to Ruby...
> I don't like the thought of adding stuff to Python just > because "language X has it." Surely somebody isn't going > to switch to Ruby solely because of blocks, but even if > they were, I'd rather not have Python cluttered with new, > non-Pythonic-feeling features just to attract a few users.
A hybrid language of Python and Ruby should probably be named "Puby": Puby the language which is really the most simple to use language! Pubys design philosophy: Puby implements each feature of each simple language that makes the simple language simple to use. Pubys slogan: "Being simpler with Puby!" Of course Pubys colors are baby-blue and baby-pink.