The Artima Developer Community
Sponsored Link

Weblogs Forum
The fate of reduce() in Python 3000

117 replies on 8 pages. Most recent reply: Jul 15, 2010 8:30 AM by Jeffry Mcdowell

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 117 replies on 8 pages [ 1 2 3 4 5 6 ... 8  | » ]
Guido van van Rossum

Posts: 359
Nickname: guido
Registered: Apr, 2003

The fate of reduce() in Python 3000 (View in Weblogs)
Posted: Mar 10, 2005 5:11 PM
Reply to this message Reply
Summary
I received an email from a compatriot lamenting the planned demise of reduce() and lambda in Python 3000. After a few exchanges I think even he agreed that they can go. Here's a summary, including my reasons for dropping lambda, map() and filter(). I expect tons of disagreement in the feedback, all from ex-Lisp-or-Scheme folks. :-)
Advertisement

About 12 years ago, Python aquired lambda, reduce(), filter() and map(), courtesy of (I believe) a Lisp hacker who missed them and submitted working patches. But, despite of the PR value, I think these features should be cut from Python 3000.

Update: lambda, filter and map will stay (the latter two with small changes, returning iterators instead of lists). Only reduce will be removed from the 3.0 standard library. You can import it from functools.

I think dropping filter() and map() is pretty uncontroversial; filter(P, S) is almost always written clearer as [x for x in S if P(x)], and this has the huge advantage that the most common usages involve predicates that are comparisons, e.g. x==42, and defining a lambda for that just requires much more effort for the reader (plus the lambda is slower than the list comprehension). Even more so for map(F, S) which becomes [F(x) for x in S]. Of course, in many cases you'd be able to use generator expressions instead.

Why drop lambda? Most Python users are unfamiliar with Lisp or Scheme, so the name is confusing; also, there is a widespread misunderstanding that lambda can do things that a nested function can't -- I still recall Laura Creighton's Aha!-erlebnis after I showed her there was no difference! Even with a better name, I think having the two choices side-by-side just requires programmers to think about making a choice that's irrelevant for their program; not having the choice streamlines the thought process. Also, once map(), filter() and reduce() are gone, there aren't a whole lot of places where you really need to write very short local functions; Tkinter callbacks come to mind, but I find that more often than not the callbacks should be methods of some state-carrying object anyway (the exception being toy programs).

So now reduce(). This is actually the one I've always hated most, because, apart from a few examples involving + or *, almost every time I see a reduce() call with a non-trivial function argument, I need to grab pen and paper to diagram what's actually being fed into that function before I understand what the reduce() is supposed to do. So in my mind, the applicability of reduce() is pretty much limited to associative operators, and in all other cases it's better to write out the accumulation loop explicitly.

There aren't a whole lot of associative operators. (Those are operators X for which (a X b) X c equals a X (b X c).) I think it's just about limited to +, *, &, |, ^, and shortcut and/or. We already have sum(); I'd happily trade reduce() for product(), so that takes care of the two most common uses. The bitwise operators are rather specialized and we can put fast versions in a library module if there's demand; for shortcut booleans I have the following proposal.

Let's add any() and all() to the standard builtins, defined as follows (but implemented more efficiently):

    def any(S):
        for x in S:
            if x:
               return True
        return False

    def all(S):
        for x in S:
            if not x:
               return False
        return True

Combine these with generator expressions, and you can write things like these::

    any(x > 42 for x in S)     # True if any elements of S are > 42
    all(x != 0 for x in S)     # True if all elements if S are nonzero

This will mostly give us back the quantifiers from ABC: ABC's IF EACH x IN s HAS p(x): becomes if all(p(x) for x in s):, its IF SOME x IN s HAS p(x): becomes if any(p(x) for x in s):, and IF NO x IN s HAS p(x): becomes if not any(p(x) for x in s):. (Except that in ABC, the variables would be bound and usable inside the IF block; if you need that in Python you'll have to write an explicit for loop and use a break statement.)

I expect that even if you disagree with dropping reduce(), you will agree that adding any() and all() is a good idea -- maybe even for Python 2.5!


Adam Collard

Posts: 1
Nickname: acollard
Registered: Mar, 2005

Re: The fate of reduce() in Python 3000 Posted: Mar 10, 2005 6:10 PM
Reply to this message Reply
+1 for any() and all() in Python! It was only last week that I wrote almost that exact code in a library of 'helper functions' for an internal software product with a Python API.

With list comprehensions and generator expressions, Python's functional programming builtins seem to grind a little, I don't think their removal in Python 3000 would be a sore loss

Jimmy Retzlaff

Posts: 4
Nickname: jretz
Registered: Feb, 2004

Re: The fate of reduce() in Python 3000 Posted: Mar 10, 2005 6:50 PM
Reply to this message Reply
I won't miss reduce() at all and I'd certainly welcome any() and all(). I've always thought I'd miss map() and filter() quite a bit and lambda slightly less. After further thought I can write my own map/filter/lambda for my typical use cases.

Where I still use map/filter over listcomps/genexps is in the degenerative case of making one call to a predefined function on each item in the sequence. It just seems clearer to me to read:

dirs = filter(os.path.isdir, os.listdir('.'))

instead of:

dirs = [pathname for pathname in os.listdir('.') if os.path.isdir(pathname)]

If I want something as complex as an expression then I switch to listcomps or genexps. filter and map can be implemented very easily in just a few lines so they'll be easy to replace if I really do miss them.

As for lambda... I do a lot of GUI programming and to reduce dependencies between different parts of my application I often pass callables around (with any parameters curried to them) instead of values. The callable can be stored in order to retrieve the value at any time. lambda reduces the penalty on the caller for this level of indirection when the full power isn't needed. For example to set the label of a button, a function is passed and the button stores that function and periodically calls it to determine if it's label has changed. For the simple case of a static label, a call looks like this:

button.setlabel(lambda: 'Click Me!')

If the function is ever more complex than returning a value/variable then I use a nested function. So I can easily have:
def wrapper(value):
def f():
return value
return f

Now I can set my button label with:

button.setlabel(wrapper('Click Me!'))

So, after further thought, I'm on board.

Tony Meyer

Posts: 1
Nickname: tonymeyer
Registered: Mar, 2005

Re: The fate of reduce() in Python 3000 Posted: Mar 10, 2005 7:18 PM
Reply to this message Reply
There was discussion about functions like any() and all() just recently on c.l.p and python-dev, e.g.:

http://mail.python.org/pipermail/python-dev/2005-February/051556.html

What about ensuring the proposed changes to itertools (and/or the consuming iterators module) go ahead, which would include any() and all(), for 2.5? They could then easily move to builtins later.

Brett C.

Posts: 4
Nickname: drifty
Registered: Jan, 2005

Re: The fate of reduce() in Python 3000 Posted: Mar 10, 2005 7:53 PM
Reply to this message Reply
I have to admit that even I was not totally sure about closures being able to be **exactly** equivalent to a lambda until a second ago (didn't think that default argument for closures were re-evaluated each time they were returned; learn something new every day).

While being able to fit stuff on a single line is fun for toy examples, it obviously is not usually the cleanest solution (I offer my one-liner for removing duplicates in an iterable that I recently posted to python-dev as an example; futzing with short-circuiting of 'or' and 'and' is never pretty). So doing away with lambda is, in my mind, a definite at this point.

And as for adding 'any', 'all', and 'product', I say go for it. And as for the loss of 'map', 'filter', and 'reduce', good riddance!

Time for me to go update PEP 3000.

Jimmy Retzlaff

Posts: 4
Nickname: jretz
Registered: Feb, 2004

Re: The fate of reduce() in Python 3000 Posted: Mar 10, 2005 7:57 PM
Reply to this message Reply
I just came across a slightly expanded lambda use in my GUI code. It's the same kind of thing as the button label above, but this time the caller wants to bind it to a member variable and have the button automatically pick up changes to that member variable. Today, leveraging nested scopes:

button.setlabel(lambda: self.nextaction)

Without lambda, still leveraging nested scopes:

def nextaction(): return self.nextaction
button.setlabel(nextaction)


Not too bad, but not quite as convenient as it is now and having a name is redundant. Here's an attempt at a reusable wrapper:
def memberwrapper(self, attr):
def f():
return getattr(self, attr)

button.setlabel(memberwrapper(self, 'nextaction'))

Close, but again it doesn't read as cleanly (assuming nested scopes are second nature).

Is there any interest in a further confined lambda like contruct that takes no parameters, but just works as a way to defer/repeat evaluation of an expression in a certain scope?

Jonas Galvez

Posts: 3
Nickname: jgalvez
Registered: Mar, 2005

Re: The fate of reduce() in Python 3000 Posted: Mar 10, 2005 8:10 PM
Reply to this message Reply
+1 for all() and any().

filter(os.path.isdir, os.listdir('.'))

would become

any(os.path.isdir(i) for i in os.listdir('.'))

very nice.

Sridhar Ratna

Posts: 3
Nickname: srid
Registered: Jan, 2005

any() and all() are redundant Posted: Mar 10, 2005 9:11 PM
Reply to this message Reply
Why not,

if True in (x > 42 for x in S):

instead of "any" and why not

if not False in (x > 42 for x in S):

instead of "all"?

Phillip J. Eby

Posts: 28
Nickname: pje
Registered: Dec, 2004

Re: The fate of reduce() in Python 3000 Posted: Mar 10, 2005 9:53 PM
Reply to this message Reply
Ouch. I won't miss reduce at all, and I can live without map and filter (although they're faster than a listcomp if the function is a builtin), but lambda I will definitely miss. It's just too convenient for working with delayed calculation idioms.

I think that maybe you're missing the fact that sometimes you need to pass a filtering or transformation function *into* something that might then be using it in a comprehension. Such functions can go beyond a simple itemgetter or attrgetter, but still fall well short of deserving a 'def' suite on a separate line.

So, if you get rid of lambda in Py3K, can we at least repurpose backquotes to allow quoting expressions inline? Or maybe allow an inline 'def' instead of lambda?

maxim khesin

Posts: 229
Nickname: xamdam
Registered: Mar, 2005

Re: The fate of reduce() in Python 3000 Posted: Mar 10, 2005 10:56 PM
Reply to this message Reply
By gosh! They will have to completely remove David Mertz's articles from IBM-developworks!

Bob Ippolito

Posts: 255
Nickname: etrepum
Registered: Nov, 2003

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 1:04 AM
Reply to this message Reply
> I just came across a slightly expanded lambda
> use in my GUI code. It's the same kind of thing as the
> button label above, but this time the caller wants to bind
> it to a member variable and have the button automatically
> pick up changes to that member variable. Today, leveraging
> nested scopes:
>
> button.setlabel(lambda: self.nextaction)
>
> Without lambda, still leveraging nested
> scopes:
>
> def nextaction(): return self.nextaction
> button.setlabel(nextaction)

>
> Not too bad, but not quite as convenient as it is now and
> having a name is redundant. Here's an attempt at a
> reusable wrapper:
>
def memberwrapper(self, attr):
> def f():
> return getattr(self, attr)
>
> button.setlabel(memberwrapper(self, 'nextaction'))

> Close, but again it doesn't read as cleanly (assuming
> nested scopes are second nature).

This is probably evil, but you could also do:

@button.setlabel
def f():
return getattr(self, attr)

Tim Lesher

Posts: 2
Nickname: leshert
Registered: Mar, 2003

Re: any() and all() are redundant Posted: Mar 11, 2005 1:38 AM
Reply to this message Reply
> Why not,
> if True in (x > 42 for x in S):
> instead of "any" and why not
> if not False in (x > 42 for x in S):
> instead of "all"?

Because "any" and "all" have shortcut semantics (they return as soon as they can determine the final result).

Sakesun

Posts: 10
Nickname: sakesun
Registered: Aug, 2003

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 2:11 AM
Reply to this message Reply
> +1 for all() and any().
>
> filter(os.path.isdir, os.listdir('.'))
>
> would become
>
> any(os.path.isdir(i) for i in
> os.listdir('.'))

>
> very nice.


I don't think that's how to use "any".
Probably you mean this

(x for x in os.listdir('.') if os.path.isdir(x))

Kay Schluehr

Posts: 302
Nickname: schluehk
Registered: Jan, 2005

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 2:30 AM
Reply to this message Reply
> I expect that even if you disagree with dropping
> reduce(),

Yes. Even in the case of a simlpe addition of two objects that are not numbers applying sum() is currently no replacement:


class X:
def __add__(self,other):
return "sum of two elements"

>>> a,b = X(),X()
>>> a+b
sum of two elements

>>> reduce(lambda x,y:x+y, (a,b))
sum of two elements

>>> sum((a,b))
Traceback (most recent call last):
File "<interactive input>", line 1, in ?
TypeError: unsupported operand type(s) for +: 'int' and 'instance'


So reduce() will obviously return into my codebase even it is not __builtin__ by default anymore, lambda will be trashed ( and replaced by operator.<method> ) and i have to implement it by myself which is no big deal:


def reduce(func,args):
res = func(args[0],args[1])
for item in args[2:]:
res = func(res,item)
return res



>you will agree that adding any() and all() is a
> good idea -- maybe even for Python 2.5!

Yes.

Regards Kay

Ka-Ping Yee

Posts: 24
Nickname: ping
Registered: Dec, 2004

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 3:20 AM
Reply to this message Reply
I ran into the above-mentioned problem with sum() over non-numbers just recently. I think sum() should just add the elements in the sequence instead trying to add 0 all the time when i'm not specifying a starter value. In other words, these should work:

sum(['foo']) -> 'foo'
sum(['a', 'b']) -> 'ab'
sum([[2], [3]]) -> [2, 3]

The current implementation of sum() fails on all these examples. Its behaviour seems unnecessarily restricted to me.

Flat View: This topic has 117 replies on 8 pages [ 1  2  3  4  5  6 | » ]
Topic: Adding Optional Static Typing to Python Previous Topic   Next Topic Topic: Reviewable, Runnable Specifications Ensure Software Quality


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us