Weblogs Forum - The fate of reduce() in Python 3000

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Weblogs Forum
The fate of reduce() in Python 3000

119 replies on 8 pages. Most recent reply: Dec 18, 2018 1:39 AM by Claudemir Moliane

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 119 replies on 8 pages [ « | 1 ... 2 3 4 5 6 7 8 | » ]

Oren Tirosh

Posts: 3
Nickname: orentirosh
Registered: Feb, 2005

Re: The fate of reduce() in Python 3000

Posted: Mar 23, 2005 11:21 PM

The map, filter and reduce function can easily be implemented in a few lines of python code. It would be nice if Python 3000 had a standard library module (implemented in Python) where they can be found:

from compatibility import map, filter, reduce

Lambda, however, is syntax so it can not be easily "imported". But we do "import" syntax for things like:

from __future__ import generators

So how about:

from __past__ import lambdas

I agree that lambda and the functional builtins don't quite have the Python spirit, but backward compatibility should not be taken lightly. I think this approach makes it clear that the feature is not recommended for new code but helps maintain compatibility with existing codebase.

Elizabeth Wiethoff

Posts: 89
Nickname: ewiethoff
Registered: Mar, 2005

Re: Syntax for "filter only" list comprehensions

Posted: Mar 26, 2005 7:11 PM

> [x in list if x > 0]
> (which is currently undefined) be exactly equal to:
> [x for x in list if x > 0]

Python's list comprehension syntax nicely parallels mathematical set theory syntax. Let's say we are interested in the positive integers: P = { n | n ε Z and n > 0 } where 'ε' is the symbol that looks like an epsilon and means "element of" or "in". Notice that 'n' appears three times in this set syntax. Although Python is a fun, versatile language, it's not about to generate all the integers, Z. So let's say that Z is a short Python list: Z = [-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5]. Then in Pythonic list comprehension syntax we have P = [n for n in Z if n > 0]. Again, 'n' appears three times.

Ka-Ping Yee would like to eliminate the first 'n' from Python's syntax when the list comprehension applies a predicate--an 'if'--a filter. The first 'n', however, is hardly redundant when an interesting function or mapping is also applied. Suppose we are interested in positive cubes: C = { n<sup>3</sup> | n ε Z and n > 0 } where 'n<sup>3</sup>' means "n raised to the 3rd power". In Python C = [pow(n, 3) for n in Z if n > 0]. Here the first 'n' appears in a function, 'pow(n, 3)'. Yet n's first appearance in the P list comprehension is also in a function, 'n', which is merely a very simple function, a simple mapping.

Python's list comprehension syntax doesn't _need_ to parallel the syntax of set theory. Yet I think it would be a shame if Python's syntax were no longer to do so. Why? Because of some "Aha!" moments I've had while reading math. Last year while I was learning Python, I suddenly grokked list comprehensions and Python's syntax while reading about axiomatic set theory sans math notation. Then last month, with my Python background, I suddenly appreciated math notation (and Python) while reading a book about sets and groups.

Python's map() and filter() might get dropped because Python doesn't _need_ to make Lispers and FP fans happy. Yet I'd hate to see Python's list comprehension syntax changed because Python doesn't _need_ to make math lovers happy.

Alexander Belopolsky

Posts: 3
Nickname: alex
Registered: Mar, 2005

Re: The fate of reduce() in Python 3000

Posted: Mar 27, 2005 7:17 PM

>>> ''.join(['a','b'])
'ab'

>>> sum([[1],[2]],[])
[1, 2]

Eduard Hiti

Posts: 1
Nickname: kitkat
Registered: Mar, 2005

Re: The fate of reduce() in Python 3000

Posted: Mar 28, 2005 5:16 AM

There might be ways to keep the cake (anonymous functions) and eat it too (removing the lambda keyword).

I'm currently studying the Mozart environment/Oz language and there is the concept of "nesting marker" which allows turing a statement into an expression.

From the documentation:
"The nesting marker $ turns any statement into an expression. The expressions value is what is at the position indicated by the nesting marker."

This makes following Oz code fragments defining the procedure Id equivalent:

proc {Id X} X end <-> Id = proc {$ X} X end

The naive translation of this example to Python would be

id = def $(x): x

A slightly better example:

filter(def $(x): x.endswith("stuff"), strings)

Just a thought, Eduard

Alexander Belopolsky

Posts: 3
Nickname: alex
Registered: Mar, 2005

Re: The fate of reduce() in Python 3000

Posted: Mar 28, 2005 8:14 PM

I agree that reduce is not the most intuitive name,
but it is better than `foldr' :-)

I would suggest to add another optional argument
to sum if reduce has to go:

sum(sequence, start=0, add=operator.add)

I think it is fairly intuitive that the add argument
is intended to replace `+' in a sum. Although
equivalent to reduce, this form is less prone to
abuse because users are not likely to want to replace
`+' with a non-associative operator.

Another alternative is `accumulate' as in C++ STL,
but I think reusing `sum' is better.

Similarly, I think that lambda can stay in Python 3000
if better syntax is found. Since functions are first class
objects in Python, I don't see why Python 3000 could not
have a display form for anonymous functions.

For example

x,y -> x+y

could be a natural alternative to lambda(x,y): x+y

Another concise syntax is used by the K language:

{[x,y] x+y}

which may also be a natural choice since Python already
uses {} in the display form of dictionaries.

Gheorghe Milas

Posts: 1
Nickname: gheorghe
Registered: Apr, 2005

Re: The fate of reduce() in Python 3000

Posted: Apr 5, 2005 2:20 PM

I would like to see real inline functions and I would not mind using something like:


add = def _(x,y): return x+y
inc = def _(y=1):
    return def _(x): 
        return x + y

Actually so many return's kind of irritates my taste but what I'm suggesting is _ which we already use and in my honest opinion is better then inline def or something like that.

David G

Posts: 1
Nickname: davidlg
Registered: Apr, 2005

Re: The fate of reduce() in Python 3000

Posted: Apr 7, 2005 8:21 PM

I agree with all of these changes in some form except getting rid of lambdas, a.k.a. anonymous inline functions.

If you can build a list or dictionary out of parts inline, why not a function? Imagine if people had to come up with names for their data structures every time. No more calling foo([bar]), you have to name that one-item list on a separate line!

I can understand objections to the name "lambda", however, which is a greek letter and means nothing to someone unfamiliar with lambda calculus. Why not just change it to something like "anon"?

-- David

Matthew Russell

Posts: 1
Nickname: horizon5
Registered: Apr, 2005

Re: The fate of reduce() in Python 3000

Posted: Apr 10, 2005 10:16 PM

i used to use map and filter a lot, but since
list expr's were introduced i've never needed them.
so +1 for removing them.
i've never been able to grok reduce, +1 for removing that also,

lambda readable_in_most_cases: False (imho)

Tim Nash

Posts: 11
Nickname: tim99
Registered: Apr, 2005

Re: The fate of reduce() in Python 3000

Posted: Apr 13, 2005 4:59 PM

Please keep map() reduce(). I was actually hoping that map and reduce would evolve into something more. I've noticed that there are papers which now point to google's use of map/reduce as an important shift in the development of distributed applications. See the paper from microsoft research/berkley/et. al "Scientific Data Management in the Coming Decade"
http://66.102.7.104/search?q=cache:OGIip8iD7xwJ:www.cs.berkeley.edu/~dtliu/pubs/sci_datamgmt_TR-2005-10.doc+google+map+reduce&hl=en
It would be "ironic" if python dropped map/reduce and C# put it in!

If you are interested in reading how google use map/reduce , check out the paper on line:
http://labs.google.com/papers/mapreduce.html

BTW, google loves to hire python programmers, does anyone know if they are using python's map reduce in development?

Guido van van Rossum

Posts: 359
Nickname: guido
Registered: Apr, 2003

Re: The fate of reduce() in Python 3000

Posted: Apr 13, 2005 5:58 PM

I've read the Google MapReduce paper and it bears no relationship to Python's map and reduce.

Michael Nedzelsky

Posts: 1
Nickname: michaeln
Registered: Apr, 2005

Re: The fate of reduce() in Python 3000

Posted: Apr 14, 2005 2:45 PM

I have a list of some records. With lambda I can do:

my_list.sort(key = lambda x: x.name)
# or
my_list.sort(key = lambda x: x.age)
# or
my_list.sort(key = lambda x: x.address)

Can I sort list by different keys very easy without lambda?

Tim Nash

Posts: 11
Nickname: tim99
Registered: Apr, 2005

Re: The fate of reduce() in Python 3000

Posted: Apr 14, 2005 3:09 PM

Thanks for the response Guido. You say that you are mostly expecting objections from functional programmers, but I'm not a functional programmer. Actually I'm a biologist/MBA type (let the arrows fly!) but in addition to all the other great things about python, it has also opened my eyes to FP techniques. This blog brings a couple of other things to mind:
1) does having map/reduce in python inspire programmers to come up with innovative solutions? Is python an important glue between the FP world and the IP world?
2) could python's map/reduce evolve to give similiar functionality as google's? Pythons reduce() could reduce to a single value in some contexts and perhaps to a single key associated with a list in a google like context?
3) python makes imperative programming accessible, can it help make functional programming accessible?
4) can python help make parallel programming accessible?

From the google article:
"Our use of a functional model with user specified map and reduce operations allows us to parallelize large computations easily and to use re-execution as the primary mechanism for fault tolerance."
"This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system."

from Microsoft article:
"From our perspective, the key aspect of Google Map-Reduce is that it applies thousands of processors and disks to explore large datasets in parallel. That system has a very simple data model appropriate for the Google processing, but we imagine it could evolve over the next decade to be quite general."

Whatever you decide, I'll still be using Python. It has made programming easier and more fun for me. I just hope python 3000 allows me to play in the important sandboxes of the future. If I have to "import goopy" to use map/reduce it is no big deal, but I would prefer python evolved the other way and made FP easier.

Guido van van Rossum

Posts: 359
Nickname: guido
Registered: Apr, 2003

Re: The fate of reduce() in Python 3000

Posted: Apr 14, 2005 4:10 PM

> 1) does having map/reduce in python inspire programmers to
> come up with innovative solutions?

I doubt that the presence/absence of map() and reduce() as builtins makes a difference. These built-ins are trivially rewritten using pure Python code; the enabling feature is that Python treats all callables (not just functions or bound methods) as first-class objects. These days map() is almost always rewritten as a list comprehension or generator expression. reduce() is just a simple for loop away; it saves a little typing but costs a lot more time to read for many of us (myself included) when used with a non-standard callable.

> Is python an important
> glue between the FP world and the IP world?

I kind of doubt it -- Python's extremely dynamic nature is quite the opposite from what you see in the FP world, where the parallellism is exploited because the compiler knows exactly what you are trying to accomplish. In Python the compiler can't know anything -- that's pretty much in the language definition.

> 2) could python's map/reduce evolve to give similiar
> functionality as google's? Pythons reduce() could reduce
> to a single value in some contexts and perhaps to a single
> key associated with a list in a google like context?

I very much doubt it (see above for rationale). But of course (and I wouldn't be surprised if Google had already done this) it's easy enough to write a Python version of Google's MapReduce system -- it just wouldn't use the built-in map() and reduce() functions.

> 3) python makes imperative programming accessible, can it
> help make functional programming accessible?

Unlikely beyond some very trivial cases, see above.

> 4) can python help make parallel programming accessible?

Ditto.

Jamie Turner

Posts: 2
Nickname: jamwt
Registered: Apr, 2005

Re: The fate of reduce() in Python 3000

Posted: Apr 21, 2005 3:24 PM

List comprehensions seem nice for the most common cases, but I still think sometimes filter() and map() look cleaner.. particularly map() for some things.

Also, how do you cleanly achieve lists that are a function of multiple sequences with list comprehensions? How do you do this, for example:

>>> map(lambda x, y: (x, y, x*y), range(6), range(6))
[(0, 0, 0), (1, 1, 1), (2, 2, 4), (3, 3, 9), (4, 4, 16), (5, 5, 25)]

Oh yes, and lambda.. ouch. We need some kind of other syntax for anonymous functions, then--the event driven network programmers will be hung out to dry.

I wonder, because things like map(), filter(), and lambda are doing very good things for a lot of people, and they cause no significant harm to those who don't use them, what's the benefit of removing them?

Especially with things like partial application coming through the door...

map(partial(math.pow, 2), seq)

Python has always been a great place to scratch little functional itches when they seem appropriate...

Jamie Turner

Posts: 2
Nickname: jamwt
Registered: Apr, 2005

Re: The fate of reduce() in Python 3000

Posted: Apr 21, 2005 3:39 PM

Also..

>>> def f1():
... t = time.time()
... a = [x for x in range(1000000) if x]
... print time.time() - t
...
>>> def f2():
... t = time.time()
... a = filter(op.truth, range(1000000))
... print time.time() - t
...
>>> f1()
0.319629907608
>>> f2()
0.172744035721

Flat View: This topic has 119 replies on 8 pages [ « | 2 3 4 5 6 7 8 | » ]

Previous Topic

Next Topic