The Artima Developer Community
Sponsored Link

Weblogs Forum
The fate of reduce() in Python 3000

117 replies on 8 pages. Most recent reply: Jul 15, 2010 8:30 AM by Jeffry Mcdowell

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 117 replies on 8 pages [ « | 1 2 3 4 5 6 7 ... 8  | » ]
Ka-Ping Yee

Posts: 24
Nickname: ping
Registered: Dec, 2004

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 3:37 AM
Reply to this message Reply
Advertisement
Hurrah for any() and all()! I've often done similar things, though i hadn't noticed the wonderful synergy with generators. Excellent.

I wouldn't miss reduce(), but i would definitely miss map(). It seems painfully tedious to have to write

[int(word) for word in line.split()]

instead of simply

map(int, line.split())

If filter() has to go, could we make its list comprehension less burdensome? Having to type three copies of a superfluous identifier is a bit much:

[item for item in list if item > 0]

All the extra words in there distract from the meaning of the line, which is really just "filter(list, > 0)". The reader has to read and parse eight chunks of information in order to absorb three.

What do you think of allowing:

[item in list if item > 0]

It's not ambiguous, since it's currently a syntax error.

Michael Hudson

Posts: 8
Nickname: mwh
Registered: Jul, 2003

nit! Posted: Mar 11, 2005 4:02 AM
Reply to this message Reply
You claim that it's only sensible to pass associative operations into reduce() (which I agree with) and then include commutativity in your definition of associativity (which isn't necessary for a reduce() call to be comprehensible).

Anthony Tarlano

Posts: 9
Nickname: tarlano
Registered: Jun, 2003

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 4:17 AM
Reply to this message Reply
It is truely a sad day for Python! I think the loss of lambda, filter and map will be terrible.

Why remove these builtins for the small minority of people who don't know now use them and thus don't care. I and others use them all the time.

We should add builtins to the language and remove them when they are not used, but these are used.

If the test for removing builtins is whether there is another what to do something, then I just have to disagree, because there is always another way.

Anthony

Paul Moore

Posts: 1
Nickname: pmoore
Registered: Dec, 2004

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 5:58 AM
Reply to this message Reply
I've no problem losing reduce, map or filter. I like any and all. But I'd find the loss of lambda a pain. I understand that a nested function is equivalent, but sometimes having to invent a name detracts from the readability, rather than adding to it.

As Philip Eby pointed out, lambda is often used to pass an "unevaluated expression" into a function - in theory this is the same as an "anonymous function", but in practice, the intention is different.

Compiler support for "placeholder" variables (or Philip's suggestion for backquotes) would cover this sort of use - something like:

getattr(_1,'attr','default')

as an equivalent to

lambda obj: getattr(obj,'attr','default')

I'm not sure how much this gains over lambda, but I do think it gains over

def foo(obj):
getattr(obj,'attr','default')
foo

(where I can easily imagine ending up with a name as meaningless as 'foo'...)

John Crichton

Posts: 1
Nickname: crichton
Registered: Mar, 2005

Re: any() and all() are redundant Posted: Mar 11, 2005 6:37 AM
Reply to this message Reply
> > Why not,
> > if True in (x > 42 for x in S):
> > instead of "any" and why not
> > if not False in (x > 42 for x in S):
> > instead of "all"?
>
> Because "any" and "all" have shortcut semantics (they
> return as soon as they can determine the final result).


Even if "any" and "all" are not strictly redundant I think that keeping the core language small would be a valuable design goal.

Christos Georgiou

Posts: 7
Nickname: tzotzioy
Registered: Mar, 2005

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 8:07 AM
Reply to this message Reply
Alex Martelli two years ago (in thread http://groups-beta.google.com/group/comp.lang.python/browse_frm/thread/adb13fc92cf85eab/c2e916087143c9cd ) voted "no" for inclusion of any and all predicates in the standard library; I don't know if he would vote "yes" for inclusion in __builtin__, or if he changed his mind since.

In that thread I proposed these as classes, and the functionality proposed by Guido was implemented in the __bool__ method (no shortcutting in that code, but that can be changed, esp. if the objects are implemented in C). That implementation allowed for a poor-man's gencomp à la

All(iterable)+1

yielding the iterable incremented by one etc. I don't know if such functionality is needed anymore, but I thought pointing to that thread was very relevant to the topic.

In any case, I surely vote +1 for both, even if only for bool tests, since they tend to make code more concise and readable.

TAL

Posts: 1
Nickname: tal00r
Registered: Mar, 2005

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 8:09 AM
Reply to this message Reply
I'd like to see reduce() and filter() go too. But I use map and lambda a lot. I did a quick grep through one of my programs for them. Some examples:

key, value = map(str.strip, line.split('=', 1))

I guess that becomes:

key, value = line.split('=', 1)
key = key.strip()
value = value.strip()

def provides(context, type): return type in map(str, context.targets)

Becomes:

def provides(context, type): return type in (str(x) for x in context.targets)

There are loads of lambdas. The most popular is for getters and setters:

handlers[option] = (
lambda: str(spin.get_value()),
lambda: spin.set_value(option.int_value))

Throwing away callback arguments is popular too:

register(lambda src, cond: self.read_ready())

entry.connect('activate', lambda widget: self.save_to_file_in_entry())

Also, aren't list comprehensions rather slow? I have a game that does its redraw using something like:

map(apply, plot, tiles, locations)

This was hugely faster than using a for loop (plot is a C function in pygame, tiles is the state of the board and locations is a pregenerated (X, Y) for each tile).

Michael Chermside

Posts: 17
Nickname: mcherm
Registered: Jan, 2004

Syntax for "filter only" list comprehensions Posted: Mar 11, 2005 8:30 AM
Reply to this message Reply
I'd just like to draw attention to Ka-Ping's interesting suggestion above for simplifying the syntax of list comprehensions that just do filtering. He proposes that

[x in list if x > 0]

(which is currently undefined) be exactly equal to:

[x for x in list if x > 0]

I'm not certain that this is "worth it", but I have to admit, every time I write out one of these list comprehensions that repeats its variable 3 times I feel like I'm being overly repetative and redundant, and his alternative is really quite readable.

Of course, I'm still expecting that a closer look will reveal that the parsing is ambiguous, or that it won't work for generator expressions, or some other impediment like that... but I haven't noticed what it is yet. Does anyone else see the lurking problem?

Christos Georgiou

Posts: 7
Nickname: tzotzioy
Registered: Mar, 2005

Re: any() and all() are redundant Posted: Mar 11, 2005 8:41 AM
Reply to this message Reply
> Even if "any" and "all" are not strictly redundant I think
> that keeping the core language small would be a valuable
> design goal.

There's a difference between core language and std library. `all' and `any' are example recipes in the itertools documentation, so they would be added to the itertools module, logically (at least if we are supposed to have opinions consistent in time...)

Christos Georgiou

Posts: 7
Nickname: tzotzioy
Registered: Mar, 2005

Re: Syntax for "filter only" list comprehensions Posted: Mar 11, 2005 9:45 AM
Reply to this message Reply
[snip]
> [x in list if x > 0]
> (which is currently undefined) be exactly equal to:
> [x for x in list if x > 0]
[snip]

> Of course, I'm still expecting that a closer look will
> reveal that the parsing is ambiguous, or that it won't
> work for generator expressions, or some other impediment
> like that... but I haven't noticed what it is yet. Does
> anyone else see the lurking problem?

It depends on the parser. When it sees

x in list

does it immediately understand

LOAD_<STH> x
LOAD_<STH> list
COMPARE_OP 6

?

Beni Cherniavsky

Posts: 1
Nickname: cben
Registered: Mar, 2005

any/all not equivallent to and/or reduction -- and they shouldn't be Posted: Mar 11, 2005 9:51 AM
Reply to this message Reply
My reply is mostly in a I-propose-something-but-it-doesn't-work-so-I-don't-propose-it mode but this is to motivate a (hopefully better) proposal, so bear with me :-)

> Let's add any() and all() to the standard builtins,
> defined as follows (but implemented more efficiently):
>

> def any(S):
> for x in S:
> if x:
> return True
> return False
>
> def all(S):
> for x in S:
> if not x:
> return False
> return True
>


This is not equivallent to reducing with the short-cirquiting or/and operator. These functions always return True or False, while a chain of or/and returns the first dissenting item or the last one. So my first sentiment was to redefine them like this:


def any(S):
for x in S:
if x:
return x # x, not True
return x # last item

def all(S):
for x in S:
if not x:
return x # x, not False
return x # last item


But now I can't now think of any good example where this change will be useful. The best ones are of this kind:

any(d.get(key) for d in dicts)
# fallback between dictionaries, assumes true keys

any((n % i == 0 and i) for i in range(2, n))
# first divisor

All such uses amount to getting the first item that satisfies some condition. For this to work, it requires the item to have true boolean value, which is not always part of the desired condition (this is a variation on the a and b or c buf). It sounds more robust to move the condition into an if clause and just get the first item, e.g.:

(d[key] for d in dicts if key in d).next()
# good but raises StopIteration if n is prime

(i for i in range(2, n) if n % i == 0).next()


Also, note that all() can give nothing more interesting than a boolean result because it finds the first false item and Python has just a few false values. I can't imagine a situation where you will want to know whether it was 0, an empty string or an empty dictionary... You can only attempt to abuse it to get the last item but then you are not writing what you mean.

Finally, the above versions crash on an empty sequence (x is never set). If we stick to and/or behavior of always returning one of the arguments, there can be no consistent behaviour for an empty sequence.

Conclusion: any() and all() should stay as Guido proposed - always returning True or False. But in combination with comrehensions with an if clause, two additional functions would useful: first() and last(). They should probably take an optional second argument - the default value to return for empty sequences. Probably they are too marginal for builtins and should go into itertools.

Nick Coghlan

Posts: 13
Nickname: ncoghlan
Registered: Dec, 2004

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 10:07 AM
Reply to this message Reply
> Why remove these builtins for the small minority of people
> who don't know now use them and thus don't care. I and
> others use them all the time.

Which is the problem. Everyone else then gets stuck reading them.

Getting rid of reduce() - yay! Never the clearest way to write things - write a function with a loop in it instead so normal people can read the code.

filter() - don't care much about this one.

map() - I'll miss this one a bit. "map(str, seq)" is still more readable (and faster) than "[str(x) for x in seq]". Still, I'm willing to trade that for never again seeing "map(lambda x: <expr>, seq)"

Thomas Heller

Posts: 2
Nickname: theller
Registered: Mar, 2005

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 11:00 AM
Reply to this message Reply
I'm with those people that don't want to loose lambda. Here's some use cases I have (all from unittests):

self.assertRaises(Exception, lambda: [].prop = 42)
self.assertRaises(Exception, lambda: my_func("spam"))

Compare this to:

self.assertRaises(Exception, setattr, [], "prop", 42)
self.assertRaises(Exception, my_func, "spam")

Thomas Heller

Posts: 2
Nickname: theller
Registered: Mar, 2005

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 11:02 AM
Reply to this message Reply
Blech, lambda: [].prop = 42 is a SyntaxError. Shame on me - but you get the idea ;-)

Guido van van Rossum

Posts: 359
Nickname: guido
Registered: Apr, 2003

Re: The fate of reduce() in Python 3000 Posted: Mar 11, 2005 11:29 AM
Reply to this message Reply
Here's my take on the key issues brought up (this is a copy of an email I sent to python-dev, where an independent discussion of the same issues is happening):

Alternative names anytrue(), alltrue(): before I posted to my blog I
played with these names (actually anyTrue(), allTrue(), anyFalse(),
allFalse()). But I realized (1) any() and all() read much better in
their natural context (an if statement), and there's no confusion
there; (2) anyFalse() and allFalse() are redundant, so there's no need
to distinguish between anyTrue() and anyFalse(). (To the person who
wondered why we default to True: because 'if' tests for True, duh. :-)

Whether to always return True and False or the first faling / passing
element? I played with that too before blogging, and realized that the
end case (if the sequence is empty or if all elements fail the test)
can never be made to work satisfactory: picking None feels weird if
the argument is an iterable of bools, and picking False feels weird if
the argument is an iterable of non-bool objects.

Whether to use each() and some() instead of all() and any(), to
preserve variable namespace: each() in particular (and some() to some
extent) emphasizes the individual element, while all() emphasizes the
whole set. As long as we accept that the return value should not
return one of the arguments elements, I prefer emphasizing the group.
The namespace argument doesn't weigh much in my mind; there's no
backwards incompatibility, and there are plenty of builtins that are
routinely reused as variables (e.g. str, file, id, type) without
apparently affecting readability. Context helps a lot!

Whether to segregate these into a separate module: they are really a
very small amount of syntactic sugat, and I expect that in most cases,
instead of importing that module (which totally makes me lose my
context while editing) I would probably just write the extra line that
it takes to get the same effect with an explicit for loop and if
statement.

What worries me a bit about doing a PEP for this simple proposal is
that it might accidentally have the wrong outcome: a compromise that
can carry a majority rather than the "right" solution because nobody
could "sell" it. Yet, if people really feel strongly that there are
more issues that need to be discussed, and they think it's worth their
time, let someone stand up now to own the PEP and guide the
discussion. But personally, I think it's more efficient if Raymond
just checks in his code now. :-)

BTW I definitely expect having to defend removing
map/filter/reduce/lambda with a PEP; that's much more controversial
because it's *removing* something and hence by definition breaking
code. The PEP, in addition to making a really strong case for each of
the removals, will have to deal with a deprecation period, possibly
moving the functions to a "functional programming" module, etc.

PS in the blog responses, a problem with sum() was pointed out --
unless you use the second argument, it will only work for numbers. Now
that string concatenation is apparently O(N) rather than O(N**2) (is
that right, Raymond?) maybe this could be revised.

Flat View: This topic has 117 replies on 8 pages [ « | 1  2  3  4  5  6  7 | » ]
Topic: Adding Optional Static Typing to Python Previous Topic   Next Topic Topic: Reviewable, Runnable Specifications Ensure Software Quality


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us