The Artima Developer Community
Sponsored Link

Weblogs Forum
Python Decorators II: Decorator Arguments

19 replies on 2 pages. Most recent reply: Feb 9, 2012 6:47 PM by Jake Griffin

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 19 replies on 2 pages [ 1 2 | » ]
Bruce Eckel

Posts: 874
Nickname: beckel
Registered: Jun, 2003

Python Decorators II: Decorator Arguments (View in Weblogs)
Posted: Oct 19, 2008 3:42 PM
Reply to this message Reply
Summary
The decorator mechanism behaves quite differently when you pass arguments to the decorator.
Advertisement

(This is part II of a series that will become a chapter in Python 3 Patterns & Idioms; you can find part I here).

Review: Decorators without Arguments

In part I, I showed how to use decorators without arguments, primarily using classes as decorators because I find them easier to think about.

If we create a decorator without arguments, the function to be decorated is passed to the constructor, and the __call__() method is called whenever the decorated function is invoked:

class decoratorWithoutArguments(object):

    def __init__(self, f):
        """
        If there are no decorator arguments, the function
        to be decorated is passed to the constructor.
        """
        print "Inside __init__()"
        self.f = f

    def __call__(self, *args):
        """
        The __call__ method is not called until the
        decorated function is called.
        """
        print "Inside __call__()"
        self.f(*args)
        print "After self.f(*args)"

@decoratorWithoutArguments
def sayHello(a1, a2, a3, a4):
    print 'sayHello arguments:', a1, a2, a3, a4

print "After decoration"

print "Preparing to call sayHello()"
sayHello("say", "hello", "argument", "list")
print "After first sayHello() call"
sayHello("a", "different", "set of", "arguments")
print "After second sayHello() call"

Any arguments for the decorated function are just passed to __call__(). The output is:

Inside __init__()
After decoration
Preparing to call sayHello()
Inside __call__()
sayHello arguments: say hello argument list
After self.f(*args)
After first sayHello() call
Inside __call__()
sayHello arguments: a different set of arguments
After self.f(*args)
After second sayHello() call

Notice that __init__() is the only method called to perform decoration, and __call__() is called every time you call the decorated sayHello().

Decorators with Arguments

Now let's modify the above example to see what happens when we add arguments to the decorator:

class decoratorWithArguments(object):

    def __init__(self, arg1, arg2, arg3):
        """
        If there are decorator arguments, the function
        to be decorated is not passed to the constructor!
        """
        print "Inside __init__()"
        self.arg1 = arg1
        self.arg2 = arg2
        self.arg3 = arg3

    def __call__(self, f):
        """
        If there are decorator arguments, __call__() is only called
        once, as part of the decoration process! You can only give
        it a single argument, which is the function object.
        """
        print "Inside __call__()"
        def wrapped_f(*args):
            print "Inside wrapped_f()"
            print "Decorator arguments:", self.arg1, self.arg2, self.arg3
            f(*args)
            print "After f(*args)"
        return wrapped_f

@decoratorWithArguments("hello", "world", 42)
def sayHello(a1, a2, a3, a4):
    print 'sayHello arguments:', a1, a2, a3, a4

print "After decoration"

print "Preparing to call sayHello()"
sayHello("say", "hello", "argument", "list")
print "after first sayHello() call"
sayHello("a", "different", "set of", "arguments")
print "after second sayHello() call"

From the output, we can see that the behavior changes quite significantly:

Inside __init__()
Inside __call__()
After decoration
Preparing to call sayHello()
Inside wrapped_f()
Decorator arguments: hello world 42
sayHello arguments: say hello argument list
After f(*args)
after first sayHello() call
Inside wrapped_f()
Decorator arguments: hello world 42
sayHello arguments: a different set of arguments
After f(*args)
after second sayHello() call

Now the process of decoration calls the constructor and then immediately invokes __call__(), which can only take a single argument (the function object) and must return the decorated function object that replaces the original. Notice that __call__() is now only invoked once, during decoration, and after that the decorated function that you return from __call__() is used for the actual calls.

Although this behavior makes sense -- the constructor is now used to capture the decorator arguments, but the object __call__() can no longer be used as the decorated function call, so you must instead use __call__() to perform the decoration -- it is nonetheless surprising the first time you see it because it's acting so much differently than the no-argument case, and you must code the decorator very differently from the no-argument case.

Decorator Functions with Decorator Arguments

Finally, let's look at the more complex decorator function implementation, where you have to do everything all at once:

def decoratorFunctionWithArguments(arg1, arg2, arg3):
    def wrap(f):
        print "Inside wrap()"
        def wrapped_f(*args):
            print "Inside wrapped_f()"
            print "Decorator arguments:", arg1, arg2, arg3
            f(*args)
            print "After f(*args)"
        return wrapped_f
    return wrap

@decoratorFunctionWithArguments("hello", "world", 42)
def sayHello(a1, a2, a3, a4):
    print 'sayHello arguments:', a1, a2, a3, a4

print "After decoration"

print "Preparing to call sayHello()"
sayHello("say", "hello", "argument", "list")
print "after first sayHello() call"
sayHello("a", "different", "set of", "arguments")
print "after second sayHello() call"

Here's the output:

Inside wrap()
After decoration
Preparing to call sayHello()
Inside wrapped_f()
Decorator arguments: hello world 42
sayHello arguments: say hello argument list
After f(*args)
after first sayHello() call
Inside wrapped_f()
Decorator arguments: hello world 42
sayHello arguments: a different set of arguments
After f(*args)
after second sayHello() call

The return value of the decorator function must be a function used to wrap the function to be decorated. That is, Python will take the returned function and call it at decoration time, passing the function to be decorated. That's why we have three levels of functions; the inner one is the actual replacement function.

Because of closures, wrapped_f() has access to the decorator arguments arg1, arg2 and arg3, without having to explicitly store them as in the class version. However, this is a case where I find "explicit is better than implicit," so even though the function version is more succinct I find the class version easier to understand and thus to modify and maintain.

In the next installment I'll show a practical example of decorators -- a build system created atop Python -- and in the final installment we'll look at class decorators.


phil jones

Posts: 3
Nickname: nooranch
Registered: Oct, 2008

Re: Python Decorators II: Decorator Arguments Posted: Oct 19, 2008 6:40 PM
Reply to this message Reply
You might be interested in some of the discussion over here : http://stackoverflow.com/questions/215689/why-python-decorators-rather-than-closures

Decorators with and without arguments are so different that it's quite confusing. The best way I find to think about it is that with no arguments, you're just calling a decorating function on the function. With arguments you're calling a decorator "factory" that's going to return a decorator on the function.

Kay Schluehr

Posts: 302
Nickname: schluehk
Registered: Jan, 2005

Re: Python Decorators II: Decorator Arguments Posted: Oct 20, 2008 2:02 AM
Reply to this message Reply
> You might be interested in some of the discussion over
> here :
> http://stackoverflow.com/questions/215689/why-python-decora
> tors-rather-than-closures
>
> Decorators with and without arguments are so different
> that it's quite confusing. The best way I find to think
> about it is that with no arguments, you're just calling a
> decorating function on the function. With arguments you're
> calling a decorator "factory" that's going to return a
> decorator on the function.

I'm not sure here is the space to elaborate on this but IMO the correct perspective on decorators needs to evolve from a simple observation about the relationship of two functions F and F1 where the function F1 can be considered as a subroutine that is called by F.

The nature of the coupling between F and F1 can be twofold: either F1 is passed to F explicitly as a parameter which requires passing around two functions F and F1 everywhere F shall be called in application code or F is a closure referring to F1 which means that the coupling is fixed. So we have either full coupling of full decoupling. Both have their costs.

In a first step a decorator we can consider a decorator as a factory function G that produces closures F depending on F1.

def G(F1):
def F(*args, **kwd):
...
F1(x,y,...)
...
return F

The function returned when decorating F1 by G can now be considered as a fixed calling context F parametrized by F1 and reassigned to F1. But this means that we have now coupled a fixed calling context F to F1. We have succeeded in liberating F from a fixed subroutine F1 but not in liberating F1 from a fixed calling context F.

In a next step we parametrize also the calling contexts F by using arbitrary parameters a, b, c... Those become closure variables of our original decorator G:

def H(a,b,c,...):
def G(F1):
def F(*args, **kwd):
...
F1(x,y,...)
...
return F
return G

In practice this will establish a meaningful variation of F not something completely arbitrary like binding F1 to two entirely unrelated contexts F and F' depending on some parameter(s).

So in essence a decorator is a functional design pattern that is about modularization of functions ( or callables to use a more general term prevalent in Python programming ) and adjusting the coupling of functions.

Alan Green

Posts: 104
Nickname: alang
Registered: Jun, 2003

Re: Python Decorators II: Decorator Arguments Posted: Oct 20, 2008 2:57 AM
Reply to this message Reply
Please ignore this if you find it impertinent. I'm blaming the glass of wine I had with dinner.

I can't help but think that this explanation is over-complicated.

A decorator expression consists of an identifier and an optional, possibly empty argument list. If the identifier has an argument list - empty or not - the value referenced by the identifier is called to obtain the decorator. If the identifier does not have an argument list, the value referenced is used as-is.

Another way to say it is that, if there are parens after the identifier, then the identifier is called to return the actual decorator. If there are no parens, then the identifier is the decorator.

Another factor complicating the explanation is the class implementation of decorators. While I understand that it looks neater, I find keeping track of the object lifecycle complicates the picture a little to much. Decorators are defined in terms of callables, and the straight function implementation is a better match to that.

In summary, I think it's important to explain the distinction between the following two cases as a stepping stone to explaining how to use arguments with decorators.


@X
def f():
...

- the value of X is used as the decorator


@X()
def f():
...

- the result of calling X is used as the decorator

Michele Simionato

Posts: 222
Nickname: micheles
Registered: Jun, 2008

Re: Python Decorators II: Decorator Arguments Posted: Oct 20, 2008 3:33 AM
Reply to this message Reply
FWIW, version 2.3 of my decorator module (http://www.phyast.pitt.edu/~micheles/python/documentation.html) added specific
support for decorator factories. Your example would read as follows:

<pre>
from decorator import decorator

@decorator
class decoratorWithArguments(object):
def __init__(self, arg1, arg2, arg3):
print "Inside __init__()"
self.arg1 = arg1
self.arg2 = arg2
self.arg3 = arg3

def call(self, f, *args):
print "Inside call()"
print "Decorator arguments:", self.arg1, self.arg2, self.arg3
res = f(*args)
print "After f(*args)"
return res
</pre>

Eli Courtwright

Posts: 14
Nickname: eliandrewc
Registered: Jan, 2006

Re: Python Decorators II: Decorator Arguments Posted: Oct 20, 2008 10:37 AM
Reply to this message Reply
This is nit-picky, but is there any reason why you're using camelCase for your variable and method names instead of separating_with_underscores?

Bruce Eckel

Posts: 874
Nickname: beckel
Registered: Jun, 2003

Re: Python Decorators II: Decorator Arguments Posted: Oct 20, 2008 1:15 PM
Reply to this message Reply
> This is nit-picky, but is there any reason why you're
> using camelCase for your variable and method
> names instead of separating_with_underscores?

Actually, that's an important issue that I have to decide for the Python book. Aesthetically I prefer camelcasing; it feels more OO (comes from Smalltalk, I believe) while underscore separation feels like C to me.

On the other hand, Python has historically used underscores and I think PEP8 says that's coding standard.

I definitely need a consistent coding standard for the book. Of course these kinds of discussions have no way to win -- I just have to be the BDFL for the book and decide.

(One argument might be that camelcasing could appeal more to Java programmers. One could also say that camelcasing emphasizes a break between pre-Python 3 and Python 3.)

Eli Courtwright

Posts: 14
Nickname: eliandrewc
Registered: Jan, 2006

Re: Python Decorators II: Decorator Arguments Posted: Oct 21, 2008 1:10 PM
Reply to this message Reply
If I recall correctly, the Python standard library is moving more and more towards separating_with_underscores in Python 3.0. It still won't be perfectly PEP8 compliant, but they're making it a little closer, based on some threads I've seen on the dev mailing lists.

So I'd personally like to see more Pythonic code formatting, but I don't think very many Python programmers will care all that much. Perhaps Java programmers would care more, in which case that might be the better move.

Michele Simionato

Posts: 222
Nickname: micheles
Registered: Jun, 2008

Re: Python Decorators II: Decorator Arguments Posted: Oct 21, 2008 1:27 PM
Reply to this message Reply
> So I'd personally like to see more Pythonic code formatting, but I don't think very many Python programmers will care all that much.

I think instead that most Pythonistas will care a lot about consistency with the official style guide. Many projects are already migrating to the convention with underscores.
I personally would not recommend a recent text book using a deprecated code style.

Maarten Hazewinkel

Posts: 32
Nickname: terkans
Registered: Jan, 2005

Re: Python Decorators II: Decorator Arguments Posted: Oct 22, 2008 2:08 PM
Reply to this message Reply
> Another way to say it is that, if there are parens after
> the identifier, then the identifier is called to return
> the actual decorator. If there are no parens, then the
> identifier is the decorator.

Thanks Alan, that makes the whole thing make sense again to me.

Berco Beute

Posts: 72
Nickname: berco
Registered: Jan, 2002

Re: Python Decorators II: Decorator Arguments Posted: Oct 23, 2008 4:30 PM
Reply to this message Reply
> I think instead that most Pythonistas will care a lot
> about consistency with the official style guide. Many
> projects are already migrating to the convention with
> underscores.

I'm not sure about that. I'm seeing more and more camelCase Python code on the web. For me personally the underscores are one of the very few official style guides that I ignore simply because I find underscores pretty hard to read and camelCase saves me one keystroke (a_b = 4 strokes, aB=3 strokes).

I thinks camelCase should be officially supported.

2B

Matthew Wilson

Posts: 4
Nickname: mw44118
Registered: Sep, 2008

Re: Python Decorators II: Decorator Arguments Posted: Oct 25, 2008 1:28 PM
Reply to this message Reply
I gotta chime in with support for Michele Simionato's decorator module.

It preserves the function signature and it makes dealing with decorator arguments MUCH more straightforward. Otherwise, you end up with a function that returns a function that returns a function, and that's just ridiculous. Throw that on top of some metaclasses, and you're guaranteed to trap python in an elitist prison for eternity :)

Also, I think the term "decorator factories" is sometimes more helpful when talking about decorators that take arguments. Strictly speaking, the arguments don't even go to a decorator. They go to a function that then returns a decorator.

I'm giving a talk at PyWorks all about decorators, and my material is available here: http://scratch.tplus1.com/decoratortalk/. I'll be updating a few sections soon, now that 2.6 is standard and I welcome feedback.

Finally, when I read PEP8, I don't really see a hard rule about underscores vs camel-case. I see it as stating a preference. The first section emphasizes that stylistic consistency is just not that important. I tend to use camel-case for anything that starts with a upper-case letter for the first letter, like exceptions and classes, and I use underscores for everything else, but I don't really care one way or the other.

Bruce Eckel

Posts: 874
Nickname: beckel
Registered: Jun, 2003

Re: Python Decorators II: Decorator Arguments Posted: Oct 25, 2008 2:33 PM
Reply to this message Reply
Thanks for all the links and suggestions. I've added these to "Further Reading" in the Decorators chapter.

I'm trying to sort out all the questionable and confusing issues before posting the first (very raw) version of the book. The naming issue is important because things need to be consistent throughout the book, so I'm inclined to fall back on PEP8 as much as possible.

Except, as I mentioned before, for camelcasing. As Matthew said, it seems to be a preference rather than a hard-and fast rule, and camelcasing *feels* more like OO to me, as if we are emphasizing the design here (which I want to do) and putting less focus on the C-ish nature that can be expressed in Python.

Even then, it doesn't always work. A decorator like the ``@rule()`` I'm creating for the build system is implemented as a class, but just seems like it should be lowercased because we are not focusing on its class-ness but rather its behavior as a decorator.

Ian Bicking

Posts: 849
Nickname: ianb
Registered: Apr, 2003

Re: Python Decorators II: Decorator Arguments Posted: Oct 25, 2008 2:54 PM
Reply to this message Reply
I think you may be seeing more camel case Python because there are more people using Python coming from Java, etc. Among people who have been programming Python for some time camel case is quite uncommon -- at least in the communities I spend time in. The only exceptions in web-related libraries that I can think of are some of the Zope things, which are simply old APIs and I don't think reflect new Zope APIs.

A few years ago when there wasn't a clear convention in Python I found it quite annoying -- half the libraries would use one style, the other half the other style, and code that mixed the two would look rather ugly. And everytime you started code you'd have to decide which set of libraries the code was more closely aligned to, so as to keep the style in sync. I'm happy that's no longer the case, and that's no longer the case because we have one accepted form: underscores. If our one accepted form had been camel case that would have been fine too. But the ambiguity is really quite annoying.

So I think you should reconsider camel case, so we can keep everyone on the same page with respect to this style issue. There's nothing more OO about one or the other, they are just two ways of laying out words. There is a historical aspect I suppose -- underscores are more dominant in C and Unix, camel case in Smalltalk and Java, and camel case with leading capital letters in Microsoft APIs. But that's just history.

Michele Simionato

Posts: 222
Nickname: micheles
Registered: Jun, 2008

Re: Python Decorators II: Decorator Arguments Posted: Oct 25, 2008 11:35 PM
Reply to this message Reply
> A decorator like the
> ``@rule()`` I'm creating for the build system is
> implemented as a class, but just seems like it should be
> lowercased because we are not focusing on its class-ness
> but rather its behavior as a decorator.

Whereas regular classes should be CamelCase, classes used to implement decorator should be lowercase. Perhaps this is not explicitly stated in PEP 8, but it is the Pythonic thing to do, since there are lots of precedents, like staticmethod, classmethod and property. So you are doing the right thing there. The not-so-good thing is to use camelCase for method names.

Flat View: This topic has 19 replies on 2 pages [ 1  2 | » ]
Topic: Scrum will never be the same Previous Topic   Next Topic Topic: WebSockets: A Glimpse of the Future


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us