The Artima Developer Community
Sponsored Link

Weblogs Forum
Decorators I: Introduction to Python Decorators

34 replies on 3 pages. Most recent reply: Sep 11, 2014 6:32 PM by Gary Mason

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
    Next Topic
Flat View: This topic has 34 replies on 3 pages [ 1 2 3 | » ]
Bruce Eckel

Posts: 874
Nickname: beckel
Registered: Jun, 2003

Decorators I: Introduction to Python Decorators (View in Weblogs)
Posted: Oct 18, 2008 3:21 PM
Reply to this message Reply
Summary
This amazing feature appeared in the language almost apologetically and with concern that it might not be that useful.
Advertisement

I predict that in time it will be seen as one of the more powerful features in the language. The problem is that all the introductions to decorators that I have seen have been rather confusing, so I will try to rectify that here.

(This series of articles will be incorporated into the open-source book Python 3 Patterns & Idioms).

Decorators vs. the Decorator Pattern

First, you need to understand that the word "decorator" was used with some trepidation, because there was concern that it would be completely confused with the Decorator pattern from the Design Patterns book. At one point other terms were considered for the feature, but "decorator" seems to be the one that sticks.

Indeed, you can use Python decorators to implement the Decorator pattern, but that's an extremely limited use of it. Python decorators, I think, are best equated to macros.

History of Macros

The macro has a long history, but most people will probably have had experience with C preprocessor macros. The problems with C macros were (1) they were in a different language (not C) and (2) the behavior was sometimes bizarre, and often inconsistent with the behavior of the rest of C.

Both Java and C# have added annotations, which allow you to do some things to elements of the language. Both of these have the problems that (1) to do what you want, you sometimes have to jump through some enormous and untenable hoops, which follows from (2) these annotation features have their hands tied by the bondage-and-discipline (or as Martin Fowler gently puts it: "Directing") nature of those languages.

In a slightly different vein, many C++ programmers (myself included) have noted the generative abilities of C++ templates and have used that feature in a macro-like fashion.

Many other languages have incorporated macros, but without knowing much about it I will go out on a limb and say that Python decorators are similar to Lisp macros in power and possibility.

The Goal of Macros

I think it's safe to say that the goal of macros in a language is to provide a way to modify elements of the language. That's what decorators do in Python -- they modify functions, and in the case of class decorators, entire classes. This is why they usually provide a simpler alternative to metaclasses.

The major failings of most language's self-modification approaches are that they are too restrictive and that they require a different language (I'm going to say that Java annotations with all the hoops you must jump through to produce an interesting annotation comprises a "different language").

Python falls into Fowler's category of "enabling" languages, so if you want to do modifications, why create a different or restricted language? Why not just use Python itself? And that's what Python decorators do.

What Can You Do With Decorators?

Decorators allow you to inject or modify code in functions or classes. Sounds a bit like Aspect-Oriented Programming (AOP) in Java, doesn't it? Except that it's both much simpler and (as a result) much more powerful. For example, suppose you'd like to do something at the entry and exit points of a function (such as perform some kind of security, tracing, locking, etc. -- all the standard arguments for AOP). With decorators, it looks like this:

@entryExit
def func1():
    print "inside func1()"

@entryExit
def func2():
    print "inside func2()"

The @ indicates the application of the decorator.

Function Decorators

A function decorator is applied to a function definition by placing it on the line before that function definition begins. For example:

@myDecorator
def aFunction():
    print "inside aFunction"

When the compiler passes over this code, aFunction() is compiled and the resulting function object is passed to the myDecorator code, which does something to produce a function-like object that is then substituted for the original aFunction().

What does the myDecorator code look like? Well, most introductory examples show this as a function, but I've found that it's easier to start understanding decorators by using classes as decoration mechanisms instead of functions. In addition, it's more powerful.

The only constraint upon the object returned by the decorator is that it can be used as a function -- which basically means it must be callable. Thus, any classes we use as decorators must implement __call__.

What should the decorator do? Well, it can do anything but usually you expect the original function code to be used at some point. This is not required, however:

class myDecorator(object):

    def __init__(self, f):
        print "inside myDecorator.__init__()"
        f() # Prove that function definition has completed

    def __call__(self):
        print "inside myDecorator.__call__()"

@myDecorator
def aFunction():
    print "inside aFunction()"

print "Finished decorating aFunction()"

aFunction()

When you run this code, you see:

inside myDecorator.__init__()
inside aFunction()
Finished decorating aFunction()
inside myDecorator.__call__()

Notice that the constructor for myDecorator is executed at the point of decoration of the function. Since we can call f() inside __init__(), it shows that the creation of f() is complete before the decorator is called. Note also that the decorator constructor receives the function object being decorated. Typically, you'll capture the function object in the constructor and later use it in the __call__() method (the fact that decoration and calling are two clear phases when using classes is why I argue that it's easier and more powerful this way).

When aFunction() is called after it has been decorated, we get completely different behavior; the myDecorator.__call__() method is called instead of the original code. That's because the act of decoration replaces the original function object with the result of the decoration -- in our case, the myDecorator object replaces aFunction. Indeed, before decorators were added you had to do something much less elegant to achieve the same thing:

def foo(): pass
foo = staticmethod(foo)

With the addition of the @ decoration operator, you now get the same result by saying:

@staticmethod
def foo(): pass

This is the reason why people argued against decorators, because the @ is just a little syntax sugar meaning "pass a function object through another function and assign the result to the original function."

The reason I think decorators will have such a big impact is because this little bit of syntax sugar changes the way you think about programming. Indeed, it brings the idea of "applying code to other code" (i.e.: macros) into mainstream thinking by formalizing it as a language construct.

Slightly More Useful

Now let's go back and implement the first example. Here, we'll do the more typical thing and actually use the code in the decorated functions:

class entryExit(object):

    def __init__(self, f):
        self.f = f

    def __call__(self):
        print "Entering", self.f.__name__
        self.f()
        print "Exited", self.f.__name__

@entryExit
def func1():
    print "inside func1()"

@entryExit
def func2():
    print "inside func2()"

func1()
func2()

The output is:

Entering func1
inside func1()
Exited func1
Entering func2
inside func2()
Exited func2

You can see that the decorated functions now have the "Entering" and "Exited" trace statements around the call.

The constructor stores the argument, which is the function object. In the call, we use the __name__ attribute of the function to display that function's name, then call the function itself.

Using Functions as Decorators

The only constraint on the result of a decorator is that it be callable, so it can properly replace the decorated function. In the above examples, I've replaced the original function with an object of a class that has a __call__() method. But a function object is also callable, so we can rewrite the previous example using a function instead of a class, like this:

def entryExit(f):
    def new_f():
        print "Entering", f.__name__
        f()
        print "Exited", f.__name__
    return new_f

@entryExit
def func1():
    print "inside func1()"

@entryExit
def func2():
    print "inside func2()"

func1()
func2()
print func1.__name__

new_f() is defined within the body of entryExit(), so it is created and returned when entryExit() is called. Note that new_f() is a closure, because it captures the actual value of f.

Once new_f() has been defined, it is returned from entryExit() so that the decorator mechanism can assign the result as the decorated function.

The output of the line print func1.__name__ is new_f, because the new_f function has been substituted for the original function during decoration. If this is a problem you can change the name of the decorator function before you return it:

def entryExit(f):
    def new_f():
        print "Entering", f.__name__
        f()
        print "Exited", f.__name__
    new_f.__name__ = f.__name__
    return new_f

The information you can dynamically get about functions, and the modifications you can make to those functions, are quite powerful in Python.

More Examples

Now that you have the basics, you can look at some more examples of decorators here. Note the number of these examples that use classes rather than functions as decorators.

In this article I have intentionally avoided dealing with the arguments of the decorated function, which I will look at in the next article.


Evan Cofsky

Posts: 9
Nickname: theunixman
Registered: Jun, 2006

Decorators and Metaclasses Posted: Oct 18, 2008 3:46 PM
Reply to this message Reply
Both of these are powerful forms in Python. You can rewrite the semantics of the language on the fly to do things automatically that would normally need to be left to the rules of using a framework properly.

Database transactions, persistence, logging, debugging, warning... And it's metaprogramming using the same syntax as the programming language itself. These can fill the same roles as templates in C++, or JNI and annotations in Java, but being able to express orthogonal things completely orthogonally in code, but with the same idioms in the core flow of the system is so much more powerful.

These I think make Python as expressive as LISP.

Bruce Eckel

Posts: 874
Nickname: beckel
Registered: Jun, 2003

Re: Decorators and Metaclasses Posted: Oct 18, 2008 7:51 PM
Reply to this message Reply
Both of us appear to have tried to avoid using the word "awesome" but you might as well have.

Evan Cofsky

Posts: 9
Nickname: theunixman
Registered: Jun, 2006

Awesome Posted: Oct 18, 2008 8:15 PM
Reply to this message Reply
We agree again. But there is something fundamentally awesome about being able to change the language to fit the problem instead of changing the problem to fit the language. And to do it using the language itself.

Michele Simionato

Posts: 222
Nickname: micheles
Registered: Jun, 2008

Re: Decorators and Metaclasses Posted: Oct 18, 2008 10:41 PM
Reply to this message Reply
> Python decorators are similar to Lisp macros in power and possibility.

This is a tempting association (decorators <=> macros) but unfortunately it is not true. Lisp macros are strictly
*more* powerful than Python decorators because:

1. macros can change the syntax of the language (decorators cannot)

2. macros work at compile time (decorators cannot)

For most practical purposes decorators are powerful enough
(the fact that they work at runtime is even an advantage
in some cases) but there are things you cannot implement
with decorators but only with macros.

For instance, an impressive usage of macros in the Scheme world is the "Typed Scheme" project: here you take a dynamically typed language such as Scheme and you implement on top
of it a statically typed language where types are checked at compile time (see http://docs.plt-scheme.org/typed-scheme/). You could not implement that on top of Python.

Evan Cofsky

Posts: 9
Nickname: theunixman
Registered: Jun, 2006

Re: Decorators and Metaclasses Posted: Oct 19, 2008 12:11 AM
Reply to this message Reply
Decorators can't change the syntax of Python, of course. But python doesn't have the same separation between compile-time and run-time, even though it has equivalents to what LISP has. But decorators and metaclasses can change fundamental behaviors of the language, and do it at the equivalent point in the python world as macros in LISP.

While not being able to change the syntax of the language does limit their power, I think it limits their power in ways that make them much more useful. Python always looks like Python, and always behaves like python inside of a block of code. But you can easily change the behavior at the block boundaries, whether it's by using context managers for anonymous blocks, decorators for functions, or metaclasses for types. And since the changes to execution are explicitly managed with Python code, these mechanisms are much simpler to unravel for less experienced Python users.

Evan Cofsky

Posts: 9
Nickname: theunixman
Registered: Jun, 2006

Typed Python Posted: Oct 19, 2008 12:22 AM
Reply to this message Reply
It is actually possible to use decorators and/or metaclasses to implement something similar to Typed Scheme (http://docs.plt-scheme.org/typed-scheme/). However, the key difference is that Python doesn't have the same kind of "compile-time" that Scheme does, and there is no way at all to do early type checking.

However it is possible to tag functions with decorators that enforce type checking at the time of the call in Python, and even provide automatic conversion for types that should be equivalent. Then it's much easier to produce informative errors when something does go wrong. Since decorators are bound to each function only once, then it's even possible to change from expensive runtime behavior to "free" non-checking runtime behavior by changing a global setting that tells the type-checking decorator what should be done. It's even easy to enable checking for some modules or functions without globally enabling or disabling checking too. Again, since the decorators are bound when the functions are constructed, once this is done the only cost is when a decorator returns a modified function.

The mechanism may not be exactly the same as LISP macros, but LISP and Python are fundamentally different languages. But decorators and metaclasses happen at the closest equivalent Python has to LISP's "compile" time.

Kay Schluehr

Posts: 302
Nickname: schluehk
Registered: Jan, 2005

Re: Decorators I: Introduction to Python Decorators Posted: Oct 19, 2008 5:42 AM
Reply to this message Reply
Macros... macros? Seems like macros have become a matter of sloppy theoretical discourse about DSLs rather than a living but thorny programming practice. Decorators don't need to be hyped as some kinds of macros for Python that lacks them. Notice that I'm not a fan of Lispy macros for Python either which reflects my experiences with implementing a macro system for the language that had its complexities. Evaluation of code defined inside of the macro body at compile time or suspension of evaluation therein leads often to bugs that are hard to track. The behaviour of the code is generally not easy to understand and I'm not sure this is a value in itself. I've started to work on something that is simpler and more straightforward to use. Otherwise it was fun to create a recursive macro that produced the n-th Fibonacci as syntactically correct source code.

Compared to fancy macros decorators are a rather convenient composition and modularization technique for functions. It is so appealing because it is just a small step apart from techniques we have used for ages and still use every day.

David Easley

Posts: 1
Nickname: easleydp
Registered: Jan, 2006

Re: Decorators I: Introduction to Python Decorators Posted: Oct 19, 2008 12:29 PM
Reply to this message Reply
I don't know Python so I've probably got the wrong end of the stick, but the examples give the impression that the source code of the thing to be decorated must be edited. That is, if I want to decorate func1, I have to put @entryExit just before the definition. Is it possible to decorate func1 without having to edit its source file?

Kay Schluehr

Posts: 302
Nickname: schluehk
Registered: Jan, 2005

Re: Decorators I: Introduction to Python Decorators Posted: Oct 19, 2008 12:54 PM
Reply to this message Reply
> I don't know Python so I've probably got the wrong end of
> the stick, but the examples give the impression that the
> source code of the thing to be decorated must be edited.
> That is, if I want to decorate func1, I have to put
> @entryExit just before the definition. Is it possible to
> decorate func1 without having to edit its source file?

The @deco decoration is just gives a visual clue for the reassignment foo = deco(foo) that has to be written after foo was defined. So it's possible to load a module and decorate each of the functions without prior manipulation of the source.

Michele Simionato

Posts: 222
Nickname: micheles
Registered: Jun, 2008

Re: Decorators I: Introduction to Python Decorators Posted: Oct 20, 2008 8:09 AM
Reply to this message Reply
Just today I have discovered a nice little library based on decorators to profile functions: http://mg.pov.lt/profilehooks/
It makes a good example of the usefulness of decorators for people not knowing the concept, and you may want to keep it in mind for your book.

Ian Bicking

Posts: 851
Nickname: ianb
Registered: Apr, 2003

Re: Decorators I: Introduction to Python Decorators Posted: Oct 21, 2008 12:18 PM
Reply to this message Reply
It would be nice if you used a style more in line with the stdlib/PEP 8 -- specifically my_decorator instead of myDecorator. (What to do with the class is a little unclear -- PEP 8 strongly suggests MyDecorator, but in some ways the class is an artifact of the implementation and not the interface, which itself is indistinguishable from a function -- I'd stick with lower case.)

Evan Cofsky

Posts: 9
Nickname: theunixman
Registered: Jun, 2006

Re: Decorators I: Introduction to Python Decorators Posted: Oct 21, 2008 2:48 PM
Reply to this message Reply
entryExit.__call__ should probably return the result of the function call. Otherwise the decorator will cause all calls to the function to return None. In general, when delegating calls, it's safest to always return the result of the delegated call.


class entryExit(object):

def __init__(self, f):
self.f = f

def __call__(self):
print "Entering", self.f.__name__
r = self.f()
print "Exited", self.f.__name__
return r

Achilleas Margaritis

Posts: 674
Nickname: achilleas
Registered: Feb, 2005

Re: Decorators I: Introduction to Python Decorators Posted: Oct 22, 2008 7:43 AM
Reply to this message Reply
Why decorators and not full compile-time programming?

nes

Posts: 137
Nickname: nn
Registered: Jul, 2004

Re: Decorators I: Introduction to Python Decorators Posted: Oct 22, 2008 4:26 PM
Reply to this message Reply
> Why decorators and not full compile-time programming?

Probably because Python is a scripting language: one of the source file of your program might be used for "configuration purposes". A sysadmin somewhere may even have a cronjob that modifies one of the source code files of your program while the program is running and does a periodic "eval" or "exec" on it.

Flat View: This topic has 34 replies on 3 pages [ 1  2  3 | » ]
    Next Topic Topic: For the Sake of Simplicity


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us