The Artima Developer Community
Sponsored Link

Weblogs Forum
Python Decorators III: A Decorator-Based Build System

19 replies on 2 pages. Most recent reply: Apr 29, 2014 6:23 AM by Andy Doddington

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 19 replies on 2 pages [ 1 2 | » ]
Bruce Eckel

Posts: 875
Nickname: beckel
Registered: Jun, 2003

Python Decorators III: A Decorator-Based Build System (View in Weblogs)
Posted: Oct 26, 2008 12:50 PM
Reply to this message Reply
Summary
Most build systems start out with dependencies, then realize they need language features and eventually discover they should have started with language design.
Advertisement

I've used make for many years. I only used ant because it produced faster Java builds. But both build systems started out thinking the problem was simple, and only later discovered that you really need a programming language to solve the build problem. By then it was too late. As a result you have to jump through annoying hoops to get things done.

There have been efforts to create build systems on top of languages. Rake is a fairly successful domain-specific language (DSL) built atop Ruby. And a number of projects have been created with Python.

For years I've wanted a system that was just a thin veneer on Python, so you get some support for dependencies but effectively everything else is Python. This way, you don't need to shift back and forth between Python and some language other than Python; it's less of a mental distraction.

It turns out that decorators are perfect for this purpose. The design I present here is just a first cut, but it's easy to add new features and I've already started using it as the build system for The Python Book, so I'll probably need to add more features. Most importantly, I know I'll be able to do anything that I want, which is not always true with make or ant (yes, you can extend ant but the cost of entry is often not worth the benefit).

While the rest of the book has a Creative Commons Attribution-Share Alike license, this program only has a Creative Commons Attribution license, because I'd like people to be able to use it under any circumstances. Obviously, it would be ideal if you make any improvements that you'd contribute them back to the project, but this is not a prerequisite for using or modifying the code.

Syntax

The most important and convenient thing provided by a build system is dependencies. You tell it what depends on what, and how to update those dependencies. Taken together, this is called a rule, so the decorator will also be called rule. The first argument of the decorator is the target (the thing that needs to be updated) and the remaining arguments are the dependencies. If the target is out of date with the dependencies, the function code is run to bring it up to date.

Here's a simple example that shows the basic syntax:

@rule("file1.txt")
def file1():
    "File doesn't exist; run rule"
    file("file1.txt", 'w')

The name of the rule is file1 because that's the function name. In this case, the target is "file1.txt" and there are no dependencies, so the rule only checks to see whether file1.txt exists, and if it doesn't it runs the function code, which brings it up to date.

Note the use of the docstring; this is captured by the build system and describes the rule on the command line when you say build help (or anything else the builder doesn't understand).

The @rule decorators only affect the functions they are attached to, so you can easily mix regular code with rules in the same build file. Here's a function that updates the date stamp on a file, or creates the file if it doesn't exist:

def touchOrCreate(f): # Ordinary function
    "Bring file up to date; creates it if it doesn't exist"
    if os.path.exists(f):
        os.utime(f, None)
    else:
        file(f, 'w')

A more typical rule is one that associates a target file with one or more dependent files:

@rule("target1.txt","dependency1.txt","dependency2.txt","dependency3.txt")
def target1():
    "Brings target1.txt up to date with its dependencies"
    touchOrCreate("target1.txt")

This build system also allows multiple targets, by putting the targets in a list:

@rule(["target1.txt", "target2.txt"], "dependency1.txt", "dependency2.txt")
def multipleBoth():
    "Multiple targets and dependencies"
    [touchOrCreate(f) for f in ["target1.txt", "target2.txt"]]

If there is no target or dependencies, the rule is always executed:

@rule()
def clean():
    "Remove all created files"
    [os.remove(f) for f in allFiles if os.path.exists(f)]

The alFiles array is seen in the example, shown later.

You can write rules that depend on other rules:

@rule(None, target1, target2)
def target3():
    "Always brings target1 and target2 up to date"
    print target3

Since None is the target, there's nothing to compare to but in the process of checking the rules target1 and target2, those are both brought up to date. This is especially useful when writing "all" rules, as you will see in the example.

Builder Code

By using decorators and a few appropriate design patterns, the code becomes quite succinct. Note that the __main__ code creates an example build.py file (containing the examples that you see above and more), and the first time you run a build it creates a build.bat file for Windows and a build command file for Unix/Linux/Cygwin. A complete explanation follows the code:

# builder.py
import sys, os, stat
"""
Adds build rules atop Python, to replace make, etc.
by Bruce Eckel
License: Creative Commons with Attribution.
"""

def reportError(msg):
    print >> sys.stderr, "Error:", msg
    sys.exit(1)

class Dependency(object):
    "Created by the decorator to represent a single dependency relation"

    changed = True
    unchanged = False

    @staticmethod
    def show(flag):
        if flag: return "Updated"
        return "Unchanged"

    def __init__(self, target, dependency):
        self.target = target
        self.dependency = dependency

    def __str__(self):
        return "target: %s, dependency: %s" % (self.target, self.dependency)

    @staticmethod
    def create(target, dependency): # Simple Factory
        if target == None:
            return NoTarget(dependency)
        if type(target) == str: # String means file name
            if dependency == None:
                return FileToNone(target, None)
            if type(dependency) == str:
                return FileToFile(target, dependency)
            if type(dependency) == Dependency:
                return FileToDependency(target, dependency)
        reportError("No match found in create() for target: %s, dependency: %s"
            % (target,  dependency))

    def updated(self):
        """
        Call to determine whether this is up to date.
        Returns 'changed' if it had to update itself.
        """
        assert False, "Must override Dependency.updated() in derived class"

class NoTarget(Dependency): # Always call updated() on dependency
    def __init__(self, dependency):
        Dependency.__init__(self, None, dependency)
    def updated(self):
        if not self.dependency:
            return Dependency.changed # (None, None) -> always run rule
        return self.dependency.updated() # Must be a Dependency or subclass

class FileToNone(Dependency): # Run rule if file doesn't exist
    def updated(self):
        if not os.path.exists(self.target):
            return Dependency.changed
        return Dependency.unchanged

class FileToFile(Dependency): # Compare file datestamps
    def updated(self):
        if not os.path.exists(self.dependency):
            reportError("%s does not exist" % self.dependency)
        if not os.path.exists(self.target):
            return Dependency.changed # If it doesn't exist it needs to be made
        if os.path.getmtime(self.dependency) > os.path.getmtime(self.target):
            return Dependency.changed
        return Dependency.unchanged

class FileToDependency(Dependency): # Update if dependency object has changed
    def updated(self):
        if self.dependency.updated():
            return Dependency.changed
        if not os.path.exists(self.target):
            return Dependency.changed # If it doesn't exist it needs to be made
        return Dependency.unchanged

class rule(object):
    """
    Decorator that turns a function into a build rule. First file or object in
    decorator arglist is the target, remainder are dependencies.
    """
    rules = []
    default = None

    class _Rule(object):
        """
        Command pattern. name, dependencies, ruleUpdater and description are
        all injected by class rule.
        """

        def updated(self):
            if Dependency.changed in [d.updated() for d in self.dependencies]:
                self.ruleUpdater()
                return Dependency.changed
            return Dependency.unchanged

        def __str__(self): return self.description

    def __init__(self, *decoratorArgs):
        """
        This constructor is called first when the decorated function is
        defined, and captures the arguments passed to the decorator itself.
        (Note Builder pattern)
        """
        self._rule = rule._Rule()
        decoratorArgs = list(decoratorArgs)
        if decoratorArgs:
            if len(decoratorArgs) == 1:
                decoratorArgs.append(None)
            target = decoratorArgs.pop(0)
            if type(target) != list:
                target = [target]
            self._rule.dependencies = [Dependency.create(targ, dep)
                for targ in target for dep in decoratorArgs]
        else: # No arguments
            self._rule.dependencies = [Dependency.create(None, None)]

    def __call__(self, func):
        """
        This is called right after the constructor, and is passed the function
        object being decorated. The returned _rule object replaces the original
        function.
        """
        if func.__name__ in [r.name for r in rule.rules]:
            reportError("@rule name %s must be unique" % func.__name__)
        self._rule.name = func.__name__
        self._rule.description = func.__doc__ or ""
        self._rule.ruleUpdater = func
        rule.rules.append(self._rule)
        return self._rule # This is substituted as the decorated function

    @staticmethod
    def update(x):
        if x == 0:
            if rule.default:
                return rule.default.updated()
            else:
                return rule.rules[0].updated()
        # Look up by name
        for r in rule.rules:
            if x == r.name:
                return r.updated()
        raise KeyError

    @staticmethod
    def main():
        """
        Produce command-line behavior
        """
        if len(sys.argv) == 1:
            print Dependency.show(rule.update(0))
        try:
            for arg in sys.argv[1:]:
                print Dependency.show(rule.update(arg))
        except KeyError:
            print "Available rules are:\n"
            for r in rule.rules:
                if r == rule.default:
                    newline = " (Default if no rule is specified)\n"
                else:
                    newline = "\n"
                print "%s:%s\t%s\n" % (r.name, newline, r)
            print "(Multiple targets will be updated in order)"
        # Create "build" commands for Windows and Unix:
        if not os.path.exists("build.bat"):
            file("build.bat", 'w').write("python build.py %1 %2 %3 %4 %5 %6 %7")
        if not os.path.exists("build"):
            # Unless you can detect cygwin independently of Windows
            file("build", 'w').write("python build.py $*")
            os.chmod("build", stat.S_IEXEC)

############### Test/Usage Examples ###############

if __name__ == "__main__":
    if not os.path.exists("build.py"):
        file("build.py", 'w').write('''\
# Use cases: both test code and usage examples
from builder import rule
import os

@rule("file1.txt")
def file1():
    "File doesn't exist; run rule"
    file("file1.txt", 'w')

def touchOrCreate(f): # Ordinary function
    "Bring file up to date; creates it if it doesn't exist"
    if os.path.exists(f):
        os.utime(f, None)
    else:
        file(f, 'w')

dependencies = ["dependency1.txt", "dependency2.txt",
                "dependency3.txt", "dependency4.txt"]

targets = ["file1.txt", "target1.txt", "target2.txt"]

allFiles = targets + dependencies

@rule(allFiles)
def multipleTargets():
    "Multiple files don't exist; run rule"
    [file(f, 'w') for f in allFiles if not os.path.exists(f)]

@rule(["target1.txt", "target2.txt"], "dependency1.txt", "dependency2.txt")
def multipleBoth():
    "Multiple targets and dependencies"
    [touchOrCreate(f) for f in ["target1.txt", "target2.txt"]]

@rule("target1.txt","dependency1.txt","dependency2.txt","dependency3.txt")
def target1():
    "Brings target1.txt up to date with its dependencies"
    touchOrCreate("target1.txt")

@rule()
def updateDependency():
    "Updates the timestamp on all dependency.* files"
    [touchOrCreate(f) for f in allFiles if f.startswith("dependency")]

@rule()
def clean():
    "Remove all created files"
    [os.remove(f) for f in allFiles if os.path.exists(f)]

@rule()
def cleanTargets():
    "Remove all target files"
    [os.remove(f) for f in targets if os.path.exists(f)]

@rule("target2.txt", "dependency2.txt", "dependency4.txt")
def target2():
    "Brings target2.txt up to date with its dependencies, or creates it"
    touchOrCreate("target2.txt")

@rule(None, target1, target2)
def target3():
    "Always brings target1 and target2 up to date"
    print target3

@rule(None, clean, file1, multipleTargets, multipleBoth, target1,
      updateDependency, target2, target3)
def all():
    "Brings everything up to date"
    print all

rule.default = all
rule.main() # Does the build, handles command-line arguments
''')

The first group of classes manage dependencies between different types of objects. The base class contains some common code, including the constructor which you'll note is automatically called if it is not explicitly redefined in a derived class (a nice, code-saving feature in Python).

Classes derived from Dependency manage particular types of dependency relationships, and redefine the updated() method to decide whether the target should be brought up to date with the dependent. This is an example of the Template Method design pattern, where updated() is the template method and _Rule is the context.

If you want to create a new type of dependency -- say, the addition of wildcards on dependencies and/or targets -- you define new Dependency subclasses. You'll see that the rest of the code doesn't require changes, which is a positive indicator for the design (future changes are isolated).

Dependency.create() is what I call a Simple Factory Method, because all it does is localize the creation of all the subtypes of Dependency. Note that forward referencing is not a problem here as it is in some languages, so using the full implementation of Factory Method given in GoF is not necessary and also more complex (this doesn't mean there aren't cases that justify the full-fledged Factory Method).

Note that in FileToDependency we could assert that self.dependency is a subtype of Dependency, but this type check happens (in effect) when updated() is called.

The rule Decorator

The rule decorator uses the Builder design pattern, which makes sense because the creation of a rule happens in two steps: the constructor captures the decorator arguments, and the __call__() method captures the function.

The Builder product is a _Rule object, which, like the Dependency classes, contains an updated() method. Each _Rule object contains a list of dependencies and a ruleUpdater() method which is called if any of the dependencies is out of date. The _Rule also contains a name (which is the decorated function name) and a description (the decorated function's docstring). (The _Rule object is an example of the Command pattern).

What's unusual about _Rule is that you don't see any code in the class which initializes dependencies, ruleUpdater(), name, and description. These are initialized by rule during the Builder process, using Injection. The typical alternative to this is to create setter methods, but since _Rule is nested inside rule, rule effectively "owns" _Rule and Injection seems much more straightforward.

The rule constructor first creates the product _Rule object, then handles the decorator arguments. It converts decoratorArgs to a list because we need it to be modifiable, and decoratorArgs comes in as a tuple. If there is only one argument it means the user has only specified the target and no dependencies. Because Dependency.create() requires two arguments, we append None to the list.

The target is always the first argument, so pop(0) pulls it off and the remainder of the list is dependencies. To accommodate the possibility that the target is a list, single targets are turned into lists.

Now Dependency.create() is called for each possible target-dependency combination, and the resulting list is injected into the _Rule object. For the special case when there are no arguments, a None to None Dependency is created.

Notice that the only thing the rule constructor does is sort out the arguments; it has no knowledge of particular relationships. This keeps special knowledge within the Dependency hierarchy, so adding a new Dependency is isolated within that hierarchy.

A similar guideline is followed for the __call__() method, which captures the decorated function. We keep the _Rule object in a static list called rules, and the first thing to check is whether any of the rule names are duplicated. Then we capture and inject the name, documentation string, and the function itself.

Note that the Builder "product", the _Rule object, is returned as the result of rule.__call__(), which means that this object -- which doesn't have a __call__() method -- is substituted for the decorated function. This is a slightly unusual use of decorators; normally the decorated function is called directly, but in this case the decorated function is never called directly, but only via the _Rule object.

Running a Build

The static method main() in rule manages the build process, using the helper method update(). If you provide no command-line arguments, main() passes 0 to update(), which calls the default rule if one has been set, otherwise it calls the first rule that was defined. If you provide command-line arguments, it passes each one (in order) to update().

If you give it an incorrect argument (typically help is reserved for this), it prints each of the rules along with their docstrings.

Finally, it checks to see that a build.bat and build command file exists, and creates them if it doesn't.

The build.py produced when you run builder.py the first time can act as a starting point for your build file.

Improvements

As it stands, this system only satisfies the basic needs; it doesn't have, for example, all the features that make does when it comes to manipulating dependencies. On the other hand, because it's built atop a full-powered programming language, you can do anything else you need quite easily. If you find yourself writing the same code over and over, you can modify rule() to reduce the duplicated effort. If you have permission, please submit such modifications back for possible inclusion.

In the last installment of this series (chapter), we'll look at class decorators and whether you can decorate an object.


Evan Cofsky

Posts: 9
Nickname: theunixman
Registered: Jun, 2006

SCons Posted: Oct 26, 2008 1:35 PM
Reply to this message Reply
The dependency specification is somewhat similar to SCons and its build descriptions. It's probably worth taking a look at, even if only for ideas.

Dmitry Cheryasov

Posts: 16
Nickname: dch
Registered: Apr, 2007

Re: Python Decorators III: A Decorator-Based Build System Posted: Oct 26, 2008 2:06 PM
Reply to this message Reply
The idea of using decorators is neat.

One problem that bothers me is verbosity. You often have to mention the same file both as a @rule() argument and as a name within the decorated function. This can lead to annoying discrepancies if a typo sneaks in one of the names that are supposed to be identical.

Probably I'd mark the "important" files somehow so that they could be reused within the decorated function. Consider:

@rule(("source.xml", "style.xsl"), "img1.png", "img2.png")
def createPdf(implicit_names):
xml, xsl = implicit_names
xsltTransform(xml, xsl)

Note the first parameter. It is always passed as the tuple of names that comes as the first argument to the @rule(). The function becomes more name-agnostic, it's like using "%" in makefiles, only better.

If the first argument to @rule() is not a tuple, an empty tuple is passed to the decorated function.

Implementing this would take rather small changes to your code, probably.

Bruce Eckel

Posts: 875
Nickname: beckel
Registered: Jun, 2003

Re: Python Decorators III: A Decorator-Based Build System Posted: Oct 26, 2008 2:26 PM
Reply to this message Reply
Yes, it had already occurred to me that some kind of automatic identifier for targets and dependencies would probably be helpful. Yours seems a plausible approach; I'll think on it a bit.

David Laban

Posts: 1
Nickname: alsuren
Registered: Oct, 2008

Re: Python Decorators III: A Decorator-Based Build System Posted: Oct 26, 2008 4:02 PM
Reply to this message Reply
I think my first reaction when looking at a build script of this form would be:

"@rule(...)? Is that some obscure reference to make rules? What's the argument order for them again?"

A more readable interface might be something like:

@creates("target1.txt", "target2.txt")
@depends("dependency1.txt", "dependency2.txt")
def multipleBoth(inputs, outputs):
"Multiple targets and dependencies"
[touchOrCreate(f) for f in outputs]

Where inputs and outputs are lists of filenames (possibly passed in as keyword arguments, to make the order of decoration less important). This would let you demonstrate even more of the shiny things that you can do with decorators.

Also, with a bit of glob-style find/replace magic in the implementation, users could create more general rules with the same function bodies, by writing:
@depends("dependency*.txt") 
@creates("target*.txt")

Bruce Eckel

Posts: 875
Nickname: beckel
Registered: Jun, 2003

Re: Python Decorators III: A Decorator-Based Build System Posted: Oct 27, 2008 7:51 AM
Reply to this message Reply
The globbing had occurred to me, but the @creates and @depends do seem clearer. Yes, I was thinking of make rules, so I didn't reexamine it. Also, the dual decorators might make a more interesting example. Worth thinking about.

Andrew Montalenti

Posts: 1
Nickname: pixelmonke
Registered: Oct, 2008

Re: Python Decorators III: A Decorator-Based Build System Posted: Oct 27, 2008 9:56 AM
Reply to this message Reply
Hi Bruce,

Interesting post. I am working on a project where we built a decorator-based build system in Python for our project. We didn't take exactly the same approach, but using decorators allowed us to do some interesting things, for example making it possible to declare dependencies on 'targets' (a la Ant) and thus make the invocation of one target automatically invoke dependent targets first. We also layered things like logging and resume into the decorators.

One system you may want to look at, and which I only discovered recently, is Zed Shaw's "Vellum", which seems like a rake-inspired build system for Python that actually takes the approach of a mixed external / internal DSL:

http://www.zedshaw.com/projects/vellum/

Eli Courtwright

Posts: 14
Nickname: eliandrewc
Registered: Jan, 2006

Re: Python Decorators III: A Decorator-Based Build System Posted: Oct 27, 2008 10:23 AM
Reply to this message Reply
Minor nitpick: the file builtin is deprecated and no longer exists in Python 3.0 and you should use the open builtin instead.

Also, I second the suggestion to look at SCons, if only for ideas. I've used SCons as a replacement for Make and have been extremely happy with it.

Bruce Eckel

Posts: 875
Nickname: beckel
Registered: Jun, 2003

Re: Python Decorators III: A Decorator-Based Build System Posted: Oct 27, 2008 11:36 AM
Reply to this message Reply
> Minor nitpick: the file builtin is deprecated
> and no longer exists in Python 3.0 and you should use the
> open builtin instead.

That explains why I've been seeing it more. I swear I remember open being the original version, then file was added later, and I liked file because it seemed more explicit.

Matt Doar

Posts: 9
Nickname: mdoar
Registered: Feb, 2004

Re: Python Decorators III: A Decorator-Based Build System Posted: Oct 27, 2008 11:42 AM
Reply to this message Reply
+1 on SCons

Kevin Teague

Posts: 15
Nickname: wheat
Registered: Mar, 2006

Re: Python Decorators III: A Decorator-Based Build System Posted: Oct 28, 2008 1:49 AM
Reply to this message Reply
It was considered that only io.open() would be used for file opening in Python 3, and the open() built-in would be dropped, making usage of open() for files more explicit (which met with generally good reception but I guess was too much of a change):

http://mail.python.org/pipermail/python-3000/2007-May/007765.html

Paparipote Tamparantan

Posts: 4
Nickname: paparipote
Registered: Jan, 2006

Re: Python Decorators III: A Decorator-Based Build System Posted: Oct 28, 2008 6:05 AM
Reply to this message Reply
Hi, I am new with decorators. I would like to know if it is possible to solve next situation with this: when I build an error message after some evaluation I am tempted to display the message on the console if the application is under desktop, display using html if the application is under a web browser or save the error in a log file. In any case I need to do a previous evaluation of an indicator I fill when invoking the program: if indicator == "console" display error, if indicator == "web" generate an html error, if indicator == "log" write error in a file, if indicator == "console-log" display error and write error in a file, etc.
How can i solve this with decorators? May be a good example of use?
Best regards.

Evan Cofsky

Posts: 9
Nickname: theunixman
Registered: Jun, 2006

Decorators for Error Reporting Posted: Oct 28, 2008 7:31 AM
Reply to this message Reply
You should check out the logging and the cgitb modules to handle the actual reporting. Then you can create wrapper functions for each that catch exceptions and then report them. Then you can create a decorator that wraps an application function and selects the correct reporting function based on the configuration setting.

Bruce Eckel

Posts: 875
Nickname: beckel
Registered: Jun, 2003

Re: Python Decorators III: A Decorator-Based Build System Posted: Oct 29, 2008 11:02 AM
Reply to this message Reply
I'm planning a rewrite based on some of the suggestions given here.

Gregg Tavares

Posts: 1
Nickname: greggman
Registered: Nov, 2008

Re: Python Decorators III: A Decorator-Based Build System Posted: Nov 3, 2008 2:02 PM
Reply to this message Reply
There's a problem inherent in the assumption that everything I want to build fits these dependency rules. Ie, I know what I want to build and all the things it needs to be built. This rarely fits game creation. We don't generally start with "I need a .exe" file, we start from the other side. I have "lavalevel.mb" (a source file), build everything it references and pull it all together into a level pak.

Everytime the lavalevel.mb changes, so does the list of things it needs built. Or to put it backwards "lavalevel.pak" is dependent on "lavalevel.mb" and an unknown number of assets referenced by "lavalevel.mb". You don't get to know what assets are needed until the rule that takes "lavalevel.mb" as input is run. Most build systems have trouble with that concept. They want to know all the dependencies before building has started.

Worse, getting that dependency list (ie, running the rule that takes "lavalevel.mb") could easily take 4+ minutes (launch maya, export file). Way too long to run every time. That means an intermediate dependency list needs to be generated. Lets call it "lavalevel.dep" Now, if lavalevel.dep is newer than lavalevel.mb use the dependencies listed in lavalevel.dep, otherwise rebuild lavalevel.dep from lavalevel.mb. Most build systems again are able to express that.

Please don't go building yet another build system that is unaware that some projects build stuff other than code.

Flat View: This topic has 19 replies on 2 pages [ 1  2 | » ]
Topic: The Cathedral and the Pirate Previous Topic   Next Topic Topic: Windows 8 is ... Not So Bad

Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2019 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use