...Wherein we examine what ant fundamentally lacks, and
possible ways to address the important problem of build control...
In a previous blog on today.java.net I
said some fairly strong things about ant, and I think I ought to
say something more, both good and bad. This seems like the place
to do it.
First, let us give ant its due. Ant is designed to be a portable
way to replace make. Ant was to be platform independent, so the
obvious implementation choice was Java, which is fine, mostly.
And the obvious data format for the project description -- saying
what needs to be built -- was XML, which is not fine, as
we shall see.
Ant performs this task with some power, if little aplomb. One
can build ant files that will compile Java, C, and many other
languages; perform CVS and FTP tasks; build jar files and EJB
deployments, and so forth. Ant can run the same configuration
without modification, no matter the platform. This is no mean
feat. There is a reason why it has been winning many awards.
But when I say it is "with little aplomb", I mean it. Nobody
can accuse XML of being an easy read, of course, but if that were
ant's worst problem, well, ant is hardly alone in this sin. XML
abuse is rampant, and ant is far, far from the worst
offender. Given the limitations of XML syntax, ant does pretty
well. To see it really suck, try reading Graydon
Hoare's essay on XML abuse. (Ignore the odd section headings
and stick to the content.)
The real problem is the one I talked about in my brief java.net blog:
Ant is nothing more than the sum of its parts
By this I mean that ant has not learned the basic power of composition,
building things out smaller parts. This was the great insight that
our Unix forbearers bequeathed us toolsmiths, and it's pretty sad
to see it forgotten. In Unix this is done with pipes -- the output
of one program can become the input of another. Along with
conventions about program output, this allows you to build up
something that is more than the sum of its parts. A utility that
finds file in the file system can generate a list of files for
any purpose: removal, editing, rebuilding, copying,
Ant has many
kinds of tasks. These are portable because they are written
in Java, and these task implementations can rely upon operating
system abstractions in the underlying ant platform.
But they almost never work together. There are some common
tools for building up lists of files, but if you want a list of
files from any other source, good luck. This is why there are two
different tasks (one optional) that calculate list of Java class
dependencies, and they are incompatible. And if you want a list
of Java class dependencies for a task that one of these two tasks
can't handle, good luck.
A common quote in our biz is the observation that a good software
tool will do things the author never expected1. I doubt that writers
of ant tasks are very often surprised.
In fact, I will assume that the value of composable tools is
so obvious, I will not belabor it further. Unless you ask...
My experience is that the primary legibility problem with
Makefiles are strings and variables. The string operations are
primitive, ugly, and limited. So to do anything at all interesting
requires cryptic and hobbled work. After that, the next worst problem
is the lack of flow control. Loops are put into the shell commands
because make doesn't have loops, or (shudder) people use recursion
as a replacement. Ugh.
The primary portability problem is obvious: Different system
have different commands, and make just executes commands.
So where would one go to leverage existing tools that already
know how to do these things well? Scripting languages, of course.
Perl, python/jython, tcl, and so on live, breath, eat, and excrete
strings for a living, and they are programming languages with real
logic. They are ported to every platform that matters (more than
Java runs on).
All they are missing are tools to express dependencies and how
to apply them to building things. One of the responses to my post
pointed me at scons,
which does this in Python. It seems just the right direction,
although I haven't had a chance to do more than play with scons.
I don't claim it is the right answer, but it sure is moving
down the right path.
One could pick any language, of course, and choosing a winner
here would antagonize folks who would just prove they could do it
better in the "right" language. I have my preferred scripting
languages, but compared to scripting in XML itself (which is where
the ant folks seem to be going), no major possibility is worse.
I intend to poke harder at scons, and if it looks good, dive
in fully to find out. You'll be seeing a blog here about scons
and the lessons to learn from it.
 Google has failed me. Can the death of
the internet be far behind? I cannot find an attribution for this
quote, despite the obvious set of words and the fact that it's a
computer quote that I've heard a billion times.
[Note: URL for Graydon Hoare's XML writeup has been updated; 7/1/2005]
At my old job we ended up with Ant files so large we split them into multiple physical files, and used XML Entities to include the subfiles in a main file. A lot of the targets were copy-and-paste because we couldn't think of a good way to factor out the differences. Some targets existed solely to check timestamps and skip other targets. Imagine maintaining all that ... in XML.
A better approach might have been to factor the many targets into wholly independent Ant files. Even better, a lot of platform-specific targets should have been ported to Makefiles, shell scripts, or .BAT files.
Another candidate along the same lines is a-a-p by Bram Moolenar of vim (vi-alike editor) fame. It just went 1.0 this month the Friday before O'Reilly OSCon. Bram gave a session on a-a-p there which I was unable to attend. I've requested he post his slides for the talk at
The syntax is much more akin to make (simple colon separated dependencies), and python is great. I was hoping he would do it in java/jython to take advantage of the ant recipes. However, perhaps ant could adopt the a-a-p syntax and provide a compatibility the other way.
Also, it occurs to me that perl6 (if it ever comes to be) could well self-contain make functionality with its new grammar construct. I'm not a perl hack (unfortunately) but the grammar keyword and support suggests to me that perl6 will be great for creating new mini-languages. The grammar regex support is tremendous, and perl6 would probably deal with any make-like syntax handily. Just a thought for the future.
A brilliant observation. I was always a bit afraid to criticize Ant since I thought perhaps I just didn't fully understand the issues, but now it seems that I do.
Readability of anything in programming is important, and yet is often lost "because we have to blah blah." When moving databases around, XML is great because it's far more readable than proprietary database formats. But to then turn around and say "XML for everything!" repeats the usual silly mistake we all make periodically, of finding the one true answer.
I've spent years with 'make', and have pushed the boundaries and found at least some of the limitations. Make is nice because it's structured and succinct; it seems to keep the amount of verbiage down to a minimum, something I immediately noticed that Ant does not do (Ant is verbose for the sake of supporting XML). But there are places where make quickly breaks down -- sometimes the automated creation of lists, sometimes complex dependencies. This has caused me hours of frustration trying to get make to do things that it very probably just can't do.
I believe problems like this have caused a rather large number of projects in the last couple of years, to build the "ultimate" build tool. Some of those tools seem to have produced interesting ideas, but I think what I want is something that (A) is simple and obvious to use (I don't want to relive the 'make' learning curve) (B) is succinct and structured. I should be able to accrete my build description without tons of redundant verbiage. (C) doesn't limit me because the designer ran out of energy.
I imagine the best solution to this problem is "The simplest thing that could possibly work." 'Make' is a language, and it falls into the Unix trap of everybody making up their own languages. I don't want to have to learn a language every time I pick up a new tool. Perl is valuable because it combines all the various Unix tools into a single unified language, although it unfortunately also combines the cruft from those Unix tools.
So instead of someone inventing yet another new build language for me to learn, a language which will inevitably be limited because the author will not go to the trouble of creating a general purpose language, I think the best solution would be to create some kind of framework on top of my favorite language (Python). Perhaps it would even have the basic syntax of make, but the point would be that the simple things would be taken care of using the fundamental structure of the thing (that is, I don't have to write a bunch of Python just to get my build configured). But if I need to do something more complex, like create and manipulate a list of files, I can easily drop into the full power of Python just by writing a Python expression (perhaps escaped as is done with JSPs in web pages).
So an example implementation of this system might use the fundamental makefile structure (so I don't have to re-learn that) for basic dependencies, but as soon as you want to do anything fancy you just write Python code, rather than trying to use the weird and whacky make doodads. This approach would also have the benefit of allowing easier translation of existing makefiles, and possibly even conversion of makefile-building tools.
I use Java for all of my builds "scripts". It works great!! Since Ant tasks are written in Java, they are easily leveraged. Also, most of my build scripts are reusable between projects. This is easy if you structure all of your projects in a similar way.
> A common quote in our biz is the observation that a good software > tool will do things the author never expected. I doubt that writers > of ant tasks are very often surprised. > > Google has failed me. Can the death of > the internet be far behind? I cannot find an attribution for this > quote, despite the obvious set of words and the fact that it's a > computer quote that I've heard a billion times.
I don't have the book at hand to verify, but I believe this is from The UNIX Programming Environment by Brian W. Kernighan, Rob Pike. If not, it's from the Bell Labs Journal devoted to the announcement of unix.
Have you looked at the Groovy scripting language (http://groovy.codehaus.org/) which is a Java-ized, JVM & Java bytecode compatible scripting language? Groovy basically tries to take Ruby (which itself tries to take the best of Perl, Python and Smalltalk) and give it a Java-like syntax.
One of the things that make Ant nice to use is that its fairly declarative. Like an XSLT stylesheet for you source code to transform it into a built result (easier to read than XSLT, though not by much). Yes, it kind of sucks as a scripting language, and yes, XML in general sucks as a scripting medium.
But build scripts are often rules. A big part of Make wasn't as much running lists of commands after each target, it was defining how to turn a *.c into a *.o, and a *.o into a *.a, etc. A big part of build scripts are defining these rules. And ant not only has a number of these rules defined for easy use, it has enough re-usable objects in it so that writing you r own task can be pretty easy.
But I can say this, but at the same time I've written "simple" ant scripts that have swollen to 4,000 lines. With build scripts, while the majority of work can get done with a few clean lines, the exceptions and fiddly bits make everything more complicated. Thats when you really need a scripting language, and thats when ant begins to chafe.
Ant slowly evolves, the addition of macros has given some hope of slimming down the monolithic ant files, and include files keep us from using horrible ENTITY includes. But this is probably leading to a nightmare of hard to navigate and debug includes of includes, like CPP macros run amock, or worse, imake.
What I'm trying to say is that a build scripts are different, and so they need different languages. But then they aren't different enough, so they need regular scripting languages...damn, I have to go. The builds broken again.