The open-source movement and agile programming may be converging. While reading Martin Fowler's excellent book "Refactoring", I realized development by refactoring is a sharp description of the normal style of open-source hackers. In this essay (which Martin Fowler and Kent Beck and I discussed in draft) I explore the connection further.
In 2001, there was a history-making conference of software-engineering
thinkers in Snowbird, Colorado. The product of that meeting was a remarkable
document called the Agile Manifesto,
a call to overturn many of the assumptions of traditional software development.
I was invited to be at Snowbird, but couldn't make it.
Ever since, though, I've been sensing a growing convergence between
agile programming and the open-source movement. I've seen agile
concepts and terminology being adopted rapidly and enthusiastically by
my colleagues in open-source-landespecially ideas like
refactoring, unit testing, and design from stories and personas. From
the other side, key agile-movement figures like Kent Beck and Martin
Fowler have expressed strong interest in open source both in published
works and to me personally. Fowler has gone so far as to include
open source on his list of agile-movement schools.
I agree that we belong on that list. But I also agree with
Fowler's description of of open source as a style, rather than a
process. I think his reservations as to whether open source can be
described as just another agile school are well-founded. There is
something more complicated and interesting going on here. and I
realized when I read Fowler's description of open source that at some
point I was going to have to do some hard thinking and writing in an
effort to sort it all out.
While doing research for my forthcoming book, The Art of Unix
Programming, I read one particular passage in Fowler's
Refactoring that finally brought it all home. He
One argument is that refactoring can be an alternative to up-front
design. In this scenario, you don't do any design at all. You just
code the first approach that comes into your head, get it working, and
then refactor it into shape. Actually, this approach can work. I've
seen people do this and come out with a very well-defined piece of
software. Those who support Extreme Programming often are portrayed
as advocating this approach.
I read this, and had one of those moments where everything comes
together in your head with a great ringing crash and the world assumes
a new shapea moment not unlike the one I had in late 1996
when I got the central insight that turned into The Cathedral
and the Bazaar. In the remainder of this essay I'm going to
try to articulate what I now think I understand about open source,
agile programming, how they are related, and why the connection should
be interesting even to programmers with no stake in either movement.
Now I need to set a little background here, because I'm going
to need to have to talk about several different categories which are
contingently but not necessarily related.
First, there is Unix programmer. Unix is the operating
system with the longest living tradition of programming and design.
It has an unusually strong and mature technical culture around it, a
culture which originated or popularized many of the core ideas and
tools of modern software design. The Art of Unix
Programming is a concerted attempt to capture the craft wisdom
of this culture, one to which I have successfully enlisted quite a few
of its founding elders.
Second, there is hacker. This is a very complex term, but
more than anything else, it describes an attitudean
intentional stance that relates hackers to programming and other
disciplines in a particular way. I have described the hacker stance
and its cultural correlates in detail in How To Become A
Third, there is open-source programmer. Open source is a
programming style with strong roots in the Unix tradition and the
hacker culture. I wrote the modern manifesto for it in 1997, The
Cathedral and the Bazaar, building on earlier thinking by
Richard Stallman and others.
These three categories are historically closely related. It is
significant that a single person (accidentally, me) wrote touchstone
documents for the second and third and is attempting a summum
bonum of the first. That personal coincidence reflects a larger
social reality that in 2003 these categories are becoming increasingly
merged essentially, the hacker community has become the core
of the open-source community, which is rapidly re-assimilating the
parts of the Unix culture that got away from the hackers during
the ten bad years after the AT&T divestiture in 1984.
But the relationship is not logically entailed; we can imagine
a hacker culture speaking a common tongue other than Unix and C (in
the far past its common tongue was Lisp), and we can imagine an
explicit ideology of open source developing within a cultural and
technical context other than Unix (as indeed nearly happened several
With this scene-setting done, I can explain that my first take on
Fowler's statement was to think "Dude, you've just described
I mean something specific and powerful by this. Throwing together
a prototype and refactoring it into shape is a rather precise
description of the normal working practice of hackers since that
culture began to self-define in the 1960s. Not a complete one, but it
captures the most salient feature of how hackers relate to code. The
open-source community has inherited and elaborated this practice,
building on similar tendencies within the Unix tradition.
The way Fowler writes about design-by-refactoring has two huge
implications for the relationship between open source and agile
First, Fowler writes as though he didn't know he was describing
hacking. In the passage, he appears unaware that design by
repeated refactoring is not just a recent practice semi-accidentally
stumbled on by a handful of agile programmers, but one which hundreds
of thousands of hackers have accumulated experience with for over three
decades and have in their bones. There is a substantial folklore, an
entire craft practice, around this!
Second, in that passage Fowler described the practice of hacking
better than hackers themselves have done. Now, admittedly,
the hacker culture has simply not had that many theoreticians, and if
you list the ones that are strongly focused on development methodology
you lose Richard Stallman and are left with, basically, myself and
maybe Larry Wall (author of Perl and occasional funny and illuminating
ruminations on the art of hacking). But the fact that we don't have a
lot of theoreticians is itself an important datum; we have always
tended to develop our most important wisdoms as unconscious and
unarticulated craft practice.
These two observations imply an enormous mutual potential, a gap
across which an arc of enlightenment may be beginning to blaze. It
implies two things:
First, people who are excited by agile-programming ideas can
look to open source and the Unix tradition and the hackers for the
lessons of experience. We've been doing a lot of the stuff the
agile movement is talking about for a long time. Doing it in a
clumsy, unconscious, learned-by-osmosis way, but doing it
nevertheless. I believe that we have learned things that you agile
guys need to know to give your methodologies groundedness. Things
like (as Fowler himself observes) how to manage communication and
hierarchy issues in distributed teams.
Second, open-source hackers can learn from agile programmers
how to wake up. The terminology and conceptual framework of
agile programming sharpens and articulates our instincts. Learning to
speak the language of open source, peer review, many eyeballs, and
rapid iterations gave us a tremendous unifying boost in the late
1990s; I think becoming similarly conscious about agile-movement ideas
like refactoring, unit testing, and story-centered design could be
just as important for us in the new century.
I've already given an example of what the agile movement has to
teach the hackers, in pointing out that repeated redesign by
refactoring is a precise description of hacking. Another thing we can
stand to learn from agile-movement folks is how to behave so that we
can actually develop requirements and deliver on them when the
customer isn't, ultimately, ourselves.
For the flip side, consider Fowler's anecdote on page 68-69, which
ends "Even if you know exactly what is going on in your system,
measure performance, don't speculate. You'll learn something, and
nine times out of ten it won't be that you were right." The Unix guy
in me wants to respond "Well, duh!". In my tribe, profiling
before you speculate is DNA; we have a strong tradition of
this that goes back to the 1970s. From the point of view of any old
Unix hand, the fact that Fowler thought he had to write this down is a
sign of severe naivete in either Fowler or his readership or both.
In reading Refactoring, I several times had the
experience of thinking "What!?! That's obvious!" closely followed
by "But Fowler explains it better than Unix traditions do..." This may
be because he relies less on the very rich shared explanatory context
that Unix provides.
How deep do the similarities run? Let's take a look at what the
Agile Manifesto says:
Individuals and interactions over processes and tools. Yeah,
that sounds like us, all right. Open-source developers will toss out
a process that isn't working in a nanosecond, and frequently do, and take
gleeful delight in doing so. In fact, the reaction against heavyweight
process has a key part of our self-identification as hackers for
at least the last quarter century, if not longer.
Working software over comprehensive documentation. That's
us, too. In fact, the radical hacker position is that source code of
a working system is its documentation. We, more than any
other culture of software engineering, emphasize program source code as
human-to-human communication that is expected to bind together
communities of cooperation and understanding distributed through time
and space. In this, too, we build on and amplify Unix tradition.
Customer collaboration over contract negotiation. In the
open-source world, the line between "developer" and "customer" blurs
and often disappears. Non-technical end users are represented by
developers who are proxies for their interestsas when, for
example, companies that run large websites second developers to
work on Apache Software Foundation projects.
Responding to change over following a plan. Absolutely.
Our whole development style encourages this. It's fairly unusual for
any of our projects to have any plan more elaborate than "fix
the current bugs and chase the next shiny thing we see".
With these as main points, it's hardly surprising that so many of
behind the Agile Manifesto read like Unix-tradition and hacker
gospel. "Deliver working software frequently, from a couple of weeks
to a couple of months, with a preference to the shorter timescale.
Well, yeahwe pioneered this. Or "Simplicitythe art of
maximizing the amount of work not doneis essential." That's
Unix-tradition holy writ, there. Or "The best architectures,
requirements, and designs emerge from self-organizing teams."
This is stone-obvious stuff to any hacker, and exactly the sort of
subversive thinking that most panics managers attached to big plans,
big budgets, big up-front design, and big rigid command-and-control
structures. Which may, in fact, be a key part of its appeal to
hackers and agile developersbecause at least one thing that points
agile-movement and open-source people in the same direction is a drive
to take control of our art back from the suits and get out from under
big dumb management.
The most important difference I see between the hackers and the
agile-movement crowd is this: the hackers are the people who never
surrendered to big dumb management they either bailed out of the
system or forted up in academia or inustrial R&D labs or
technical-specialty areas where pointy-haired bosses weren't permitted
to do as much damage. The agile crowd, on the other hand, seems to be
composed largely of people who were swallowed into the belly of the
beast (waterfall-model projects, Windows, the entire conventional
corporate-development hell so vividly described in David Yourdon's
books) and have been smart enough not just to claw their way out but
to formulate an ideology to justify not getting sucked back in.
Both groups are in revolt against the same set of organizational
assumptions. And both are winning because those assumptions are
obsolete, yesterday's adaptations to a world of expensive machines and
expensive communications. But software development doesn't need big
concentrations of capital and resources anymore, and doesn't need the
control structures and hierarchies and secrecy and elaborate rituals
that go with managing big capital concentrations either. In fact, in
a world of rapid change, these things are nothing but a drag. Thus
agile techniques. Thus, open source. Converging paths to the same
destination, which is not just software that doesn't suck but a
software-development process that doesn't suck.
When I think about how the tribal wisdom of the hackers and the
sharp cut-the-bullshit insights of the agile movement seem to be
coming together, my mind keeps circling back to Phil Greenspun's brief
but trenchant essay Redefining
Professionalism for Software Engineers. Greenspun proposes,
provocatively but I think correctly, that the shift towards
open-source development is a key part of the transformation of
software engineering into a mature profession, with the dedication to
excellence and ethos of service that accompanies professionalism. I
have elsewhere suggested that we are seeing a close historical analog
of the transition from alchemy to chemistry. Secrets leak out, but
skill sustains; the necessity to stop relying on craft secrecy is one
of the crises that occupational groups normally face as they attain
I'm beginning to think that from the wreckage of the software
industry big dumb management made, I can see the outline of a mature,
humane discipline of software engineering emerging and
that it will be in large part a fusion of the responsiveness and
customer focus of the agile movement with the wisdom and groundedness
of the Unix tradition, expressed in open source.
> > If you could make an analogy then, XP is industrial > quality hacking. Hacking on the other hand, is what any > good developer does instinctively, it benefits the > individual as opposed to the team.
That was exactly my feeling about this too. The project I'm working on is using XP and I'm pushing the unit-testing stuff very hard.
Nice essay! For a lot of the essay I nodded my head in agreement. At the risk of sounding argumentative and in the hope of providing useful feedback, I'll point out where I wasn't in agreement.
my first take on Fowler's statement was to think "Dude, you've just described hacking!"
I've had that impression from the beginning regarding XP and Agile programming. Except I came from the other end of the scale -- that is, not out of excitement for the formalization and legitimization of the methods, but out of disappointment. My biggest issue is that prototypes don't get thrown away. They become legacy, and prevent other decisions from being made.
I am being won over to a number of XP/Agile methodologies ... especially test driven development and you-aint-gonna-need-it. I have always accepted refactoring as a necessary evil, though now less evil than previously.
I think the biggest exception I take with XP and Agile is using it as an excuse to not try to think through problems beforehand.
people who are excited by agile-programming ideas can look to open source and the Unix tradition and the hackers for the lessons of experience.
While I agree with this statement, I got the feeling through this essay that you were not being distinct between Unix tradition, hackers, and open source, and this line is probably the beginning of that feeling.
Open-source developers will toss out a process that isn't working in a nanosecond, and frequently do, and take gleeful delight in doing so.
I notice you don't point out that open-source developers can also hang on to dysfunctional processes. It's probably worth noting that the set of dysfunctional processes is probably much larger than functional ones.
the radical hacker position is that source code of a working system is its documentation.
This depends on what you mean by documentation. The source code documents what the system does, but usually not why it does it that way or how to interact with it.
More specifically, the source code is the specification of the (presumably) working system. I'm growing fond of simpler languages and tools that provide a view the structure of a piece of software.
Responding to change over following a plan. Absolutely. Our whole development style encourages this.
Again, which community are you referring to by "Our"? To say that all open-source projects are hackers is incorrect, even if the correlation is relatively high.
Greenspun proposes [...] the shift towards open-source development is a key part of the transformation of software engineering into a mature profession
I don't think I agree with this. I don't think that open source focuses on excellence in all categories. I think open source succeeds because it excels in some number of Greenspun's criteria. I don't think that means that open source is helping causing a shift of software engineering into a mature profession. The percentage of open source projects that make a huge dent in Greenspun's list of 7 objectives is (from my experience) pretty small.
I have elsewhere suggested that we are seeing a close historical analog of the transition from alchemy to chemistry.
Or art to science? You don't repeat your logic (or link to it), so I don't know exactly what you're referring to. But if you are correct, I don't see the link from hacking and open source to science and engineering.
We are seeing a close historical analog of the transition from alchemy to chemistry because that, too, involved giving up secrecy in favor of process transparency and peer review. That change was necessary in order for the study of chemical phenomena to move from obfuscation to engineering, from black art practiced by cryptic masters to a discipline with repeatable results. So too with software.
I think if you re-read my essay, you'll find my reasons for not bothering to distinguish those three categories of people are quite explicitly stated.
Finally, I recommend you reread Phil Greenspun's essay and note where he explicitly says "open-source is the one true path for a professional software engineer". OK, let's concede that the percentage of open source projects that make a huge dent in Greenspun's list of 7 objectives is, as you say, pretty small. All that establishes is that open source is not a sufficient condition for professionalism. But neither Phil nor I have ever so claimed; we are saying rather that open source is a necessary condition that must be supplemented by other behaviors.
I am not sure I agree with the proposed parallel of the hacking-unix culture and the agile methodologies. As the basis of his conclusion Eric extracts some concepts (individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, responding to change over following a plan, ...) out of the Agile Manifesto to make the comparison, using himself as the reference for the hacking / open source movement. (I don't dare to disagree :-))
It is strange how the gist of the manifesto for me is completly different from his 'main points'. I believe the key phrases to be 'face-to-face conversation', 'business people and developers must work together daily throughout the project' and 'simplicity--the art of maximizing the amount of work not done--is essential'. A group of people, programmers and users, that interact as close, in time and space, as possible is at the core of agile development. For an agile developer all of the rest ( like small iterations, not too much up front design, ... ) is merely a tool, a mean to a achive a goal.
I also don't like Eric's use of the phrase "big dumb management": isn't that the real cultural gap between hackers and agile developers: we (I'll classify myself with the latter for now...) try to align development with business needs, we try to add value to the company / community / ... , while hackers are in it for technical mastrubation, peer appreciation or pushing mankind another great step forward.
Actually, the conference took place in Snowbird, Utah. Sorry, for the nitpick, but just want to make sure Utah gets put on the map, as a few of the significant contributions to the Agile movement have happened in Utah.
I just finished reading the Unix Haters Handbook (http://research.microsoft.com/~daniel/uhh-download.html). One of its main complaints is that Unix warts are not refactored; people just learn to work around them and rationalize the non-refactoring with "we're keeping backward compatibility".
I use Unix occasionally and I do not hate it. But I wonder how much refactoring has actually ocurred in open-source Unix. Clean code is a tenet of agile programming. From what I've seen of hacker code it's a very mixed bag.
I was MIT Press's first-line technical reviewer for the Unix Hater's Handbook. While their case is not completely without merit, the book is basically an exercise in flaming and hysteria. The authors wanted it that way; they staunchly resisted all attempts by myself and others to press them into writing a more substantive critique. This is a shame; some of their arguments deserved to be taken seriously.
A lot of the "we're keeping backward compatibility" cruft was an adaptive response to closed-source Unix, having lots of stuff around you couldn't modify and rebuild. Open-source practice has since gone much further in the direction of scrap-and-rebuild (or refactor).
Nevertheless you're right. Hacker code is a mixed bag. But that's not because the hard-core types don't understand code cleanliness and good practice, it's because a lot of the community is amateurs and autodidacts who are learning as they go. (I consider this a good thing; I prefer cultures like this to cultures that are bogged down in credentialism and exclusivity.)
My new book is aimed straight at these people. They're willing to do the work to write clean code, if anyone will teach them how. The hacker culture is healthy that way, there is a lot of social reinforcement for coding not just the expedient thing but the Right Thing.
You looked at the Agile principles & practices from an hacker point of view and realized there is a connection. More than a year ago a realized the same connection but the other way round looking at you 'The Cathedral and the Bazaar' from an Agile point of view: http://c2.com/cgi/wiki?OpenSourceAsAgileProcess :)