The Artima Developer Community
Sponsored Link

Weblogs Forum
Hacking and Refactoring

16 replies on 2 pages. Most recent reply: Sep 28, 2005 12:17 PM by Christopher L. Marshall

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 16 replies on 2 pages [ 1 2 | » ]
Eric S. Raymond

Posts: 6
Nickname: esr
Registered: Jun, 2003

Hacking and Refactoring (View in Weblogs)
Posted: Jun 14, 2003 8:38 PM
Reply to this message Reply
Summary
The open-source movement and agile programming may be converging. While reading Martin Fowler's excellent book "Refactoring", I realized development by refactoring is a sharp description of the normal style of open-source hackers. In this essay (which Martin Fowler and Kent Beck and I discussed in draft) I explore the connection further.
Advertisement

In 2001, there was a history-making conference of software-engineering thinkers in Snowbird, Colorado. The product of that meeting was a remarkable document called the Agile Manifesto, a call to overturn many of the assumptions of traditional software development. I was invited to be at Snowbird, but couldn't make it.

Ever since, though, I've been sensing a growing convergence between agile programming and the open-source movement. I've seen agile concepts and terminology being adopted rapidly and enthusiastically by my colleagues in open-source-land—especially ideas like refactoring, unit testing, and design from stories and personas. From the other side, key agile-movement figures like Kent Beck and Martin Fowler have expressed strong interest in open source both in published works and to me personally. Fowler has gone so far as to include open source on his list of agile-movement schools.

I agree that we belong on that list. But I also agree with Fowler's description of of open source as a style, rather than a process. I think his reservations as to whether open source can be described as just another agile school are well-founded. There is something more complicated and interesting going on here. and I realized when I read Fowler's description of open source that at some point I was going to have to do some hard thinking and writing in an effort to sort it all out.

While doing research for my forthcoming book, The Art of Unix Programming, I read one particular passage in Fowler's Refactoring that finally brought it all home. He writes:

One argument is that refactoring can be an alternative to up-front design. In this scenario, you don't do any design at all. You just code the first approach that comes into your head, get it working, and then refactor it into shape. Actually, this approach can work. I've seen people do this and come out with a very well-defined piece of software. Those who support Extreme Programming often are portrayed as advocating this approach.

I read this, and had one of those moments where everything comes together in your head with a great ringing crash and the world assumes a new shape—a moment not unlike the one I had in late 1996 when I got the central insight that turned into The Cathedral and the Bazaar. In the remainder of this essay I'm going to try to articulate what I now think I understand about open source, agile programming, how they are related, and why the connection should be interesting even to programmers with no stake in either movement.

Now I need to set a little background here, because I'm going to need to have to talk about several different categories which are contingently but not necessarily related.

First, there is Unix programmer. Unix is the operating system with the longest living tradition of programming and design. It has an unusually strong and mature technical culture around it, a culture which originated or popularized many of the core ideas and tools of modern software design. The Art of Unix Programming is a concerted attempt to capture the craft wisdom of this culture, one to which I have successfully enlisted quite a few of its founding elders.

Second, there is hacker. This is a very complex term, but more than anything else, it describes an attitude—an intentional stance that relates hackers to programming and other disciplines in a particular way. I have described the hacker stance and its cultural correlates in detail in How To Become A Hacker.

Third, there is open-source programmer. Open source is a programming style with strong roots in the Unix tradition and the hacker culture. I wrote the modern manifesto for it in 1997, The Cathedral and the Bazaar, building on earlier thinking by Richard Stallman and others.

These three categories are historically closely related. It is significant that a single person (accidentally, me) wrote touchstone documents for the second and third and is attempting a summum bonum of the first. That personal coincidence reflects a larger social reality that in 2003 these categories are becoming increasingly merged — essentially, the hacker community has become the core of the open-source community, which is rapidly re-assimilating the parts of the Unix culture that got away from the hackers during the ten bad years after the AT&T divestiture in 1984.

But the relationship is not logically entailed; we can imagine a hacker culture speaking a common tongue other than Unix and C (in the far past its common tongue was Lisp), and we can imagine an explicit ideology of open source developing within a cultural and technical context other than Unix (as indeed nearly happened several different times).

With this scene-setting done, I can explain that my first take on Fowler's statement was to think "Dude, you've just described hacking!"

I mean something specific and powerful by this. Throwing together a prototype and refactoring it into shape is a rather precise description of the normal working practice of hackers since that culture began to self-define in the 1960s. Not a complete one, but it captures the most salient feature of how hackers relate to code. The open-source community has inherited and elaborated this practice, building on similar tendencies within the Unix tradition.

The way Fowler writes about design-by-refactoring has two huge implications for the relationship between open source and agile programming:

First, Fowler writes as though he didn't know he was describing hacking. In the passage, he appears unaware that design by repeated refactoring is not just a recent practice semi-accidentally stumbled on by a handful of agile programmers, but one which hundreds of thousands of hackers have accumulated experience with for over three decades and have in their bones. There is a substantial folklore, an entire craft practice, around this!

Second, in that passage Fowler described the practice of hacking better than hackers themselves have done. Now, admittedly, the hacker culture has simply not had that many theoreticians, and if you list the ones that are strongly focused on development methodology you lose Richard Stallman and are left with, basically, myself and maybe Larry Wall (author of Perl and occasional funny and illuminating ruminations on the art of hacking). But the fact that we don't have a lot of theoreticians is itself an important datum; we have always tended to develop our most important wisdoms as unconscious and unarticulated craft practice.

These two observations imply an enormous mutual potential, a gap across which an arc of enlightenment may be beginning to blaze. It implies two things:

First, people who are excited by agile-programming ideas can look to open source and the Unix tradition and the hackers for the lessons of experience. We've been doing a lot of the stuff the agile movement is talking about for a long time. Doing it in a clumsy, unconscious, learned-by-osmosis way, but doing it nevertheless. I believe that we have learned things that you agile guys need to know to give your methodologies groundedness. Things like (as Fowler himself observes) how to manage communication and hierarchy issues in distributed teams.

Second, open-source hackers can learn from agile programmers how to wake up. The terminology and conceptual framework of agile programming sharpens and articulates our instincts. Learning to speak the language of open source, peer review, many eyeballs, and rapid iterations gave us a tremendous unifying boost in the late 1990s; I think becoming similarly conscious about agile-movement ideas like refactoring, unit testing, and story-centered design could be just as important for us in the new century.

I've already given an example of what the agile movement has to teach the hackers, in pointing out that repeated redesign by refactoring is a precise description of hacking. Another thing we can stand to learn from agile-movement folks is how to behave so that we can actually develop requirements and deliver on them when the customer isn't, ultimately, ourselves.

For the flip side, consider Fowler's anecdote on page 68-69, which ends "Even if you know exactly what is going on in your system, measure performance, don't speculate. You'll learn something, and nine times out of ten it won't be that you were right." The Unix guy in me wants to respond "Well, duh!". In my tribe, profiling before you speculate is DNA; we have a strong tradition of this that goes back to the 1970s. From the point of view of any old Unix hand, the fact that Fowler thought he had to write this down is a sign of severe naivete in either Fowler or his readership or both.

In reading Refactoring, I several times had the experience of thinking "What!?! That's obvious!" closely followed by "But Fowler explains it better than Unix traditions do..." This may be because he relies less on the very rich shared explanatory context that Unix provides.

How deep do the similarities run? Let's take a look at what the Agile Manifesto says:

Individuals and interactions over processes and tools. Yeah, that sounds like us, all right. Open-source developers will toss out a process that isn't working in a nanosecond, and frequently do, and take gleeful delight in doing so. In fact, the reaction against heavyweight process has a key part of our self-identification as hackers for at least the last quarter century, if not longer.

Working software over comprehensive documentation. That's us, too. In fact, the radical hacker position is that source code of a working system is its documentation. We, more than any other culture of software engineering, emphasize program source code as human-to-human communication that is expected to bind together communities of cooperation and understanding distributed through time and space. In this, too, we build on and amplify Unix tradition.

Customer collaboration over contract negotiation. In the open-source world, the line between "developer" and "customer" blurs and often disappears. Non-technical end users are represented by developers who are proxies for their interests—as when, for example, companies that run large websites second developers to work on Apache Software Foundation projects.

Responding to change over following a plan. Absolutely. Our whole development style encourages this. It's fairly unusual for any of our projects to have any plan more elaborate than "fix the current bugs and chase the next shiny thing we see".

With these as main points, it's hardly surprising that so many of the Principles behind the Agile Manifesto read like Unix-tradition and hacker gospel. "Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Well, yeah—we pioneered this. Or "Simplicity—the art of maximizing the amount of work not done—is essential." That's Unix-tradition holy writ, there. Or "The best architectures, requirements, and designs emerge from self-organizing teams."

This is stone-obvious stuff to any hacker, and exactly the sort of subversive thinking that most panics managers attached to big plans, big budgets, big up-front design, and big rigid command-and-control structures. Which may, in fact, be a key part of its appeal to hackers and agile developers—because at least one thing that points agile-movement and open-source people in the same direction is a drive to take control of our art back from the suits and get out from under big dumb management.

The most important difference I see between the hackers and the agile-movement crowd is this: the hackers are the people who never surrendered to big dumb management — they either bailed out of the system or forted up in academia or inustrial R&D labs or technical-specialty areas where pointy-haired bosses weren't permitted to do as much damage. The agile crowd, on the other hand, seems to be composed largely of people who were swallowed into the belly of the beast (waterfall-model projects, Windows, the entire conventional corporate-development hell so vividly described in David Yourdon's books) and have been smart enough not just to claw their way out but to formulate an ideology to justify not getting sucked back in.

Both groups are in revolt against the same set of organizational assumptions. And both are winning because those assumptions are obsolete, yesterday's adaptations to a world of expensive machines and expensive communications. But software development doesn't need big concentrations of capital and resources anymore, and doesn't need the control structures and hierarchies and secrecy and elaborate rituals that go with managing big capital concentrations either. In fact, in a world of rapid change, these things are nothing but a drag. Thus agile techniques. Thus, open source. Converging paths to the same destination, which is not just software that doesn't suck but a software-development process that doesn't suck.

When I think about how the tribal wisdom of the hackers and the sharp cut-the-bullshit insights of the agile movement seem to be coming together, my mind keeps circling back to Phil Greenspun's brief but trenchant essay Redefining Professionalism for Software Engineers. Greenspun proposes, provocatively but I think correctly, that the shift towards open-source development is a key part of the transformation of software engineering into a mature profession, with the dedication to excellence and ethos of service that accompanies professionalism. I have elsewhere suggested that we are seeing a close historical analog of the transition from alchemy to chemistry. Secrets leak out, but skill sustains; the necessity to stop relying on craft secrecy is one of the crises that occupational groups normally face as they attain professional standing.

I'm beginning to think that from the wreckage of the software industry big dumb management made, I can see the outline of a mature, humane discipline of software engineering emerging — and that it will be in large part a fusion of the responsiveness and customer focus of the agile movement with the wisdom and groundedness of the Unix tradition, expressed in open source.


Joseph Grace

Posts: 6
Nickname: occam
Registered: Mar, 2003

Wow, grand unification theory. Posted: Jun 15, 2003 1:35 AM
Reply to this message Reply
Yikes, the end meets the beginning. Nice article, mind-bender.

Carlos Perez

Posts: 153
Nickname: ceperez
Registered: Jan, 2003

Re: Hacking and Refactoring Posted: Jun 16, 2003 9:31 AM
Reply to this message Reply
No grand unification occurring here.

The analogies are nice, but they're a bit out of context. Here's why:

Agile methodologies center around team development and the falliability of people. That is why there's something called test driven development. TDD is conspicously absent in the activity of hacking.

Simply because hacking centers around the individual, and hacking implicit assumes the prowess of the individual.

If you could make an analogy then, XP is industrial quality hacking. Hacking on the other hand, is what any good developer does instinctively, it benefits the individual as opposed to the team.

Chris Hartjes

Posts: 12
Nickname: bloodshot
Registered: May, 2003

Re: Hacking and Refactoring Posted: Jun 16, 2003 12:10 PM
Reply to this message Reply
>
> If you could make an analogy then, XP is industrial
> quality hacking. Hacking on the other hand, is what any
> good developer does instinctively, it benefits the
> individual as opposed to the team.

That was exactly my feeling about this too. The project I'm working on is using XP and I'm pushing the unit-testing stuff very hard.

Johan Lindström

Posts: 3
Nickname: jplindstro
Registered: Jun, 2003

Re: Hacking and Refactoring Posted: Jun 16, 2003 6:45 PM
Reply to this message Reply
"TDD is conspicously absent in the activity of hacking."

I was thinking kind of the same thing.

But on the other hand, I was introduced to the culture of unit and regression testing through Perl and CPAN. That's what got me interested in XP practices.

So, more examples of TDD and unit testing in OSS anyone?


/J - incidentally with Fowler's Refactoring on the night stand :)

Eric S. Raymond

Posts: 6
Nickname: esr
Registered: Jun, 2003

Re: Hacking and Refactoring Posted: Jun 17, 2003 10:51 AM
Reply to this message Reply
TDD is beginning to take hold in the Python community as well.

Arnaldo Riquelme

Posts: 1
Nickname: arnaldo
Registered: Jun, 2003

Re: Hacking and Refactoring Posted: Jun 17, 2003 1:41 PM
Reply to this message Reply
The open source project Mono http://go-mono.com/testing.html
is also using TDD extensively.

-ajr

Chris Dailey

Posts: 56
Nickname: mouse
Registered: Dec, 2002

Re: Hacking and Refactoring Posted: Jun 18, 2003 12:30 AM
Reply to this message Reply
Greetings, Eric,

Nice essay! For a lot of the essay I nodded my head in agreement. At the risk of sounding argumentative and in the hope of providing useful feedback, I'll point out where I wasn't in agreement.

my first take on Fowler's statement was to think "Dude, you've just described hacking!"

I've had that impression from the beginning regarding XP and Agile programming. Except I came from the other end of the scale -- that is, not out of excitement for the formalization and legitimization of the methods, but out of disappointment. My biggest issue is that prototypes don't get thrown away. They become legacy, and prevent other decisions from being made.

I am being won over to a number of XP/Agile methodologies ... especially test driven development and you-aint-gonna-need-it. I have always accepted refactoring as a necessary evil, though now less evil than previously.

I think the biggest exception I take with XP and Agile is using it as an excuse to not try to think through problems beforehand.

people who are excited by agile-programming ideas can look to open source and the Unix tradition and the hackers for the lessons of experience.

While I agree with this statement, I got the feeling through this essay that you were not being distinct between Unix tradition, hackers, and open source, and this line is probably the beginning of that feeling.

Open-source developers will toss out a process that isn't working in a nanosecond, and frequently do, and take gleeful delight in doing so.

I notice you don't point out that open-source developers can also hang on to dysfunctional processes. It's probably worth noting that the set of dysfunctional processes is probably much larger than functional ones.

the radical hacker position is that source code of a working system is its documentation.

This depends on what you mean by documentation. The source code documents what the system does, but usually not why it does it that way or how to interact with it.

More specifically, the source code is the specification of the (presumably) working system. I'm growing fond of simpler languages and tools that provide a view the structure of a piece of software.

Responding to change over following a plan. Absolutely. Our whole development style encourages this.

Again, which community are you referring to by "Our"? To say that all open-source projects are hackers is incorrect, even if the correlation is relatively high.

Greenspun proposes [...] the shift towards open-source development is a key part of the transformation of software engineering into a mature profession

I don't think I agree with this. I don't think that open source focuses on excellence in all categories. I think open source succeeds because it excels in some number of Greenspun's criteria. I don't think that means that open source is helping causing a shift of software engineering into a mature profession. The percentage of open source projects that make a huge dent in Greenspun's list of 7 objectives is (from my experience) pretty small.

I have elsewhere suggested that we are seeing a close historical analog of the transition from alchemy to chemistry.

Or art to science? You don't repeat your logic (or link to it), so I don't know exactly what you're referring to. But if you are correct, I don't see the link from hacking and open source to science and engineering.

Thanks for your insightful essay!

Eric S. Raymond

Posts: 6
Nickname: esr
Registered: Jun, 2003

Re: Hacking and Refactoring Posted: Jun 18, 2003 5:53 AM
Reply to this message Reply
We are seeing a close historical analog of the transition from alchemy to chemistry because that, too, involved giving
up secrecy in favor of process transparency and peer review. That change was necessary in order for the study of chemical phenomena to move from obfuscation to engineering, from black art practiced by cryptic masters to a discipline with repeatable results. So too with software.

I think if you re-read my essay, you'll find my reasons for not bothering to distinguish those three categories of people are quite explicitly stated.

Finally, I recommend you reread Phil Greenspun's essay and note where he explicitly says "open-source is the one true path for a professional software engineer". OK, let's concede that the percentage of open source projects that make a huge dent in Greenspun's list of 7 objectives is, as you say, pretty small. All that establishes is that open source is not a sufficient condition for professionalism. But neither Phil nor I have ever so claimed; we are saying rather that open source is a necessary condition that must be supplemented by other behaviors.

Mik Lernout

Posts: 10
Nickname: miklernout
Registered: Jan, 2003

Re: Hacking and Refactoring Posted: Jun 18, 2003 9:44 AM
Reply to this message Reply
I am not sure I agree with the proposed parallel of the hacking-unix culture and the agile methodologies. As the basis of his conclusion Eric extracts some concepts (individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, responding to change over following a plan, ...) out of the Agile Manifesto to make the comparison, using himself as the reference for the hacking / open source movement. (I don't dare to disagree :-))

It is strange how the gist of the manifesto for me is completly different from his 'main points'. I believe the key phrases to be 'face-to-face conversation', 'business people and developers must work together daily throughout the project' and 'simplicity--the art of maximizing the amount of work not done--is essential'. A group of people, programmers and users, that interact as close, in time and space, as possible is at the core of agile development. For an agile developer all of the rest ( like small iterations, not too much up front design, ... ) is merely a tool, a mean to a achive a goal.

I also don't like Eric's use of the phrase "big dumb management": isn't that the real cultural gap between hackers and agile developers: we (I'll classify myself with the latter for now...) try to align development with business needs, we try to add value to the company / community / ... , while hackers are in it for technical mastrubation, peer appreciation or pushing mankind another great step forward.

Rod Schmidt

Posts: 1
Nickname: schromosan
Registered: Jun, 2003

Re: Hacking and Refactoring Posted: Jun 19, 2003 5:29 PM
Reply to this message Reply
Actually, the conference took place in Snowbird, Utah. Sorry, for the nitpick, but just want to make sure Utah gets put on the map, as a few of the significant contributions to the Agile movement have happened in Utah.

Ian Rae

Posts: 21
Nickname: ianrae
Registered: May, 2003

Re: Hacking and Refactoring Posted: Jun 20, 2003 3:54 PM
Reply to this message Reply
I just finished reading the Unix Haters Handbook (http://research.microsoft.com/~daniel/uhh-download.html). One of its main complaints is that Unix warts are not refactored; people just learn to work around them and rationalize the non-refactoring with "we're keeping backward compatibility".

I use Unix occasionally and I do not hate it. But I wonder how much refactoring has actually ocurred in open-source Unix. Clean code is a tenet of agile programming. From what I've seen of hacker code it's a very mixed bag.

Eric S. Raymond

Posts: 6
Nickname: esr
Registered: Jun, 2003

Unix-Hater's Handbook Posted: Jun 20, 2003 4:40 PM
Reply to this message Reply
I was MIT Press's first-line technical reviewer for the Unix Hater's Handbook. While their case is not completely without merit, the book is basically an exercise in flaming and hysteria. The authors wanted it that way; they staunchly resisted all attempts by myself and others to press them into writing a more substantive critique. This is a shame; some of their arguments deserved to be taken seriously.

A lot of the "we're keeping backward compatibility" cruft was an adaptive response to closed-source Unix, having lots of stuff around you couldn't modify and rebuild. Open-source practice has since gone much further in the direction of scrap-and-rebuild (or refactor).

Nevertheless you're right. Hacker code is a mixed bag. But that's not because the hard-core types don't understand code cleanliness and good practice, it's because a lot of the community is amateurs and autodidacts who are learning as they go. (I consider this a good thing; I prefer cultures like this to cultures that are bogged down in credentialism and exclusivity.)

My new book is aimed straight at these people. They're willing to do the work to write clean code, if anyone will teach them how. The hacker culture is healthy that way, there is a lot of social reinforcement for coding not just the expedient thing but the Right Thing.

Marco Abis

Posts: 325
Nickname: marcoabis
Registered: Jun, 2003

Re: Hacking and Refactoring Posted: Jun 25, 2003 8:14 AM
Reply to this message Reply
You looked at the Agile principles & practices from an hacker point of view and realized there is a connection. More than a year ago a realized the same connection but the other way round looking at you 'The Cathedral and the Bazaar' from an Agile point of view: http://c2.com/cgi/wiki?OpenSourceAsAgileProcess :)

Mike McClure

Posts: 2
Nickname: mcclure
Registered: Nov, 2003

Re: Hacking and Refactoring Posted: Nov 24, 2003 12:09 AM
Reply to this message Reply
TDD is conspicously absent in the activity of hacking.

imho tdd is conspicuously central to the activity of hacking insofar that hacking is usually done to a achieve a very specific purpose, generally with a binary view of success.

so where is the testing? the proof is (as they say) in the pudding - does the software do what i want it to, or not?

admittedly it is not a sophisticated form of testing such as that to which we are becoming accustomed but it is there and oh-so-very central.

Flat View: This topic has 16 replies on 2 pages [ 1  2 | » ]
Topic: Representing Units of Measure Previous Topic   Next Topic Topic: XML down a slippery slope


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us