How much thought did you put into the tradeoffs of the last technique you brought into your organization: Ajax, TDD, On-Site Customer, or other buzzwords? Did you research its track record? Or did you go to the buzzword yellow pages? Checklists and ceremonies border on religion, and developers seem to hold to them with religious fervor.
I've been packaging my learnings about TDD from the past 20 years and taking them on the road these days. The reaction has been visceral and hardly subtle. I think that's good, as this is a topic on which we need much dialog. The dialog has kept me busy, searching 'blogs and entering the discussion:
This 'blog was inspired by Grig Gheorghiu's 'blog on on Agile Testing that agonized over the question of whether to do TDD or acceptance testing.
My initial reply, on that site, provides the foundation on which the following post builds.
The Wrong Question
We seem to be looking for the right checklist that will give us quality. The checklist at hand has TDD and acceptance testing on it and we're asked to check zero, one, or two boxes.
I think it's the wrong question.
The objective is to deliver what the customer wanted while optimizing quality; the broader goal is to create value. Quality suffers if you misunderstand what the customer wanted, or if the code has internal interactions that are difficult to foresee, or if local code violates array boundaries or uses untethered pointers. It also suffers if the interface makes it possible for the user to commit errors, or if it takes too many keystrokes to enter a small amount of common information. Testing covers only a fraction of these and does so in the least efficient way known to our industry.
For some reason, we have invented and are following religions, or perhaps even cults, intent on ceremonies and rituals rather than thinking or inspection. We sometimes even throw in platitudes that are to pass for "values," but there is little talk about the kind of value that an end user would care about. We now even need to make it explicit that there are methodologies that encourage "inspect and adapt," having long ago forgotten that the word "software" contains the word "soft." One role of religion is to remind us what is important in matters of identity and relationship which, curiously enough, both figure prominently in software development. After my talk at JavaZone Trygve Reenskaug came up to me and said, "You just gave a talk where you highlighted the importance of the user interface and of domain knowledge. And you had to tell them that!"
Discussions in this area inspire passion. Why? If I were to attack 40-hour week people would just roll their eyes. The sober and sobering evaluations of On-Site Customer haven't led to a revolt. Why TDD? Gertrud Bjørnvig and I were talking about this the other day and agree that it could be because TDD is the one aspect of Agile that is personal. People have tied their Agile success to their introduction of TDD practices into their work. It's the one thing they can do as individuals rather than as an enterprise. Pair programming comes close, and if it weren't for the fact that it works at least half the time, any research that criticised it would likely be received with equal emotional content. It becomes a religious issue.
Religion can be grounded in ceremonies and rituals that long have lost any tie to value, if they ever had such ties at all. The industry is at an uncomfortable juncture where we cannot and should not take some things for granted, and yet we do. We are told that you have to do TDD to be a professional (in another great Trygve moment and in response to this passionate plea, Trygve and I gleefully agreed that we must not be professionals) and yet are given no reason to believe this, no substantiation, and no evidence. Just believe.
(Of course, one problem with religions is that there are so many of them. In our recent explorations of TDD we have found that it means so many things to so many different people that it's no surprise that there is resentment and confusion. We simply aren't communicating that. Watch for Roy Osherove to pick up that thread in his 'blog.)
These days, all things testing seem to be moving to the forefront of these religious debates. The 'blog I pointed to above that led me to write this one found itself in the middle of this debate: Are you of the test-last religion or the TDD religion? The answer (credible, but unsupported by data) ended being a universal platitude of: "Do both." Why are we focusing so much on testing, or even on TDD in particular, which even Kent
Beck defines as a state-of-mind kind of thing that extends to concepts such as spending a week iterating a paper design and then test-driving the code? (Yes, he uses exactly those words.)
Raising the Bar
I can tell you where my current experience--including what I've garnered from a broad range of clients and from academic publications alike--points.
Focus on quality assurance instead of testing. That means using Use Cases (which are an Agile way of engaging customers—in the sense of your full constituency, not just the sacrificial victim the company sends as the On-Site Customer) instead of XP-style user stories (so you understand feature interactions up-front), doing pair programming or desk-check code inspections, doing design by contract, driving the process with usability experts and doing good usability testing, and a host of other things.
On one hand we should be encouraged to use every weapon in our arsenal; on the other hand, we'll never get done if we try to do everything. It then becomes a matter of cost effectiveness. Integration and system testing have long been demonstrated to be the least efficient way of finding bugs. Recent studies (Siniaalto and Abrahamsson) of TDD show that it may have no benefits over traditional test-last development and that in some cases has deteriorated the code and that it has other alarming (their word) effects. The one that worries me the most is that it deteriorates the architecture. And acceptance testing is orders of magnitude less efficient than good old-fashioned code inspections, extremely expensive, and comes too late to keep the design clean.
Jef Raskin often said that the "interface is the program." If we're going to do testing, that's where the focus should lie. The interface is our link to the end user. Agile is about engaging that user; TDD, about engaging the code. TDD is a methodology. TDD is about tools and processes over people and interactions. Woops. Agile pundits go so far as to marginalize the end-user-facing side of software (read the first and foremost point of the articulated Philosophy. Perhaps Philosophy is supposed to be more objective than religion).
Whither Systems Engineering?
Sorting these things out requires a system view. It used to be called "systems engineering," but the New Age Agile movements consistently move away from such practices. It shows in the quality of our interfaces, in the incredible code bloat we have today relative to two decades ago, and in the fact that we keep getting stuck in practices that look good from a simple analysis, but which fall apart when you look at them closer. Pair programming has been getting mixed results (summarization by Abrahamsson); TDD is under attack (North, who calls TDD "a series of blind alleys"; Siniaalto and Abrahamsson, who call the findings about TDD "alarming"; Bjørnvig, Coplien and Harrison, who suggest practices that can help); on-site customer has also been at least partially discredited (Martin and Noble).
The answer for industry, I think, is to focus on Quality and to find the techniques that work best and are the most cost-effective for your project. Just answering "both" to the "TDD-or-acceptance" question is not only at the wrong scope, it is missing the whole point.
The Amazing Shrinking Academic Collective Mind
It's about thinking, not about checklists. I think that one root of the problem lies in modern education, which has increased its focus on technique and reduced its focus on thinking. I also think that because of a hot job market more and more people are getting into the industry before having had enough exposure to academia to properly develop their thinking tools. Fewer and fewer job applicants have Masters' degrees; fewer and fewer students in Denmark wait to finish even their undergraduate before succumbing to the lure of money and their chance to make a mark on the world. Also, fewer and fewer programmers understand software history. Not knowing design-by-contract or Clean Room, they never know to even ask the question of whether TDD can be as effective as they can in exactly the same improvements to code.
There was a time when good schools afforded their graduates at least the basics of systems engineering thinking. The notion of a "liberal education" used to be valued, even for computer scientists. I haven't heard these words in a good long while. At North Central College, where I was recently a full-time Associate Professor, management's most-discussed idea of progress was to move to a Java curriculum. I think their idea of thinking skills reduced largely to a course in first-order logic. Students in first-year programming classes taught by my colleagues have stronger memories of being corrected on their commenting conventions and on exactly when to use --v and when to use v-- than they recall anything about logical design.
...and a place for industry?
Such grounding, of course, makes it difficult for industry to rise to the challenge I posed above: to focus on Quality and on the thought process rather than on techniques. But industry itself is much to blame for the state of academia. Middle-tier academics pride themselves not on academic excellence (which is impossible to measure, in spite of anyone who would claim to the contrary) but on their placement of graduates. As someone who has been both on the hiring end and on the "brain-factory" side, I can tell you that placement has much to do with unwritten contracts between industry and academica. North Central made a big deal about its relationship with local industry, and it appealed annually to local companies to ask what skill sets would make new hires employable. That's where Java curricula come from (or C++ curricula, or C# curricula). It almost makes one take one's hat off in salute to those institutions who base their instruction on Scheme.
I don't think there is much we can do to help the poor ambassadors that industry sends to academia to ask schools to do their bidding. I do think that as academics, we can educate the ambassadors. We can also hold our ground. That requires that time-honored but fading academic value of courage. It takes more courage to be a good educator these days than perhaps it did in the 1960s, when anything that could be labeled as technologically intellectual passed for good academics. It is probably as difficult as it was for Newton who, in the face of predominate views in a broad and powerful religion, had difficulty prevailing with new insights on the working of the universe. In fighting the broad misunderstandings that industry has chosen to adopt from selected interpretations of Agile, SOA, and programming language buzzwords, we face no less a religion, and no less threat to our cozy industry-financed academic positions, than he did.
I have sympathy with those who comment (as Grig does in his 'blog) that this is not an academic problem but a problem of industry. I simply think it's too late at that point, and I don't know of any good solutions when things have come that far. One solution lies in industrial training programs. Many of those are in the hands of consultants brought on board to teach a given tool or testing technique, and the likelihood that such engagements will lead to a culture of thinking are small. Too few organizations have license to freely explore alternative techniques and to truly evolve. There are places where inspect-and-adapt doesn't work; it is such places where paradigm shifts have their place. Just as many interesting advances in biology took place not through the classical Darwinian model but through discontinuities, so we need platforms that enable practitioners to drop what they are doing and to completely re-direct their efforts. That happens infrequently enough in industrial projects for me to not make it my first focus.
Oh, and I'm terribly sorry that this post did not give you a checklist.
The objective is to deliver what the customer wanted while optimizing quality; the broader goal is to create value.
The customer never wants the source code and is also not interested in maintaing it.
TDD has no intrinsic advantage over testing late but without TDD it's not unlikely testing falls apart - the programmer knows his code and has debugged it; time and budget resources are always scarce and where do management can cut budget if not at the end of the schedule? So TDD is foremost a means to an end of having UTs in the hands of programmers at all.
I've observed people's hunger for the "one right way" and their treatment of the heretics who might question their version of it for more than 25 years. While I'm not sure it's a "newfound" restraint, it is a wide spread problem and your post does an admirable job of bringing it to light.
There's nothing wrong with capturing a set of techniques that someone found helped them be more productive under some circumstances. Unfortunately, it's easier for people to focus on the techniques and forget about those pesky circumstances.
All of the buzzwords you mention have value. Most of the things people write about them focus on that value. Knowing when not to use a technique is just as valuable as knowing when to use it. What I'd love to see is more written on various techniques' limitations or even what we don't yet know about their limitations. Unfortunately, too many of us interpret identifying limitations as an attack on the dogma du jour.
I am happy that you talked about "The Amazing Shrinking Academic Collective Mind" When i mentioned this in a presentation (The jouneyman tale) in the "professional developer" track in jaoo, they accused me to be violent, and hostile. You know, in nature...
It is sometimes the case that the customer actually wants the code. When I was at Bell Labs we had a department-sized project to develop a programming environment that would allow our Bell South users to program the routing of certain calls and network traffic as a function of call type and time of day. The Applied Research organization in which I was embedded did some heavy investigation into "customer-programmable systems." This programming went all the way down to object-oriented programming in C++ and all the way up to parameterization. The most useful part of what we delivered was an application-specific language and a development environment for it. Their code integrated with ours so, yes, we delivered code.
Most flavours of TDD claim not to be about testing, but about design, so your second paragraph leaves me a bit confused. I'm all for testing, as long as it isn't the central focus of your quality program. But TDD as people teach it has little more to do with testing than compilation does. Hmm. Perhaps TDD is one way of compensating for the lack of static type checking (which Perry's research has established as a powerful way of reducing errors) in languages like Smalltalk.
As for unit testing, I think it's great for bottom-up procedural programming, and we used it a lot back inthe 1970s and 1980s. As we moved into OO we found that it had less and less value because the scaffolding necessary for an equally meaningful test started to be larger than the code itself. These days some of that scaffolding has been factored into xUnit, Casper and friends, but that's just boilerplate: I find that individual class tests these days say a lot less about their well-being than unit testing could prognosticate for functions. I think unit testing makes more and more sense as you move to higher and higher units of integration, where the APIs look more and more like procedural interfaces. Otherwise, you should use professional testing techniques under the guidance of a professional tester.
> What I'd love to see is more > written on various techniques' limitations or even what we > don't yet know about their limitations. Unfortunately, too > many of us interpret identifying limitations as an attack > on the dogma du jour.
An old friend of mine, Dennis DeBruler, bemoaned the fact that there are so few academic papers entitled "Blind Alley Discovered Here!" This is why I applaud the courage of Maria Siniaalto in her honest exploration of TDD and Angela Martin in her studies of On-Site Customer. This is yet another academic foible.
A great metaphor for academia, in my opinion, is "The Name of the Rose" by Umberto Eco. Under the pretense of being a repository of world knowledge, the monastery at Melk in fact had succeeded in hiding humankind's greatest intellectual achievement for centuries. Whenever I read the book with this metaphor in mind, it makes me angry, and sad.
I, too, would like to thank you for your essay. I always find your writing refreshing and interesting, as you _think for yourself_, rather than following dogma and fads.
You're also one of the few I know about who push for a more scientific approach to programming. As someone said to me at the last ACCU conference (where I also attended your very interesting session on multi-paradigm programming), when we talked about the poor state of our industry: "We [our field] haven't found our language, yet." Mathematics have its language; so has physics, but in "computer science", we've yet to find a reasonable comprehensible set of principles and techniques for understanding and developing systems.
Anyway, enough rambling and praise: I was curious about this one:
> Perhaps TDD is one way of > compensating for the lack of static type checking (which > Perry's research has established as a powerful way of > reducing errors) in languages like Smalltalk.
Who's this "Perry"? (It's not mentioned in the essay, and I dind't find much with a search), could you give a reference or link?
> Who's this "Perry"? (It's not mentioned in the essay, and > I didn't find much with a search), could you give a > reference or link?
DeWayne Perry is a well-known member of the ICSE community who has done much empirically-inspired research over the past twenty years. We worked together in the Software Production Research center at Bell Labs. I had lost track of him but Google found him in Texas (http://users.ece.utexas.edu/~perry/). I can't find the paper on this topic right now, but if you Google "Perry typing quality" or something like that, you'll get a large number of hits.
Since my blog post inspired this article, I feel compelled to throw in my 2 cents. First of all, I didn't 'agonize' at all over the question of whether to do either TDD or acceptance testing or both. I have a pretty clear picture of what I think a sound testing strategy should be, and I go at length on that strategy in another post, which I linked to, but which Jim never bother to read.
I call it 'holistic testing', which means you have to test your application from as many angles as possible -- unit tests, functional tests (both at the business logic level and at the GUI level), acceptance tests, and also performance/load/stress tests, security tests....and the list can go on.
I agree with Jim that usability testing is woefully neglected, and sorely needed. But assuming you know WHAT to write in terms of your application requirements (and I admit that's a big assumptions), techniques such as TDD show you HOW to write it. I personally haven't used TDD per se, but something I call TED for Test-Enhanced Development: write some code, write unit tests for it, change the code, etc.
I find it amusing that I'm accused of preaching a religion here. I consider myself to be pragmatic, and to do 'whatever works' to get the job done.
I'm also highly skeptical of studies conducted by people who never actually took the time to participate in agile projects, except for studying them 'from the outside in'. I prefer experience reports from people who have been in the trenches -- from 'pigs' if you want, in the Scrum sense, as opposed to 'chickens'.
I looked at http://mango2.vtt.fi/virtual/agile/publications.html and I shuddered. Looks like your typical publication list written by people isolated in an ivory tower, that have to publish or perish. A self-referential community which doesn't contribute one bit to the adoption of agile practices and methods in the larger software development community.
> I'm also highly skeptical of studies conducted by people > who never actually took the time to participate in agile > projects, except for studying them 'from the outside in'. > I prefer experience reports from people who have been in > the trenches -- from 'pigs' if you want, in the Scrum > sense, as opposed to 'chickens'. >
Chickens, of course, do participate in scrum projects although they're not responsible for implementations the way pigs are.
Although "testimonials" can be interesting and informative, they're really not a very objective way of evaluating a practice.
So without evaluating the quality of the specific research you link to, I'd be far more likely to believe "outside-in" research than that of true believers.
> > TDD has no intrinsic advantage over testing late but > without TDD it's not unlikely testing falls apart - the > programmer knows his code and has debugged it; time and > budget resources are always scarce and where do management > can cut budget if not at the end of the schedule? So TDD > is foremost a means to an end of having UTs in the hands > of programmers at all.
I've heard this argument before, but I don't buy it. When comparing TDD with post-coding unit testing you can't just assume that the latter won't happen.
If time and budget constraints motivate management to say "get coding ASAP - we don't have time to do this TDD stuff" should we then conclude that TDD is a poor form of unit testing because it won't happen?
No, if we want to legitimately compare these approaches we must assume that both would be carried out.
I think a good case can be made that TDD is less efficient simply because, in general, more iterations are required. In TDD you run tests against what is known to be incomplete and non-release-candidate code.
Testing after coding has a "just in time" quality.
..One also quickly discovers that the routines/units that one wants to test are not easily tested.
What happens then? Refactoring for testability I suppose.
I find that, more often than not, objections to TDD are made by those who've had little training and/or experience doing it properly. If no one teaches you how to make lemonade (unless you are somewhat persistent, or lucky), you are just left with water, sugar and lemons and the distinct impression that lemonade isn't very good.
In my experience at ThoughtWorks, TDD resulted in simpler designs that still delivered the customers' desired business value. The mantra, "Red. Green. Refactor." is what keeps the design from deteriorating.
The end of the iteration retrospective and client demo are what keeps the client involved in the UI part of the development.
I guess ultimately it comes down to the old disclaimer, "Your mileage may vary."
> > > > TDD has no intrinsic advantage over testing late but > > without TDD it's not unlikely testing falls apart - the > > programmer knows his code and has debugged it; time and > > budget resources are always scarce and where do > management > > can cut budget if not at the end of the schedule? So > TDD > > is foremost a means to an end of having UTs in the > hands > > of programmers at all. > > I've heard this argument before, but I don't buy it. When > comparing TDD with post-coding unit testing you can't just > assume that the latter won't happen.
That the latter won't happen is not just an arbitrary theoretical assumption. You can go making field research and just count the number of projects without having a proper testbase. For obvious reasons none of them uses TDD.
A defense of TDD might not even come from practical considerations. It is just the most consistent way of placing tests into the development process without running danger to lose them in the end. But it is also somewhat artificial and a heavy methodology which has caused sentiments against it. It doesn't feel agile but more as something being invented to fit perfectly into a development methodology.
I guess most people who claim applying TDD are actually coding a little piece and add one or more testcases. This turns out to be observationally equivalent with test-first development but isn't. From a pragmatical point of view I sympathize with this kind of development more than I do with applying TDD strictly. The reason is that a practice of accompanied testing can reduce additional costs of testing ( the transformational costs you were talking about ) to almost a minimum while preserving many of the good features of TDD.
Some Accompanied Testing ( to coin a new buzzword ) approaches are so cheap because each interaction with a system leads to a number of regression tests. The developer is reifying feedback loops. Regression tests are those that help stabilizing code and make changes visible if they fail. The fallout of regression tests in Accompanied Testing is based on recording and replaying interactions ( not unlike acceptance test tools for GUIs ). Classical UT frameworks like XUnit are not appropriate for them but solutions like doctest or [/i]consoletest[/i] naming just two being available for Python development:
> > > > > > TDD has no intrinsic advantage over testing late but > > > without TDD it's not unlikely testing falls apart - > the > > > programmer knows his code and has debugged it; time > and > > > budget resources are always scarce and where do > > management > > > can cut budget if not at the end of the schedule? So > > TDD > > > is foremost a means to an end of having UTs in the > > hands > > > of programmers at all. > > > > I've heard this argument before, but I don't buy it. > When > > comparing TDD with post-coding unit testing you can't > just > > assume that the latter won't happen. > > That the latter won't happen is not just an arbitrary > theoretical assumption. You can go making field research > and just count the number of projects without having a > proper testbase. For obvious reasons none of them uses > TDD. >
And they obviously aren't performing post-coding unit tests either. What does that prove about the advantages of one over the other? Nothing.
Just as its absurd to say that TDD fails by pointing to projects that don't follow it, you can't conclude that a post-coding testing process fails by pointing to projects that don't actually do it.
This seems to be a a recurrent theme in pro-agile arguments: that those who follow the agile way will, by definition, do exactly what they are supposed to do while those using other methodologies suffer from human frailty and screw things up.
We know that any methodology carried out by humans will be done imperfectly. Thus there is no doubt that some projects done using TDD will have poor testbases just as some projects using other methods do.
Flat View: This topic has 55 replies
on 4 pages