The Artima Developer Community
Sponsored Link

Weblogs Forum
Testing Genes, Test-Infection, and the Future of Developer Testing

30 replies on 3 pages. Most recent reply: Feb 4, 2007 11:08 AM by nguyen Phuc Hai

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 30 replies on 3 pages [ 1 2 3 | » ]
Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Testing Genes, Test-Infection, and the Future of Developer Testing (View in Weblogs)
Posted: Jan 24, 2007 11:27 AM
Reply to this message Reply
Summary
Some developers are easily test-infected - they take to unit testing like a duck to water. Others need some time and encouragement, but eventually "get it". A third group appears to have immunity to test infection. I invent a test-gene model to categorize these groups and look at its implications for the future of developer/unit testing.
Advertisement

Developer testing is the simple, but logical, idea that developers should take responsibility for playing an active role in testing their own code. We are nearing the 10th anniversary of JUnit - the unit testing framework that kick-started the developer testing revolution. But, despite its obvious benefits and broad agreement that "it's the right thing to do", developer testing is still only practiced by a minority of developers in a minority of organizations.

I gave talks on JUnit and the developer testing revolution at two JavaOne conferences (2004 and 2006) where I talked about the three possible outcomes for the future of developer/unit testing:

Long-Term Defeat: The idea of developer testing was a noble, notable, but ultimately short-lived blip in the history of software development. Almost all developers went back to the old way of thinking: testing is for QA.

Minor Victory: Developer testing does not disappear, but it remains a niche activity. Practiced only by a devout minority of test-infected developers.

Major Victory: Developer testing becomes a standard software development practice. The rule, instead of the exception. Releasing software without unit/developer testing is as unthinkable as releasing software without QA.

I am rooting, of course, for the major victory, but I am starting to have my doubts, and wondering why progress is so slow. Below is my analysis of what I believe is happening and I'd be very interested in your opinion.

The Developer Testing Gene

In the past few years, I've noticed that some developers are easily "test-infected", others require longer exposure and some encouragement but eventually "get it" , and one group seems to have a built-in immunity.

It's as if there are 3 basic types of test genes:

T1 - Very susceptible to test infection. Show them a single JUnit example and they get it immediately - and start using it regularly and with great fervor. When time pressure hits, they fight hard for enough time to test and would rather quit than produce code without tests. They often have framed pictures of Kent Beck on their desk :-).

T2 - Somewhat susceptible to test infection. Show them how to use JUnit/TDD, give them some encouragement and time to appreciate the benefits and rewards; eventually they will become regular users. When time pressure hits, however, they often revert to no-test mode without too much kicking and screaming.

T3 - Immune to test infection. There's no amount of showing and convincing that will get them to practice developer testing on their own with any regularity - if at all. Many of them would rather quit than have to write tests on a regular basis.

I would love to be able to say that these genes are somewhat evenly distributed. A rough Gaussian distribution. Something like this:

T1: 30% T2: 40% T3: 30%

But my experience is that developers with the T3 gene outnumber the combined number of those with T2 and T1 gene.

This spells trouble because I have also found that when developers become managers they bring with them their attitudes toward developer testing. And a T3 manager is not going to provide an environment where the T2 (the ones that require some time and encouragement) can become test infected. And this might explain why developer testing is still practiced by a minority of developers in a minority of organizations.

What do you think? Am I too pessimistic? Is predisposition to testing more a matter or nature or nurture?

Alberto


Jason Yip

Posts: 31
Nickname: jchyip
Registered: Mar, 2003

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 24, 2007 3:37 PM
Reply to this message Reply
I'd put the Major Victory as "Developer testing becomes a commonly accepted part of Quality Assurance. The rule, instead of the exception. Releasing software without unit/developer testing is as unthinkable as releasing software without doing any Quality Control."

I'm not pessimistic about there being more T3 than T1 or T2 developers. Just so long as the T1 and T2 developers end up generally more successful, I'm sure the change is inevitable.

nitin chander

Posts: 1
Nickname: nchander
Registered: Jan, 2007

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 24, 2007 3:41 PM
Reply to this message Reply
Alberto,
I agree that T3 outnumbers T1 + T2. At least thats the feeling I have after speaking to a number of developers.

One of the things I would like to discuss with the community is "how feasible TDD really is?" Writing tests for every trivial piece of code seems tedious and boring and I have a strong feeling that this may be something that prevents a lot of developers from continuing writing unit tests. It would be interesting to hear what others think about this...

I would also like to hear from developers who practice TDD and if they find the practice really rewarding...

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 24, 2007 4:20 PM
Reply to this message Reply
>One of the things I would like to discuss with
>the community is "how feasible TDD really is?"

Hi Nitin,

I should clarify that in this case I am talking about ANY type of developer/unit testing, not just TDD. I am counting developers who write and use unit tests with some consistency; whether in test-driven fashion, during, or even after development. The numbers would be considerably smaller for the TDD gene.

I think that regular test infection (i.e. not the TDD variety) is an easier path to achieve broader overall adoption of developer testing. Most un-infected and un-exposed developers I talk to understand unit testing and why they should do it. Like diet and exercise, knowing what's good for you does not mean that you'll actually do it. But at least they understand and agree that they should be doing it.

On the other hand, when I explain and demonstrate TDD I get mostly confused or quizzical looks and comments: "Say that again: "You want me to write the tests before I write the code? That's crazy talk!"

Alberto

Vincent O'Sullivan

Posts: 724
Nickname: vincent
Registered: Nov, 2002

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 24, 2007 4:43 PM
Reply to this message Reply
I think the answer is simply that the majority of programmers learn their trade "on the job" or, if they're lucky, get sent on a one week basic introductory course to whatever language they are going to use. After that they never seem to seek nor get offered any further training.

They are 'journeymen'. They do what they do for a living and give it no more thought once they leave the office. In a few years time they'll be promoted to managers or transfer to personnel or whatever. To them programming is just another office task.

The ones who have a genuine interest in programming as a skill, or a hobby, are a surprisingly small minority. They may make up almost the entire on-line population, or be everyone you meet at a conference, but don't let that fool you; they're still a minority. Even the concept of limiting the size of a method is alien to many of these people. In my company, it's still not particularly unusual to see five-hundred line methods. In the absence of peer review then who's going to stop it happening?

Alternatively, look at any coding problems forums. Questions are either trivially simple - and get piles of duplicated answers - or genuinely taxing - and get largely ignored. (The third category is homework questions where the recipient has no interest in either doing the work or looking at your answer but just wants something that they can show their tutor.)

Set against this, I'd say a 30% take-up rate for unit testing is amazingly high not low.

Erik Engbrecht

Posts: 210
Nickname: eengbrec
Registered: Apr, 2006

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 24, 2007 5:01 PM
Reply to this message Reply
> <p><strong>Major Victory</strong>: Developer testing
> becomes a standard software development
> practice. The rule, instead of the exception. Releasing
> software without
> unit/developer testing is as unthinkable as releasing
> software without
> QA.</p>

Hmmmm...that would imply that in the majority of environments releasing code without QA is unthinkable...

I think you're so far ahead of the pack that they can no longer see you anymore.

Anyway, I went through 3 phases with TDD:
1. Huh?
2. Jeeze, that looks like a lot of work...
3. Oh, it's what I've always done + lots of dogma

This isn't a religion. It's common sense.

Kevin Lawrence

Posts: 13
Nickname: kevlaw
Registered: Feb, 2005

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 24, 2007 5:31 PM
Reply to this message Reply
> On the other hand, when I explain and demonstrate TDD I get mostly confused or quizzical looks and comments:


And yet ... I would wager that, of the people who are doing unit testing regularly, the percentage who do TDD is quite high. Part of the paradox is that TDD sounds crazy but the alternative - the "sensible way" (test last) - is hard and tedious and boring and people stop doing it after a while.

As long as you have an 'optional' part of development that is hard and tedious and boring people will not do it. TDD addresses this problem by making testing a core part of the development cycle...and it actually becomes fun. More fun than debugging anyway.

Kevin
www.junitfactory.com
Send us Java. Get JUnit back. For free.

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 24, 2007 6:34 PM
Reply to this message Reply
>They are 'journeymen'. They do what they do for a
>living and give it no more thought once they
>leave the office.

Hi Vincent. Thank you for adding that perspective. Living in Silicon Valley (land of the geeks), it's very easy to forget that the majority of people doing software development were not born geeks drawn to it by passion, but because it's a pretty darn good way to make a living.

However, my guesstimates for the T1/T2/T3 ratios would not change much if I counted only the true alpha-geeks with a passion and a true talent for programming. As a matter of fact, these are the people I deal, and have dealt, with most of the time (having been fortunate enough to have worked at places like Sun, Google, and Agitar). I started thinking about the test gene theory not by looking at a uniform slice of the developer community, but at the developers in my "backyard" - most of whom don't qualify as 'journeyman'.

As a matter of fact, when we started Agitar our VCs had us do due-diligence with some "big name" software developers; People who have made major contributions to programming languages, operating systems, etc. I still remember when one of the biggest contributors to Java (no, not James Gosling nor Bill Joy) told us straight to our face something along the lines: "I am a developer. I don't write tests." That was a short and very uncomfortable meeting.

Bottom line. I agree 100% with your important distinction and distribution of "true" developers vs. "journeymen", but I don't think it changes the distribution of T1, T2, and T3s much.

Alberto

Derek Parnell

Posts: 22
Nickname: derekp
Registered: Feb, 2005

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 24, 2007 6:34 PM
Reply to this message Reply
My organisation found that the type of 'encouragement' that worked was a combination of carrot and stick.

Sticks:
(a) If the peer review does not see unit testing code then it is rejected and recorded as such.
(b) If QA does not see unit testing results when the unit was handed over to them, it is rejected and recorded as such.
(c) Persistent failures in these two areas mean that coders are given less responsible (i.e. interesting) tasks until they prove their worth to the project/company again.

Carrots:
(a) High pass rate through Review and QA process is rewarded, both to the individual coder and the team they are in.
(b) Senior managers and Line managers are also rewarded by having successful coders in their teams.
(c) Our customers are also pleased with the quality improvement, such that some of them award 'bonuses' to the team working on their issues.

The result was that within a few months, unit testing was seen as normal and desirable by all parties concerned. Even though more time is devoted to peer reviews and coding unit tests (with schedules planned accordingly), and there are more administration overheads; the plesant feeling of producing quality and being recognized as a company asset, overcome these.


Note that unit testing does not automatically improve quality, but it certainly improves the probablity of achieving high quality.

Writing unit tests before writing implementation code can detect errors in the specification and/or requirements. Writing unit tests after the implemenation tends to test what has been implmented rather than test the specification, and often leads to supporting the impression that bad specifications are actually okay, which are then costly to fix after it has left the coder.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 24, 2007 9:34 PM
Reply to this message Reply
> One of the things I would like to discuss with the
> community is "how feasible TDD really is?" Writing tests
> for every trivial piece of code seems tedious and boring
> and I have a strong feeling that this may be something
> that prevents a lot of developers from continuing writing
> unit tests. It would be interesting to hear what others
> think about this...

My personal feeling is that unit tests a really good idea for a small portion of the code I write. And actually, it's the trivial pieces that it works best for. The larger portion of the code I work doesn't rely only on the correctness of the component pieces (or units) but their interaction together.

I think a lot of people are writing 'unit tests' to test these higher-level interactions but I think there's a difference and the way to test them is different too. What I call this is regression testing and personally I think that the testing community has put much too little effort into making it easy and automated. In the work I've done lately, you basically have a process that takes some sort of input and based on a lot of environment related resources and factors produces some sort of document. Unit testing frameworks are really not very helpful with this. What you need to test this is a repository of correct outputs related to canned inputs. If you find a bug in production that your test didn't catch, you add tests. If you change the code, you need to test with filters (ignoring expected changes) and then eventually update the output files. Once I got this going, unit-tests seemed really rudimentary.

Jeff Ratcliff

Posts: 242
Nickname: jr1
Registered: Feb, 2006

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 24, 2007 11:16 PM
Reply to this message Reply
> Living in Silicon Valley (land of the geeks), it's very
> y easy to forget that the majority of people doing
> software development were not born geeks drawn to it by
> passion, but because it's a pretty darn good way to make a
> living.

Wow, two unsupported claims in one sentence:
1) A higher percentage of Silicon Valley geeks are doing it out of passion.
2) The majority of people doing software development aren't doing it out of passion.

Is this just sloppy thinking or is there an ad hominem argument implied here: that those without the "testing gene" are less geeky than those who have it?

Jerry Kurian

Posts: 6
Nickname: javaj
Registered: Jan, 2007

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 25, 2007 5:59 AM
Reply to this message Reply
I think one way to encourage T2 to become T1 would be through better tool support in IDEs. As of now IDEs are mostly concentrating on making coding easier and very little on generating tests.

Vincent O'Sullivan

Posts: 724
Nickname: vincent
Registered: Nov, 2002

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 25, 2007 7:33 AM
Reply to this message Reply
> I think one way to encourage T2 to become T1 would be
> through better tool support in IDEs. As of now IDEs are
> mostly concentrating on making coding easier and very
> little on generating tests.

What do you mean by this? Do you think that IDEs should generate tests from the code that you've written? That would seem to mean that the tests come after the code. Also; being generated from the code, I would imagine that those tests would fail to find any bugs since any such bugs would form part of the definition of the tests.

Given that the tests should be produced before the code; what form of assistance would you expect from the IDE?

Ravi Venkataraman

Posts: 80
Nickname: raviv
Registered: Sep, 2004

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 25, 2007 8:17 AM
Reply to this message Reply
I typically spend 1.5 to 2 hours in testing for every hour of code development. Using my methods I've developed complex financial applications that have run in production without any bugs at all!

Yet, I do not follow TDD. TDD is simply a buzzword driven fad that will soon be confined to the dustbin of history, along with many other fads (such as Java and C#).

Can TDD always lead to the most appropriate algorithm for doing things? Does not TDD focus at too low a level?

A contrarian view to stir up the pot, and hopefully force the religious adherents of TDD to think about their dogmatic views.

Ravi

Jerry Kurian

Posts: 6
Nickname: javaj
Registered: Jan, 2007

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 25, 2007 8:30 AM
Reply to this message Reply
>> what form of assistance would you expect from the IDE?

I would imaging IDE can generate tests using some sort of graphical representation of requirements and the signature of the method under test.
The test definitely should not be generated from the code, which will not be there in any case as per TDD.

Flat View: This topic has 30 replies on 3 pages [ 1  2  3 | » ]
Topic: Testing Genes, Test-Infection, and the Future of Developer Testing Previous Topic   Next Topic Topic: Interface Implementations Not Just From Factories


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us