The Artima Developer Community
Sponsored Link

Weblogs Forum
How Much Unit Test Coverage Do You Need? - The Testivus Answer

7 replies on 1 page. Most recent reply: Aug 24, 2010 11:21 AM by William Dye

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 7 replies on 1 page
Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

How Much Unit Test Coverage Do You Need? - The Testivus Answer (View in Weblogs)
Posted: May 4, 2007 10:30 AM
Reply to this message Reply
Summary
Answer to the question: "What is Testivus' wisdom concerning the proper percentage of test coverage?"
Advertisement

Referring to "The Way of Testivus" entry:

http://www.artima.com/weblogs/viewpost.jsp?thread=203994

Morgan Conrad asked: "What is Testivus' wisdom concerning the proper percentage of test coverage?"

Here you go Morgan.

Testivus On Test Coverage

Early one morning, a programmer asked the great master:

“I am ready to write some unit tests. What code coverage should I aim for?”

The great master replied:

“Don’t worry about coverage, just write some good tests.”

The programmer smiled, bowed, and left.

...

Later that day, a second programmer asked the same question.

The great master pointed at a pot of boiling water and said:

“How many grains of rice should put in that pot?”

The programmer, looking puzzled, replied:

“How can I possibly tell you? It depends on how many people you need to feed, how hungry they are, what other food you are serving, how much rice you have available, and so on.”

“Exactly,” said the great master.

The second programmer smiled, bowed, and left.

...

Toward the end of the day, a third programmer came and asked the same question about code coverage.

“Eighty percent and no less!” Replied the master in a stern voice, pounding his fist on the table.

The third programmer smiled, bowed, and left.

...

After this last reply, a young apprentice approached the great master:

“Great master, today I overheard you answer the same question about code coverage with three different answers. Why?”

The great master stood up from his chair:

“Come get some fresh tea with me and let’s talk about it.”

After they filled their cups with smoking hot green tea, the great master began to answer:

“The first programmer is new and just getting started with testing. Right now he has a lot of code and no tests. He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.”

“The second programmer, on the other hand, is quite experience both at programming and testing. When I replied by asking her how many grains of rice I should put in a pot, I helped her realize that the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.”

“I see,” said the young apprentice, “but if there is no single simple answer, then why did you answer the third programmer ‘Eighty percent and no less’?”

The great master laughed so hard and loud that his belly, evidence that he drank more than just green tea, flopped up and down.

“The third programmer wants only simple answers – even when there are no simple answers … and then does not follow them anyway.”

The young apprentice and the grizzled great master finished drinking their tea in contemplative silence.


Raoul Duke

Posts: 127
Nickname: raoulduke
Registered: Apr, 2006

Re: How Much Unit Test Coverage Do You Need? - The Testivus Answer Posted: May 4, 2007 6:25 PM
Reply to this message Reply
You need to turn all of this into a sit-com. Really. "Office Space meets Master of Zen."

Anit Agarwal

Posts: 7
Nickname: anit
Registered: Feb, 2007

Re: How Much Unit Test Coverage Do You Need? - The Testivus Answer Posted: May 4, 2007 8:08 PM
Reply to this message Reply
Hi Alberto,

This was a hilarious and insightful article,

best,

--anit

Jacob Northey

Posts: 1
Nickname: jwnorthey
Registered: May, 2007

Re: How Much Unit Test Coverage Do You Need? - The Testivus Answer Posted: May 6, 2007 2:15 AM
Reply to this message Reply
Alberto,

A common trait of great masters is their ability to teach through telling a story.

This was definitely one of the funniest posts I've read in a while (So much so that it compelled me to post a comment for the first time). Who came up with the arbitrary 80% test coverage anyway?

One thing that has been bugging me for a while in terms of testing is the ability to write a test around an interface. This may seem odd at first, but what would be useful is to extend the contract of an interface to the rules of what it must do.

For instance the Java Comparator class has the rules that x.equals(y) implies y.equals(x) and compare(x, y) = -compare(y, x). How would you suggest a developer implement a test so that concrete classes must adhere to the contract laid out in the interface test?

Thank you,
Jake

Morgan Conrad

Posts: 307
Nickname: miata71
Registered: Mar, 2006

Re: How Much Unit Test Coverage Do You Need? - The Testivus Answer Posted: May 7, 2007 2:25 PM
Reply to this message Reply
> For instance the Java Comparator class has the rules that
> x.equals(y) implies y.equals(x) and compare(x, y) =
> -compare(y, x). How would you suggest a developer
> implement a test

I wrote a helper test method

runEquals(Object object, Object equalObject, Object differentObject)

that runs equals rules such as
an Object never == null
an Object always equals itself
x.equals(y) implies y.equals(x)
x.equals(y) implies x.hashCode() == y.hashCode();

This makes it much easier for developers to test equals() consistently. (We caught a lot of hashCode() problems!)

I didn't do anything with Comparators but you could add that.

IMO, it really helps propel TDD if you make it "easy" for developers to test "monotonous" things like equals().



I'm not sure where Alberto got the 80%, but I was on a project where that was the goal too.


One last point about improving test coverage that might make a good "guru" quote from testivus. One of the best ways to increase test coverage is to GET RID OF BAD, REPETITIVE CODE. I sometimes joke with my colleages that I was "really productive yesterday", I wrote -1234 lines of code.

Eric Armstrong

Posts: 207
Nickname: cooltools
Registered: Apr, 2003

Re: How Much Unit Test Coverage Do You Need? - The Testivus Answer Posted: May 8, 2007 11:05 AM
Reply to this message Reply
I *love* that story. It reminds me so much of my martial arts master, too! Love it, love it, love it.
:_)

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: How Much Unit Test Coverage Do You Need? - The Testivus Answer Posted: May 14, 2007 1:33 PM
Reply to this message Reply
Jacob,

Sorry for taking so long to reply. I was busy with JavaOne where, by the way, we were giving away hard copies of "The Way of Testivus" - which were very popular. I even had people quote my own epigrams back to me - weird.

Anyway, on the matter of interface testing ...

When Java was still called Oak (and I was working at SunLabs, rubbing elbows with the Java fathers and mothers) I remember a meeting where I proposed adding semantics to interfaces. The Java architects seem interested (somewhat) but other time-to-market pressures too priorities and we lost a great chance.

This is definitely something I'd like to see happen and I plan to play an active role in making it happen.

In JUnit Factory (http://www.junitfactory.com - which is free) and AgitarOne we have a mechanism for specifying and testing class and method invariants (i.e. things that should always be true for any instance of that class or invocation of that method).

If I write the following method in a JUnit Factory test helper class:

...
public void assertInvariantFooMustBeBar(Foo aFoo) {
assertTrue(isBar(aFoo));
}
...

The test generator will automatically insert the assertion:

assertTrue(isBar(X));

everytime it has an object of type X in the test.

I will talk to the team today about extending this type of automated invariant assertions applied to interfaces.

In other words, the test helper method:

...
public void assertInvariantFoo(Foo aFoo) {
assertTrue(Y(aFoo));
}
...

Where Foo is an interface, will insert and automatically test the Y-ness of any object that implements the interface Foo.

Would this be going in the direction you are thinking of.

Alberto

William Dye

Posts: 1
Nickname: willdye
Registered: Aug, 2010

Re: How Much Unit Test Coverage Do You Need? - The Testivus Answer Posted: Aug 24, 2010 11:21 AM
Reply to this message Reply
You wrote "is quite experience both at programming and testing". I think you mean "experienced", not "experience".

Flat View: This topic has 7 replies on 1 page
Topic: Memory debugging in Python -- fighting the GC Previous Topic   Next Topic Topic: Have Generics Killed Java?


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us