The Artima Developer Community
Sponsored Link

Weblogs Forum
One Assertion Per Test

22 replies on 2 pages. Most recent reply: Mar 1, 2004 1:37 PM by Dave Astels

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 22 replies on 2 pages [ 1 2 | » ]
Dave Astels

Posts: 32
Nickname: dastels
Registered: Mar, 2003

One Assertion Per Test (View in Weblogs)
Posted: Feb 23, 2004 4:09 PM
Reply to this message Reply
Summary
For some time I've been thinking about how TDD tests can be as simple, as expressive, and as elegant as possible. This article explores a bit about what it's like to make tests as simple and decomposed as possible: aiming for a single assertion in each test.
Advertisement

A while ago there was a bit of fuss on the testdrivendevelopment Yahoo group about the idea of limiting yourself to one assertion per test method, which is a guideline that others and I offer for TDD work.

An address parser was the example of a situation where it was argued that multiple assertions per test made sense. Date was formatted with one address per line, each in one of the following formats:

  1. ADDR1$ADDR2$CSP$COUNTRY
  2. ADDR1$ADDR2$CSP
  3. ADDR1$CSP$COUNTRY
  4. ADDR1$CSP

The poster went on to say: My first inclination is/was to write a test like this:

	a = CreateObject("Address")
	a.GetAddressParts("ADDR1$ADDR2$CITY IL 60563$COUNTRY")
	AssertEquals("ADDR1", a.Addr1)
	AssertEquals("ADDR2", a.Addr2)
	AssertEquals("CITY IL 60563", a.CityStatePostalCd)
	AssertEquals("Country", a.Country)

They didn't see how to achieve this with one assertion per test, as there are obviously four things to test in this case. I decided that rather than simply reply, I would write some tests and code to illustrate my view on the matter, and offer a solid response.

For this problem, I chose Squeak Smalltalk (see www.squeak.org) and Java. For the sake of conciseness, I'll omit any required accessors.

So, where to start? Well, when doing TDD it often makes sense to start with something simple to quickly and easily get some code written and working. Then it can be extended and evolved in response to further test driving. Here the simplest case is: ADDR1$CSP. There are two requirements in the parsing of this example: that the ADDR1 was recognized, and that the CSP was recognized. Viewed this way, we need two tests. We start with one for ADDR1:

Squeak:

    testAddr1
        | anAddress |
        anAddress := Address from: 'ADDR1$CITY IL 60563'.
        self assert: anAddress addr1 equals: 'ADDR1'

Java:

    public void testAddr1() throws Exception {
        Address anAddress = new Address("ADDR1$CITY IL 60563");
        assertEquals("ADDR1",  anAddress.getAddr1());
    }

To get this to pass we need an Address class and a from: factory method, which creates an instance and has it parse the address string. For brevity, I'll skip the "return literal" step.

Squeak:

    Object subclass: #Address
        instanceVariableNames: 'addr1'
        classVariableNames: ''
        poolDictionaries: ''
        category: 'Adaption-One Assertion Test'

    (class method)
    from: aString
        ^self new parse: aString;
            yourself

    parse: aString
        | parts |
        parts := aString findTokens: '$'.
        addr1 := (parts at: 1)

Java:

    public class Address {

        private String addr1;

        public Address(String aString) {
            parse(aString);
        }

        private void parse(String aString) {
            StringTokenizer parts = new StringTokenizer(aString, "$");
            addr1 = parts.nextToken();
        }
    }

That's well & good. The next test is for CSP.

Squeak:

    testCsp
        | anAddress |
        anAddress := Address from: 'ADDR1$CITY IL 60563'.
        self assert: anAddress csp equals: 'CITY IL 60563'

Java:

    public void testCsp() throws Exception {
        Address anAddress = new Address("ADDR1$CITY IL 60563");
        assertEquals("CITY IL 60563",  anAddress.getCsp());
    }

Address>>parse: will need to be extended (and we need to add a csp instance variable and accessors):

Squeak:

    parse: aString
        | parts |
        parts := aString findTokens: '$'.
        addr1 := (parts at: 1).
        csp := (parts at: 2)

Java:

    private void parse(String aString) {
        StringTokenizer parts = new StringTokenizer(aString, "$");
        addr1 = parts.nextToken();
        csp = parts.nextToken();
    }

So. We have two tests for this one situation. Notice the duplication in the tests... the creation of the instance of Address that is being probed. This is the fixture. After refactoring, we have:

Squeak:

    TestCase subclass: #Addr1CspTests
        instanceVariableNames: 'anAddress '
        classVariableNames: ''
        poolDictionaries: ''
        category: 'Adaption-One Assertion Test'!

    setUp
        anAddress := Address from: 'ADDR1$CITY IL 60563'

    testAddr1
        self assert: anAddress addr1 equals: 'ADDR1'

    testCsp
        self assert: anAddress csp equals: 'CITY IL 60563'

Java:

    public class Addr1CspTests extends TestCase {
    
        private Address anAddress;
    
        protected void setUp() throws Exception {
            anAddress = new Address("ADDR1$CITY IL 60563");
        }

        public void testAddr1() throws Exception {
            assertEquals("ADDR1",  anAddress.getAddr1());
        }
    
        public void testCsp() throws Exception {
            assertEquals("CITY IL 60563",  anAddress.getCsp());
        }
    }

So, a fixture that creates the Address instance from the string, and very simple tests that focus on each aspect of that fixture.

The next simplest case is the obvious choice for the next fixture:

Squeak:

    setUp
        anAddress := Address from: 'ADDR1$CITY IL 60563$COUNTRY'

Java:

    protected void setUp() throws Exception {
        anAddress = new Address("ADDR1$CITY IL 60563$COUNTRY");
    }

This set of tests will include ones for addr1 and csp as before (refactoring this to remove that duplication is left to the reader) as well as a new test for country:

Squeak:

    testCountry
        self assert: anAddress country equals: 'COUNTRY'

Java:

    public void testCountry() throws Exception {
        assertEquals("COUNTRY",  anAddress.getCountry());
    }

As before, an instance variable and associated accessors need to be added to the Address class.

This drives Address>>parse: to evolve:

Squeak:

    parse: aString
        | parts |
        parts := aString findTokens: '$'.
        addr1 := (parts at: 1).
        csp := (parts at: 2).
        country := (parts at: 3 ifAbsent: [''])

Java:

    private void parse(String aString) {
        StringTokenizer parts = new StringTokenizer(aString, "$");
        addr1 = parts.nextToken();
        csp = parts.nextToken();
        country = parts.hasMoreTokens() ? parts.nextToken() : "";
    }

From here on, the evolution gets a bit more complex, as we add the ADDR2 option to the mix.

Conclusion

So we took a situation that was thought to require multiple assertions in a test and did it in such as way as to have only one assertion per test.

The key is that instead of using a single TestCase subclass with a complex (i.e. multiple assertion) tests for each situation, we made each of those situations into a separate fixture. Each fixture is implemented by a separate subclass of TestCase. Now each test focuses on a very small, specific aspect of that particular fixture.

I'm convinced writing tests like this is a useful approach. One advantage is that the resulting tests simpler and easier to understand. Just as important, and maybe more so, is that by adding the specification of the behavior one tiny piece at a time, you drive toward evolving the code in small, controllable, understandable steps.

It also fits better into the test fixture centered approach that is the recommended way to organize your tests. We set up the object to test in the setUp method, and tested each aspect of it in individual tests methods.

As I've been writing this, something clicked. I see these test methods as specifications of tiny facets of the required behavior. Thus, it makes sense to me to be as gradual as possible about it, driving the evolution of the code in the smallest steps possible. Striving for one assertion per test is a way to do that.

If, however, you view test methods as strictly performing verification, then I can see how it might be seen to make sense to invoke some code and then test all the postconditions. But this view is not TDD, and doesn't buy you all of the benefits of TDD. I contend that central to TDD is this notion of working in the smallest steps possible, both for the finest-grained long-term verification, and for the most flexible design evolution. Furthermore, this is best done by striving to keep tests as small, focused and simple as possible. Aiming for one assertion per test is one way to get there.


Ian Bicking

Posts: 851
Nickname: ianb
Registered: Apr, 2003

Re: One Assertion Per Test Posted: Feb 23, 2004 4:35 PM
Reply to this message Reply
I have just been trying to polish up an extension to the Python unittest framework to make this kind of testing easier (at svn://colorstudy.com/trunk/DataTest) -- i.e., allow for small tests with different kinds of context. The next extension of your example is to feed a bunch of addresses and look for the expected results, but after adding this and other customizations to a TestCase subclass I find the framework becomes rather difficult to work with. I wrote about it a while ago (http://blog.colorstudy.com/ianb/weblog/2003/10/10.html), though I don't know how much applies to Java or Smalltalk (I got the impression that the Python implementation was modeled on those language's implementations, but maybe not).

There's really a wide variety of loops and other structures to a test, all of which would be nice to have accessible from outside, particularly when using TDD. When you start your code with lots of failing tests, you want to be able to focus on a few tests and knock them down one by one. The granularity of tests that I've worked with is usually at the level of individual subclasses of TestCase, but often I want to write twenty inputs for one test, and when all I have is one subclass, I end up rearranging the tests themselves so that the most interesting failure is the first failure. That's a pretty crude way to do it, obviously.

That said, writing tests can be a pain, and I don't want to burden myself with too much structure for tests. Disincentives too easily prey on my inclination to skip the tests ;) So decomposing the test into small pieces has to be easy and not require an excess of typing or clicking. Most unit testing examples have scary amounts of typing involved.

Dave Astels

Posts: 32
Nickname: dastels
Registered: Mar, 2003

Re: One Assertion Per Test Posted: Feb 23, 2004 5:16 PM
Reply to this message Reply
> I don't know how much applies to Java or Smalltalk
> (I got the impression that the Python implementation was
> modeled on those language's implementations, but maybe
> not).

I've used PyUnit a bit but not unittest. If it's part of the family of "Beckian" test frameworks, then yes... it all started with Smalltalk (like so many good things). JUnit is the Java flavour... also authored initially by Kent Beck along with Erich Gamma (of GoF and now Eclipse fame).

> When you start your code with lots of failing tests

This is not the way to do TDD. You should only have one test failing at a time... the one you just wrote. If you have a bunch of tests failing, it means you've broken something and you are obliged to fix that before doing anything else (especially before checking in code)

> inputs for one test, and when all I have is one subclass,
> I end up rearranging the tests themselves so that the most
> interesting failure is the first failure.

One of the tenets of TDD is that tests are order independant.

Generally with TDD, you don't start with the most intertesting test, but rather the simplest... gradually specifying more complex behaviour until you have the functionallity you need.

> That said, writing tests can be a pain, and I don't want
> to burden myself with too much structure for tests.
> Disincentives too easily prey on my inclination to skip
> p the tests ;) So decomposing the test into small pieces
> has to be easy and not require an excess of typing or
> clicking. Most unit testing examples have scary amounts
> of typing involved.

Automation in IDEs can help with this. This is one area I'd like to explore. Peter van Rooijen has done some work in that direction. See http://www.esug.org/summerschools/2001_Essen/presentations/EnsuDemo/

Yes, writing tests is work. But a) writing them first is less work than writing them later, and b) writing them is less work than dealing with code that has no tests.

Dave

Malte Finsterwalder

Posts: 20
Nickname: jufoda
Registered: Aug, 2003

Re: One Assertion Per Test Posted: Feb 24, 2004 3:57 AM
Reply to this message Reply
One thing that would interest me:

How do you refactor your test code to remove duplication that spawns several TestCases?
I often find that I have more or less complex setUp-code that creates an environment for running my test. If I split my testcode in very many TestCases, as you describe, I will need the same setup-code for several TestCases

Will Newton

Posts: 1
Nickname: willn
Registered: Feb, 2004

Re: One Assertion Per Test Posted: Feb 24, 2004 5:08 AM
Reply to this message Reply
I may be taking the article too literally, but I think that asserting pre-conditions as well as post-conditions can be valuable. e.g.

AssertEquals(null, a.Addr1)
a.GetAddressParts("ADDR1$ADDR2$CITY IL 60563$COUNTRY")
AssertEquals("ADDR1", a.Addr1)

It makes it clear which action, in this case GetAddressParts, sets the value of Addr1.

Robert Watkins

Posts: 2
Nickname: robertdw
Registered: Jun, 2003

Re: One Assertion Per Test Posted: Feb 24, 2004 5:22 AM
Reply to this message Reply
While this is good advice to follow, there may sometimes be exceptions.

One example would be assertions that make it easier to know what's happening with a failing test. For example, consider a test that verifies that a factory method works correctly. It will obviously check to see if the resulting object is set up as per spec, and this is the assertion that the test revolves around.

However, if the factory method isn't working, it may return a null object. Having a simple assertion in the test to verify it's not null (as well as a separate test to verify it's not null!) will help prevent a bit of mess if the factory method does break. After all, if the factory method breaks, you'll probably have several failing tests, and the "testFactoryDoesnNotReturnNull" test will be buried in all the rest of the noise.

Other than that (very minor) point, this is a great idea, Dave. Do you have this one covered in your book?

Robert Watkins

Posts: 2
Nickname: robertdw
Registered: Jun, 2003

Re: One Assertion Per Test Posted: Feb 24, 2004 5:24 AM
Reply to this message Reply
> I often find that I have more or less complex setUp-code
> that creates an environment for running my test. If I
> split my testcode in very many TestCases, as you describe,
> I will need the same setup-code for several TestCases

Well, you could always create TestSuites to contain that setup code... or a test hierarchy, where you inherit setup code from a parent test case.

Vincent O'Sullivan

Posts: 724
Nickname: vincent
Registered: Nov, 2002

Re: One Assertion Per Test Posted: Feb 24, 2004 5:45 AM
Reply to this message Reply
One assertion per test! Read this and weep.

In another thread I recently posted an example of a small unit test from from I project I recently joined. Here's another (bigger but by no means the biggest). Is it a comprehensive test? Could be, but it's only one of several similar and overlapping tests in this particular TestCase. Am I going to refactor it? Absolutely not - I'm still getting my head around the eclipse editor they use here and besides I've got my own deadlines.

(It's a good indicator of the coding style in general though.)

Vince.

public void testKioskObjectCompare()
    {
        System.out.println( "===testKioskCompare()===" );
        // First a simple create
        Kiosk test = new Kiosk("name","userid");
        assert(test.getPrinterName().equals("userid"));
        assert(test.getKioskName().equals("name"));
        Kiosk test2 = new Kiosk("name","userid");
        assert(test2.getPrinterName().equals("userid"));
        assert(test2.getKioskName().equals("name"));
        int i = test.compare(test,test2);
        assert(i==0);
        System.out.println("compare equal");
        i=test.compareTo( test2 );
        assert(i==0);
        System.out.println("compareTo equal");
        
        test2 = new Kiosk("aname","userid");
        i = test.compare(test,test2);
        assert(i>0);
        i=test.compareTo( test2 );
        assert(i>0);
        System.out.println("compareTo greaterthan");
        
        test2 = new Kiosk("zname","userid");
        i = test.compare(test,test2);
        assert(i<0);
        i=test.compareTo( test2 );
        assert(i<0);
        System.out.println("compareTo lessthan");
        
        test2 = new Kiosk("name","a");
        i = test.compare(test,test2);
        System.out.println("test="+test+" test2="+test2+" i="+i);
        assert(i>0);
        i=test.compareTo( test2 );
        assert(i>0);
        System.out.println("compareTo greaterthan");
        
        test2.setPrinterName("z");
        i = test.compare(test,test2);
        assert(i<0);
        i=test.compareTo( test2 );
        assert(i<0);
        System.out.println("compareTo lessthan");
        
        System.out.println( "===after Kiosk compare test===" );
    }

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: One Assertion Per Test Posted: Feb 24, 2004 8:41 AM
Reply to this message Reply
> One assertion per test! Read this and weep.
> ...
> Am I going to refactor it? Absolutely not -
> I'm still getting my head around the eclipse editor they
> use here and besides I've got my own deadlines.

Forget the test.

I can appreciate the kind of pressure that goes into "I've got my own deadlines" but, not knowing your situation, from here it sounds like you're new to the team or everyone thinks that way and there is no team. If the people around the project don't act like a team, its usually a bigger deal than any testing problem.

Sven Gorts

Posts: 2
Nickname: gorowitch
Registered: Feb, 2004

Re: One Assertion Per Test Posted: Feb 24, 2004 9:48 AM
Reply to this message Reply
With that kind of attitude, how do you think the situation is going to improve ?

Dave Astels

Posts: 32
Nickname: dastels
Registered: Mar, 2003

Re: One Assertion Per Test Posted: Feb 24, 2004 11:35 AM
Reply to this message Reply
> One thing that would interest me:
>
> How do you refactor your test code to remove duplication
> that spawns several TestCases?
> I often find that I have more or less complex setUp-code
> that creates an environment for running my test. If I
> split my testcode in very many TestCases, as you describe,
> I will need the same setup-code for several TestCases


You could factor common setup into a common baseclass.. sort of a super-fixture. Be careful not to put test methods there as they will be inherited by each concrete fixture class and run with each one.

Dave

Dave Astels

Posts: 32
Nickname: dastels
Registered: Mar, 2003

Re: One Assertion Per Test Posted: Feb 24, 2004 11:38 AM
Reply to this message Reply
> I may be taking the article too literally

Possibly. It's more of something to be thought provoking. I don't do this all the time in practice, but I do keep it in mind as a worthy goal and aspire to reach it as often as possible/reasonable.

> but I think
> that asserting pre-conditions as well as post-conditions
> can be valuable. e.g.
>
> AssertEquals(null, a.Addr1)
> a.GetAddressParts("ADDR1$ADDR2$CITY IL 60563$COUNTRY")
> AssertEquals("ADDR1", a.Addr1)
>
> It makes it clear which action, in this case
> GetAddressParts, sets the value of Addr1.

I don't know how useful this is, but it can add communication value to the test. As you say it makes it very clear what the side effects of the call is. Maybe that's the key... it's good for making side effects explicit.

Hmm... nice.

Dave

Dave Astels

Posts: 32
Nickname: dastels
Registered: Mar, 2003

Re: One Assertion Per Test Posted: Feb 24, 2004 11:42 AM
Reply to this message Reply
> While this is good advice to follow, there may sometimes
> be exceptions.

Yes. There are always exceptions.

> However, if the factory method isn't working, it may
> return a null object. Having a simple assertion in the
> test to verify it's not null (as well as a separate test
> to verify it's not null!) will help prevent a bit of mess
> if the factory method does break. After all, if the
> factory method breaks, you'll probably have several
> failing tests, and the "testFactoryDoesnNotReturnNull"
> test will be buried in all the rest of the noise.

I'm not completely convinced about the "buried in the rest of the noise" bit.

> Other than that (very minor) point, this is a great idea,
> Dave.

Thanks.

> Do you have this one covered in your book?

Not explicitly.. well at least not at length (it's been a while since I wrote/read it :), but in I do talk about the idea of decomposing tests to a greater extent than most people do in practice.

Dave

Dave Astels

Posts: 32
Nickname: dastels
Registered: Mar, 2003

Re: One Assertion Per Test Posted: Feb 24, 2004 1:34 PM
Reply to this message Reply
> One assertion per test! Read this and weep.

All to typical, though.

> Am I going to refactor it? Absolutely not -
> I'm still getting my head around the eclipse editor they
> use here and besides I've got my own deadlines.

a) refactoring is easy/safe in Eclipse
b) refactoring should improve the clarity of the tests and make it easier to understand.
c) the author of this needs guidance on how to write tests... start some seminars.. official or not.

Dave

Vincent O'Sullivan

Posts: 724
Nickname: vincent
Registered: Nov, 2002

Re: One Assertion Per Test Posted: Feb 24, 2004 4:14 PM
Reply to this message Reply
By aspiring to Ghandi's famous edict: "Be the change you're trying to create."

Nevertheless, if I had time to go and rewrite all the code that other's had written and I though I could improve, whilst still meeting my own deadlines then there'd be something seriously wrong with the deadlines. Not to mention the sheer vanity (and vainness) of joining a project and telling everyone else where they were going wrong whilst demonstrating an impressive incapability to get eclipse set up properly.

I'll need to have my feet under the table here a bit longer before I venture down that road.

Vince.

PS: Things I have had my fingers rapped for in the past:
1) Writing PL1 using lower case. PL1 code doesn't distinguish between upper and lower case (a significant failing in Java) but this project wrote everything in upper case. They were so used to it that they found the lower case code hard to read. I was even asked if I'd tried to compile it, they were so sure it was wrong.
2) (Same project) Writing PL1 functions that returned values. The project had never used functions only subroutines that didn't return values but modified one or more of the parameters passed in. I was accused showing off by writing deliberately obscure code!

So you'll maybe appreciate why I have "that kind of attitude".

Flat View: This topic has 22 replies on 2 pages [ 1  2 | » ]
Topic: Thinking out loud, and in public... Previous Topic   Next Topic Topic: Yahoo groups are enormously popular. That's probably an understatement.


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us