On his website, Cafe au Lait, Elliotte Rusty Harold writes:

Martin Fowler points out some really weird behavior in JUnit. Apparently JUnit creates a new test object for each test. I understand why it does that, and I agree with the principles behind it. But this has some other implications:

• If this is the case why the hell bother with a setUp() method? Why not just use the constructor? tearDown() makes a tad more sense, but not much; and I've never actually needed to use it in practice.
• If something's going to take a long time to set up, you should make it a local variable in every test that uses it, rather than a field, as long as some tests don't use it.

Discover JUnit's semantics

I discovered that JUnit creates a new TestCase instance to run each test method a few years ago while looking through JUnit's source code. I was reading JUnit's code in an attempt to figure out how to automatically generate API signature tests for JUnit while creating a conformance test kit (CTK) for the ServiceUI API. A signature test verifies the artifacts of an API's signature, such as name, type, parameters, access level, and other modifiers, but does not verify the API's semantics.

When I discovered JUnit was creating a new TestCase instance for each test method, I was suprised. I hadn't understood that by reading the documentation, and it seemed very non-intuitive to me. The documentation emphasized that I should use setUp() to create my fixture to avoid side effects, which to me implied that JUnit was creating one instance of each TestCase, and invoking all test methods on that serially. And as Elliotte Rusty Harold put it, if you're going to create a new TestCase instance for each test method, "why the hell bother with a setUp() method?" You can just use the TestCase constructor.

I've heard Bruce Eckel point out that there is one subtle difference between creating your fixture in setUp() versus creating it in the TestCase constructor. JUnit creates all the TestCase instances up front, and then for each instance, calls setup(), the test method, and tearDown(). In other words, the subtle difference is that constructors are all invoked in batch up front, whereas the setUp() method is called right before each test method. But this seems to be not that useful a difference in practice.

What does "test" mean?

My discovery that JUnit created a new TestCase instance for each test method did help me get some insight into one other aspect of JUnit's API that had confused me: what does "test" mean in the API? Is a test a test method? One would think so, but then what's a TestCase? A TestCase seemed to be a suite of tests that all use the same fixture, so a TestCase is a suite of related tests. Hmm, "TestCase" seems like an odd name for a suite of tests. But wait, TestCase implements Test, which is also implemented by TestSuite. So a TestCase is a Test, and a TestSuite is a Test, and a TestCase is a suite of tests.

When I discovered by looking at the code that JUnit's runner was instantiating a new instance of each TestCase for each test method, I realized that a TestCase did actually map to a test at runtime. Now I finally thought I could perhaps understand where the name TestCase had come from. A TestCase instance was one test and one fixture. Unfortunately, this meant that in JUnit's API, the TestCase class represents one concept (a suite of tests), whereas a TestCase instance represents a different concept (a test). I found that to be very non-intuitive.

When I threw up my hands and wrote SuiteRunner, I collapsed Test, TestCase, and TestSuite into one class called Suite. A Suite represents a conceptual suite of tests. A test method is a kind of test, but not necessarily the only kind. Instead of using the composite pattern as JUnit does with Test, TestCase, and TestSuite, SuiteRunner just uses plain old composition. A Suite can hold references to sub-Suites. This approach requires fewer types in the public API, and uses type names that are consistent with underlying concepts. And that makes the API easier to understand and use.

Ask a Suite to execute itself

Another thing that bothered me about JUnit's design is that the code that decides how to execute a suite of tests is in the runner. It seemed more intuitive and object-oriented to me to ask a suite to execute its own tests. That way the API could provide a default way to execute tests, which the users could override at their option. In JUnit's design, if you want to execute tests differently, you must write a new runner.

In SuiteRunner, therefore, class Suite has an execute() method. Suite's implementation of execute() uses reflection to discover any test methods on itself, invokes them one after another, and invokes execute() on any sub-Suites. If you prefer to create a separate instance for each test method a la JUnit, you could override execute() in a Suite subclass and do just that. In fact, you can do anything you want in execute(). For example, SuiteRunner has a package access Suite subclass called JUnitSuite, whose execute() method uses reflection and dynamic proxies to run a suite of JUnit tests. That's how SuiteRunner can act as a JUnit runner to run JUnit tests.

Conclusion: Design for usability

I originally wrote SuiteRunner to solve a specific problem, a CTK for ServiceUI. One reason I did the extra work (with some help from my friends) to make SuiteRunner a general tool was to demonstrate something I think is important to good design: usability. No matter how you arrive at a design, be it UML modeling, TDD micro-steps, or "applying patterns, one after another, until you have the architecture of the system," you should take a step back and look at the public API as a whole from a usability perspective. Are there ways to minimize the number of types and methods in the public interface? Are types and methods named consistently with each other and with the concepts they represent? Is the behavior intuitive? How easy is the API as a whole to learn, understand and use?

Resources

JUnit's home page:
http://www.junit.org

Test Infected, by Kent Beck and Erich Gamma, introduced JUnit to the world:
http://junit.sourceforge.net/doc/testinfected/testing.htm

Artima SuiteRunner is a free open source testing toolkit and JUnit runner:
http://www.artima.com/suiterunner/index.html

Why We Refactored JUnit
http://www.artima.com/suiterunner/why.html

Runnning JUnit Tests with Artima SuiteRunner, how to use Artima SuiteRunner as a JUnit runner to run your existing JUnit test suites:
http://www.artima.com/suiterunner/junit.html

	[TestFixture]
	public class ServerTester
	{
		private IList list = new ArrayList();
		[Test]
		public void first() {
			list.Add(1);
			Assert.AreEqual(1, list.Count);
		}
		[Test]
		public void second() {
			Assert.AreEqual(0, list.Count);
		}
	}