The Artima Developer Community
Sponsored Link

Weblogs Forum
Design for Usability

4 replies on 1 page. Most recent reply: Sep 13, 2004 11:21 AM by Bill Venners

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 4 replies on 1 page
Bill Venners

Posts: 2242
Nickname: bv
Registered: Jan, 2002

Design for Usability (View in Weblogs)
Posted: Sep 13, 2004 5:15 AM
Reply to this message Reply
Summary
No matter how you arrive at a design, be it UML modeling, TDD micro-steps, or applying patterns, you should take a step back and look at the public API as a whole from a usability perspective. How easy is the API as a whole to learn, understand and use?
Advertisement

On his website, Cafe au Lait, Elliotte Rusty Harold writes:

Martin Fowler points out some really weird behavior in JUnit. Apparently JUnit creates a new test object for each test. I understand why it does that, and I agree with the principles behind it. But this has some other implications:
  • If this is the case why the hell bother with a setUp() method? Why not just use the constructor? tearDown() makes a tad more sense, but not much; and I've never actually needed to use it in practice.
  • If something's going to take a long time to set up, you should make it a local variable in every test that uses it, rather than a field, as long as some tests don't use it.

Discover JUnit's semantics

I discovered that JUnit creates a new TestCase instance to run each test method a few years ago while looking through JUnit's source code. I was reading JUnit's code in an attempt to figure out how to automatically generate API signature tests for JUnit while creating a conformance test kit (CTK) for the ServiceUI API. A signature test verifies the artifacts of an API's signature, such as name, type, parameters, access level, and other modifiers, but does not verify the API's semantics.

When I discovered JUnit was creating a new TestCase instance for each test method, I was suprised. I hadn't understood that by reading the documentation, and it seemed very non-intuitive to me. The documentation emphasized that I should use setUp() to create my fixture to avoid side effects, which to me implied that JUnit was creating one instance of each TestCase, and invoking all test methods on that serially. And as Elliotte Rusty Harold put it, if you're going to create a new TestCase instance for each test method, "why the hell bother with a setUp() method?" You can just use the TestCase constructor.

I've heard Bruce Eckel point out that there is one subtle difference between creating your fixture in setUp() versus creating it in the TestCase constructor. JUnit creates all the TestCase instances up front, and then for each instance, calls setup(), the test method, and tearDown(). In other words, the subtle difference is that constructors are all invoked in batch up front, whereas the setUp() method is called right before each test method. But this seems to be not that useful a difference in practice.

What does "test" mean?

My discovery that JUnit created a new TestCase instance for each test method did help me get some insight into one other aspect of JUnit's API that had confused me: what does "test" mean in the API? Is a test a test method? One would think so, but then what's a TestCase? A TestCase seemed to be a suite of tests that all use the same fixture, so a TestCase is a suite of related tests. Hmm, "TestCase" seems like an odd name for a suite of tests. But wait, TestCase implements Test, which is also implemented by TestSuite. So a TestCase is a Test, and a TestSuite is a Test, and a TestCase is a suite of tests.

When I discovered by looking at the code that JUnit's runner was instantiating a new instance of each TestCase for each test method, I realized that a TestCase did actually map to a test at runtime. Now I finally thought I could perhaps understand where the name TestCase had come from. A TestCase instance was one test and one fixture. Unfortunately, this meant that in JUnit's API, the TestCase class represents one concept (a suite of tests), whereas a TestCase instance represents a different concept (a test). I found that to be very non-intuitive.

When I threw up my hands and wrote SuiteRunner, I collapsed Test, TestCase, and TestSuite into one class called Suite. A Suite represents a conceptual suite of tests. A test method is a kind of test, but not necessarily the only kind. Instead of using the composite pattern as JUnit does with Test, TestCase, and TestSuite, SuiteRunner just uses plain old composition. A Suite can hold references to sub-Suites. This approach requires fewer types in the public API, and uses type names that are consistent with underlying concepts. And that makes the API easier to understand and use.

Ask a Suite to execute itself

Another thing that bothered me about JUnit's design is that the code that decides how to execute a suite of tests is in the runner. It seemed more intuitive and object-oriented to me to ask a suite to execute its own tests. That way the API could provide a default way to execute tests, which the users could override at their option. In JUnit's design, if you want to execute tests differently, you must write a new runner.

In SuiteRunner, therefore, class Suite has an execute() method. Suite's implementation of execute() uses reflection to discover any test methods on itself, invokes them one after another, and invokes execute() on any sub-Suites. If you prefer to create a separate instance for each test method a la JUnit, you could override execute() in a Suite subclass and do just that. In fact, you can do anything you want in execute(). For example, SuiteRunner has a package access Suite subclass called JUnitSuite, whose execute() method uses reflection and dynamic proxies to run a suite of JUnit tests. That's how SuiteRunner can act as a JUnit runner to run JUnit tests.

Conclusion: Design for usability

I originally wrote SuiteRunner to solve a specific problem, a CTK for ServiceUI. One reason I did the extra work (with some help from my friends) to make SuiteRunner a general tool was to demonstrate something I think is important to good design: usability. No matter how you arrive at a design, be it UML modeling, TDD micro-steps, or "applying patterns, one after another, until you have the architecture of the system," you should take a step back and look at the public API as a whole from a usability perspective. Are there ways to minimize the number of types and methods in the public interface? Are types and methods named consistently with each other and with the concepts they represent? Is the behavior intuitive? How easy is the API as a whole to learn, understand and use?

Resources

JUnit's home page:
http://www.junit.org

Test Infected, by Kent Beck and Erich Gamma, introduced JUnit to the world:
http://junit.sourceforge.net/doc/testinfected/testing.htm

Artima SuiteRunner is a free open source testing toolkit and JUnit runner:
http://www.artima.com/suiterunner/index.html

Why We Refactored JUnit
http://www.artima.com/suiterunner/why.html

Runnning JUnit Tests with Artima SuiteRunner, how to use Artima SuiteRunner as a JUnit runner to run your existing JUnit test suites:
http://www.artima.com/suiterunner/junit.html

The ServiceUI API defines a standard way to attach user interfaces to Jini services:
http://www.artima.com/jini/serviceui/index.html


Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: Design for Usability Posted: Sep 13, 2004 8:00 AM
Reply to this message Reply
I was mystified by the name "TestCase" too until I figured it out. The word "TestCase" describes the object that is created, not the text of the class.

Each TestCase, at run time, is configured to run exactly one test method so it really is a test case. The fact that subclasses of TestCase can each have many test methods doesn't make "TestCase" a suite. Individual test cases don't ever use all those methods.

The name describes the object, just like the name "Account" is intended to describe each "Account" object. Seeing TestCases as suites is kind of like wanting to call an "Account" class "Accounts".

Interestingly, if the same instance is reused, the name "TestCase" doesn't seem as accurate any more.

Bill Venners

Posts: 2242
Nickname: bv
Registered: Jan, 2002

Re: Design for Usability Posted: Sep 13, 2004 10:40 AM
Reply to this message Reply
> The name [TestCase] describes the object, just like the name
> "Account" is intended to describe each "Account" object.
> Seeing TestCases as suites is kind of like wanting to
> call an "Account" class "Accounts".
>
I disagree. In just about every other class I've seen, the concept of the class is the same as the concept of the instance. In class Account, for example, each instance represents one account concept, and the class represents the blueprint for one account. In class TestCase, each instance represents one test, but the class represents the blueprint for many tests. That's what makes TestCase an unusual design for me--not "wrong," just unusual, and therefore surprising and non-intuitive. The design tradeoff with that choice is that you get the isolation that prevents side affects, but you make the API a bit harder to learn and understand. It is a small dissonance, but most JUnit users have probably had to scratch their head about it at some point.

Jason Diamond

Posts: 1
Nickname: jdiamond
Registered: Sep, 2004

Re: Design for Usability Posted: Sep 13, 2004 11:03 AM
Reply to this message Reply
> That's what makes TestCase an unusual
> design for me--not "wrong," just unusual, and therefore
> surprising and non-intuitive. The design tradeoff with
> that choice is that you get the isolation that prevents
> side affects, but you make the API a bit harder to learn
> and understand. It is a small dissonance, but most JUnit
> users have probably had to scratch their head about it at
> some point.

NUnit 1.0 followed the same design and was just as confusing (to me).

When NUnit 2.0 switched to using attributes instead of inheritance, they made things clearer by requiring a [TestFixture] attribute on classes that previously derived from TestCase.

I can't say if they create a test fixture new instance for each test method defined in the class or not but that seems like an implementation detail that I don't need to concern myself with.

When it comes to using JUnit, I think of the TestCase class like this:

A TestCase derivative with only one test method in it defines a simple test case. A TestCase derivative with multiple test methods in it defines a test fixture.

Bill Venners

Posts: 2242
Nickname: bv
Registered: Jan, 2002

Re: Design for Usability Posted: Sep 13, 2004 11:21 AM
Reply to this message Reply
> I can't say if [NUnit] creates a test fixture new instance for
> each test method defined in the class or not but that
> seems like an implementation detail that I don't need to
> concern myself with.
>
According to Martin Fowler, NUnit doesn't create a new instance for each test method:


...The popular NUnit implementation doesn't do it. The following NUnit test causes a failure.



[TestFixture]
public class ServerTester
{
private IList list = new ArrayList();
[Test]
public void first() {
list.Add(1);
Assert.AreEqual(1, list.Count);
}
[Test]
public void second() {
Assert.AreEqual(0, list.Count);
}
}


From:

http://martinfowler.com/bliki/JunitNewInstance.html

Flat View: This topic has 4 replies on 1 page
Topic: Announcing Artima Developer Spotlight Previous Topic   Next Topic Topic: Thurston Howell III and Open Source Economics


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us