Sponsored Link •
Artima SuiteRunner, a free, open source unit and conformance testing toolkit for Java, was released this week. Find out why the authors felt the need to create Artima SuiteRunner and learn how you can use it to run JUnit test suites.
This week Matt Gerrans, Frank Sommers, and I released Artima SuiteRunner, a free open source testing toolkit for Java. You can use this tool with JUnit to run existing JUnit test suites, or standalone to create unit and conformance tests for Java APIs. JUnit, originally written by Erich Gamma and Kent Beck, is very popular among Java developers and especially the extreme programming (XP) community. Like most Java developers, when we needed to create tests we first turned to JUnit. But as we started using JUnit in our own projects, we encountered some limitations. We created a new tool, Artima SuiteRunner, that better addressed our testing needs. In this article, we explain why JUnit didn't fit the bill, and how Artima SuiteRunner was created.
As leader of the Jini community's ServiceUI project, I wrote a specification and implemented an API that defines a standard way to attach user interfaces to Jini services. Within the Jini community, the ServiceUI API quickly became a de facto standard, but I wanted to make it official. To propose the ServiceUI API as an official Jini community standard through the Jini Decision Process, however, I needed a conformance test kit (CTK). To help me build the ServiceUI CTK, I turned to JUnit.
My initial experience with JUnit was both pleasant and frustrating. I quickly learned how to create a test case and write test methods, but found it difficult to run a test suite. In fact, I spent three hours sitting with my laptop at a coffee shop in Ft. Collins, Colorado, trying hard not to pull my hair out in frustration. But once I set up JUnit to run my test suite, I was quite happy. I liked the simplicity of creating JUnit test cases and adding test methods. I liked the red or green status bar of JUnit's graphical runner. Except for one special signature test, I proceeded contentedly to write the entire ServiceUI CTK with JUnit.
Before long I had only one missing piece remaining in my ServiceUI CTK: a signature test. A signature test verifies the artifacts of an API's signature, such as name, type, parameters, access level, and other modifiers. It does not verify the API's semantics. Although not necessary for unit testing, a signature test is a critical part of any conformance test. A signature test ensures that each type, field, and method required by an API specification is actually present in an implementation.
Terence Parr, creator of ANTLR, once said, "Why write something in five days that you can spend five years automating?" When it was time to write the signature test portion of the ServiceUI CTK, I decided rather than write it by hand, I would write a generic tool to automatically generate signature tests. I figured I would design this tool as an add-on to JUnit, and release it open source, primarily for the benefit of the Jini community.
So I wrote a tool to generate signature tests, and then attempted to make it work with JUnit. Once again, my hair was in danger of being pulled, but this time it was worse. My previous difficulty was primarily in figuring out how to use JUnit as an application. Now I was trying to figure out how to use JUnit as an API. At this point I was 9000 feet above sea level, in Crested Butte, Colorado. Perhaps the thin air affected my cognitive abilities, but at one point I just threw up my hands and said to myself, "Why is this so hard? It would be easier to rewrite JUnit than figure it out."
Just some friendly advice: if you ever find yourself saying it would be easier to rewrite something rather than figure it out, slap yourself. Chances are you are wrong. I certainly was. Creating SuiteRunner was a huge amount of work. Despite my frustrations with JUnit, it would have been orders of magnitude easier to decipher JUnit's source code than to create a brand new testing toolkit.
Nevertheless, I was fairly far along in the process of creating Artima SuiteRunner before I realized this, and along the way two of my friends, Matt Gerrans and Frank Sommers, became involved in the project. So despite many moments of doubt, including one decision to cancel the project entirely, ultimately Matt, Frank, and I decided to finish what we had started.