Artima SuiteRunner is a free open source testing toolkit for Java released under the Open Software License. You can use this tool with JUnit to run existing JUnit test suites, or standalone to create unit and conformance tests for Java APIs. To get you up and running quickly, the Artima SuiteRunner distribution ZIP file includes a simple example. This article uses the example to show you how to use Artima SuiteRunner to run existing JUnit test suites.
When you unzip the Artima SuiteRunner distribution ZIP file, you get the following items:
||Recipe file for the example|
||Recipe file for the JUnit example (Added in 1.0beta4)|
||Directory of Java source and class files for the example|
||Directory containing Javadoc documentation for the API|
||The obligatory README file|
||Artima SuiteRunner executable JAR file|
||ZIP file of Artima SuiteRunner source code|
||Recipe file for the XMLReporter example (Added in 1.0beta5)|
Artima SuiteRunner's executable JAR file is suiterunner-[release].jar. The source and class files for the example are found in the
example directory. A recipe file that runs the example JUnit test suite is contained in
accountjunit.srj, which was added to the distribution in 1.0beta4. If you don't see
accountjunit.srj, then you need to download the latest version of Artima SuiteRunner.
This article shows screenshots of Artima SuiteRunner version 1.0beta4, as it appeared on Mac OS X. To run the example JUnit test suite, type this command in the
suiterunner-[release] subdirectory of the directory in which you unzipped the Artima SuiteRunner distribution ZIP file:
java -jar suiterunner-[release].jar accountjunit.srj
When you type the previous command, you should see:
Figure 1. Without
junit.jar in the runpath, the run aborts.
When you first start Artima SuiteRunner with the previous command, it attempts to run the
TestCase mentioned in the recipe file. As shown in Figure 1, this initial run attempt aborts. To see the information shown in Figure 1 in the Details text area, click on "Run Aborted" in the Reports list. The details indicate that Artima SuiteRunner aborted your requested run because it couldn't find class
java -jar suiterunner-[release].jar accountjunit.srj
A recipe file is a Java properties file that supplies information needed by Artima SuiteRunner to run a suite of tests. Recipe files contain values for three properties:
org.suiterunner.Reporters. The contents of
org.suiterunner.Runpath=-p "example" org.suiterunner.Suites=-s com.artima.examples.account.ex6test.AccountTestCase org.suiterunner.Reporters=-g
-p "example") specifies a runpath with a single directory,
-s com.artima.examples.account.ex6test.AccountTestCase) indicates that Artima SuiteRunner should load the specified class, a subclass of
junit.framework.TestCase, and run it.
-g) indicates that Artima SuiteRunner will display its graphical user interface (GUI) and show the results of the run there.
When invoked via the previous command that specifies
accountjunit.srj as a command line parameter, Artima SuiteRunner will:
URLClassLoaderthat can load classes from the
exampledirectory, the directory specified in the recipe file's
URLClassLoader, attempt to load class
com.artima.examples.account.ex6test.AccountTestCase, the class specified in in the recipe file's
com.artima.examples.account.ex6test.AccountTestCaseclass can't be loaded because its superclass,
junit.framework.TestCase, can't be found.
NoClassDefFoundErrorto the user.
junit.jarto the Runpath
When you type the following command, Artima SuiteRunner aborts because it can't load class
java -jar suiterunner-[release].jar accountjunit.srj
To enable Artima SuiteRunner to load
junit.framework.TestCase and the rest of the JUnit classes, you must tell Artima SuiteRunner where to find
junit.jar. JUnit is not included in the Artima SuiteRunner distribution ZIP file. Presumably, if you want to use Artima SuiteRunner as a JUnit runner, you have already JUnit installed on your system. If you don't, you can download JUnit from junit.org. Once you locate
junit.jar on your system, you can add it to Artima SuiteRunner's runpath by editing the recipe. To edit the recipe, select File-->Edit Recipe, as shown in Figure 2.
Figure 2. Editing the recipe.
The Edit Recipe dialog allows you to edit all three components of the recipe: suites, runpath, and reporters. To add
junit.jar to the runpath, you must select the Runpath tab, as shown in Figure 3.
Figure 3. Selecting the Runpath tab.
The easiest way to add
junit.jar to the runpath is by pressing on the Select button on the right hand side of the Runpath tab's dialog. Pressing Select brings up a file chooser dialog with the title "Select JAR File or Directory to Add." Use this dialog box to navigate to your copy of
junit.jar, as shown in Figure 4, then press Choose.
Figure 4. Selecting
junit.jar to add to the runpath.
On my Macintosh,
junit.jar is located in the
/Users/bv/nobkp/junit directory. As Figure 5 shows,
junit.jar is now in the runpath. When you press OK, Artima SuiteRunner will change the runpath per your request, and attempt to run
Figure 5. File
junit.jar is now in the runpath.
When you press the OK button of the Edit Recipe dialog, the
AccountTestCase should run and give the results shown in Figure 6. Don't be disheartened by all the red you see in the Reports list. The example is supposed to fail initially. It is left as an exercise for you to fix the bugs in the account example to get the test to run successfully. The existence of the Run Starting, Test Failed, and Run Completed reports in the Reports list indicates that Artima SuiteRunner successfully loaded the JUnit classes and executed the example JUnit test suite.
Figure 6. With
junit.jar in the runpath, the
AccountTestCase example runs.
In the previous step, you used Artima SuiteRunner as a JUnit runner to run the example JUnit test suite included in the Artima SuiteRunner distribution. Now I'll show you how to use Artima SuiteRunner to run your own existing JUnit test suite. To do so, you must edit the recipe and change the runpath and suites list.
To edit the runpath, select File-->Edit Recipe and click on the Runpath tab. The Runpath tab's dialog should still show the
example directory and the
junit.jar file. To run your existing JUnit test suite, you must keep
junit.jar in the runpath, but you can get rid of the
example directory. To remove the
example directory from the runpath, click on
example and press the Remove button, as shown in Figure 7.
Figure 7. Removing
example from the runpath.
After pressing the Remove button, only
junit.jar remains in the runpath, as shown in Figure 8.
Figure 8. The
example directory is no longer in the runpath.
At this point, you must add to the runpath all JAR files and directories needed by JUnit to run your existing test suite. If you use JUnit's Swing runner, for example, you will likely need to add to the runpath all the files and directories that appear in the classpath when you run JUnit. To load classes for your existing JUnit test suite, Artima SuiteRunner will create a
URLClassLoader that will look in each JAR file and directory in the order they appear in the runpath.
In my case, I have a JUnit-based conformance test suite for the ServiceUI API sitting in the directory
/Users/bv/projects/scslsrc/serviceui. (This is the very JUnit test suite that gave me sufficient frustration that I went off and started writing what eventually became Artima SuiteRunner, as described in Why We Refactored JUnit.) Because both the classes that make up the ServiceUI API conformance test suite and the ServiceUI API classes being tested can be loaded from the
/Users/bv/projects/scslsrc/serviceui directory, all I need to do is to add the
/Users/bv/projects/scslsrc/serviceui directory to the runpath. To add this directory, I press the Select button on the Runpath tab's dialog, navigate to the
/Users/bv/projects/scslsrc directory, select
serviceui, and press the Choose button, as shown in Figure 9.
Figure 9. Selecting the
serviceui directory to add to the runpath.
As shown in Figure 10, my runpath now includes
junit.jar and the directory from which Artima SuiteRunner can load both the class files for the JUnit-based ServiceUI API conformance test suite and the actual ServiceUI API classes they test.
Figure 10. The
serviceui directory is now in the runpath.
At this point, my runpath will enable all the classes needed by the JUnit-based ServiceUI API conformance test suite to be loaded, but my recipe is still not quite ready. The recipe's suites list still indicates Artima SuiteRunner should run class
com.artima.examples.ex6.ex6test.AccountTestCase, the class from the example JUnit test suite specified in
accountjunit.srj. Hopefully, you have built the correct runpath for your own existing JUnit test suite. Your only remaining step is to specify the appropriate suites list.
As a first step, you can remove the
com.artima.examples.ex6.ex6test.AccountTestCase class from the suites list. If you press the Suites tab in the Edit Recipe dialog, you will see the current suites list, as shown in Figure 11.
Figure 11. Removing
AccountTestCase from the suites list.
AccountTestCase in the suites list and press the Remove button. This should leave no suites specified, as shown in Figure 12. Pressing the Edit Recipe's OK button will cause Artima SuiteRunner to change the recipe as you specified, with a new runpath and empty suites list.
Figure 12. The suites list is now empty.
At this point, the easiest way to select a JUnit
TestCase to run is by pressing the Select button at the top of Artima SuiteRunner's main window. As shown in Figure 13, Artima SuiteRunner will search the runpath for any
org.suiterunner.Suite classes or
junit.framework.TestCase classes and display them in the "Select a Suite to Run" dialog's Suites list box.
If you don't see the
TestCase you want to run in this list, that means you have an error in your runpath. Make sure the JAR file or directory that contains the
TestCase class you want to run appears in the runpath.
In my case, Artima SuiteRunner discovers all the
TestCases in the JUnit-based ServiceUI API conformance test suite, because they are all available in the directory I added to the runpath,
/Users/bv/projects/scslsrc/serviceui. To indicate to Artima SuiteRunner that I want to run
LocalesTestCase, I select
LocalesTestCase in the Suites list and press OK, as shown in Figure 13.
Figure 13. Selecting
LocalesTestCase via the Suite Selector dialog.
As soon as I press the Select a Suite to Run dialog's OK button, Artima SuiteRunner attempts to load the class I selected,
LocalesTestCase, and run it. As shown in Figure 14, Artima SuiteRunner is able to load and run this JUnit
TestCase, and what's more, the test itself succeeds with a big satisfying green bar.
Figure 14. The
LocalesTestCase runs with no errors.
Once you get the recipe correct, Artima SuiteRunner will run your existing JUnit test suite. To make it easier to run this test suite again in the future, you should save the current recipe in a file. The current recipe file name, which appears in the title bar of Artima SuiteRunner's main window, is
accountjunit.srj. This recipe file, which you specified when you started Artima SuiteRunner, contains the recipe for running the account JUnit test suite example. If you select File-->Save, you will overwrite
accountjunit.srj with your new recipe. Most likely, you'll want to create a new recipe file by selecting File-->Save As, as shown in Figure 15.
Figure 15. Saving the current recipe in a file.
You can name the recipe file anything you want, but the default extension for a recipe file is
.srj (for SuiteRunner for Java). In my case, I decided to name the file
serviceui.srj, as shown in Figure 16.
Figure 16. Naming the recipe file
Once you save your new recipe as a file, the new file name will appear in the title bar of Artima SuiteRunner's main window. In my case, the name
serviceui.srj appears in Artima SuiteRunner's title bar, as shown in Figure 17.
Figure 17. Running from the
serviceui.srj recipe file.
Once you have saved your recipe in a file, you can rerun the test specified by the recipe by starting Artima SuiteRunner with the name of your new recipe file as the first argument. If you associate the extension
.srj with the command
java -jar suiterunner-[release].jar on your desktop, then you can rerun tests by simply double clicking on the recipe file.
This article gives an introduction to using Artima SuiteRunner as a JUnit runner to run existing JUnit test suites, but doesn't cover all possible ways to use Artima SuiteRunner in conjunction with JUnit. For example, I could have used the following command to run my existing JUnit-based ServiceUI API conformance test suite completely from the classpath with no runpath specified:
java -classpath ./suiterunner-1.0beta4.jar:/Users/bv/nobkp/junit/junit.jar:/Users/bv/projects/scslsrc/serviceui org.suiterunner.Runner -s com.artima.test.serviceui.ui.attribute.LocalesTestCase
To find out more about how Artima SuiteRunner can help you create and run unit and conformance tests, please see the Artima SuiteRunner Tutorial.
For help with Artima SuiteRunner, please post to the SuiteRunner Forum.
JUnit is available at:
Why We Refactored JUnit
Artima SuiteRunner Tutorial, Building Conformance and Unit Tests with Artima SuiteRunner:
Create an XML Reporter for Your Unit Tests, how to create a customer reporter for Artima SuiteRunner that formats unit test results in XML:
Artima SuiteRunner home page:
Artima SuiteRunner download page (You must log onto Artima.com to download the release):
The SuiteRunner Forum:
Have an opinion? Be the first to post a comment about this article.
Bill Venners is president of Artima Software, Inc. and editor-in-chief of Artima.com. He is author of the book, Inside the Java Virtual Machine, a programmer-oriented survey of the Java platform's architecture and internals. His popular columns in JavaWorld magazine covered Java internals, object-oriented design, and Jini. Bill has been active in the Jini Community since its inception. He led the Jini Community's ServiceUI project that produced the ServiceUI API. The ServiceUI became the de facto standard way to associate user interfaces to Jini services, and was the first Jini community standard approved via the Jini Decision Process. Bill also serves as an elected member of the Jini Community's initial Technical Oversight Committee (TOC), and in this role helped to define the governance process for the community. He currently devotes most of his energy to building Artima.com into an ever more useful resource for developers.