Application that runs a suite of tests. The application accepts command line arguments that specify optional user-defined properties, an optional runpath, zero to many Reporters, optional lists of test groups to include and/or exclude, zero to many Suite class names, zero to many "members-only" Suite paths, zero to many "wildcard" Suite paths, and zero to many TestNG XML config file paths. All of these arguments are described in more detail below. Here's a summary:

scala [-classpath scalatest-<version>.jar:...] org.scalatest.tools.Runner [-D<key>=<value> [...]] [-p <runpath>] [reporter [...]] [-n <includes>] [-x <excludes>] [-c] [-s <suite class name> [...]] [-m <members-only suite path> [...]] [-w <wildcard suite path> [...]] [-t <TestNG config file path> [...]]

The simplest way to start Runner is to specify the directory containing your compiled tests as the sole element of the runpath, for example:

scala -classpath scalatest-<version>.jar org.scalatest.tools.Runner -p compiled_tests

Given the previous command, Runner will discover and execute all Suites in the compiled_tests directory and its subdirectories, and show results in graphical user interface (GUI).

Specifying user-defined properties

A user-defined property consists of a key and a value. The key may not begin with "org.scalatest.". User-defined properties may be specified on the command line. Each property is denoted with a "-D", followed immediately by the key string, an "=", and the value string. For example:

-Ddbname=testdb -Dserver=192.168.1.188

Specifying a runpath

A runpath is the list of filenames, directory paths, and/or URLs that Runner uses to load classes for the running test. If runpath is specified, Runner creates a custom class loader to load classes available on the runpath. The graphical user interface reloads the test classes anew for each run by creating and using a new instance of the custom class loader for each run. The classes that comprise the test may also be made available on the classpath, in which case no runpath need be specified.

The runpath is specified with the -p option. The -p must be followed by a space, a double quote ("), a white-space-separated list of paths and URLs, and a double quote. If specifying only one element in the runpath, you can leave off the double quotes, which only serve to combine a white-space separated list of strings into one command line argument. Here's an example:

-p "serviceuitest-1.1beta4.jar myjini http://myhost:9998/myfile.jar"

Specifying reporters

Reporters can be specified on the command line in any of the following ways:

• -g[configs...] - causes display of a graphical user interface that allows tests to be run and results to be investigated
• -f[configs...] <filename> - causes test results to be written to the named file
• -o[configs...] - causes test results to be written to the standard output
• -e[configs...] - causes test results to be written to the standard error
• -r[configs...] <reporterclass> - causes test results to be reported to an instance of the specified fully qualified Reporter class name

The [configs...] parameter, which is used to configure reporters, is described in the next section.

The -r option causes the reporter specified in <reporterclass> to be instantiated. Each reporter class specified with a -r option must be public, implement org.scalatest.Reporter, and have a public no-arg constructor. Reporter classes must be specified with fully qualified names. The specified reporter classes may be deployed on the classpath. If a runpath is specified with the -p option, the specified reporter classes may also be loaded from the runpath. All specified reporter classes will be loaded and instantiated via their no-arg constructor.

For example, to run a suite named MySuite from the mydir directory using two reporters, the graphical reporter and a file reporter writing to a file named "test.out", you would type:

java -jar scalatest.jar -p mydir -g -f test.out -s MySuite

The -g, -o, or -e options can appear at most once each in any single command line. Multiple appearances of -f and -r result in multiple reporters unless the specified <filename> or <reporterclass> is repeated. If any of -g, -o, -e, <filename> or <reporterclass> are repeated on the command line, the Runner will print an error message and not run the tests.

Runner adds the reporters specified on the command line to a dispatch reporter, which will dispatch each method invocation to each contained reporter. Runner will pass the dispatch reporter to executed suites. As a result, every specified reporter will receive every report generated by the running suite of tests. If no reporters are specified, a graphical runner will be displayed that provides a graphical report of executed suites.

Configuring Reporters

Each reporter specification on the command line can include configuration characters. Configuration characters are specified immediately following the -g, -o, -e, -f, or -r. Valid configuration characters are:

• Y - present runStarting method invocations
• Z - present testStarting method invocations
• T - present testSucceeded method invocations
• F - present testFailed method invocations
• G - present testIgnored method invocations
• U - present suiteStarting method invocations
• P - present suiteCompleted method invocations
• B - present suiteAborted method invocations
• I - present infoProvided method invocations
• S - present runStopped method invocations
• A - present runAborted method invocations
• R - present runCompleted method invocations

Each reporter class has a default configuration. If no configuration is specified on the command line for a particular reporter, that reporter uses its default configuration. If a configuration is specified, Runner will present to the configured reporter only those report types mentioned in the configuration characters. If the command line includes argument -oFAB, for example, only testFailed, runAborted, and suiteAborted events will be reported to the standard output reporter.

For example, to run a suite using two reporters, the graphical reporter (using its default configuration) and a standard error reporter configured to print only test failures, run aborts, and suite aborts, you would type:

scala -classpath scalatest-<version>.jar -p mydir -g -eFAB -s MySuite

Note that no white space is allowed between the reporter option and the initial configuration parameters. So "-e FAB" will not work, "-eFAB" will work.

Specifying includes and excludes

You can specify named groups of tests to include or exclude from a run. To specify includes, use -n followed by a white-space-separated list of group names to include, surrounded by double quotes. (The double quotes are not needed if specifying just one group.) Similarly, to specify excludes, use -x followed by a white-space-separated list of group names to exclude, surrounded by double quotes. (As before, the double quotes are not needed if specifying just one group.) If includes is not specified, then all tests except those mentioned in the excludes group (and in the Ignore group), will be executed. (In other words, an empty includes list is like a wildcard, indicating all tests be included.) If includes is specified, then only those tests in groups mentioned in the argument following -n and not mentioned in the excludes group, will be executed. For more information on test groups, see the documentation for Suite. Here are some examples:

• -n CheckinTests
• -n FunctionalTests -x SlowTests
• -n "CheckinTests FunctionalTests"-x "SlowTests NetworkTests"

Executing Suites concurrently

With the proliferation of multi-core architectures, and the often parallelizable nature of tests, it is useful to be able to run tests concurrently. If you include -c on the command line, Runner will pass a Distributor to the Suites you specify with -s. Runner will set up a thread pool to execute any Suites passed to the Distributor's put method concurrently. Trait Suite's implementation of runNestedSuites will place any nested Suites into this Distributor. Thus, if you have a Suite of tests that must be executed sequentially, you should override runNestedSuites as described in the documentation for Distributor.

Specifying Suites

Suites are specified on the command line with a -s followed by the fully qualified name of a Suite subclass, as in:

-s com.artima.serviceuitest.ServiceUITestkit

Each specified suite class must be public, a subclass of org.scalatest.Suite, and contain a public no-arg constructor. Suite classes must be specified with fully qualified names. The specified Suite classes may be loaded from the classpath. If a runpath is specified with the -p option, specified Suite classes may also be loaded from the runpath. All specified Suite classes will be loaded and instantiated via their no-arg constructor.

The runner will invoke execute on each instantiated org.scalatest.Suite, passing in the dispatch reporter to each execute method.

Runner is intended to be used from the command line. It is included in org.scalatest package as a convenience for the user. If this package is incorporated into tools, such as IDEs, which take over the role of runner, object org.scalatest.tools.Runner may be excluded from that implementation of the package. All other public types declared in package org.scalatest.tools.Runner should be included in any such usage, however, so client software can count on them being available.

Specifying "members-only" and "wildcard" Suite paths

If you specify Suite path names with -m or -w, Runner will automatically discover and execute accessible Suites in the runpath that are either a member of (in the case of -m) or enclosed by (in the case of -w) the specified path. As used in this context, a path is a portion of a fully qualified name. For example, the fully qualifed name com.example.webapp.MySuite contains paths com, com.example, and com.example.webapp. The fully qualifed name com.example.webapp.MyObject.NestedSuite contains paths com, com.example, com.example.webapp, and com.example.webapp.MyObject. An accessible Suite is a public class that extends org.scalatest.Suite and defines a public no-arg constructor. Note that Suites defined inside classes and traits do not have no-arg constructors, and therefore won't be discovered. Suites defined inside singleton objects, however, do get a no-arg constructor by default, thus they can be discovered.

For example, if you specify -m com.example.webapp on the command line, and you've placed com.example.webapp.RedSuite and com.example.webapp.BlueSuite on the runpath, then Runner will instantiate and execute both of those Suites. The difference between -m and -w is that for -m, only Suites that are direct members of the named path will be discovered. For -w, any Suites whose fully qualified name begins with the specified path will be discovered. Thus, if com.example.webapp.controllers.GreenSuite exists on the runpath, invoking Runner with -w com.example.webapp will cause GreenSuite to be discovered, because its fully qualifed name begins with "com.example.webapp". But if you invoke Runner with -m com.example.webapp, GreenSuite will not be discovered because it is directly a member of com.example.webapp.controllers, not com.example.webapp.

If you specify no -s, -m, or -w arguments on the command line to Runner, it will discover and execute all accessible Suites in the runpath.

Specifying TestNG XML config file paths

If you specify one or more file paths with -t, Runner will create a org.scalatest.testng.TestNGWrapperSuite, passing in a List of the specified paths. When executed, the TestNGWrapperSuite will create one TestNG instance and pass each specified file path to it for running. If you include -t arguments, you must include TestNG's jar file on the class path or runpath. The -t argument will enable you to run existing TestNG tests, including tests written in Java, as part of a ScalaTest run. You need not use -t to run suites written in Scala that extend TestNGSuite. You can simply run such suites with -s, -m, or -w parameters.