A suite of tests. A Suite instance encapsulates a conceptual suite (i.e., a collection) of tests. This trait defines a default way to create and execute tests, which involves writing test methods. This approach will likely suffice in the vast majority of applications, but if desired, subtypes can override certain methods to define other ways to create and execute tests.

The easiest way to use this trait is to use its default approach: Simply create classes that extend Suite and define test methods. Test methods have names of the form testX, where X is some unique, hopefully meaningful, string. A test method must be public and can have any result type, but the most common result type is Unit. Here's an example:

 import org.scalatest.Suite

 class MySuite extends Suite {

   def testAddition() {
     val sum = 1 + 1
     assert(sum === 2)
     assert(sum + 2 === 4)
   }

   def testSubtraction() {
     val diff = 4 - 1
     assert(diff === 3)
     assert(diff - 2 === 1)
   }
 }
 

You run a Suite by invoking on it one of three overloaded execute methods. Two of these execute methods are intended to serve as a convenient way to run tests from within the Scala interpreter. For example, to run MySuite from within the Scala interpreter, you could write:

 scala> (new MySuite).execute()
 

Or, to run just the testAddition method, you could write:

 scala> (new MySuite).execute("testAddition")
 

These two execute methods print test results to the standard output. If you try these two examples from within the Scala interpeter, you should see reports that indicate the tests were started and successfully completed.

The third overloaded execute method takes seven parameters, so it is a bit unwieldy to invoke from within the Scala interpreter. Instead, this execute method is intended to be invoked indirectly by a test runner, such as Runner or an IDE. See the documentation for Runner for more detail.

Assertions and ===

Inside test methods, you can write assertions by invoking assert and passing in a Boolean expression, such as:

 val left = 2
 val right = 1
 assert(left == right)
 

If the passed expression is true, assert will return normally. If false, assert will complete abruptly with an AssertionError. This exception is usually not caught by the test method, which means the test method itself will complete abruptly by throwing the AssertionError. Any test method that completes abruptly with an AssertionError or any Exception is considered a failed test. A test method that returns normally is considered a successful test.

If you pass a Boolean expression to assert, a failed assertion will be reported, but without reporting the left and right values. You can alternatively encode these values in a String passed as a second argument to assert, as in:

 val left = 2
 val right = 1
 assert(left == right, left + " did not equal " + right)
 

Using this form of assert, the failure report will include the left and right values, thereby helping you debug the problem. However, Suite provides the === operator to make this easier. You use it like this:

 val left = 2
 val right = 1
 assert(left === right)
 

Because you use === here instead of ==, the failure report will include the left and right values. For example, the detail message in the thrown AssertionErrorm from the assert shown previously will include, "2 did not equal 1". From this message you will know that the operand on the left had the value 2, and the operand on the right had the value 1.

If you're familiar with JUnit, you would use === in a ScalaTest Suite where you'd use assertEquals in a JUnit TestCase. The === operator is made possible by an implicit conversion from Any to Equalizer. If you're curious to understand the mechanics, see the documentation for Equalizer and Suite's convertToEqualizer method.

Expected results

 val a = 5
 val b = 2
 expect(2) {
   a - b
 }
 

In this case, the expected value is 2, and the code being tested is a - b. This expectation will fail, and the detail message in the AssertionError will read, "Expected 2, but got 3."

Intercepted exceptions

Sometimes you need to test whether a method throws an expected exception under certain circumstances, such as when invalid arguments are passed to the method. You can do this in the JUnit style, like this:

 val s = "hi"
 try {
   s.charAt(-1)
   fail()
 }
 catch {
   case e: IndexOutOfBoundsException => // Expected, so continue
 }
 

If charAt throws IndexOutOfBoundsException as left, control will transfer to the catch case, which does nothing. If, however, charAt fails to throw an exception, the next statement, fail(), will be executed. The fail method always completes abruptly with an AssertionError, thereby signaling a failed test.

To make this common use case easier to express and read, Suite provides an intercept method. You use it like this:

 val s = "hi"
 intercept(classOf[IndexOutOfBoundsException]) {
   s.charAt(-1)
 }
 

This code behaves much like the previous example. If charAt throws an instance of IndexOutOfBoundsException, intercept will return that exception. But if charAt completes normally, or throws a different exception, intercept will complete abruptly with an AssertionError. intercept returns the caught exception so that you can inspect it further if you wish, for example, to ensure that data contained inside the exception has the expected values.

Nested suites

A Suite can refer to a collection of other Suites, which are called nested Suites. Those nested Suites can in turn have their own nested Suites, and so on. Large test suites can be organized, therefore, as a tree of nested Suites. This trait's execute method, in addition to invoking its test methods, invokes execute on each of its nested Suites.

A List of a Suite's nested Suites can be obtained by invoking its nestedSuites method. If you wish to create a Suite that serves as a container for nested Suites, whether or not it has test methods of its own, simply override nestedSuites to return a List of the nested Suites. Because this is a common use case, ScalaTest provides a convenience SuperSuite class, which takes a List of nested Suites as a constructor parameter. Here's an example:

 import org.scalatet.Suite

 class ASuite extends Suite
 class BSuite extends Suite
 class CSuite extends Suite

 class AlphabetSuite extends SuperSuite(
   List(
     new ASuite,
     new BSuite,
     new CSuite
   )
 )
 

If you now run AlphabetSuite, for example from the interpreter:

 scala> (new AlphabetSuite).execute()
 

You will see reports printed to the standard output that indicate nested suites—ASuite, BSuite, and CSuite—were run.

Note that Runner can discover Suites automatically, so you need not necessarily specify SuperSuites explicitly. See the documentation for Runner for more information.

Test fixtures

A test fixture is objects or other artifacts (such as files, sockets, database connections, etc.) used by tests to do their work. If a fixture is used by only one test method, then the definitions of the fixture objects should be local to the method, such as the objects assigned to sum and diff of the earlier MySuite examples. If multiple methods need to share a fixture, the best approach is to assign them to instance variables. Here's a (very contrived) example, in which the object assigned to shared is used by multiple test methods:

 import org.scalatest.Suite

 class MySuite extends Suite {

   // Sharing fixture objects via instance variables
   val shared = 5

   def testAddition() {
     val sum = 2 + 3
     assert(sum === shared)
   }

   def testSubtraction() {
     val diff = 7 - 2
     assert(diff === shared)
   }
 }
 

In some cases, however, a shared fixture may be changed by a test method such that it needs to be recreated or reinitialized before each test. It may also need to be cleaned up after each test. JUnit offers methods setup and tearDown for this purpose. In ScalaTest, you can avoid vars by writing a createFixture method that returns a new instance of the fixture object (or returns a tuple of new instances of fixture objects) each time it is called. You can then call createFixture at the beginning of each test method that needs the fixture, storing the fixture object or objects in local variables. Here's an example:

 import org.scalatest._
 import scala.collection.mutable.ListBuffer

 class MySuite extends Suite {

   def createFixture = (new StringBuilder("ScalaTest is "), new ListBuffer[String])

   def testEasy() {
     val (sb, lb) = createFixture
     sb.append("easy!")
     assert(sb.toString === "ScalaTest is easy!")
     assert(lb.isEmpty)
     lb += "sweet"
   }

   def testFun() {
     val (sb, lb) = createFixture
     sb.append("fun!")
     assert(sb.toString === "ScalaTest is fun!")
     assert(lb.isEmpty)
   }
 }
 

Another approach to fixtures that avoids vars is to use traits FunSuite1 through FunSuite9. If you prefer instead to reassign variables to reinitialize a fixture, however, one approach is to override runTest. Here's an example:

 import org.scalatest._

 class MySuite extends Suite {

   var sb: StringBuilder = _

   override protected def runTest(testName: String, reporter: Reporter,
       stopper: Stopper, properties: Map[String, Any]) {

     // First, initialize the fixture. This
     // would be in JUnit's setup() method.
     sb = new StringBuilder("ScalaTest is ")

     // Then call super.runTest to run the test 
     super.runTest(testName, reporter, stopper, properties)

     // Then perform cleanup. This would be in
     // JUnit's teardown() method. If needed, you could 
     // also do cleanup in a finally clause.
     sb.setLength(0)
   }

   def testEasy() {
     sb.append("easy!")
     assert(sb.toString === "ScalaTest is easy!")
   }

   def testFun() {
     sb.append("fun!")
     assert(sb.toString === "ScalaTest is fun!")
   }
 }
 

In this example, the instance variable sb is a var, so it can be reinitialized between tests. The runTest method, which is invoked by execute for each test method to run, is overriden here such that sb is first initialized. Then super.runTest is invoked, which causes the test method to be run. Lastly, the fixture is “cleaned up” by clearing the buffer.

Properties

In some cases you may need to pass information from a suite to its nested suites. For example, perhaps a main suite needs to open a database connection that is then used by all of its nested suites. You can accomplish this in ScalaTest by using properties, which are passed to execute as a Map[String, Any]. This trait's execute method calls two other methods, both of which you can override:

• runNestedSuites - responsible for running this Suite's nested Suites
• runTests - responsible for running this Suite's tests

To pass custom properties to nested Suites, simply override runNestedSuites. Here's an example:

 import org.scalatest._
 import java.io.FileWriter
 
 class NestedSuite extends Suite {
   override def execute(testName: Option[String], reporter: Reporter, stopper: Stopper,
       includes: Set[String], excludes: Set[String], properties: Map[String, Any], distributor: Option[Distributor]) {
 
     val w = properties("fixture.FileWriter").asInstanceOf[FileWriter]
     w.write("hi there\n")
   }
 }
 
 class MainSuite extends SuperSuite(new NestedSuite :: Nil) {
 
   override def runNestedSuites(reporter: Reporter, stopper: Stopper, includes: Set[String],
       excludes: Set[String], properties: Map[String, Any], distributor: Option[Distributor]) {
 
     val w = new FileWriter("fixture.txt")
     try {
       val myProps = properties + ("fixture.FileWriter" -> w)
       super.runNestedSuites(reporter, stopper, includes, excludes, myProps, distributor)
     }
     finally {
       w.close()
     }
   }
 }
 

In this example, MainSuite's runNestedSuites method opens a file for writing, then passes the FileWriter to its NestedSuite via the properties Map. The NestedSuite grabs the FileWriter from the properties Map and writes a friendly message to the file.

Suite set up and clean up

The previous example gives a hint at how you would accomplish the kind of intialization and cleanup in ScalaTest that you would do with @BeforeClass and @AfterClass annotations in JUnit 4. If you want to do something before and/or after all tests in a Suite, you override runTests. Inside runTests, you first do what you need to do before all tests, then call super.runTests, then do what you need to after all tests. For example, if you must open a file before the tests are run, and close it after all the tests have completed, you could override the method like this:

 import org.scalatest._
 import java.io.FileReader
 
 class MySuite extends Suite {
 
   var reader: FileReader = _
 
   protected override def runTests(testName: Option[String], reporter: Reporter, stopper: Stopper,
       includes: Set[String], excludes: Set[String], properties: Map[String, Any]) {
 
     reader = new FileReader("filename.txt")
     try {
       super.runTests(testName, reporter, stopper, includes, excludes, properties)
     }
     finally {
       reader.close()
     }
   }
 }
 

If you want to do something before and after both the tests and the nested Suites, then override execute itself.

Test groups

A Suite's tests may be classified into named groups. When executing a Suite, groups of tests can optionally be included and/or excluded. In this trait's implementation, groups are indicated by annotations attached to the test method. To create a group, simply define a new Java annotation. (Currently, for annotations to be visible in Scala programs via Java reflection, the annotations themselves must be written in Java.) For example, to create a group named SlowAsMolasses, to use to mark slow tests, you would write in Java:

 import java.lang.annotation.*; 
 
 @Retention(RetentionPolicy.RUNTIME)
 @Target({ElementType.METHOD, ElementType.TYPE})
 public @interface SlowAsMolasses {}
 

Given this new annotation, you could place methods into the SlowAsMolasses group like this:

 @SlowAsMolasses
 def testSleeping() = sleep(1000000)
 

The primary execute method takes two Set[String]s called includes and excludes. If includes is empty, all tests will be executed except those those belonging to groups listed in the excludes Set. If includes is non-empty, only tests belonging to groups mentioned in includes, and not mentioned in excludes, will be executed.

Ignored tests

Another common use case is that tests must be “temporarily” disabled, with the good intention of resurrecting the test at a later time. ScalaTest provides an Ignore annotation for this purpose. You use it like this:

 import org.scalatest.Suite
 import org.scalatest.Ignore

 class MySuite extends Suite {

   def testAddition() {
     val sum = 1 + 1
     assert(sum === 2)
     assert(sum + 2 === 4)
   }

   @Ignore
   def testSubtraction() {
     val diff = 4 - 1
     assert(diff === 3)
     assert(diff - 2 === 1)
   }
 }
 

If you run this version of MySuite with:

 scala> (new MySuite).execute()
 

It will run only testAddition and report that testSubtraction was ignored.

Ignore is implemented as a group. The execute method that takes no parameters adds org.scalatest.Ignore to the excludes Set it passes to the primary execute method, as does Runner. The only difference between org.scalatest.Ignore and the groups you may define and exclude is that ScalaTest reports ignored tests to the Reporter. The reason ScalaTest reports ignored tests is as a feeble attempt to encourage ignored tests to be eventually fixed and added back into the active suite of tests.

Reporters

One of the parameters to the primary execute method is a Reporter, which will collect and report information about the running suite of tests. Information about suites and tests that were run, whether tests succeeded or failed, and tests that were ignored will be passed to the Reporter as the suite runs. Most often the reporting done by default by Suite's methods will be sufficient, but occasionally you may wish to provide custom information to the Reporter from a test method. For this purpose, you can optionally include a Reporter parameter in a test method, and then pass the extra information to the Reporter's infoProvided method. Here's an example:

 import org.scalatest._
 
 class MySuite extends Suite {
   def testAddition(reporter: Reporter) {
     assert(1 + 1 === 2)
     val report =
       new Report("MySuite.testAddition(Reporter)", "Addition seems to work.")
     reporter.infoProvided(report)
   }
 }
 

 Info Provided: MySuite.testAddition: Addition seems to work.
 

Executing suites concurrently

The primary execute method takes as its last parameter an optional Distributor. If a Distributor is passed in, this trait's implementation of execute puts its nested Suites into the distributor rather than executing them directly. The caller of execute is responsible for ensuring that some entity executes the Suites placed into the distributor. The -c command line parameter to Runner, for example, will cause Suites put into the Distributor to be executed concurrently via a pool of threads.

Extensibility

Trait Suite provides default implementations of its methods that should be sufficient for most applications, but many methods can be overridden when desired. Here's a summary of the methods that are intended to be overridden, and why you might want to do so:

• execute - override this method to define custom ways to executes suites of tests. A common use case is to set up a fixture before running a suite of tests, and if needed, clean it up afterwords.
• runTest - override this method to define custom ways to execute a single named test. A common use case is to set up a fixture before running each test in a suite, and if needed, clean it up afterwords.
• testNames - override this method to specify the Suite's test names in a custom way, such as in a different order.
• groups - override this method to specify the Suite's test groups in a custom way.
• nestedSuites - override this method to specify the Suite's nested Suites in a custom way.
• suiteName - override this method to specify the Suite's name in a custom way.
• expectedTestCount - override this method to count this Suite's expected tests in a custom way.

For example, this trait's implementation of testNames performs reflection to discover methods starting with test, and places these in a Set whose iterator returns the names in alphabetical order. If you wish to run tests in a different order in a particular Suite, perhaps because a test named testAlpha can only succeed after a test named testBeta has run, you can override testNames so that it returns a Set whose iterator returns testBeta before testAlpha. (This trait's implementation of execute will invoke tests in the order they come out of the testNames Set iterator.)

Alternatively, you may not like starting your test methods with test, and prefer using @Test annotations in the style of Java's JUnit 4 or TestNG. If so, you can override testNames to discover tests using either of these two APIs @Test annotations, or one of your own invention.

Moreover, test in ScalaTest does not necessarily mean test method. A test can be anything that can be given a name, that starts and either succeeds or fails, and can be ignored. In a Scala test framework called Rehersal, for example, tests are represented as function values. This approach might look foreign to JUnit users, but may feel more natural to programmers with a functional programming background. If you prefer this approach, you can override testNames, runTest, and execute such that you can define tests as function values.

You can also model JUnit, JUnit 4, or TestNG tests as suites of tests, thereby incorporating existing Java tests into a ScalaTest suite. If this turns out to be a common use case, we may later add wrapper suites to the API that will make this easy. The point here is that no matter what legacy tests you may have, it is likely you can create our use an existing Suite subclass that allows you to model those tests as ScalaTest suites and tests, to incorporate them into a ScalaTest suite. You can then write new tests in Scala and continue supporting older tests in Java.