The Artima Developer Community
Articles | News | Weblogs | Buzz | Chapters | Forums

ScalaTest
A tool for testing Scala and Java software
OSI Certified Open Source Software

ScalaTest is a free, open-source testing tool for Scala and Java programmers. It is written in Scala, and enables you to write tests in Scala to test either Scala or Java code. It is released under the Apache 2.0 open source license.

Because different developers take different approaches to creating software, no single approach to testing is a good fit for everyone. In light of this reality, ScalaTest is designed to facilitate different styles of testing. ScalaTest provides several traits that you can mix together into whatever combination makes you feel the most productive.

For behavior-driven developers

For example, if you like the behavior-driven development style of testing, in which tests are viewed as informal specifications of the code under test, you might mix trait Spec with trait ShouldMatchers. Your tests would look something like: 

class StackSpec extends Spec with ShouldMatchers {

  describe("A Stack") {

    it("should pop values in last-in-first-out order") {
      val stack = new Stack[Int]
      stack.push(1)
      stack.push(2)
      stack.pop() should equal (2)
      stack.pop() should equal (1)
    }

    it("should be empty when created") {
      val stack = new Stack[Int]
      stack should be ('empty)
    }
  }
}

When you run this Spec its output will be structured as an easy-to-read informal specification of the Stack it tests. ScalaTest can format the specification output in its GUI or in text. In text, it would look like: 

A Stack
- should pop values in last-in-first-out order
- should be empty when created

ScalaTest's ShouldMatchers provide an elegant DSL for expressing assertions, which can make the code more readable and failure messages easier to understand. Some other examples are: 

map should (contain key ("one") and not contain value (10))
result should be >= (0)
book should have (title ("Programming in Scala"))
tempFile should not be a ('directory)
quotient should be (1.0 plusOrMinus 0.2)

For more traditional testers

As elegant and readable as the matcher expressions may seem to some, you may find them too verbose for your taste. You may also prefer to think of tests as, well, tests rather than as specifications, and you also may find you often need to test private methods. If this describes you, you might mix trait FunSuite with trait PrivateMethodTester, yielding tests that look like: 

class StackSuite extends FunSuite with PrivateMethodTester {

  test("Stack should pop values in last-in-first-out order") {
    val stack = new Stack[Int]
    stack.push(1)
    stack.push(2)
    assert(stack.pop() === (2))
    assert(stack.pop() === (1))
  }

  test("Stack should throw NoSuchElementException if an empty stack is popped") {
    val emptyStack = new Stack[String]
    intercept[NoSuchElementException] {
      emptyStack.pop()
    }
  }

  test("Stack's private decorate method should place quotes around strings") {
    val stack = new Stack[String]
    val decorate = PrivateMethod[String]('decorate)
    val result = stack invokePrivate decorate("hello")
    assert(result === "\"hello\"")
  } 
}

For TestNG enthusiasts

Or perhaps you've been using TestNG in Java, and you want to continue using it in Scala. You like the matchers DSL, but prefer the word "must" to "should". If this sounds like you, you can mix trait TestNGSuite with trait MustMatchers, yielding TestNG tests (i.e., TestNG will be happy to run them, as will ScalaTest) that look like this: 

class StackSuite extends TestNGSuite with MustMatchers {

  @Test def mustPopInLifoOrder() {
    val stack = new Stack[Int]
    stack.push(1)
    stack.push(2)
    stack.pop() must equal (2)
    stack.pop() must equal (1)
  }

  @Test def mustBeEmptyWhenCreated() {
    val stack = new Stack[Int]
    stack must be ('empty)
  }
}

For JUnit and ScalaCheck users

ScalaTest also provides traits that support writing JUnit tests in Java, combining the benefits of using the established defacto standard testing tool for the Java community with the more concise, readable syntax of ScalaTest assertions and matchers. If you want to write JUnit tests in Scala, but also really like the bang for the buck you get by writing ScalaCheck property checks, you might mix trait JUnit3Suite with trait Checkers. (Trait Checkers can be mixed into any testing trait, including the Spec, FunSuite, and TestNGSuite traits shown previously, to include ScalaCheck property checks among traditional assertions.) This would give you tests that look like: 

class StackSuite extends JUnit3Suite with Checkers {

  def testPopsInLifoOrder() {
    val stack = new Stack[Int]
    stack.push(1)
    stack.push(2)
    assert(stack.pop() === (2))
    assert(stack.pop() === (1))
  }

  def testIsEmptyWhenCreated() {
    val stack = new Stack[Int]
    assert(stack.isEmpty)
  }

  def testElementsHasAllObjectsPushedInOrder() { // uses a ScalaCheck property check
    check(
      (list: List[Int]) => {
        val stack = new Stack[Int]
        for (element <- list) stack.push(element)
        stack.elements.toList == list
      }
    )
  }
}

And more...

If you have some other preference, ScalaTest is designed to be extended, so you can create your own testing traits that look and act exactly as you want. In short, ScalaTest is all about letting you work the way you like to work. You can mix traits together into whatever combination creates the testing tool you find the most fun and productive to use.

Getting started with ScalaTest

The current release of ScalaTest is version 0.9.5. Although it sports a pre-1.0 version number, ScalaTest is mature, fully featured, and thoroughly documented. Download and unzip scalatest-0.9.5.zip, and you'll get a scalatest-0.9.5 directory that contains all the good stuff.

You can also download it from the scala-tools.org Maven repository:

If you don't have it already, you'll also need Scala 2.7.3.final, which you can get from here:

Note: ScalaTest 0.9.5 was compiled under Scala version 2.7.3.final, but seems to work just fine under Scala 2.7.2.final. So if you don't want to upgrade yet to Scala 2.7.3.final, you can still give ScalaTest a try today.

For help, you can post to the scalatest-users mailing list (Google group) at:

Or, you can post to the ScalaTest Forum at:

Quick overview

ScalaTest is a testing API that also includes a Runner application. You can use the API to create tests and the application to run them. The ScalaTest API is in package org.scalatest and subpackages. The main concepts in this API are represented by these three types:

One significant difference between the xUnit frameworks and ScalaTest is that in ScalaTest, the framework does not execute tests directly. Instead, each suite of tests is responsible for executing itself. To ask a suite of tests to execute itself, the framework invokes its execute method. In fact, you can execute a suite of tests yourself in the Scala interpreter, like this: 

scala> (new StackSpec).execute()
A Stack
- should pop values in last-in-first-out order
- should be empty when created

The execute method invoked by the ScalaTest framework takes several arguments, including a Reporter to collect and present results. The no-arg execute method invoked in the above interpreter example turns around and invokes the other execute method, passing in default values for all arguments, including a Reporter that prints to the standard output. The ability for each suite of tests to decide how to execute itself is key to ScalaTest's ability to facilitate different styles of testing.

If you are new to Scala, the easiest way to get started writing tests is to subclass org.scalatest.Suite (Suite) and define test methods. A test method is any public method with a name that starts with "test". A Suite can hold references to other, "nested" Suites. You organize a large suite of tests by building a tree of Suites. The base Suite in the tree has nested Suites, each of which may have nested Suites, and so on. When you execute the base Suite, it makes sure all Suites in the tree are executed.

For more detail, please check out the ScalaDoc documentation for Suite. This will give you the rundown on how to use a Suite. After that, take a look at the ScalaDoc documentation for Runner. This will explain the command line used to run tests. If you'd like to learn more about the traits shown in previous examples, you can find their documention here: Spec, ShouldMatchers, FunSuite, PrivateMethodTester, TestNGSuite, MustMatchers, JUnit3Suite, Checkers. If you want the big picture, you can also simply:

Trying it out

To try it out, you can use ScalaTest to run some of its own tests, i.e., tests used to test ScalaTest itself. This command will run and just print results to the standard output:

scala -classpath scalatest-0.9.5.jar org.scalatest.tools.Runner -p "scalatest-0.9.5-tests.jar" -o -s org.scalatest.SuiteSuite

This command will run the GUI:

scala -classpath scalatest-0.9.5.jar org.scalatest.tools.Runner -p "scalatest-0.9.5-tests.jar" -g -s org.scalatest.SuiteSuite

You should see something that looks like this:

ScalaTest screenshot

If you want to run the suites concurrently, add a -c parameter:

scala -classpath scalatest-0.9.5.jar org.scalatest.tools.Runner -p "scalatest-0.9.5-tests.jar" -g -c -s org.scalatest.SuiteSuite

ScalaTest has been tested with Scala version 2.7.3.final.

ScalaTest Ant task

ScalaTest provides an ant task to more conveniently run ScalaTest from Ant. To use it, you must first define the task in your ant file using taskdef: 

 <path id="scalatest.classpath">
   <pathelement location="${lib}/scalatest.jar"/>
   <pathelement location="${lib}/scala-library-2.6.1-final.jar"/>
 </path>

 <target name="main" depends="dist">
   <taskdef name="scalatest" classname="org.scalatest.tools.ScalaTestTask">
     <classpath refid="scalatest.classpath"/>
   </taskdef>

   <scalatest ...
 </target>

You can specify user-defined properties using nested <property> elements: 

  <scalatest>
    <property name="dbname" value="testdb"/>
    <property name="server" value="192.168.1.188"/>

You can specify a runpath using either a runpath attribute and/or nested <runpath> elements, using standard ant path notation: 

  <scalatest runpath="serviceuitest-1.1beta4.jar:myjini">
or 
  <scalatest>
    <runpath>
      <pathelement location="serviceuitest-1.1beta4.jar"/>
      <pathelement location="myjini"/>
    </runpath>

To add a URL to your runpath, use a <runpathurl> element (since ant paths don't support URLs): 

  <scalatest>
    <runpathurl url="http://foo.com/bar.jar"/>

You can specify reporters using nested <reporter> elements, where the type attribute must be one of the following:

Each may include a config attribute to specify the reporter configuration. Types file and reporterclass require additional attributes filename and classname, respectively: 

  <scalatest>
    <reporter type="stdout"        config="FAB"/>
    <reporter type="file"          filename="test.out"/>
    <reporter type="reporterclass" classname="my.ReporterClass"/>

You can specify group includes and excludes using <includes> and <excludes> elements: 

  <scalatest>
    <includes>
        CheckinTests
        FunctionalTests
    </includes>

    <excludes>
        SlowTests
        NetworkTests
    </excludes>

You can specify suites using either a suite attribute or nested <suite> elements: 

  <scalatest suite="com.artima.serviceuitest.ServiceUITestkit">

or 

  <scalatest>
    <suite classname="com.artima.serviceuitest.ServiceUITestkit"/>

To specify suites using members-only or wildcard package names, use either the membersonly or wildcard attributes, or nested <membersonly> or <wildcard> elements: 

  <scalatest membersonly="com.artima.serviceuitest">

or 

  <scalatest wildcard="com.artima.joker">

or 

  <scalatest>
    <membersonly package="com.artima.serviceuitest"/>
    <wildcard package="com.artima.joker"/>

Changes in 0.9.5

Changes in 0.9.4

Changes in 0.9.3

Changes in 0.9.2

Changes in 0.9.1

Known issues

  1. Reloading classes from JAR files between runs in the GUI doesn't work, because URLClassLoader uses a JAR cache. For now, you'll have to restart the app to pick up new classes in JAR files. But a better alternative is to simply point ScalaTest to the build directory containing .class files produced by the Scala and Java compiler. URLClassLoader will not cache these files, so changes to them will be picked up each time you press the Run or Rerun buttons.

About ScalaTest

ScalaTest was written by Bill Venners, Josh Cough, and George Berger, starting in late 2007. ScalaTest, which is almost exclusively written in Scala, follows and improves upon the Java code and design of SuiteRunner, a testing tool also written primarily by Bill Venners, starting in 2001. Over the years a few other people have contributed to SuiteRunner as well, including:

You can get the source code from SVN here:

https://scalatest.dev.java.net/source/browse/scalatest/

The command, if you don't have a java.net account, is: 

svn checkout https://scalatest.dev.java.net/svn/scalatest/trunk/app scalatest --username guest

The password for guest is empty, so just hit return if prompted for a password. The command: 

$ ant test

will build and run most of the tests, which requires 1Gb of memory on my Mac. So I always source this script first: 

JAVA_OPTS="-Xmx1024M"
export JAVA_OPTS
ANT_OPTS="-Xmx256m -Dscala.home=$SCALA_HOME"
export ANT_OPTS

The reason I use fsc instead of scalac is that I could not find a way to get scalac to not run out of memory. The scalac ant task does not support forking, so fsc was a convenient way to get the compile running in a different VM from the build. The fsc ant task sometimes can have trouble finding the file that gets dropped indicating the port fsc is running on, which is why I define scala.home for ant before running it. If you successfully build, and want that gigabyte of memory back, just type: 

$ fsc -shutdown

Thanks, and enjoy.



Google
  Web Artima.com   
Copyright © 1996-2009 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us