The Artima Developer Community

ScalaTest
A tool for testing Scala and Java software
OSI Certified Open Source Software

ScalaTest is a free, open-source testing tool for Scala and Java programmers. It is written in Scala, and enables you to write tests in Scala to test either Scala or Java code. It is released under the Apache 2.0 open source license.

The current release of ScalaTest is version 0.9.2. It is functional (and object-oriented), so you can already use it to write and run tests in Scala. It is also quite thoroughly documented. Download and unzip scalatest-0.9.2.zip, and you'll get a scalatest-0.9.2 directory that contains all the good stuff.

If you don't have it already, you'll also need Scala 2.7.1.final, which you can get from here:

Quick overview

ScalaTest is a testing API that also includes a Runner application. You can use the API to create tests and the application to run them. The ScalaTest API is in package org.scalatest and subpackages. The main concepts in this API are represented by these three types:

ScalaTest provides a variety of traits that facilitate different styles of writing tests. The most straightforward way to create a test suite is to subclass org.scalatest.Suite (Suite) and define test methods. A test method is any public method with a name that starts with "test". A Suite can hold references to other, "nested" Suites. You organize a large suite of tests by building a tree of Suites. The base Suite in the tree has nested Suites, each of which may have nested Suites, and so on. When you execute the base Suite, it makes sure all Suites in the tree are executed.

For more detail, please check out the ScalaDoc documentation for Suite. This will give you the run down on how to use a Suite. After that, take a look at the ScalaDoc documentation for Runner. This will explain the command line used to run tests. If you want the big picture, you can also:

Trying it out

To try it out, you can use ScalaTest to run some of its own tests, i.e., tests used to test ScalaTest itself. This command will run and just print results to the standard output:

scala -classpath scalatest-0.9.2.jar org.scalatest.tools.Runner -p "scalatest-0.9.2-tests.jar" -o -s org.scalatest.SuiteSuite

This command will run the GUI:

scala -classpath scalatest-0.9.2.jar org.scalatest.tools.Runner -p "scalatest-0.9.2-tests.jar" -g -s org.scalatest.SuiteSuite

You should see something that looks like this:

ScalaTest screenshot

If you want to run the suites concurrently, add a -c parameter:

scala -classpath scalatest-0.9.2.jar org.scalatest.tools.Runner -p "scalatest-0.9.2-tests.jar" -g -c -s org.scalatest.SuiteSuite

ScalaTest has been tested with Scala version 2.7.1.final.

Tests as function values

ScalaTest also provides a family of traits, FunSuite and FunSuite1 through FunSuite9, which allow you to express tests as function values. The “Fun ”in FunSuite stands for functional. Here's an example FunSuite:
import org.scalatest.fun.FunSuite

class MySuite extends FunSuite {

  test("addition") {
    val sum = 1 + 1
    assert(sum === 2)
    assert(sum + 2 === 4)
  }

  test("subtraction") {
    val diff = 4 - 1
    assert(diff === 3)
    assert(diff - 2 === 1)
  }
}

test” is a method defined in FunSuite, which will be invoked by the primary constructor of MySuite. You specify the name of the test as a string between the parentheses, and the test code itself between curly braces. The test code is a function passed as a by-name parameter to test, which registers it for later execution. One benefit of FunSuite compared to Suite is you need not name all your tests starting with “test.” In addition, you can more easily give long names to your tests, because you need not encode them in camel case, as you must do with test methods.

FunSuite1 through FunSuite9 facilitate the sharing of mutable fixture objects among tests. If you need three mutable fixture objects, for example, you'd subclass FunSuite3. You will need to parameterize FunSuite3 with the types of each of the fixture objects. Most often you'll do this explicitly in your subclass's declaration. Here's an example:

class MySuite extends FunSuite3[Float, Int, Long] {

  testWithFixture("example test") {
    (f, i, x) => {
      // test code that uses the passed fixture objects...
    }
  }

  def withFixture(f: (Float, Int, Long) => Unit) {

     // Create the fixture objects (as in JUnit 3's setUp method)
     val f = 1.1f
val i = 7
val x = 21L // Pass the fixture objects to the test function f(f, i, x) // If need be, perform any cleanup (as in JUnit 3's tearDown method) } }

This is a contrived example, because normally you would only pass fixture objects in this manner if they are mutable. (The reason we used immutable objects in this example was just to make the type names fit more easily on the page.) If they aren't mutable, then the fixture objects can simply be referenced from instance variables and shared by all test methods that need them. Since they aren't mutable, they can't be "corrupted" by one test and rendered unusable by the next. For mutable fixture objects, though, this trait allows you to reinitialize and pass in a fresh set of fixture objects to each test function that needs them. It is essentially a functional alternative to the setUp and tearDown approach of JUnit and TestNG, which require reasssigning instance variables between tests.

JUnit integration

ScalaTest also provides trait JUnit3Suite, which allows you to write tests that that can be run with either JUnit 3 or ScalaTest. JUnit3Suite allows you to write JUnit 3 tests with ScalaTest's more concise assertion syntax as well as JUnit's assertions (assertEquals, etc.). You create tests in JUnit 3 fashion, by defining methods that start with test, and can create fixtures with methods named setUp and tearDown. For example:
import org.scalatest.junit.JUnit3Suite
import scala.collection.mutable.ListBuffer

class TwoSuite extends JUnit3Suite {

  var sb: StringBuilder = _
  var lb: ListBuffer[String] = _

  override def setUp() {
    sb = new StringBuilder("ScalaTest is ")
    lb = new ListBuffer[String]
  }

  def testEasy() {
    sb.append("easy!")
    assert(sb.toString === "ScalaTest is easy!")
    assert(lb.isEmpty)
    lb += "sweet"
  }

  def testFun() {
    sb.append("fun!")
    assert(sb.toString === "ScalaTest is fun!")
    assert(lb.isEmpty)
  }
}

In this version of ScalaTest, the JUnit integration is in its early stages. JUnit3Suite isn't yet complete, but can still be used for the most typical use cases. In a future release of ScalaTest, we will add support for groups, nested suites, and concurrent execution to JUnit3Suite and also provide a JUnit3WrapperSuite that will allow you to run existing JUnit 3 tests written in Java with ScalaTest. We also plan to provide a JUnit4Suite and JUnit4WrapperSuite that provides similar integration for JUnit 4.

To execute JUnitSuites with ScalaTest's Runner, you must include JUnit3's jar file on the class path or runpath. This version of JUnitSuite was tested with JUnit version 3.8.1.

TestNG integration

ScalaTest also provides trait TestNGSuite, which allows you to write tests that can be run with either TestNG or ScalaTest. TestNGSuite allows you to mark any method as a test using TestNG's @Test annotation, and supports all other TestNG annotations. Here's an example:

import org.scalatest.testng.TestNGSuite
import org.testng.annotations.Test
import org.testng.annotations.Configuration
import scala.collection.mutable.ListBuffer

class MySuite extends TestNGSuite {

  var sb: StringBuilder = _
  var lb: ListBuffer[String] = _

  @Configuration { val beforeTestMethod = true }
  def setUpFixture() {
    sb = new StringBuilder("ScalaTest is ")
    lb = new ListBuffer[String]
  }

  @Test { val invocationCount = 3 }
  def easyTest() {
    sb.append("easy!")
    assert(sb.toString === "ScalaTest is easy!")
    assert(lb.isEmpty)
    lb += "sweet"
  }

  @Test { val groups = Array("com.mycompany.groups.SlowTest") }
  def funTest() {
    sb.append("fun!")
    assert(sb.toString === "ScalaTest is fun!")
    assert(lb.isEmpty)
  }
}

In addition, ScalaTest provides TestNGWrapperSuite, which wraps existing TestNG test suites described by TestNG XML files. TestNGWrapperSuite allows existing TestNG tests written in Java to be run by ScalaTest. One way to use TestNGWrapperSuite is to extend it and specify a list of one or more names of TestNG XML suite file names to run. Here's an example:

class MyWrapperSuite extends TestNGWrapperSuite(
  List("oneTest.xml", "twoTest.xml", "redTest.xml", "blueTest.xml")
)

You can also specify TestNG XML files on Runner's command line with -t parameters. See the documentation for Runner for more information.

To execute TestNGSuites or TestNGWrapperSuites with ScalaTest's Runner, you must include TestNG's jar file on the class path or runpath. This version of TestNGSuite was tested with TestNG version 5.7.

ScalaCheck integration

ScalaTest provides trait CheckSuite, which contains several “check” methods that can be used to perform ScalaCheck property checks. A ScalaCheck property check will be reported as ScalaTest test failure if ScalaCheck finds a test case for which the property doesn't hold. To use ScalaCheck, you specify properties and, in some cases, generators that generate test data. You need not always create generators because ScalaCheck provides many default generators for you, which can be used in many situations. ScalaCheck will use the generators to generate test data and with that data run tests that check that the property holds. Property-based tests can, therefore, give you a lot more testing for a lot less code than asertion-based tests. Here's an example of using ScalaCheck from a ScalaTest suite:

import org.scalatest.scalacheck.CheckSuite
import org.scalacheck.Arbitrary._
import org.scalacheck.Prop._

class MySuite extends CheckSuite {
  def testConcat() {
    checkProperty(
      (a: List[Int], b: List[Int]) => {
        a.size + b.size == (a ::: b).size
      }
    )
  }
}

The checkProperty method, defined in CheckSuite, makes it easy to write property-based tests inside ScalaTest, JUnit, and TestNG test suites. This example specifies a property that List's ::: method should obey. ScalaCheck properties are expressed as function values that take as parameters the required test data, which will be generated by ScalaCheck. In this case, the test data is composed of lists of integer named a and b. Inside the body of the function, you see:

a.size + b.size == (a ::: b).size

The property in this case is a Boolean expression that will yield true if the size of the concatenated list is equal to the size of each individual list added together. With this small amount of code, ScalaCheck will generate possibly hundreds of values for a and b and test each one, looking for a value for which the property doesn't hold. If the property holds true for every value ScalaCheck tries, checkProperty returns normally. Otherwise, checkProperty will complete abruptly with an AssertionError that contains information including the values for a and b that caused the failure.

To execute CheckSuites with ScalaTest's Runner, you must include ScalaCheck's jar file on the class path or runpath. This version of CheckSuite was tested with ScalaCheck version 1.1.1.

ScalaTest Ant task

ScalaTest provides an ant task to more conveniently run ScalaTest from Ant. To use it, you must first define the task in your ant file using taskdef:

 <path id="scalatest.classpath">
   <pathelement location="${lib}/scalatest.jar"/>
   <pathelement location="${lib}/scala-library-2.6.1-final.jar"/>
 </path>

 <target name="main" depends="dist">
   <taskdef name="scalatest" classname="org.scalatest.tools.ScalaTestTask">
     <classpath refid="scalatest.classpath"/>
   </taskdef>

   <scalatest ...
 </target>

You can specify user-defined properties using nested <property> elements:

  <scalatest>
    <property name="dbname" value="testdb"/>
    <property name="server" value="192.168.1.188"/>

You can specify a runpath using either a runpath attribute and/or nested <runpath> elements, using standard ant path notation:

  <scalatest runpath="serviceuitest-1.1beta4.jar:myjini">
or
  <scalatest>
    <runpath>
      <pathelement location="serviceuitest-1.1beta4.jar"/>
      <pathelement location="myjini"/>
    </runpath>

To add a URL to your runpath, use a <runpathurl> element (since ant paths don't support URLs):

  <scalatest>
    <runpathurl url="http://foo.com/bar.jar"/>

You can specify reporters using nested <reporter> elements, where the type attribute must be one of the following:

Each may include a config attribute to specify the reporter configuration. Types file and reporterclass require additional attributes filename and classname, respectively:

  <scalatest>
    <reporter type="stdout"        config="FAB"/>
    <reporter type="file"          filename="test.out"/>
    <reporter type="reporterclass" classname="my.ReporterClass"/>

You can specify group includes and excludes using <includes> and <excludes> elements:

  <scalatest>
    <includes>
        CheckinTests
        FunctionalTests
    </includes>

    <excludes>
        SlowTests
        NetworkTests
    </excludes>

You can specify suites using either a suite attribute or nested <suite> elements:

  <scalatest suite="com.artima.serviceuitest.ServiceUITestkit">

or

  <scalatest>
    <suite classname="com.artima.serviceuitest.ServiceUITestkit"/>

To specify suites using members-only or wildcard package names, use either the membersonly or wildcard attributes, or nested <membersonly> or <wildcard> elements:

  <scalatest membersonly="com.artima.serviceuitest">

or

  <scalatest wildcard="com.artima.joker">

or

  <scalatest>
    <membersonly package="com.artima.serviceuitest"/>
    <wildcard package="com.artima.joker"/>

Changes in 0.9.2

Changes in 0.9.1

Still to do

ScalaTest 0.9.2 contains many features, but being feature-holics we'd like add a few more for the 1.0 release:

  1. Reloading classes from JAR files between runs in the GUI doesn't work, because URLClassLoader uses a JAR cache. For now, you'll have to restart the app to pick up new classes in JAR files. But a better alternative is to simply point ScalaTest to the build directory containing .class files produced by the Scala and Java compiler. URLClassLoader will not cache these files, so changes to them will be picked up each time you press the Run or Rerun buttons.
  2. Integration JUnit 4 and a wrapper suite for JUnit 3.

After the 1.0 release, we'd like to attempt an integration with Eric Torreborre's specs that would allow specs assertions to be easily used in ScalaTest suites, and we'd like to integrate with popular Java mocking frameworks.

About ScalaTest

ScalaTest was written by Bill Venners, Josh Cough, and George Berger, starting in late 2007. ScalaTest, which is almost exclusively written in Scala, follows and improves upon the Java code and design of SuiteRunner, a testing tool also written primarily by Bill Venners, starting in 2001. Over the years a few other people have contributed to SuiteRunner as well, including:

Our goal is to have ScalaTest released 1.0 by the first week of May, 2008, just in time for JavaOne. Please submit feedback in the ScalaTest Forum.

We're in the process of creating a java.net project for scalatest. You can now get the source code from SVN here:

https://scalatest.dev.java.net/source/browse/scalatest/

Thanks, and enjoy.





Google
  Web Artima.com   
Copyright © 1996-2007 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us