|
|
|
ScalaTest is a free, open-source testing tool for Scala and Java programmers. It is written in Scala, and enables you to write tests in Scala to test either Scala or Java code. It is released under the Apache 2.0 open source license.
Because different developers take different approaches to creating software, no single approach to testing is a good fit for everyone. In light of this reality, ScalaTest is designed to facilitate different styles of testing. ScalaTest provides several traits that you can mix together into whatever combination makes you feel the most productive.
For example, if you like the behavior-driven development style of testing, in which tests are viewed as
informal specifications of the code under test, you might mix trait Spec with trait
ShouldMatchers. Your tests would look something like:
class StackSpec extends Spec with ShouldMatchers {
describe("A Stack") {
it("should pop values in last-in-first-out order") {
val stack = new Stack[Int]
stack.push(1)
stack.push(2)
stack.pop() should equal (2)
stack.pop() should equal (1)
}
it("should be empty when created") {
val stack = new Stack[Int]
stack should be ('empty)
}
}
}
When you run this Spec its output will be structured as an easy-to-read informal specification
of the Stack it tests. ScalaTest can format the specification output in its GUI or in text. In
text, it would look like:
A Stack - should pop values in last-in-first-out order - should be empty when created
ScalaTest's ShouldMatchers provide an elegant DSL for expressing
assertions, which can make the code more readable and failure messages easier to understand.
Some other examples are:
map should (contain key ("one") and not contain value (10))
result should be >= (0)
book should have (title ("Programming in Scala"))
tempFile should not be a ('directory)
quotient should be (1.0 plusOrMinus 0.2)
As elegant and readable as the matcher expressions may seem to some, you may find them too verbose for your taste. You may
also prefer to think of tests as, well, tests rather than as specifications, and you also may find you
often need to test private methods. If this describes you, you might mix trait FunSuite
with trait PrivateMethodTester, yielding tests that look like:
class StackSuite extends FunSuite with PrivateMethodTester {
test("Stack should pop values in last-in-first-out order") {
val stack = new Stack[Int]
stack.push(1)
stack.push(2)
assert(stack.pop() === (2))
assert(stack.pop() === (1))
}
test("Stack should throw NoSuchElementException if an empty stack is popped") {
val emptyStack = new Stack[String]
intercept[NoSuchElementException] {
emptyStack.pop()
}
}
test("Stack's private decorate method should place quotes around strings") {
val stack = new Stack[String]
val decorate = PrivateMethod[String]('decorate)
val result = stack invokePrivate decorate("hello")
assert(result === "\"hello\"")
}
}
Or perhaps you've been using TestNG in Java, and you want to continue using it in Scala. You like the matchers DSL,
but prefer the word "must" to "should". If this sounds like you, you can mix trait TestNGSuite with
trait MustMatchers, yielding TestNG tests (i.e., TestNG will be happy to run them, as
will ScalaTest) that look like this:
class StackSuite extends TestNGSuite with MustMatchers {
@Test def mustPopInLifoOrder() {
val stack = new Stack[Int]
stack.push(1)
stack.push(2)
stack.pop() must equal (2)
stack.pop() must equal (1)
}
@Test def mustBeEmptyWhenCreated() {
val stack = new Stack[Int]
stack must be ('empty)
}
}
ScalaTest also provides traits that support writing JUnit tests in Java, combining the benefits of
using the established defacto standard testing tool for the Java community with the more concise,
readable syntax of ScalaTest assertions and matchers. If you want to write JUnit tests in Scala, but
also really like the bang for the buck you get by writing ScalaCheck property checks, you might
mix trait JUnit3Suite with trait Checkers. (Trait Checkers can
be mixed into any testing trait, including the Spec, FunSuite, and
TestNGSuite traits shown previously, to include ScalaCheck property checks among
traditional assertions.) This would give you tests
that look like:
class StackSuite extends JUnit3Suite with Checkers {
def testPopsInLifoOrder() {
val stack = new Stack[Int]
stack.push(1)
stack.push(2)
assert(stack.pop() === (2))
assert(stack.pop() === (1))
}
def testIsEmptyWhenCreated() {
val stack = new Stack[Int]
assert(stack.isEmpty)
}
def testElementsHasAllObjectsPushedInOrder() { // uses a ScalaCheck property check
check(
(list: List[Int]) => {
val stack = new Stack[Int]
for (element <- list) stack.push(element)
stack.elements.toList == list
}
)
}
}
The current release of ScalaTest is version 0.9.5. Although it sports a pre-1.0 version number, ScalaTest is mature, fully featured, and thoroughly documented. Download and unzip scalatest-0.9.5.zip, and you'll get a scalatest-0.9.5 directory that contains all the good stuff.
You can also download it from the scala-tools.org Maven repository:
If you don't have it already, you'll also need Scala 2.7.3.final, which you can get from here:
Note: ScalaTest 0.9.5 was compiled under Scala version 2.7.3.final, but seems to work just fine under Scala 2.7.2.final. So if you don't want to upgrade yet to Scala 2.7.3.final, you can still give ScalaTest a try today.
For help, you can post to the scalatest-users mailing list (Google group) at:
Or, you can post to the ScalaTest Forum at:
ScalaTest is a testing API that also includes a Runner application. You can use the API to create tests and the application
to run them. The ScalaTest API is in package org.scalatest and subpackages.
The main concepts in this API are represented by these three types:
org.scalatest.Suite - a trait whose instances represent test suites (one to many tests)org.scalatest.Reporter - a trait implemented by objects that collect test results and present them to the userorg.scalatest.tools.Runner - a Scala application that runs test suites
One significant difference between the xUnit frameworks and ScalaTest is that in ScalaTest, the framework does not
execute tests directly. Instead, each suite of tests is responsible for executing itself. To ask a suite of tests
to execute itself, the framework invokes its execute method. In fact, you can execute a suite of tests
yourself in the Scala interpreter, like this:
scala> (new StackSpec).execute() A Stack - should pop values in last-in-first-out order - should be empty when created
The execute method invoked by the ScalaTest framework takes several arguments, including a Reporter
to collect and present results. The no-arg execute method invoked in the above
interpreter example turns around and invokes the other execute method, passing in default values for all
arguments, including a Reporter that prints to the standard output. The ability for each suite of
tests to decide how to execute itself is key to ScalaTest's ability to facilitate different styles of testing.
If you are new to Scala, the easiest way to get started writing tests is to subclass org.scalatest.Suite (Suite) and define test methods.
A test method is any public method with a name that starts with "test".
A Suite can hold references to other, "nested" Suites.
You organize a large suite of tests by building a tree of Suites. The base Suite in the
tree has nested Suites, each of which may have
nested Suites, and so on. When you execute the base Suite, it makes sure all Suites
in the tree are executed.
For more detail, please check out the ScalaDoc documentation for Suite. This will
give you the rundown on how to use a Suite. After that, take a look at the ScalaDoc documentation for
Runner. This will explain the command line used to run tests.
If you'd like to learn more about the traits shown in previous examples, you can find their documention here:
Spec,
ShouldMatchers,
FunSuite,
PrivateMethodTester,
TestNGSuite,
MustMatchers,
JUnit3Suite,
Checkers.
If you want the big picture, you can also simply:
To try it out, you can use ScalaTest to run some of its own tests, i.e., tests used to test ScalaTest itself. This command will run and just print results to the standard output:
scala -classpath scalatest-0.9.5.jar org.scalatest.tools.Runner -p "scalatest-0.9.5-tests.jar" -o -s org.scalatest.SuiteSuite
This command will run the GUI:
scala -classpath scalatest-0.9.5.jar org.scalatest.tools.Runner -p "scalatest-0.9.5-tests.jar" -g -s org.scalatest.SuiteSuite
You should see something that looks like this:
ScalaTest screenshot
If you want to run the suites concurrently, add a -c parameter:
scala -classpath scalatest-0.9.5.jar org.scalatest.tools.Runner -p "scalatest-0.9.5-tests.jar" -g -c -s org.scalatest.SuiteSuite
ScalaTest has been tested with Scala version 2.7.3.final.
ScalaTest provides an ant task to more conveniently run ScalaTest from Ant.
To use it, you must first define the task in your ant file using taskdef:
<path id="scalatest.classpath">
<pathelement location="${lib}/scalatest.jar"/>
<pathelement location="${lib}/scala-library-2.6.1-final.jar"/>
</path>
<target name="main" depends="dist">
<taskdef name="scalatest" classname="org.scalatest.tools.ScalaTestTask">
<classpath refid="scalatest.classpath"/>
</taskdef>
<scalatest ...
</target>
You can specify user-defined properties using nested <property> elements:
<scalatest>
<property name="dbname" value="testdb"/>
<property name="server" value="192.168.1.188"/>
You can specify a runpath using either a runpath attribute and/or nested
<runpath> elements, using standard ant path notation:
<scalatest runpath="serviceuitest-1.1beta4.jar:myjini">or
<scalatest>
<runpath>
<pathelement location="serviceuitest-1.1beta4.jar"/>
<pathelement location="myjini"/>
</runpath>
To add a URL to your runpath, use a <runpathurl> element
(since ant paths don't support URLs):
<scalatest>
<runpathurl url="http://foo.com/bar.jar"/>
You can specify reporters using nested <reporter> elements, where the type
attribute must be one of the following:
graphic file stdout stderr reporterclass
Each may include a config attribute to specify the reporter configuration.
Types file and reporterclass require additional attributes filename
and classname, respectively:
<scalatest>
<reporter type="stdout" config="FAB"/>
<reporter type="file" filename="test.out"/>
<reporter type="reporterclass" classname="my.ReporterClass"/>
You can specify group includes and excludes using <includes> and
<excludes> elements:
<scalatest>
<includes>
CheckinTests
FunctionalTests
</includes>
<excludes>
SlowTests
NetworkTests
</excludes>
You can specify suites using either a suite attribute or nested
<suite> elements:
<scalatest suite="com.artima.serviceuitest.ServiceUITestkit">
or
<scalatest>
<suite classname="com.artima.serviceuitest.ServiceUITestkit"/>
To specify suites using members-only or wildcard package names, use
either the membersonly or wildcard attributes, or nested
<membersonly> or <wildcard> elements:
<scalatest membersonly="com.artima.serviceuitest">
or
<scalatest wildcard="com.artima.joker">
or
<scalatest>
<membersonly package="com.artima.serviceuitest"/>
<wildcard package="com.artima.joker"/>
ShouldMatchers and MustMatchers traits, plus many supporting classes
and traits.SpecReport in the GUI, so that SpecReports get displayed in
a specification style. This makes org.scalatest.Spec output look nicer in the GUI,
and also makes it possible for org.specs.Specification to send SpecReports that will
look nice in the ScalaTest GUI.TestFailedException, which allows a stack trace depth to be sent along with
test failures. This allows reports to highlight the exactly filename and line number at which a test
failed. This makes it quicker for users to find the cause of the problem. Enhanced everything to
send TestFailedExceptions where possible.TestFailedException,
which was not being propagated to the reporters before.FunSuite and Spec were not sending Rerunnables in
reports even though they could, which made rerunning not work in some cases in the GUI. They now send
a Rerunnable whenever they can.SpecDasher from the API, which also is no longer mixed in by Spec.
The only reason this was included in 0.9.4 was that it was demonstrated in Programming in
Scala. It still works and can be used, so if you liked it, you can just grab it from
0.9.4 and mix it in yourself as a local mix in.ImpSuite to BeforeAndAfter. ImpSuite remains for now but is marked as deprecated. In
a future release of ScalaTest ImpSuite will be dropped, so please rename your uses of ImpSuite to BeforeAndAfter
as soon as convenient. The reason this was changed is that the "Imp" stood for imperative, because the trait facilitates an imperative style
of testing. Also "imp" in English means "little devil," which projected a good attitude, because one goal of ScalaTest is to encourage users to adopt
a more functional style for fixture "setup" and "teardown." But it's a bit too cute and its meaning not obvious enough from its name. Plus mixing
an ImpSuite into a Spec felt like mixing metaphors.expect and intercept constructs from Suite
into their own trait, named Assertions. Suite now mixes in this trait, so that it has the same functionality as
before. The main reason this was done was so that these constructs could more easily be used outside of ScalaTest Suites, and also
so the trait could be extended by ScalaTest's matchers. (ScalaTest's matchers are not yet released, but will probably make their debut in
ScalaTest version 0.9.5.) This allows ScalaTest's matchers to assume these constructs exist, and makes it easier to use ScalaTest matchers
independently from ScalaTest Suites.intercept that takes an implicit Manifest, so that you need not explicitly pass
a class instance anymore. Thus, instead of saying this:
intercept(classOf[StringIndexOutOfBoundsException]) {
"hi".charAt(-1)
}
You can write:
intercept[StringIndexOutOfBoundsException] {
"hi".charAt(-1)
}
The old style that requires an explicit class instance are still in the API, but they are deprecated. They will be removed in some future
release of ScalaTest, so please start migrating to the new style. There are a few corner cases where the new form can't be used, which
are caused by overloading ambiguities with the old style. The two corners are passing an explicit null to intercept and throwing
an explicit exception such that the inferred type of the by-name parameter is Nothing. It is unlikely anyone would do that
anyway, and these two issues will resolve themselves once the deprecated old-style intercept methods are removed.
Spec. This trait is inspired by Ruby's RSpec BDD tool.PropSuite to FunSuite in org.scalatest.prop. This was done because it made sense
to add the extra ScalaCheck methods to Spec as well, so it seemed
more reasonable to just have traits with the same name in two different packages. So there would be a FunSuite and Spec
in both org.scalatest and org.scalatest.prop. The latter batch would already mix
in Checkers and offer the extra ScalaCheck property-taking methods, and of course, have a dependency on ScalaCheck. Thus,
PropSuite was deprecated, but it is left in the API for the time being to avoid client code breakage. PropSuite
will go away in a future release. Later, the traits that had the extra property-taking methods were judged not worth their weight, so
ScalaTest 0.9.4 does not include a Spec in package org.scalatest.prop.
However because an example using org.scalatest.prop.FunSuite was included in Programming in Scala, it was included
in 0.9.4, already deprecated. Both PropSuite and FunSuite will be removed from the org.scalatest.prop
package in a future ScalaTest release, so please start migrating to use org.scalatest.FunSuite, mixing in Checkers
and passing properties to ScalaCheck with check, as shown in this example.SpecDasher trait. This may or may not be a good idea, but
an example of the "dashes" style it enables was shown in Programming in Scala, so it was included in 0.9.4, but deprecated. It may stay or go in
a future release, depending on user feedback. Currently, Spec already mixes SpecDasher in by default, but if
it ends up staying in the API, this connection will almost certainly be severed so that users must explicitly mix it in to get the "dashes" style.
So use it with caution, but even if it is removed, if you like it you can always just add the code for it to your own project and
use it that way.SpecReport, a subclass of Report. Specs emit SpecReports when they run, and
the print reporters (standard out, err, file, etc.) recognize them and generate specification-like output. In this release the graphic
reporter does not handle SpecReport's specially, so the output looks like regular test output. In a future release the graphic
reporter will generate a specification-like output. One use case for SpecReports is integration with Eric Torreborre's specs
tool, however one piece still remains for that integration. Specs (and JUnit and TestNG) generate nested test output, whereas ScalaTest's
model is flat. In a future version ScalaTest's Report class will be enhanced to support nested reports. This will enable
specs tests run through ScalaTest to provide nice nested, specification-like reports.ExecuteAndRun trait. The purpose of this trait is to pull out abstract method signatures for execute and
runTest (and later if it turns out to be useful, runNestedSuites and runTests) so they can be
called by traits that have Suite as their self type, but extend ExecuteAndRun not Suite, such
as BeforeAndAfter. This makes such traits stackable behaviors that can be mixed into a Suite while not
being a Suite themselves.testWithInformer and ignoreWithInformer methods from FunSuite, because they were
simply too ugly to live. Unfortunately these were not deprecated, so if you used them you'll need to already make an adjustment to upgrade
to 0.9.4. What replaced them is an imperative approach, which while more error prone will work fine unless abused, and makes
for much nicer client code. There's a new method named info in FunSuite, which returns an Informer.
So where before you said:
testWithInformer("test name") {
info => {
info("send info to the reporter")
}
}
Now you'll just say:
test("test name") {
info("send info to the reporter")
}
During suite execution, the info method returns an Informer that immediately forwards information passed to its apply
methods to the reporter. During construction of the object, the info method returns an Informer that saves the information
passed to its apply methods for later sending to the reporter when the suite is executed. At any other time, the info
method will return an Informer that throws an exception.
FunSuite from the org.scalatest.fun to org.scalatest package.FunSuite1 through FunSuite9, which were also in the org.scalatest.fun package, because the surface area
they added to the API didn't seem worth the small savings in code size over creating "with" methods with meaningful
names and explicitly calling them.ImpSuite, which can be mixed into another suite to gain methods that will run before and after each suite and test.Suite and FunSuite that discuss mutable fixtures. Dropped
the suggestion to override runTest, which still works but is verbose, and instead recommended defining and explicitly calling "create" methods or
"with" methods from tests that need the mutable fixtures, or mixing in ImpSuite and using beforeEach, afterEach, etc.Informer trait, which provides a simpler way to send information to the infoProvided
method of a Reporter.Suite such that instead of being able to define test methods that take a Reporter, you can define
test methods that take an Informer. Replaced the section of scaladoc for Suite that used to explain how to
use a passed-in Reporter to now show how to use an Informer.FunSuite such that instead of being able to register tests whose function values take a Reporter, you can register
tests whose function values take an Informer. Changed the names of the registration methods from test/ignoreWithReporter to
test/ignoreWithInformer. Replaced the section of scaladoc for FunSuite that used to explain how to
use a Reporter passed in to a test function to now show how to use an Informer.org.scalatest.scalacheck.CheckSuite to org.scalatest.prop.Checkers, and renamed its checkProperty
methods to plain old check.PropSuite, a subtrait of FunSuite, which offers methods that register tests composed of a single property check.assert(... === ...) and expect(...){...} methods. Strings are now reported surrounded
by double quotes, with ends truncated where matching if very long, and with square brackets surrounding the portions that differ. Characters are reported
surrounded by single quotes, integer and floating point numbers as is, and any other object's toString result surrounded by angle brackets.Suite's scaladoc that describes using external assertions, such as JUnit's assertions, Hamcrest
matchers, or specs matchers, in ScalaTest tests.Runner from org.scalatest to org.scalatest.tools package.org.scalatest.junit.org.scalatest.testng.org.scalatest.scalacheck.FunSuite family of traits in package org.scalatest.fun.testGroups method name to groups.===, from expected === actual to left === right.2 === 1, from "Expected 1, but got 2" to "2 did not equal 1".expect method name to intercept.expectNPE and expectIAE methods.expect method, which takes an expected and actual value. expect(1) {2} would generate
the error message "Expected 1, but got 2.expect and intercept that take a message.Suite discovery mechanism in Runner, accessed via -m or -w
command-line arguments.!== method from the great Equalizer.ScalaTest was written by Bill Venners, Josh Cough, and George Berger, starting in late 2007. ScalaTest, which is almost exclusively written in Scala, follows and improves upon the Java code and design of SuiteRunner, a testing tool also written primarily by Bill Venners, starting in 2001. Over the years a few other people have contributed to SuiteRunner as well, including:
You can get the source code from SVN here:
https://scalatest.dev.java.net/source/browse/scalatest/
The command, if you don't have a java.net account, is:
svn checkout https://scalatest.dev.java.net/svn/scalatest/trunk/app scalatest --username guest
The password for guest is empty, so just hit return if prompted for a password. The command:
$ ant test
will build and run most of the tests, which requires 1Gb of memory on my Mac. So I always source this script first:
JAVA_OPTS="-Xmx1024M" export JAVA_OPTS ANT_OPTS="-Xmx256m -Dscala.home=$SCALA_HOME" export ANT_OPTS
The reason I use fsc instead of scalac is that I could not find a way to get scalac
to not run out of memory. The scalac ant task does not support forking, so fsc was a convenient way
to get the compile running in a different VM from the build. The fsc ant task sometimes can have trouble finding
the file that gets dropped indicating the port fsc is running on, which is why I define scala.home
for ant before running it. If you successfully build, and want that gigabyte of memory back, just type:
$ fsc -shutdown
Thanks, and enjoy.