The Artima Developer Community
Sponsored Link

Weblogs Forum
Testing without setup and tearDown

7 replies on 1 page. Most recent reply: Jun 9, 2008 4:48 PM by Bill Venners

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 7 replies on 1 page
Bill Venners

Posts: 2242
Nickname: bv
Registered: Jan, 2002

Testing without setup and tearDown (View in Weblogs)
Posted: Jun 3, 2008 1:38 AM
Reply to this message Reply
Summary
In the most recent release of ScalaTest, I've placed some guidelines in the documentation for how to avoid the use of vars in testing code. In this post I include those guidelines and ask for feedback.
Advertisement

This weekend I released version 0.9.3 of ScalaTest, a testing tool for Java and Scala programmers. One of the issues that has arisen as I've been working on ScalaTest is what to do about setup and tearDown. These methods, which appeared on class TestCase of the original JUnit, run before and after each test. (JUnit 4 and TestNG use "before" and "after" annotations to serve the same purpose.) This allows you to reinitialize shared mutable objects referenced from instance variables, or other mutable "fixtures" such as files, databases, or sockets, before each test. If one test destroys a fixture, the next test will get a new fixture.

In ScalaTest, I pulled this technique out into a separate trait, which I called ImpSuite. The "Imp" in ImpSuite is short for "imperative," because this technique is definitely an imperative approach to solving the problem. If you want to take this approach in ScalaTest, it is as easy as mixing in trait ImpSuite and overriding methods beforeEach and/or afterEach. However, I wanted to recommend that users consider a few other approaches that avoid reassigning variables (vars) first, because the tradition in Scala programming is to minimize the use of vars.

I wanted to get feedback on these suggestions. I'm including the relevant section of the ScalaTest's Scaladoc next. Please take a look and submit feedback in the forum discussion for this post.

Test fixtures (from ScalaTest's documentation)

A test fixture is objects or other artifacts (such as files, sockets, database connections, etc.) used by tests to do their work. If a fixture is used by only one test method, then the definitions of the fixture objects should be local to the method, such as the objects assigned to sum and diff in the previous MySuite examples. If multiple methods need to share a fixture, the best approach is to assign them to instance variables. Here's a (very contrived) example, in which the object assigned to shared is used by multiple test methods:

 import org.scalatest.Suite

 class MySuite extends Suite {

   // Sharing fixture objects via instance variables
   private val shared = 5

   def testAddition() {
     val sum = 2 + 3
     assert(sum === shared)
   }

   def testSubtraction() {
     val diff = 7 - 2
     assert(diff === shared)
   }
 }
 

In some cases, however, shared mutable fixture objects may be changed by test methods such that it needs to be recreated or reinitialized before each test. Shared resources such as files or database connections may also need to be cleaned up after each test. JUnit offers methods setup and tearDown for this purpose. In ScalaTest, you can use ImpSuite, which will be described later, to implement an approach similar to JUnit's setup and tearDown, however, this approach often involves reassigning vars between tests. Before going that route, you should consider two approaches that avoid vars. One approach is to write one or more "create" methods that return a new instance of a needed object (or a tuple of new instances of multiple objects) each time it is called. You can then call a create method at the beginning of each test method that needs the fixture, storing the fixture object or objects in local variables. Here's an example:

 import org.scalatest.Suite
 import scala.collection.mutable.ListBuffer

 class MySuite extends Suite {

   // create objects needed by tests and return as a tuple
   private def createFixture = (
     new StringBuilder("ScalaTest is "),
     new ListBuffer[String]
   )

   def testEasy() {
     val (builder, lbuf) = createFixture
     builder.append("easy!")
     assert(builder.toString === "ScalaTest is easy!")
     assert(lbuf.isEmpty)
     lbuf += "sweet"
   }

   def testFun() {
     val (builder, lbuf) = createFixture
     builder.append("fun!")
     assert(builder.toString === "ScalaTest is fun!")
     assert(lbuf.isEmpty)
   }
 }
 

Another approach to mutable fixture objects that avoids vars is to create "with" methods, which take test code as a function that takes the fixture objects as parameters, and wrap test code in calls to the "with" method. Here's an example:

 import org.scalatest.Suite
 import scala.collection.mutable.ListBuffer

 class MySuite extends Suite {

   private def withFixture(testFunction: (StringBuilder, ListBuffer[String]) => Unit) {

     // Create needed mutable objects
     val sb = new StringBuilder("ScalaTest is ")
     val lb = new ListBuffer[String]

     // Invoke the test function, passing in the mutable objects
     testFunction(sb, lb)
   }

   def testEasy() {
     withFixture {
       (builder, lbuf) => {
         builder.append("easy!")
         assert(builder.toString === "ScalaTest is easy!")
         assert(lbuf.isEmpty)
         lbuf += "sweet"
       }
     }
   }

   def testFun() {
     withFixture {
       (builder, lbuf) => {
         builder.append("fun!")
         assert(builder.toString === "ScalaTest is fun!")
         assert(lbuf.isEmpty)
       }
     }
   }
 }
One advantage of this approach compared to the create method approach shown previously is that you can more easily perform cleanup after each test executes. For example, you could create a temporary file before each test, and delete it afterwords, by doing so before and after invoking the test function in a withTempFile method. Here's an example:
 import org.scalatest.Suite
 import java.io.FileReader
 import java.io.FileWriter
 import java.io.File

 class MySuite extends Suite {

   private def withTempFile(testFunction: FileReader => Unit) {

     val FileName = "TempFile.txt"

     // Set up the temp file needed by the test
     val writer = new FileWriter(FileName)
     try {
       writer.write("Hello, test!")
     }
     finally {
       writer.close()
     }

     // Create the reader needed by the test
     val reader = new FileReader(FileName)

     try {
       // Run the test using the temp file
       testFunction(reader)
     }
     finally {
       // Close and delete the temp file
       reader.close()
       val file = new File(FileName)
       file.delete()
     }
   }

   def testReadingFromTheTempFile() {
     withTempFile {
       (reader) => {
         var builder = new StringBuilder
         var c = reader.read()
         while (c != -1) {
           builder.append(c.toChar)
           c = reader.read()
         }
         assert(builder.toString === "Hello, test!")
       }
     }
   }

   def testFirstCharOfTheTempFile() {
     withTempFile {
       (reader) => {
         assert(reader.read() === 'H')
       }
     }
   }
 }
 

If you are more comfortable with reassigning instance variables, however, you can instead use ImpSuite, a subtrait of Suite that provides methods that will be run before and after each test. ImpSuite's beforeEach method will be run before, and its afterEach method after, each test (like JUnit's setup and tearDown methods, respectively). For example, here's how you'd write the previous test that uses a temp file with an ImpSuite:

 import org.scalatest.ImpSuite
 import java.io.FileReader
 import java.io.FileWriter
 import java.io.File

 class MySuite extends ImpSuite {

   private val FileName = "TempFile.txt"
   private var reader: FileReader = _

   // Set up the temp file needed by the test
   override def beforeEach() {
     val writer = new FileWriter(FileName)
     try {
       writer.write("Hello, test!")
     }
     finally {
       writer.close()
     }

     // Create the reader needed by the test
     reader = new FileReader(FileName)
   }

   // Close and delete the temp file
   override def afterEach() {
     reader.close()
     val file = new File(FileName)
     file.delete()
   }

   def testReadingFromTheTempFile() {
     var builder = new StringBuilder
     var c = reader.read()
     while (c != -1) {
       builder.append(c.toChar)
       c = reader.read()
     }
     assert(builder.toString === "Hello, test!")
   }

   def testFirstCharOfTheTempFile() {
     assert(reader.read() === 'H')
   }
 }
 

In this example, the instance variable reader is a var, so it can be reinitialized between tests by the beforeEach method. If you want to execute code before and after all tests (and nested suites) in a suite, such as you could do with @BeforeClass and @AfterClass annotations in JUnit 4, you can use the beforeAll and afterAll methods of ImpSuite.

See the documentation for ScalaTest for more information.

What's your opinion?

Would you want to write tests this way? Would you stick with before and after? Do you see any benefit in trying to avoid vars like this?


Daniel Jimenez

Posts: 40
Nickname: djimenez
Registered: Dec, 2004

Re: Testing without setup and tearDown Posted: Jun 5, 2008 11:21 AM
Reply to this message Reply
Being very familiar with setUp/tearDown usage (and I assume it's mostly syntax that has changed with @Before/@After, not usage), I find it very frustrating to duplicate fixture code in my test methods.

Both the createFixture and withXxx alternatives force identical calls to those structures in each test method, something I consistently try to avoid by putting that code into setUp/tearDown.

I think most - certainly all of my - usage of variable assignment in setUp has a connotation of final. Additionally, the quirk of JUnit that led Cedric Beust to create TestNG (a new TestCase instance for each test method called) means that those variables are literally only assigned once (and could therefore be marked final, if getting a compiler to prove that specific situation was even remotely tractable).

That said, the withXxx alternative seems like it could be pushed into declarative syntax somehow, maybe in a way similar to TestNG's @Test[dataProvider] (which allows each test method to have its data passed to it). Any thoughts along those lines?

Bill Venners

Posts: 2242
Nickname: bv
Registered: Jan, 2002

Re: Testing without setup and tearDown Posted: Jun 5, 2008 2:25 PM
Reply to this message Reply
> Being very familiar with setUp/tearDown usage (and I
> assume it's mostly syntax that has changed with
> @Before/@After, not usage), I find it very frustrating to
> duplicate fixture code in my test methods.
>
> Both the createFixture and withXxx alternatives force
> identical calls to those structures in each test method,
> something I consistently try to avoid by putting that code
> into setUp/tearDown.
>
> I think most - certainly all of my - usage of variable
> assignment in setUp has a connotation of final.
> Additionally, the quirk of JUnit that led Cedric Beust to
> create TestNG (a new TestCase instance for each test
> method called) means that those variables are literally
> only assigned once (and could therefore be marked final,
> if getting a compiler to prove that specific situation was
> even remotely tractable).
>
> That said, the withXxx alternative seems like it could be
> pushed into declarative syntax somehow, maybe in a way
> similar to TestNG's @Test[dataProvider] (which allows each
> test method to have its data passed to it). Any thoughts
> along those lines?
>
I actually did that in 0.9.2, but took it out in 0.9.3. I could put it back. I took it out because it seemed to add a lot of surface area to the API for little gain over just doing "with" methods like I've shown here. Here's an example from the 0.9.2. scaladoc:


import org.scalatest.fun.FunSuite1

class EasySuite extends FunSuite1[StringBuilder] {

testWithFixture("easy test") {
sb => {
sb.append("easy!")
assert(sb.toString === "Testing is easy!")
}
}

testWithFixture("fun test") {
sb => {
sb.append("fun!")
assert(sb.toString === "Testing is fun!")
}
}

def withFixture(f: StringBuilder => Unit) {
val sb = new StringBuilder("Testing is ")
f(sb)
}
}


The 0.9.2. scaladoc is here:

http://www.artima.com/scalatest/doc-0.9.2/

What you'll see in the left-hand column is FunSuite1 through FunSuite9. I had one for each arity because I need the types and the number of types so that I can all the withFixture method internally to execute a test. So all those traits is the surface area. The only downside of that is you have to wade through it all those FunSuiteN traits to find the class you want when you're looking in the lower-left list. Not really a big deal, but... The real problem was that in 0.9.3 I added a PropSuite, which is a subtrait of FunSuite. So to continue along that path, I'd need a PropSuite1 through PropSuite9. I'm working now on adding some other Suite subtypes, so I'd need 9 of each of those too if I'm to be consistent. It is easy for me to do (with code generation), but it clutters up the API.

If you count the number of chars you save, it isn't that much. I also added ImpSuite in 0.9.3, so that's the alternative if you prefer to be more concise. But I hate to drive everyone to ImpSuite if what they're really after is more conciseness.

(By the way, I will get rid of the "WithInformer/WithReporter" stuff in FunSuite in the next release, so the only two methods in FunSuiteN would be test and testWithFixture. I'm going to ask Martin if there's a reason the type system couldn't be enhanced to tell the difference between a function literal that takes params and one that doesn't. It can't right now, which is why I currently would need testWithFixture. Maybe there's a good reason, but if not, perhaps eventually you could just always say "test", like this:


import org.scalatest.fun.FunSuite1

class EasySuite extends FunSuite1[StringBuilder] {

test("easy test") {
sb => {
sb.append("easy!")
assert(sb.toString === "Testing is easy!")
}
}

test("fun test") {
sb => {
sb.append("fun!")
assert(sb.toString === "Testing is fun!")
}
}

def withFixture(f: StringBuilder => Unit) {
val sb = new StringBuilder("Testing is ")
f(sb)
}
}


Actually, even if not, I could just say that every test method in a FunSuiteN takes N fixture parameters, and leave it at that. Then you could have the above syntax.

My question to you is, would you actually use such an approach over the traditional beforeEach and afterEach approach of ImpSuite?

Daniel Jimenez

Posts: 40
Nickname: djimenez
Registered: Dec, 2004

Re: Testing without setup and tearDown Posted: Jun 6, 2008 11:35 AM
Reply to this message Reply
> > That said, the withXxx alternative seems like it could
> be
> > pushed into declarative syntax somehow, maybe in a way
> > similar to TestNG's @Test[dataProvider] (which allows
> each
> > test method to have its data passed to it). Any
> thoughts
> > along those lines?
> >
> I actually did that in 0.9.2, but took it out in 0.9.3. I
> could put it back. I took it out because it seemed to add
> a lot of surface area to the API for little gain over just
> doing "with" methods like I've shown here. Here's an
> example from the 0.9.2. scaladoc:
>
> /snip example/
>
> The 0.9.2. scaladoc is here:
>
> http://www.artima.com/scalatest/doc-0.9.2/
>
> What you'll see in the left-hand column is FunSuite1
> through FunSuite9. I had one for each arity because I need
> the types and the number of types so that I can all the
> withFixture method internally to execute a test. So all
> those traits is the surface area. The only downside of
> that is you have to wade through it all those FunSuiteN
> traits to find the class you want when you're looking in
> the lower-left list. Not really a big deal, but... The
> real problem was that in 0.9.3 I added a PropSuite, which
> is a subtrait of FunSuite. So to continue along that path,
> I'd need a PropSuite1 through PropSuite9. I'm working now
> on adding some other Suite subtypes, so I'd need 9 of each
> of those too if I'm to be consistent. It is easy for me to
> do (with code generation), but it clutters up the API.
>
> If you count the number of chars you save, it isn't that
> much. I also added ImpSuite in 0.9.3, so that's the
> alternative if you prefer to be more concise. But I hate
> to drive everyone to ImpSuite if what they're really after
> is more conciseness.
>
Your reasoning here seems right to me - the added surface area isn't nearly made up for by the small increase in concision. The difference between the 0.9.3 and 0.9.2 examples is just the call to withTempFile, which serves exactly the same purpose as the withFixture override in FunSuite1-9: that truly is not much more by way of typing.

This also allows for multiple fixtures per test class; FunSuite1-9 defined the fixture type parameters directly on the class. Since setUp, the old way, creates an implicit fixture of variables, I'm not actually sure if I've needed this extra "feature." Interesting.

> /snip again/
>
> My question to you is, would you actually use such an
> approach over the traditional beforeEach and afterEach
> approach of ImpSuite?

This is, of course, the heart of the question. It seems to me that the purpose of FunSuite - the reason ImpSuite isn't just the default, like JUnit/TestNG - is the compiler checking of the final-(val-)ness of the fixture parameters. (A review of your comments on the post about the release of 0.9.1 indicates I'm in the right ballpark here: "I'd rather the framework lead people in the direction of using vals everywhere.")

Typically, most of my production Java code is dependency-inverted: constructors take "interfaces", store them in final variables, test fixtures provide stubs, and test methods ensure the proper methods are called on those interfaces. Most mutable state ends up in bean classes that are usually arguments to the production methods. In the tests, the stubs tend to be mutable, with boolean methodCalled variables, sometimes lists of arguments received, etc, so I'm used to recreating them afresh for each test method invocation.

In Scala, I don't think that's how I should code. Well, the dependency-inverted-ness seems alright, but the two places with mutable data seem out of place in a language so capably functional. Perhaps I should worry less about typing concision and consider more carefully exactly how my tests would be structured in this "new" paradigm.

I'm convinced: In answer to your question, I certainly prefer FunSuite to ImpSuite in Scala, and certainly would fight to overcome the draw of imperative familiarity to structure my tests more in line with the zeitgeist of the language (is that Scala-ic, a la Pythonic?).

You described your path to Scala-ic code in your post on the first release of ScalaTest, replacing SuiteRunner's mutable state. How did the tests you wrote around SuiteRunner/ScalaTest change as a result of your decreasing reliance on mutable state?

PS - the documentation on ScalaTest is excellent. You obviously put a lot of work into it and it shows.

Bill Venners

Posts: 2242
Nickname: bv
Registered: Jan, 2002

Re: Testing without setup and tearDown Posted: Jun 6, 2008 1:03 PM
Reply to this message Reply
> > If you count the number of chars you save, it isn't
> that
> > much. I also added ImpSuite in 0.9.3, so that's the
> > alternative if you prefer to be more concise. But I
> hate
> > to drive everyone to ImpSuite if what they're really
> after
> > is more conciseness.
> >
> Your reasoning here seems right to me - the added surface
> area isn't nearly made up for by the small increase in
> concision. The difference between the 0.9.3 and 0.9.2
> examples is just the call to withTempFile, which serves
> exactly the same purpose as the withFixture override in
> FunSuite1-9: that truly is not much more by way of
> typing.
>
> This also allows for multiple fixtures per test class;
> FunSuite1-9 defined the fixture type parameters directly
> on the class. Since setUp, the old way, creates an
> implicit fixture of variables, I'm not actually sure if
> I've needed this extra "feature." Interesting.
>
Yes, you can have multiple fixtures per class this way. I puzzled long ago over why JUnit called TestCase a TestCase, and one of my theories was that a TestCase collected together tests that share a single kind of fixture.

I have already found a use case for having multiple fixtures in a single Suite of tests. I have a suite for stacks that sometimes wants an empty stack, sometimes a full stack, sometimes an almost empty stack, sometimes an almost full stack. It seems natural to have all these tests in the same suite. In the ImpSuite approach, you'd probably just reinitialize each of these four things and only use some of them in each test.

> > /snip again/
> >
> > My question to you is, would you actually use such an
> > approach over the traditional beforeEach and afterEach
> > approach of ImpSuite?
>
> This is, of course, the heart of the question. It seems to
> me that the purpose of FunSuite - the reason ImpSuite
> isn't just the default, like JUnit/TestNG - is the
> compiler checking of the final-(val-)ness of the fixture
> parameters. (A review of your comments on the post about
> the release of 0.9.1 indicates I'm in the right ballpark
> here: "I'd rather the framework lead people in the
> direction of using vals everywhere.")
>
The main reason I didn't make ImpSuite the default is that there's are good alternatives (create methods and with methods) that feel more natural in Scala. In addition, you can mix ImpSuite into any Suite that uses runTest and it will work. So because it is a trait people can mix it into other Suites they may develop. It takes advantage of the stackable behavior traits offer to make it easier for people to create their own Suite types.

> Typically, most of my production Java code is
> dependency-inverted: constructors take "interfaces", store
> them in final variables, test fixtures provide stubs, and
> test methods ensure the proper methods are called on those
> interfaces. Most mutable state ends up in bean classes
> that are usually arguments to the production methods. In
> the tests, the stubs tend to be mutable, with boolean
> methodCalled variables, sometimes lists of arguments
> received, etc, so I'm used to recreating them afresh for
> each test method invocation.
>
> In Scala, I don't think that's how I should code. Well,
> the dependency-inverted-ness seems alright, but the two
> places with mutable data seem out of place in a language
> so capably functional. Perhaps I should worry less about
> typing concision and consider more carefully exactly how
> my tests would be structured in this "new" paradigm.
>
I use a lot of mutable stub objects still. It is really concise to write them in Scala by hand, without needing a mock API. Here's an example:

class MySuite extends Suite {
var theTestThisCalled = false
var theTestThatCalled = false
def testThis() { theTestThisCalled = true }
def testThat() { theTestThatCalled = true }
}


> I'm convinced: In answer to your question, I certainly
> prefer FunSuite to ImpSuite in Scala, and certainly would
> fight to overcome the draw of imperative familiarity to
> structure my tests more in line with the zeitgeist of the
> language (is that Scala-ic, a la Pythonic?).
>
Scala-esque? Doesn't work as well as Pythonic.

> You described your path to Scala-ic code in your post on
> the first release of ScalaTest, replacing SuiteRunner's
> mutable state. How did the tests you wrote around
> SuiteRunner/ScalaTest change as a result of your
> decreasing reliance on mutable state?
>
I'll think about this and post later.

> PS - the documentation on ScalaTest is excellent. You
> obviously put a lot of work into it and it shows.
>
Thanks. I want to try and set a good example because I think documentation is important. The Scala API's own docs are quite sparse in places. But the downside is that my insistence on waiting until the documentation is done has delayed releases by quite a bit, so there is a tradeoff.

Bill Venners

Posts: 2242
Nickname: bv
Registered: Jan, 2002

Re: Testing without setup and tearDown Posted: Jun 9, 2008 1:36 PM
Reply to this message Reply
> You described your path to Scala-ic code in your post on
> the first release of ScalaTest, replacing SuiteRunner's
> mutable state. How did the tests you wrote around
> SuiteRunner/ScalaTest change as a result of your
> decreasing reliance on mutable state?
>
OK. Basically, the way I'd summarize my current take on mutable state in Scala programs is that, like operator identifiers and implicit conversions, you should have a justification for using mutable objects and vars. There's nothing inherently evil about them at all. They come with certain trade-offs. Programmers need to be aware of the tradeoffs and learn how to use a functional style as well as they know how to use a more imperative style. Then make decisions as they code.

So by default, I'll use a val. If I have a good reason, I'll use a var. In testing, I think I've only used vars for flags in mock objects. We haven't integrated with any of the Java mocking frameworks yet, so I just was writing mocks by hand. They are very concise to write by hand in Scala. The public var property, which I'd try and avoid in general, was a very good fit for mock objects.

I rarely used setup and teardown for some reason when using Java testing tools, so that hasn't changed much. But I did need it occasionally, and not using that approach that would probably be the main difference in how I test in Scala compared to Java. I feel that setup and teardown can make an object a bit harder to understand, but not that much. It can make it more concise than the "create" and "with" methods I show in ScalaTest's documentation, but not that much. I don't think the slight increase in conciseness in the case of setup and tearDown justifies the var, so I've used "create" and "with" methods instead.

By the way, I'm leaning towards adding the FunSuiteN-like traits back into the API, because conciseness does matter. By my current count it would add 63 traits to the API. Maybe I could put them in a different API bundle, so they aren't in the same API docs as the regular ones. But that may just make it harder to find them. Either way, I think people spend so much time writing test code that if you can save 20 lines of code per suite, it is worthwhile.

Raoul Duke

Posts: 127
Nickname: raoulduke
Registered: Apr, 2006

Re: Testing without setup and tearDown Posted: Jun 9, 2008 3:31 PM
Reply to this message Reply
[i looked around but didn't find a way to send general site feedback? sorry for spamming here!] for long articles like this, it would be great if the "comments" link from the article went to the start of the comments, rather than the re-print of the article which precedes the comments, especially when the article is longer, like this one.

Bill Venners

Posts: 2242
Nickname: bv
Registered: Jan, 2002

Re: Testing without setup and tearDown Posted: Jun 9, 2008 4:48 PM
Reply to this message Reply
> [i looked around but didn't find a way to send general
> site feedback? sorry for spamming here!] for long articles
> like this, it would be great if the "comments" link from
> the article went to the start of the comments, rather than
> the re-print of the article which precedes the comments,
> especially when the article is longer, like this one.

Hey, that's a good idea. Thanks for the suggestion. I'll add it to our list of things to do.

Flat View: This topic has 7 replies on 1 page
Topic: Testing without setup and tearDown Previous Topic   Next Topic Topic: What I want to store in my code comments


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us