One of the greatest things about our software development process is our need and desire to test everything. Unfortunately, this doesn’t always translate into absolute best practices in the test code, coverage, or design, but we are always improving. Something I have been been meaning to get around to is property based (or automated specification based) testing. The traditional approach to unit testing (and TDD) is to write a test, see red, and then go fix it. If doing TDD religiously, this can lead to some annoying interstitial states where the code is patently absurd, for example returning the constant used in the unit test may satisfy the test until a second test is added. Another problem with using constants in tests is that we usually don’t explore the edge-cases in our code. We’ll only use characters from the US-ASCII charset as test strings for example, and therefore have little hope of finding encoding issues. We’ll often need to be very diligent and verbose in our tests in order to cover all these aspects, and when push comes to shove and the pressure is on to just ship it, test coverage is an easy thing to drop. So, techniques that increase both coverage and reuse can be highly valuable. With all that in mind, we recently started using ScalaCheck to try out property based testing. The results have been excellent! Arbitrariness The general idea is that you replace a fixed property of type A with an arbitrary one, created by a generator (basically a Factory in GoF speak). So if I have a test where I need some String for a value, rather that use only the specific one I rack my brain for (and usually come up with “fred”): 12345def pushAddsOne = { val stack = new Stack[String] stack push "fred" stack.peek must beEqualTo("fred") } I declare that my test should work for all Strings: 12345def pushAddsOne = Props.forAll { s: String => val stack = new Stack[String] stack push s stack.peek must beEqualTo(s) } Props.forAll takes a function for whom it can find an Arbitrary instance for all the arguments, and then runs the test by default 100 times with different arbitrary inputs! Furthermore, once it finds a failure case, it then tries variations of those patterns to see if it can reduce the failure down to the canonical inputs that cause the problem, saving you some of the trouble to work out what is the underlying cause. Using it in the Real World™ We are writing a Storage API, with the main implementation being file-system based. I wanted some comprehensive tests of this implementation that simulated real world usages. This means storing some large data in it. We are storing a mix of binary and char data, but largely character data which is what I wanted to test. Here’s an example test: 123456def findNone = Prop.forAll { hash: Hash => execute { Storage.get(hash, _.isDefined) } must beRight like { case Right(f) => f must beFalse } } […]
Previous Topic
