The Artima Developer Community
Sponsored Link

Weblogs Forum
ScalaTest 0.9 Released

58 replies on 4 pages. Most recent reply: Jan 21, 2008 6:26 AM by Bill Venners

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 58 replies on 4 pages [ 1 2 3 4 | » ]
Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

ScalaTest 0.9 Released (View in Weblogs)
Posted: Jan 15, 2008 12:56 AM
Reply to this message Reply
Summary
Today I released a testing tool written in Scala, which can be used to test Scala or Java code. It is intended primarily as a tool for testing Scala code, but can also be used as a low-risk way to get started programming in Scala, by writing tests in Scala for production code written in Java.
Advertisement

The last several months I've been working part time on a testing tool for Scala named ScalaTest. I did this for a few reasons. First, I wanted to start using Scala internally at Artima, and I felt the good place to start would be in writing tests. I looked around and didn't see a production-ready testing framework for Scala that took advantage of what Scala has to offer, so I felt there was a need. I also figured it would be a good project for me to start with, because I could get real experience in Scala by writing something “on the side”—in other words, something I didn't need to put into production right away. I wanted some experience in Scala before starting to try and integrate Scala into our web framework, in the hopes that I'd have a better chance of getting that right.

Some friends and I had written a test framework called SuiteRunner, starting in 2001. I released this tool open source in 2003, but never released it 1.0 because I realized that the Java community had settled on JUnit. Over time many of the problems that inspired me to create SuiteRunner were fixed in later versions of JUnit, and I took to recommending people use either JUnit or TestNG. Internally at Artima, however, we have used SuiteRunner for our unit testing since 2003, and it has worked quite nicely. Thus, I decided to try and “port” SuiteRunner to Scala as my first Scala project. The result is ScalaTest.

ScalaTest is not SuiteRunner code written in Scala syntax, however. Early on, when I was less experienced, I spent lengthy swaths of time staring at my Scala code trying to figure out how to write it without any “vars,” which are reassignable variables (like normal variables in Java). Over time I became better at it, and now programming with “vals,” which are like final variables in Java, feels very natural to me. I also used Option everywhere instead of null, pattern matching where useful, and even made a few new control structures using by-name parameters to get rid of code duplication. All these things were pretty straightforward for me coming from a Java background.

I also tried to eliminate mutable state. In SuiteRunner, Suite was a class that had quite a few instance variables. In ScalaTest, Suite is a trait, which by definition has no instance variables. In SuiteRunner, interface Reporter had a setConfiguration method, which pretty much pushes any class implementing that interface to maintain the most recent configuration as mutable state. ScalaTest's Reporter trait has no such method. Instead of configuring reporters directly, I wrap them internally in a FilterReporter whose constructor takes the configuration state.

Because I had a production-ready testing tool written in Java as my starting point, the result of my exercise is a testing tool written in Scala whose features rival those of JUnit 3, JUnit 4, and TestNG. ScalaTest seems to work just fine already, but I want to keep it under 1.0 (right now it is at release 0.9) for a few months of beta testing. My plan is to release it 1.0 in time for JavaOne in May.

So I encourage you to try out ScalaTest and give me feedback. I worked hard to document it thoroughly, so it should be pretty easy for you to figure out how to use. To satisfy your thirst for code, here's a simple example:

import org.scalatest.Suite

 class MySuite extends Suite {

   def testAddition() {
     val sum = 1 + 1
     assert(sum === 2)
     assert(sum + 2 === 4)
   }

   def testSubtraction() {
     val diff = 4 - 1
     assert(diff === 3)
     assert(diff - 2 === 1)
   }
 }

To figure out what that all means, the best place to start is the ScalaDoc documentation for Suite itself.

I have created a project for ScalaTest at java.net, but have not yet gotten it rolling. For the time being, please submit feedback to the ScalaTest Forum.


Jörn Zaefferer

Posts: 22
Nickname: jzaefferer
Registered: Jul, 2007

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 7:29 AM
Reply to this message Reply
Hey Bill,

you really should mention in your post that assert(x === y) is not the same as assert(x == y), rather that it is able to print both expected and actual values for failed assertions. Thats really slick and convinced me to give ScalaTest a try.

Any chance to get a Eclipse plugin similar to the JUnit plugin?

Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 11:11 AM
Reply to this message Reply
> Hey Bill,
>
> you really should mention in your post that assert(x ===
> y) is not the same as assert(x == y), rather that it is
> able to print both expected and actual values for failed
> assertions. Thats really slick and convinced me to give
> ScalaTest a try.
>
Yes, the === works with an implicit conversion. It does what assertEquals does in JUnit, but with a more natural syntax. It is a little thing, but we write so many of these assertions. In JUnit you say:


assertEquals(actual, expected)


But in ScalaTest you can say:


assert(actual === expected)


There's also a control construct called "expect", that makes checking for thrown exceptions more concise. Instead of:


try {
"hi".charAt(-1)
fail()
}
catch {
case e: IndexOutOfBoundsException => // Expected, so continue
}


You can say in ScalaTest:


expect(classOf[IndexOutOfBoundsException]) {
"hi".charAt(-1)
}


In JUnit 4 and TestNG, you can do that with an annotation, but it is more verbose because you have to make one test method for each expected exception.


@Test(expected= IndexOutOfBoundsException.class)
public void testCharAtIndexOutOfBounds() {
"hi".charAt(-1)
}


> Any chance to get a Eclipse plugin similar to the JUnit
> plugin?
>
Our next step is an ant task. Then a maven one. Then I figure we'll do some adapters and such to make it easy to integrate ScalaTest with ScalaCheck, JUnit, JUnit 4, TestNG, and SuiteRunner. The IDE tools in Scala are still forming. There's a pretty nice Eclipse plug-in for Scala. Someone is working on one for NetBeans, and hopefully Sun themselves will support this someday. Someone started a plug-in for IntelliJ. The main expected use case for ScalaTest is to write tests for Scala code, so the place support should go is into those Scala plug-ins I think. I'll try and contact those folks and suggest that to them.

Bruce Eckel

Posts: 874
Nickname: beckel
Registered: Jun, 2003

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 1:59 PM
Reply to this message Reply
Seems like you should be able to get rid of the requirement to say "test" everywhere.

Jörn Zaefferer

Posts: 22
Nickname: jzaefferer
Registered: Jul, 2007

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 2:01 PM
Reply to this message Reply
> Seems like you should be able to get rid of the
> requirement to say "test" everywhere.

Makes sense to me, too. A test shouldn't need any public non-test methods.

Another thing: A common annoying thing in JUnit is the lack of multiple inheritance forcing me to maintain several branches that subclass TestCase. Could you provide an example how ScalaTest avoids that, eg. just an example showing how to refactor my test classes?

Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 2:58 PM
Reply to this message Reply
> Seems like you should be able to get rid of the
> requirement to say "test" everywhere.

This is easy to do. If you don't like starting test methods with "test", just make a sub-trait that and override the testNames method. You could, for example, change it to discover all public methods, as Joern suggests. Though you'll need to exclude methods like assert, execute, runTests, etc, that are public and declared in trait Suite. If you prefer an @Test annotation, you could do that too. Before long we'll probably have a trait you can use to use the JUnit 4 or TestNG annotations for those who prefer that approach. You'll find some other examples if you read the "Extensibility" section in the Suite documentation:

http://www.artima.com/scalatest/doc/org/scalatest/Suite.html

The "test" approach is just the default. It can be overridden.

Bruce Eckel

Posts: 874
Nickname: beckel
Registered: Jun, 2003

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 3:26 PM
Reply to this message Reply
It seems like Scala could support discovery-based unit testing rather than just reiterating variations of what we've done in the past. See:
http://somethingaboutorange.com/mrl/projects/nose/

Bruce Eckel

Posts: 874
Nickname: beckel
Registered: Jun, 2003

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 3:31 PM
Reply to this message Reply
What I mean is that you should only have to jump through hoops if it gives you something. Just saying you have to do this because we've always done it this way in previous unit test frameworks is wasteful. One of the great benefits of Scala is eliminating the need to write unnecessary code, and it looks to me that this framework may be backsliding a bit.

Is there any way that I can just write the minimal amount of code in order to create a test? The annotated test framework I developed in Thinking in Java 4e is an example of this.

And "the simplest thing that could possibly work" should be the default behavior, not something I can get if I'm willing to work at it.

This will improve the out-of-the-box testing experience and also be another example of why Scala is great.

Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 3:57 PM
Reply to this message Reply
> It seems like Scala could support discovery-based unit
> testing rather than just reiterating variations of what
> we've done in the past. See:
> http://somethingaboutorange.com/mrl/projects/nose/

I'm not sure what they mean by discovery based testing. Perhaps you could clarify?

What is currently discovered by ScalaTest is test methods in a Suite, and what I have planned for 1.0 is discovering Suites in a package. This is described in the "Still to do" list on the ScalaTest page or README.txt. You need to organize your unit tests somehow, so you may as well put them in test methods inside Suites. But the idea is if you add a new test method, or add a new suite, you just push the button and it will be discovered and run. Moreover, the default behavior can be overridden, so if people want to do things differently they can. Suite is a trait, not a class, so it can be mixed into anything. If someone wants to put test methods right in a production class, they could mix in Suite to that production class. But I wouldn't recommend that. If they want it in the same file, they could put a separate Suite in the same file, or put the Suite as a nested class inside the class under test. All of these Suites will be discovered automatically once I get the Suite wildcard feature implemented.

Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 4:13 PM
Reply to this message Reply
> What I mean is that you should only have to jump through
> hoops if it gives you something. Just saying you have to
> do this because we've always done it this way in previous
> unit test frameworks is wasteful.
>
Are you saying that writing "test" is jumping through a hoop? I don't see it as such. It also isn't required by ScalaTest. It is just the default.

I didn't do it this way because we've always done it this way. I did it that way because I think it produces the most concise, clearest code. Compare these two:


def testExcludes() {
// ...
}


with:


@Test
def testExcludes() {
// ...
}


The latter repeats "test". I've seen TestNG examples avoid the repetition with:


@Test
def verifyExcludes() {
// ...
}


That zaps the repetition, but adds two more characters to type. If you just get rid of the verify, you get:


@Test
def excludes() {
// ...
}


Now you have just one extra character, but the method name isn't a verb anymore. To me, testExcludes is the most straightforward, easy to type, easy to read, name for that method. That's why it is the default.

> One of the great
> benefits of Scala is eliminating the need to write
> unnecessary code, and it looks to me that this framework
> may be backsliding a bit.
>
Is the unecessary code you're talking about the "test" at the beginning of a test method. Most people are doing JUnit 3, and that's what they are doing now. If by backsliding you mean that JUnit 4 and TestNG got rid of that, then yes, I "went back" to it, but because I think it is better. (I personally didn't go back, because never used the @Test annotation.) But it is only the default. If people want to define tests a different way, they can. I plan to provide a package of Suites that make it easy for people to JUnit or TestNG's @Test annotation if they want, or Rehersal's test as function value approach, etc.

Tests in Scala will be more concise than equivalent tests in Java, simply because Scala code is more concise than Java code. For an example of how Scala helped me make testing code more concise, look at the expect method in Suite. There's an example in a previous post in this form.

> Is there any way that I can just write the minimal amount
> of code in order to create a test? The annotated test
> framework I developed in Thinking in Java 4e is an example
> of this.
>
> And "the simplest thing that could possibly work" should
> be the default behavior, not something I can get if I'm
> willing to work at it.
>
> This will improve the out-of-the-box testing experience
> and also be another example of why Scala is great.
>
Can you give a specific example? I don't see the verbosity which you seem to see.

Raoul Duke

Posts: 127
Nickname: raoulduke
Registered: Apr, 2006

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 5:53 PM
Reply to this message Reply
ScalaTest sounds real nice. Reading the 'nose' stuff for Python, it makes me think that if/when ScalaTest and ScalaCheck are nicely integrated, things will be super duper sweet in the land of Scala.

(Now if only I could use things like JML and ESC/Java and PMD and FindBugs and all that super easily perfectly wonderfully well with Scala sources...)

Leo Lipelis

Posts: 111
Nickname: aeoo
Registered: Apr, 2006

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 6:32 PM
Reply to this message Reply
> The "test" approach is just the default. It can be overridden.

Assembly is just the default way of programming. You are not stuck to it, you can just write your own C compiler.

I know it's an extreme example, but the point is that software is all about reasonable defaults. If you think about it, a C compiler is as much a spreadsheet as Excel. So why do people like to use Excel? It's because Excel has reasonable defaults for accountants. In some ways a compiler is actually a better spreadsheet because it is a lot more flexible.

Providing reasonable defaults is as big a deal as providing extensibility.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 7:48 PM
Reply to this message Reply
> Hey Bill,
>
> you really should mention in your post that assert(x ===
> y) is not the same as assert(x == y), rather that it is
> able to print both expected and actual values for failed
> assertions. Thats really slick and convinced me to give
> ScalaTest a try.


That's nice. The only thing I'd offer is that the fact that === is palindrome seems to imply that the operator is symmetric.

HUnit in Haskell uses @=? as an operator with expected on the left and actual on the right.

It's kind of neat because it looks like a little placeholder for an equation with '?' holding the position of the thing we are questioning.

Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 8:56 PM
Reply to this message Reply
> ScalaTest sounds real nice. Reading the 'nose' stuff for
> Python, it makes me think that if/when ScalaTest and
> ScalaCheck are nicely integrated, things will be super
> duper sweet in the land of Scala.
>
Rickard Nilsson and I spoke a few days ago about how to do the integration. I want to make property-based tests as easy to write as assertion-based ones in ScalaTest. My next step is to make a Suite subtrait that will define and check a property, and if there's a problem complete abruptly with an AssertionError containing useful information. That way problems found by property-based tests can show up in the same ScalaTest reports as assertion-based ones. You'll be able to mix assertion and property-based tests in the same test methods.

Bill Venners

Posts: 2249
Nickname: bv
Registered: Jan, 2002

Re: ScalaTest 0.9 Released Posted: Jan 15, 2008 9:02 PM
Reply to this message Reply
> > The "test" approach is just the default. It can be
> overridden.
>
> Assembly is just the default way of programming. You are
> not stuck to it, you can just write your own C compiler.
>
> I know it's an extreme example, but the point is that
> software is all about reasonable defaults. If you think
> about it, a C compiler is as much a spreadsheet as Excel.
> So why do people like to use Excel? It's because Excel
> l has reasonable defaults for accountants. In some ways a
> compiler is actually a better spreadsheet because it is a
> lot more flexible.
>
> Providing reasonable defaults is as big a deal as
> providing extensibility.
>
I agree with that. I just think asking people to name test methods the way I expect most of them would name them anyway is a reasonable default. See my explanation in the previous post.

But on the other hand, I don't mean you need to override the behavior in every Suite. You just pick the Suite trait that makes you happy and extend that each time. You can write your own if you need to, but I expect very few people would need to. I plan to make some Suite subtraits that will allow people to use TestNG or JUnit 4 annotation if they wish, for example. One good thing about that is you could run such Suites with either ScalaTest or TestNG/JUnit 4. In the 99.99% case, I expect people will just extend an existing Suite trait.

Flat View: This topic has 58 replies on 4 pages [ 1  2  3  4 | » ]
Topic: ScalaTest 0.9 Released Previous Topic   Next Topic Topic: What Are Your Java Pain Points, Really?


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us