The Artima Developer Community
Sponsored Link

Weblogs Forum
ScalaTest 3.0 Preview

6 replies on 1 page. Most recent reply: Jan 6, 2016 11:48 AM by Bill Venners

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 6 replies on 1 page
Bill Venners

Posts: 2282
Nickname: bv
Registered: Jan, 2002

ScalaTest 3.0 Preview (View in Weblogs)
Posted: Oct 16, 2015 2:37 PM
Reply to this message Reply
Summary
The 3.0.0 release of ScalaTest is coming soon. It is a major release with significant new features. In this post, I give a preview of the new features, and ask for feedback.
Advertisement

This week I released ScalaTest and Scalactic 3.0.0-M10 for Scala 2.10 and 2.11 and Scala.js 0.9.5. If you have time please try this release out on your projects and post feedback to scalatest-users@googlegroups.com or the ScalaTest gitter channel. Here's the dependency for an sbt build:

libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.0-M10" % "test"

The main new feature of ScalaTest 3.0 is Scala.js support. We support all aspects of ScalaTest that can reasonably work on Scala.js. A few features are missing, such as the have and be matchers that take symbols--for example, bird should have ('color(blue))--and the Spec trait, which use reflection. These continue to work on the JVM as before. Other than a few such features that don't make sense or aren't possible on Scala.js, all of ScalaTest is available for testing Scala.js applications.

ScalaTest is the most full-featured test framework available on Scala.js. Its ability to run in both worlds means you test code that you run both on the JVM and on JavaScript with tests that you run both on the JVM and JavaScript.

Another major new feature in ScalaTest 3.0, which was in great part motivated by Scala.js, is async testing styles. ScalaTest 3.0 will offer an async flavor of each style trait in which tests are functions registered at construction time. For example in addition to FunSuite, there's now an AsyncFunSuite. In addition to FeatureSpec, there's now an AsyncFeatureSpec, and so on for the other test-as-function styles. The difference between FunSuite and AsyncFunSuite is that in FunSuite, the result type of a test is "Assertion", whereas in AsyncFunSuite, it is Future[Assertion].

Assertion is a type alias, new in 3.0, for org.scalatest.Succeeded.type, the type of the Succeeded singleton. Succeeded was added in ScalaTest 2.0 to represent a successful test Outcome. In 2.0 ScalaTest's withFixture method signature was modified to take a test function with result type Outcome as its parameter and to return an Outcome itself. This allows users to execute shared code at the beginning and/or end of a test, execute test bodies multiple times, or change the outcome of tests under certain condition--all without needing to catch exceptions. In 3.0, all the assertions and matcher expressions have been changed to return the Succeeded singleton instead of the Unit value singleton, which was the result in ScalaTest 1.x and 2.x. They still throw the same exceptions when a problem is detected, but when they succeed they return a different singleton object. I expect this change break no existing code in practice, because in earlier versions of ScalaTest the result type of tests was Unit or Any, essentially, and so any result of an assertion or matcher expression was normally ignored. Here's an example from the REPL using 3.0.0-M10:

scala> val x = 1
x: Int = 1

scala> assert(x > 0)
res0: org.scalatest.Assertion = Succeeded

scala> assert(x == 0)
org.scalatest.exceptions.TestFailedException: 1 did not equal 0
  at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:507)
  at org.scalatest.Matchers$.newAssertionFailedException(Matchers.scala:7743)
  at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:472)
  ... 43 elided

scala> x should be > 0
res2: org.scalatest.Assertion = Succeeded

scala> x shouldEqual 0
org.scalatest.exceptions.TestFailedException: 1 did not equal 0
  at org.scalatest.MatchersHelper$.newTestFailedException(MatchersHelper.scala:148)
  at org.scalatest.MatchersHelper$.indicateFailure(MatchersHelper.scala:358)
  at org.scalatest.Matchers$AnyShouldWrapper.shouldEqual(Matchers.scala:6792)
  ... 43 elided

You write traditional (synchronous) tests just like you always did, and all existing code should compile and run exactly as before, but in async styles you must return from your tests a Future[Assertion]. The purpose of async styles is to allow you to write tests in a non-blocking manner. On the JVM this enables you to write tests consistently with production code in reactive frameworks like Akka and Play. On Scala.js it enables testing asynchronous code that would not otherwise be possible to test using ScalaTest blocking constructs (like ScalaFutures), because in JavaScript you *can't* block.

In an async test you work in futures. Given, for example, a Future[Int] that has been returned by some code under test, you can map an assertion onto it like this:

scala> import scala.concurrent.Future
import scala.concurrent.Future

scala> import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.ExecutionContext.Implicits.global

scala> val fut = Future { 1 }
fut: scala.concurrent.Future[Int] = scala.concurrent.impl.Promise$DefaultPromise@52c293d0

scala> fut map { x => x should be > 0 }
res4: scala.concurrent.Future[org.scalatest.Assertion] = scala.concurrent.impl.Promise$DefaultPromise@1ca1e995

This value of type Future[Assertion] you can return to ScalaTest by leaving it as the result of your async test. When that future completes, the test will either succeed or fail. This future assertion would result in a successful test:

scala> res4.value
res5: Option[scala.util.Try[org.scalatest.Assertion]] = Some(Success(Succeeded))

Here's one that would result in a failed test:

scala> fut map { x => x shouldEqual 0 }
res6: scala.concurrent.Future[org.scalatest.Assertion] = scala.concurrent.impl.Promise$DefaultPromise@18daf92c

scala> res6.value
res7: Option[scala.util.Try[org.scalatest.Assertion]] =
        Some(Failure(org.scalatest.exceptions.TestFailedException: 1 did not equal 0))

You can include non-async tests that result in Assertion in an async style alongside your tests that result in Future[Assertion]. The final expression of such tests will be converted to a Future[Assertion] that has already succeeded. At run time, the test will either throw an exception or return that successful Future[Assertion].

The traditional styles like FunSuite and FeatureSpec also have changed the test result type. In ScalaTest 1.x and 2.x the result type of tests was Unit. In 3.0 it will be Assertion. However, we have a trait named Compatibility that is currently in effect by default that provides an implicit conversion from Any to Assertion. This will keep all old code working by providing a kind of value discarding similar to what the compiler does for Unit. The Compatibility "registration policy" offers convenience if you're using 3rd party tools that provide assertion syntax the returns the Unit value, such ask Akka's testkit of any of several Java mocking frameworks. If you prefer more type safety, you can mix in trait Safety, which will hide that Any => Assertion implicit. Any type errors you get you can fix by ending your test with Succeeded, or what I will recommend, simply succeed. The "succeed" construct is a new one in ScalaTest 3.0 that returns the Succeeded singleton.

scala> succeed
res8: org.scalatest.Succeeded.type = Succeeded

One issue I would like to hear user's opinions on is whether Compatibility or Safety should be the default. My feeling is that safety should be the default in async styles, because no prior code exists that would break, and I fear Compatibility might be error prone when futures are involved. If safety (AsyncSafety actually) is the default in async styles, then I am thinking Safety should be the default in traditional styles for consistency. That would break a lot of code, so I'd do that through a long deprecation cycle. But still it would require everyone to either mix in Compatibility into their base classes, if they have base classes, or actually fix the type errors one way or another. To give you one example of the kind of type error you might encounter, ScalaTest's intercept construct, if it appears last in a test body, would give a type error because its result type is Throwable, not Assertion, because it returns the caught exception if the assertion succeeds.

scala> intercept[IndexOutOfBoundsException] { "hi".charAt(-1) }
res9: IndexOutOfBoundsException = java.lang.StringIndexOutOfBoundsException: String index out of range: -1

You could fix this type error by placing "succeed" after intercept, but we added a new construct to Assertions in ScalaTest 3.0 called assertThrows, which has the same behavior as intercept except it returns the Succeeded singleton. So you could instead replace intercept with assertThrows:

scala> assertThrows[IndexOutOfBoundsException] { "hi".charAt(-1) }
res10: org.scalatest.Assertion = Succeeded

That could end up being quite being quite a few changes to get rid of deprecation warnings, so please give your opinions on whether Compatibility or Safety should be the default.

The BeforeAndAfter, BeforeAndAfterEach, and BeforeAndAfterEachTestData traits do work with async styles, but you will need to synchronize access to shared mutable state between "before" and "after" code and test code. So I will recommend people override withAsyncFixture by default instead. The withAsyncFixture method takes a test function that returns a Future[Outcome] and itself returns Future[Outcome]. You can execute "beginning-of-test" code before invoking the test function, then map "end-of-test" code onto the returned Future[Outcome], or just register the end-of-test code as a call back to be executed when the future completes.

Although tests in async styles run assertions in futures, by default tests will execute one at a time in order. The second test won't start until the first test completes. The third test won't start until the second test completes, and so on. If you want the async tests to run in parallel, you mix in ParallelTestExecution to the test class, just as with traditional styles.

ScalaTest 3.0.0-M10 has a handful of deprecations compared to the current, 2.x release, but should only have two breaking changes. First, one of the overloaded "thread" methods in Conductor would no longer overload after changing the result type of tests to Assertion. It has been renamed to threadNamed in 3.0. I will likely release a 2.2.x version of ScalaTest with that deprecated prior to the 3.0 final release. Also, if someone had been invoking the whenCompleted method of Status, that now takes a Try[Boolean] => Unit instead of just a Boolean => Unit. Otherwise no existing ScalaTest 2.x code should break (so long as you clear all deprecation warnings before upgrading). Please let me know if you see a breakage when trying 3.0.0-M10 in your projects.

Thanks and happy testing.


Bárður Árantsson

Posts: 5
Nickname: somebody
Registered: May, 2007

Re: ScalaTest 3.0 Preview Posted: Oct 16, 2015 3:37 PM
Reply to this message Reply
scala.js would be amazing. Do you have any numbers for what the footprint[1] is, or even what it would be in theory?

[1] I'm thinking in terms of extra overhead for a JS engine, etc. as compared to e.g. uTest. (Rhino has been a problem for me & my just because of the ridiculous PermGen/Metaspace needed.).

Bill Venners

Posts: 2282
Nickname: bv
Registered: Jan, 2002

Re: ScalaTest 3.0 Preview Posted: Oct 16, 2015 5:04 PM
Reply to this message Reply
> scala.js would be amazing. Do you have any numbers for
> what the footprint[1] is, or even what it would be in
> theory?
>
> [1] I'm thinking in terms of extra overhead for a JS
> engine, etc. as compared to e.g. uTest. (Rhino has been a
> problem for me & my just because of the ridiculous
> PermGen/Metaspace needed.).
>
People have been using ScalaTest 3.0 miletone releases to test on Scala.js for several months now. We haven't had any reports of memory problems to my knowledge. You can checkout the size for yourself of the jar on sonatype:

https://oss.sonatype.org/content/groups/public/org/scalatest/scalatest-all_sjs0.6_2.11/3.0.0-M10/

Konstantin Silin

Posts: 2
Nickname: kostja
Registered: Dec, 2015

Re: ScalaTest 3.0 Preview - Exception interception Posted: Dec 22, 2015 11:15 AM
Reply to this message Reply
What would be the recommended to asynchronously test for Exceptions?

Have been using `intercept[Exception]{ Await...}` until now.

Using `assertThrows` in a similar way to async `should` does not seem to work. The Exception thrown by the Future in the below code is not intercepted:

`someFuture map { r => assertThrows[Exception]{r.toString }}`

What am I missing here?

Bill Venners

Posts: 2282
Nickname: bv
Registered: Jan, 2002

Re: ScalaTest 3.0 Preview - Exception interception Posted: Dec 22, 2015 11:47 AM
Reply to this message Reply
> What would be the recommended to asynchronously test for
> Exceptions?
>
> Have been using `intercept[Exception]{ Await...}` until
> now.
>
> Using `assertThrows` in a similar way to async `should`
> does not seem to work. The Exception thrown by the Future
> in the below code is not intercepted:
>
> `someFuture map { r => assertThrows[Exception]{r.toString
> }}`
>
> What am I missing here?

Hi Konstantin,

You are missing the next 3.0 milestone release that we haven't released yet. I will make a milestone release today (3.0.0-M15) to get you going, and we'll deploy Scaladoc. I'll post here to let you know when that is deployed. What you'll find in there is two new constructs you can use when you want to asynchronously test for exceptions. Here's a copy and paste from the AsyncFunSuite Scaladoc. Same works for any async style:

If you need to test for expected exceptions in the context of futures, you can use the recoverToSucceededIf and recoverToExceptionIf methods of trait RecoverMethods. Because this trait is mixed into supertrait AsyncSuite, both of these methods are available by default in an AsyncFunSuite.

If you just want to ensure that a future fails with a particular exception type, and do not need to inspect the exception further, use recoverToSucceededIf:


recoverToSucceededIf[IllegalStateException] { // Result type: Future[Assertion]
emptyStackActor ? Peek
}


The recoverToSucceededIf method performs a job similar to assertThrows, except in the context of a future. It transforms a Future of any type into a Future[Assertion] that succeeds only if the original future fails with the specified exception. Here's an example in the REPL:


scala> import org.scalatest.RecoverMethods._
import org.scalatest.RecoverMethods._

scala> import scala.concurrent.Future
import scala.concurrent.Future

scala> import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.ExecutionContext.Implicits.global

scala> recoverToSucceededIf[IllegalStateException] {
| Future { throw new IllegalStateException }
| }
res0: scala.concurrent.Future[org.scalatest.Assertion] = ...

scala> res0.value
res1: Option[scala.util.Try[org.scalatest.Assertion]] = Some(Success(Succeeded))


Otherwise it fails with an error message similar to those given by assertThrows:


scala> recoverToSucceededIf[IllegalStateException] {
| Future { throw new RuntimeException }
| }
res2: scala.concurrent.Future[org.scalatest.Assertion] = ...

scala> res2.value
res3: Option[scala.util.Try[org.scalatest.Assertion]] =
Some(Failure(org.scalatest.exceptions.TestFailedException: Expected exception
java.lang.IllegalStateException to be thrown, but java.lang.RuntimeException
was thrown))

scala> recoverToSucceededIf[IllegalStateException] {
| Future { 42 }
| }
res4: scala.concurrent.Future[org.scalatest.Assertion] = ...

scala> res4.value
res5: Option[scala.util.Try[org.scalatest.Assertion]] =
Some(Failure(org.scalatest.exceptions.TestFailedException: Expected exception
java.lang.IllegalStateException to be thrown, but no exception was thrown))


The recoverToExceptionIf method differs from the recoverToSucceededIf in its behavior when the assertion succeeds: recoverToSucceededIf yields a Future[Assertion], whereas recoverToExceptionIf yields a Future[T], where T is the expected exception type.


recoverToExceptionIf[IllegalStateException] { // Result type: Future[IllegalStateException]
emptyStackActor ? Peek
}


In other words, recoverToExpectionIf is to intercept as recovertToSucceededIf is to assertThrows. The first one allows you to perform further assertions on the expected exception. The second one gives you a result type that will satisfy the type checker at the end of the test body. Here's an example showing recoverToExceptionIf in the REPL:


scala> val futureEx =
| recoverToExceptionIf[IllegalStateException] {
| Future { throw new IllegalStateException("hello") }
| }
futureEx: scala.concurrent.Future[IllegalStateException] = ...

scala> futureEx.value
res6: Option[scala.util.Try[IllegalStateException]] =
Some(Success(java.lang.IllegalStateException: hello))

scala> futureEx map { ex => assert(ex.getMessage == "world") }
res7: scala.concurrent.Future[org.scalatest.Assertion] = ...

scala> res7.value
res8: Option[scala.util.Try[org.scalatest.Assertion]] =
Some(Failure(org.scalatest.exceptions.TestFailedException: "[hello]" did not equal "[world]")


Bill

Konstantin Silin

Posts: 2
Nickname: kostja
Registered: Dec, 2015

Re: ScalaTest 3.0 Preview - Exception interception Posted: Jan 6, 2016 11:38 AM
Reply to this message Reply
Hi Bill

have just tried the approaches you have described with M15, works like a charm :)

many thank for the help and for creating great stuff which makes our lives better
kostja

Bill Venners

Posts: 2282
Nickname: bv
Registered: Jan, 2002

Re: ScalaTest 3.0 Preview - Exception interception Posted: Jan 6, 2016 11:48 AM
Reply to this message Reply
> Hi Bill
>
> have just tried the approaches you have described with
> M15, works like a charm :)
>
Awesome, thanks for trying it out and letting us know. So far it looks good for RC1 this weekend!

> many thank for the help and for creating great stuff which
> makes our lives better
>
You're very welcome.

Bill

Flat View: This topic has 6 replies on 1 page
Topic: Object Design Previous Topic   Next Topic Topic: The fate of reduce() in Python 3000


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2017 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us