ScalaTest/ScalaUtils Forum - TestFailedException should be an error and not a RuntimeException

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

ScalaTest/ScalaUtils Forum
TestFailedException should be an error and not a RuntimeException

0 replies on 1 page.

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 0 replies on 1 page

Andrew Norman

Posts: 2
Nickname: normana400
Registered: Mar, 2016

TestFailedException should be an error and not a RuntimeException

Posted: Aug 9, 2016 1:59 PM

it is a common coding practice in scala (as in Java) to have code that throws unchecked exceptions (ie: RuntimeException) when coding assumptions are broken. Most test libraries realize this and try to accommodate this use case by implementing their assertion / failure detection as Errors (usually as AssertionErrors) and not as RuntimeExceptions. The java / scala standard is that assertion failures are Errors and not Exceptions. This is because code should not catch Errors as Errors by their definition live outside the responsibility of the program to manage.

Please don't try to dispute this. It's a fundamental plank that the Java / Scala community is based upon.

Unfortunately ScalaTest does not implement this strategy and that unfortunately creates a large pain point for those who use scalatest to implement a negative testing (testing to ensure exceptions are thrown). It seems that the ScalaTest design didn't envision that developers would commonly be writing code that throws RuntimeExceptions and would need to develop a negative test to test that scenario.

The common way to write this sort of negative test is:

scenario("negative test that should throw a RuntimeException") {
try {
doSomething("anInvalidParameter")
fail("exception not thrown")
} catch {
case e: RuntimeException => //passes
}
}

the problem is that the scalatest fail() method throws a TestFailedException (which is an extension of a RuntimeException and not a AssumptionError). The ramifications of this is that this commonly written test pattern will always pass. This is because either the RuntimeException is thrown by the method being tested or a TestFailedException is thrown by the test and in both cases they are RuntimeExceptions. This is doubly bad because many developers who are writing tests implementing this sort of pattern will not realize that the test isn't effectively working since it will pass by default. They'd have to purposely test it against code that they know should not throw an error (like I did) to know there is an issue here.

The way to work around the scalatest class heirarchy flaw is to write the test like this:

scenario("negative test that should throw a RuntimeException") {
try {
doSomething("anInvalidParameter")
fail("exception not thrown")
} catch {
case e: TestFailedException => throw e
case e: RuntimeException => //passes
}
}

the problem here is that most developers won't realize the need to do this since it is a common fundamental plank in the java / scala community that test failures are Errors and not to be handled as Exception since test failures do not fall in the responsibility of an application to catch or handle. They truly need to live as Errors so that unexpected consequences like the scenario mentioned above do not happen!

Previous Topic

Next Topic


	Web Artima.com