I've been working on improving the behavior of executeTestMethods when exceptions get thrown from setUpFixture, tearDownFixture, and test methods, and clarifying the documentation. I'd like to ask people's opinions.
If setUpFixture throws an exception, should we not just abort execution of that entire set of test methods? Because I don't think it makes sense to invoke the test method, and it doesn't make sence to invoke tearDownFixture. That means the next thing to invoke would be setUpFixture again prior to invoking the next test method. But it just failed, so it will probably fail again.
If setUpFixture succeeds, but tearDownFixture fails, I don't feel as clear as to the correct behavior. But I'm leaning towards just aborting the entire execution of the test methods of that Suite. It is conceivable that something needs to be done in tearDownFixture for the subsequent test methods to work correctly. Also, aborting the execution of all remaining test methods would be consistent with what we do when setUpFixture throws an exception. So we could say:
If setUpFixture returns normally, tearDownFixture will be called even if the test method completes abruptly with an exception. But if either setUpFixture or tearDownFixture completes abruptly with an exception, the executeTestMethods method will report the problem and abort. It will not attempt to execute any subsequent test methods.
This sounds reasonable to me. One other possibility would be to define this as the default behavior but provide some mechanism for customizing how/when/if setUpFixture and tearDownFixture will be executed.
And don't forget, you can always wrap all the code in your setUpFixture & tearDownFixture methods in try/catch blocks and consume (by not rethrowing) any exceptions in the catch block--probably easier than overriding executeTestMethods. So I'd vote to have exceptions terminate the suite from either method and then the test writer can consume them in either method if needed.
Previous Topic
