Weblogs Forum - The Third State of your Binary JUnit Tests

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Weblogs Forum
The Third State of your Binary JUnit Tests

15 replies on 2 pages. Most recent reply: Jan 3, 2008 4:12 AM by chalam rani

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 15 replies on 2 pages [ 1 2 | » ]

Vladimir Ritz Bossicard

Posts: 4
Nickname: vladimir
Registered: Apr, 2003

The Third State of your Binary JUnit Tests (View in Weblogs)

Posted: Apr 23, 2003 7:35 PM

Summary
Passing or failing: these are the usual states of JUnit tests. But discover a third state that many use but only few acknowledge, and how you can elegantly manage these tests with JUnit-addons Runner.

Tests can either pass or fail. Thus they are binary.

Looking at several Open Source projects (and my own practice) I've noticed numerous tests whose names were altered or simply commented out only to exclude this particular test from the general execution.

But why should you ever exclude a test? Plenty of very good reasons:

test reveals a bug that you can't fix (for example in an external library)
test reveals a bug but you don't know how to fix it (or you simply don't have the time)
test is broken but you don't have time to fix it
...

And why does it matter? After the build itself, tests are the best indicators of the code's correctness. If failing tests remain in the test suite, nobody can trust them anymore: it confuses other developers (did my modification break the tests or were they already broken?) and slowing the development process. You can of course comment out the failing tests but the standard JUnit runner won't remind you that they are waiting to be fixed.

One solution can be to create another test suite for these tests but my experience is that doing so is the exception, not the rule. Fortunately the JUnit-addons Runner offers a flexible and elegant solution to this problem.

One of the features of this runner is its ability to ignore a test by simply appending _ignored to its method's name (public void testDummy becomes public void testDummy_ignored). When executing the suite, the runner outputs:

    ****

    Elapsed time: 0.005 sec (4 tests + 1 ignored)

    IGNORED

    1) testDummy_ignored (junitx.example.IgnoredTestCase)

All your tests now pass and you (and other developers) are always reminded that there is still work to be done.

The JUnit-addons Runner is available at sourceforge.net/projects/junit-addons

Guido van van Rossum

Posts: 359
Nickname: guido
Registered: Apr, 2003

Re: The Third State of your Binary JUnit Tests

Posted: Apr 24, 2003 6:57 AM

There's another reason for wanting to ignore a test,
and I think it can't be done by renaming the test:
for multi-platform software, some tests may not
apply to some platforms. This may not be a problem
for Java's lowest-common-denominator platform
abstraction, but for other languages (like Python)
that acknowledge the different capabilities of different
platforms, this is a necessity.

Python's standard library regression test framework
(which is built on top of PyUnit, a Python version of JUnit)
reserves a special exception for this purpose. A test
that cannot be run on a specific platform raises that
exception, and the test runner notes this.

In some cases we also rearrange the test suite dynamically
to include or exclude certain categories of tests, but
I'm not sure if Java/JUnit has the necessary flexibility
to do this.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: The Third State of your Binary JUnit Tests

Posted: Apr 24, 2003 8:06 AM

> There's another reason for wanting to ignore a test,
> and I think it can't be done by renaming the test:
> for multi-platform software, some tests may not
> apply to some platforms. This may not be a problem
> for Java's lowest-common-denominator platform
> abstraction, but for other languages (like Python)
> that acknowledge the different capabilities of different
> platforms, this is a necessity.
>
> Python's standard library regression test framework
> (which is built on top of PyUnit, a Python version of
> JUnit)
> reserves a special exception for this purpose. A test
> that cannot be run on a specific platform raises that
> exception, and the test runner notes this.
>
So is the test method itself responsible for knowing it can't be run on a particular platform, and throwing the exception explicitly in the test code? I would guess that's the case. If so, does PyUnit provide a convenience method that verifies that the platform is a specified platform, and if not, throws the exception?

> In some cases we also rearrange the test suite
> dynamically
> to include or exclude certain categories of tests, but
> I'm not sure if Java/JUnit has the necessary flexibility
> to do this.

That sounds interesting. Can you give some use cases? When would you exclude categories of tests dynamically, and how is it done in PyUnit? Do you mark certain tests as to be ignored from the runner?

In SuiteRunner, you can use a different recipe file for different subsets of test suites that you want to run. You would have to put the tests that you want to sometimes not run together in the same Suite or JUnit TestCase, because that's the unit you can turn on and off via the recipe file. More info about the recipe file is here:

http://www.artima.com/suiterunner/junit3.html

Guido van van Rossum

Posts: 359
Nickname: guido
Registered: Apr, 2003

Re: The Third State of your Binary JUnit Tests

Posted: Apr 24, 2003 8:20 AM

> So is the test method itself responsible for knowing it
> can't be run on a particular platform, and throwing the
> exception explicitly in the test code? I would guess
> that's the case.

Indeed.

> If so, does PyUnit provide a convenience
> method that verifies that the platform is a specified
> platform, and if not, throws the exception?

I hardly see the need. It's one line of code to say
if sys.platform.startswith("java"): raise TestSkipped

In many cases the test is not for a specific platform identifier but for the presence or absence of a particular function in the "os" package:

if not hasattr(os, "fork"): raise TestSkipped

> > In some cases we also rearrange the test suite
> > dynamically to include or exclude certain categories
> > of tests, but I'm not sure if Java/JUnit has the
> > necessary flexibility to do this.
>
> That sounds interesting. Can you give some use cases? When
> would you exclude categories of tests dynamically, and how
> is it done in PyUnit? Do you mark certain tests as to be
> ignored from the runner?

When there's a whole slew of tests that need to be skipped if a platform doesn't support certain functionality.

The way it is typically done is to have a function make_suite() in a test module which constructs the suite of tests to be run; it can test for a platform or feature and then include or exclude a test suite. E.g.

def make_suite():
    suite = unittest.TestSuite()
    suite.addTest(unittest.makeSuite(UniversalTests))
    if sys.platform.startswith("win"):
        suite.addTest(unittest.makeSuite(WindowsTests))
    return suite

> In SuiteRunner, you can use a different recipe file for
> different subsets of test suites that you want to run.

In Python we typically put the recipes in the test modules themselves.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

What is Value of Reporting Ignored Tests?

Posted: Apr 24, 2003 8:24 AM

I see the value of enabling tests to be ignored, but I'm curious what value people get out of the ignored tests being reported. I would guess that one value is that the ignored test reports serve as a reminder that some tests aren't being executed. Is that the only value?

Guido van van Rossum

Posts: 359
Nickname: guido
Registered: Apr, 2003

Re: What is Value of Reporting Ignored Tests?

Posted: Apr 24, 2003 8:30 AM

> I see the value of enabling tests to be ignored, but I'm
> curious what value people get out of the ignored tests
> being reported. I would guess that one value is that the
> ignored test reports serve as a reminder that some tests
> aren't being executed. Is that the only value?

Right. In the Python standard regression test suite, we maintain a list for each platform containing the *expected* skips for that platform. Then if a test is reported as skipped without being in the list, it can be flagged as a suspicious skip; this could indicate that the platform isn't configured right (e.g. package X is expected to exist but the test can't find it).

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: The Third State of your Binary JUnit Tests

Posted: Apr 24, 2003 8:31 AM

Vladimir, I'm curious why you chose to let users mark tests to be ignored by terminating the test method name with _ignored. Wouldn't it be better to have something at the front of the method, like ignoreTestSomething or skipTestSomething, so that the method would also not be executed if a different runner is used?

I am asking this questions because I would consider adding such a feature to SuiteRunner, but I want to make sure it is worth it to users, because every feature adds complexity as well as utility. The utility has to be worth the complexity.

Also, if I were to support this feature in SuiteRunner, I'd rather do it in a way that's compatible with JUnit Add-Ons, so that users could switch back and forth between our runners easily. But if I use _ignored, then they can't easily switch between our runners and the standard JUnit ones. That's vendor lock-in, in a way, which I would like to avoid.

Another potential way to let users mark tests as to be ignored or skipped is to provide a TestSkippedException, and have them throw that. By doing it this way, the test would actually fail when executed via generic JUnit runners (the runners that come with JUnit, the ones that are built into IDEs, etc.). That's probably not what users want, but maybe it is what users want. That's the behavior that you provide right now with your approach of having users append _ignored to the end of the method name. In the generic JUnit runners, those _ignored methods would be executed and the tests would generally fail.

The alternative, which would be provided by a front of method name marker, like skipTestSomething would be for the regular JUnit runners to silently ignore the test. I suspect that's probably closer to what the user wants. What do you think?

Vladimir Ritz Bossicard

Posts: 4
Nickname: vladimir
Registered: Apr, 2003

Re: The Third State of your Binary JUnit Tests

Posted: Apr 24, 2003 10:09 AM

> Vladimir, I'm curious why you chose to let users mark
> tests to be ignored by terminating the test method name
> with _ignored. Wouldn't it be better to have
> something at the front of the method, like
> ignoreTestSomething or
> skipTestSomething, so that the method would
> also not be executed if a different runner is used?

To be executed, a test method must first be extracted. And to be extracted, the test method must begin with test (JUnit constraint). That's why _ignored is appended to the method's name.

> That's vendor lock-in, in a way,
> which I would like to avoid.

If you use other features of JUnit-addons Runner (like accessing external resources) you're locked-in anyway. So it's not an issue for me.

> Another potential way to let users mark tests as to be
> ignored or skipped is to provide a TestSkippedException,
> and have them throw that.

That was an option, too. And I'm still open to that. In fact it's easy to add this feature to the runner since you can define your own TestResult that manages the execution of the tests. The TestListener/TestMonitor/TestResult architecture is highly flexible and totally customizable.

Also, the goal of the JUnit-addons Runner is not to replace the other runners nor to be compatible with them. It's the runner I use, not the one everyone must use. But that will be the topic of another post.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: The Third State of your Binary JUnit Tests

Posted: Apr 24, 2003 10:16 AM

> > If so, does PyUnit provide a convenience
> > method that verifies that the platform is a specified
> > platform, and if not, throws the exception?
>
> I hardly see the need. It's one line of code to say
> if sys.platform.startswith("java"): raise TestSkipped
>
Oh that's right. This is Python we're talking about. Last week I actually put into production at Artima.com a Python CGI script that was one line of code, though I suppose it would have been one line of code in many languages. It just redirected people who clicked on the Oracle ad in my newsletter last week to the appropriate page. It allowed me to put in the newsletter a short URL that wouldn't get chopped by email processors and to track clicks in my server logs.

> When there's a whole slew of tests that need to be skipped
> if a platform doesn't support certain functionality.
>
> The way it is typically done is to have a function
> make_suite() in a test module which
> constructs the suite of tests to be run; it can test for a
> platform or feature and then include or exclude a test
> suite. E.g.
>

def make_suite():
> suite = unittest.TestSuite()
> suite.addTest(unittest.makeSuite(UniversalTests))
> if sys.platform.startswith("win"):
> suite.addTest(unittest.makeSuite(WindowsTests))
> return suite

> > In SuiteRunner, you can use a different recipe file for
> > different subsets of test suites that you want to run.
>
> In Python we typically put the recipes in the test modules
> themselves.

I see. In JUnit and SuiteRunner, you can also write a method or constructor that dynamically populates a Suite or TestSuite differently depending on the environment. I had never thought of doing that before, but it makes sense and is already possible in the Java world too.

The recipe file approach I described before is really a static way to do the same thing, an approach that I now realize makes most sense when your test suite itself can't dynamically detect which subset of tests the user wants to run. In that case, the user would be able to choose which subset of tests to run by double-clicking on the appropriate recipe file.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: The Third State of your Binary JUnit Tests

Posted: Apr 24, 2003 10:24 AM

> > Vladimir, I'm curious why you chose to let users mark
> > tests to be ignored by terminating the test method name
> > with _ignored. Wouldn't it be better to
> have
> > something at the front of the method, like
> > ignoreTestSomething or
> > skipTestSomething, so that the method
> would
> > also not be executed if a different runner is used?
>
> To be executed, a test method must first be extracted.
> And to be extracted, the test method must begin with
> test (JUnit constraint). That's why
> _ignored is appended to the method's name.
>
Hmm. But in this case, the user doesn't want the test to be executed. They want it to be ignored, but reported as ignored.

Perhaps in your runner you allow JUnit to detect the test methods in its usual way, allow JUnit to attempt to execute them, but if they fail, report them differently after the fact. Is this what you do? I had imagined that your runner would prevent the tests from being attempted in the first place, but perhaps this was hard to add onto JUnit from the outside in Add-Ons.

Vladimir Ritz Bossicard

Posts: 4
Nickname: vladimir
Registered: Apr, 2003

Re: The Third State of your Binary JUnit Tests

Posted: Apr 24, 2003 3:12 PM

> Hmm. But in this case, the user doesn't want the test to
> be executed. They want it to be ignored, but reported as
> ignored.

The tests are not executed. I thought of executing them and report the "ignored - success" and "ignored - failure" separately but it's too much information IMO.

> Perhaps in your runner you allow JUnit

The first thing to understand is that what's left from JUnit are TestCase, TestSuite and TestSetup. All other classes have been rewritten.

The idea is that the already written tests stay the same, but the runner is completely different. It offers other functionalities

> but perhaps this was hard to add onto
> JUnit from the outside in Add-Ons.

There is JUnit-addons and JUnit-addons Runner (maybe I should have find another name). These are two very different things:

* JUnit-addons can be used with the standard JUnit runner
* JUnit-addons Runner is a totally new runner

JUnit-addons Runner offers features that were impossible to implement in the current state of the standard JUnit runner.

Shivprasad Mangesh Bhide

Posts: 1
Nickname: shivprasad
Registered: Feb, 2003

Re: The Third State of your Binary JUnit Tests

Posted: Apr 24, 2003 10:36 PM

Hi Vladimir,

Thank you for an interesting article.

It will be very helpful to have some sample code of creating & executing JUnit-Addons runner.

Waiting for the sample code...

Regards,
--- Shivprasad M. Bhide

Vladimir Ritz Bossicard

Posts: 4
Nickname: vladimir
Registered: Apr, 2003

Re: The Third State of your Binary JUnit Tests

Posted: May 1, 2003 11:35 AM

> It will be very helpful to have some sample code of
> creating & executing JUnit-Addons runner.

the best documentation is the code itself... A (very) short documentation is also available at:

http://www.worldwidewiki.net/wiki/JunitAddonsRunner

It's a wiki page so everyone can edit the documentation. So you don't have any excuse for not helping out :-)

Srinivasan Ranganathan

Posts: 1
Nickname: srirang
Registered: Jan, 2004

Re: The Third State of your Binary JUnit Tests

Posted: Jan 7, 2004 1:08 PM

The wiki states that to migrate to the new Runner, I just need to change junit.textui.TestRunner.run() to junitx.runner.TestRunner. But there is no run() method in junitx.runner.TestRunner. Can someone please tell me how to migrate to using the runner to report ignored tests.

Robert Parnell

Posts: 22
Nickname: robparnl
Registered: Jul, 2005

Re: The Third State of your Binary JUnit Tests

Posted: Aug 6, 2005 9:30 AM

I don't know about you? Using wikis isn't fun. They seem so dead, then people spam them.

Flat View: This topic has 15 replies on 2 pages [ 1 2 | » ]

Previous Topic

Next Topic