The Artima Developer Community
Sponsored Link

SuiteRunner Forum
Are SuiteRunner tests tied to Java Methods

3 replies on 1 page. Most recent reply: Feb 28, 2003 2:33 PM by Bill Venners

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 3 replies on 1 page
Barry Roberts

Posts: 2
Nickname: manithree
Registered: Feb, 2003

Are SuiteRunner tests tied to Java Methods Posted: Feb 28, 2003 9:54 AM
Reply to this message Reply
Advertisement
I just started with JUnit and SuiteRunner yesterday, and I have some things sort of working, but I'm having problems getting exactly what I need.

I have a TestCase-derived class that has one method that calls the same code with multiple data sets to form a regression test suite. What I would like to be able to do is add a new set of data to the xml file I use for configuration, and the next time I run JUnit or SuiteRunner, there's a new test without re-compiling my class to add a method for it.

Naturally, I would like each to test to individually report its success/failure.

I've tried implementing a countTestCases, and both JUnit and SuiteRunner report the number returned, but I don't know how to make my one testCalculation method report success/failure for multiple tests.

Am I going about this wrong, or is this even possible with SuiteRunner?

Any pointers would be greatly appreciated.

Thanks,
Barry Roberts


Barry Roberts

Posts: 2
Nickname: manithree
Registered: Feb, 2003

Re: Are SuiteRunner tests tied to Java Methods Posted: Feb 28, 2003 12:50 PM
Reply to this message Reply
Never mind. I figured it out.

It's just too bad suiterunner doesn't have a "Test Hierarchy" view like JUnit's GUI does.

This is cool stuff, though.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Re: Are SuiteRunner tests tied to Java Methods Posted: Feb 28, 2003 2:21 PM
Reply to this message Reply
I'm glad you asked your question. Even though you said you figured out a solution, I want to post an answer for others to read.

Basically, a Suite, an object that represents the conceptual notion of a suite of tests, is responsible for executing itself. To get a Suite to execute, you invoke its execute method. The default behavior of execute, i.e., the implementation of Suite.execute, is to call executeTestMethods and executeSubSuites, in that order. The default behavior of executeTestMethods is to discover test methods on itself via reflection, and invoke them. The default behavior of executeSubSuites is to invoke execute sequentially on each of this Suite's sub-Suites.

The most common way to create a test suite with Artima SuiteRunner, is to create a subclass of Suite and define test methods. However, you can also override the execute methods described above in your Suite subclass. For example, if you want to run test methods in a particular order, you can override executeTestMethods and make sure the test methods are invoked in your preferred order.

In your case, you want to create a conceptual suite of tests that is composed of the same test functionality called with different data values read from an XML file. There are several ways to do that with Artima SuiteRunner, but here's the way I would first think of doing it. Create a subclass of Suite. Define a test method that accepts the appropriate test data as a parameter.

Then, override the execute method, which takes a Reporter. Your execute method would open the XML file and read in the data. For each piece of data, call Reporter.testStarting, and invoke your test method with the appropriate value for the test data. If the test method returns normally, invoke reporter.testSucceeded. If the test method completes abruptly with an exception, invoke reporter.testFailed. And you probably want to close the XML file in a finally clause.

If you also want to have sub-Suites, you could override executeTestMethods instead of execute itself and implement it in the same I described above for overriding execute.

Basically, you want to create a conceptual suite of tests that behaves slightly differently than the default behavior provided by Suite itself. It was our intent that you be able to do that by customizing the behavior of the execute methods, execute, executeTestMethods, and executeSubSuites. So that's the way I would recommend doing it.

Bill Venners

Posts: 2284
Nickname: bv
Registered: Jan, 2002

Heirarchical View of Test Results Posted: Feb 28, 2003 2:33 PM
Reply to this message Reply
> It's just too bad suiterunner doesn't have a "Test
> Hierarchy" view like JUnit's GUI does.

Actually, the reason we don't have a hierarchy view is that in our model test results are a series of reports that aren't necessarily hierarchical. In other words, they don't necessarily form a nice tree. You can go up into the View menu and select which kinds of reports to look at, but I am a bit concerned that in large result sets this information is difficult to navigate. You can of course put results into files or database or whatever via other kinds of reporters.

One example of test results that wouldn't likely be hierarchical is a multi-threaded stress test, in which results are coming from multiple threads at the same time to a single Reporter.

Also, in JUnit, (I'm not actually positive this is the case, but it is my current best guess) the hierarchy view seems to assume the suite of tests will be a hierarchy, and it somehow matches up the results with the hierarchy of test cases. Whether that's true or not, in our model, once again, the suite of tests itself does not necessarily form a nice tree. As I described in my previous post to this topic, you can override execute and do whatever the heck you want. You could read data from an XML file and call the same test method multiple times, for example. So I can't look at the Suites themselves to get a hierarchy, I would have to find a hierarchy in the results, which are not guaranteed to be a well-formed tree.

Nevertheless, it did occur to me that I often test results will be single threaded and actually in practice a well-formed tree. So we could potentially add a tree view for results that is useful most of the time. Before I attemped to do that, I wanted to see how many people complained. I think there is room for improvement in the GUI for navigating results. I'm not sure whether a tree view would help, or if there's a better solution. If any of you have ideas or preferences, please post them to this forum.

Flat View: This topic has 3 replies on 1 page
Topic: Bug report? Suites with more than 20 test methods in them seem to hang. Previous Topic   Next Topic Topic: tearDownFixture not executed when a test fails.  Bug or

Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2019 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use