Summary
Stelligent has published the second podcast in their monthly series on early software quality. This podcast introduces test categorization as a technique for reducing software build durations.
The basic idea of test categorization is to segregate tests based on how long they take to run, and run the fast ones more often than the slow ones. This gives the benefit that you actually run the fast ones more often, i.e., you don't not run the fast ones because they are lumped in with the slow ones, and the whole bundle of tests is taking too long. Categorizing tests is easy with NUnit or TestNG, using annotations. With JUnit it can be done with naming conventions or directory partitioning.
Andy Glover, President of Stelligent, identifies three categories of tests in this podcast:
Unit Tests - verifies behavior of a single class with no dependencies anything heavy. Unit tests should run to completion in about a fraction of a second and should be run any time a build is run.
Component [integration] Tests - tests portions of a system with more limited sets of external dependencies like a database of file system. ... They take a bit longer to run and they should be run at regular intervals.
Systems Tests - exercises a complete software system. Because they exercise an entire system, they are often created towards the latter cycles in development. Also, these tests have tendencies to have lengthy runtimes in addition to prolonged setup times, i.e., deploying a WAR to a web container that has a database going.
Do you categorize tests? Do you have any other strategies for dealing with long-running tests?
I like categorizing tests along two parallel axes: who owns them, and what are the goals the tests try to achieve.
Who, as development, has goals ranging from ensuring the developer's understanding matches the implementation (unit-level tests) to allowing groups of classes to work together with no dependencies beyond memory and a "runtime" (component- or boundary-level tests) to putting together those pieces in such a way as to depend on some subset of a potential deployment environment (integration-level tests).
When who is QA, they focus tests supporting their v&v activities on ensuring compliance of the implementation with the specification, within specific deployment environments (correctness- or functional-level tests). Sometimes performance of the entire system under load has its own set of tests (stress- or load-style tests).
For the customer or its liason, the focus is almost exclusively on satisfaction with the system as a whole (acceptance-level tests). Of course, development staff often execute the action of programming the tests for QA and customers, but the ownership of what is being tested is not transferred along with the logistics of creating them.
There's other breakdowns, slews of other test types, and differing definitions of these tests, obviously, but I usually get good traction when I address the concerns each ownership group has alongside the techniques used to build the different types of tests.
Previous Topic
