Weblogs Forum - A Set of Unit Testing Rules

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Weblogs Forum
A Set of Unit Testing Rules

50 replies on 4 pages. Most recent reply: Jan 21, 2011 2:19 AM by Steve Merrick

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 50 replies on 4 pages [ « | 1 2 3 4 | » ]

Ingo Lundberg

Posts: 2
Nickname: lundbergi
Registered: Jun, 2005

Re: A Set of Unit Testing Rules

Posted: Sep 19, 2005 5:58 AM

Hi Michael.

Just curious. Are you a "mock/interaction UTer" or a "state check UTer"? (Fowler talks about these two at http://www.martinfowler.com/articles/mocksArentStubs.html)

I've been shifting from state-based testing to interaction testing lately. Could it be that interaction based UTing makes it easier to conform to your rules? I'm leaning toward that conclusion.

Regards,
Ingo

Jay Flowers

Posts: 369
Nickname: jflowers
Registered: Sep, 2005

Re: A Set of Unit Testing Rules

Posted: Sep 19, 2005 6:10 AM

I have a related post here http://jflowers.blogspot.com/2005/09/understanding-unit-testing.html.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: A Set of Unit Testing Rules

Posted: Sep 19, 2005 6:19 AM

> Just curious. Are you a "mock/interaction UTer" or a
> "state check UTer"? (Fowler talks about these two at
> http://www.martinfowler.com/articles/mocksArentStubs.html)
>
> I've been shifting from state-based testing to interaction
> testing lately. Could it be that interaction based UTing
> makes it easier to conform to your rules? I'm leaning
> toward that conclusion.

I tend more toward state checking. I've sat with a number of people who do interaction style testing and often the designs go too Demeter-ish for my tastes. But that said, it's something I feel I have to do more of to appreciate. I'm good friends with some of the people who were on the team that triggered the mock objects paper and every time I see them I talk about how I don't see it yet.

With regard to the rules, I think they really push stubs at the system boundaries but allow objects to use other objects internally. I guess I'm a state based tester if I say that I use stubs (mocks) only as a concrete response to pain.

Obie Fernandez

Posts: 608
Nickname: obie
Registered: Aug, 2005

Re: A Set of Unit Testing Rules

Posted: Sep 19, 2005 6:40 AM

After a run-in with a team that egregiously violated some of the unit test guidelines cited in your article, Aslak and I wrote the following library:

http://ashcroft.codehaus.org/

Ashcroft is implemented as a custom Java security manager and causes tests that violate proper unit testing rules to fail. The current (and only) release is being used successfully at various ThoughtWorks client sites.

Erick Reid

Posts: 7
Nickname: erickreid
Registered: May, 2003

Re: A Set of Unit Testing Rules

Posted: Sep 19, 2005 10:08 AM

> causes tests that violate proper unit testing rules to
> fail.

that'll teach 'em! :)

Erick Reid

Posts: 7
Nickname: erickreid
Registered: May, 2003

Re: A Set of Unit Testing Rules

Posted: Sep 19, 2005 10:16 AM

did my best to scan all responses, not wanting to be redundant...

i had trouble with understanding unit v. functional in years past (perhaps still do, but it feels better nowadays).

a wacky coworker put this reminder to us, "if your unit test 'poops in the sandbox' it is not a unit test". meaning, if the unit test altered info in a db, then it is a functional test.

i was initially incredulous/confused, "but, but, but, i've <b>got</b> to see if it works or not!"

in order to keep the sandbox clean, and keep the unit tests true to intent, i've had a good time using mockrunner (http://mockrunner.sourceforge.net/).

Dave Nicolette

Posts: 6
Nickname: dnicolet
Registered: Sep, 2005

Re: A Set of Unit Testing Rules

Posted: Sep 23, 2005 9:00 AM

I'm with Michael Feathers and Ed Gibbs. The 'unit' under test shouldn't have any external dependencies. If it does, the test may be valid and necessary, but it isn't a unit test.

For Java, you have tools available like HSQLDB and DBUnit that will let you set up an in-memory database for a test case and extend JUnit to run tests against it. Works with straight JDBC or with o/r tools like Hibernate. There's no need (excuse?) to access an external database in a unit test. Doing so always seems to cause problems.

Network and filesystem accesses can be mocked or mimicked using stream classes, including stdout/stderr and including any external config files the class under test might access.

You can avoid most other dependencies, too. There are testing tools like smtpUnit and who knows how many others that let you mock various external services your application might be using.

Isolating the class under test isn't that hard to do, yields a lot of value, and keeps your unit tests from becoming fragile due to external dependencies outside your control.

It's worth the effort to get into these habits. Michael's checklist is right on the mark.

Dave Nicolette

Posts: 6
Nickname: dnicolet
Registered: Sep, 2005

Re: A Set of Unit Testing Rules

Posted: Sep 23, 2005 9:11 AM

> There is also a philosophical question you're touching on:
> When is testing a developer function, and when is it a
> software tester function?

It's not just a philosophical question. At our company we're involved right now in hashing out exactly what sort of testing ought to be done on development projects and whose responsibility each piece really is.

We're at a point in integrating Agile methods into the mainstream IT department where there are collisions between different groups and disagreement about fundamentals, and nailing down these issues is important to make sure the ball isn't dropped somewhere along the line.

For instance, developers tend to think they needn't take unit testing as far as UI testing, say with tools like httpUnit or htmlUnit. They consider it functional testing, and in the jurisdiction of the testing group. But from a holistic point of view, code that had been unit tested up to the UI layer would certainly be more solid and functional testing would go more smoothly. We've already had problems with sloppy output from webapps that would have been nipped in the bud had developers taken unit testing just one more layer up.

Unit testing at that level can follow Michael's guidelines quite easily and successfully. There's no need to unit test with a real browser any more than there is with a real file or a real database or a real mail server.

Anyway, I understand Michael's comments as pertaining specifically to unit testing, which is clearly a developer function. Some of the other points people have raise are perfectly valid, but they don't pertain specifically to unit testing.

Rick O'Shay

Posts: 2
Nickname: velocity
Registered: Apr, 2006

Re: A Set of Unit Testing Rules

Posted: Apr 13, 2006 7:01 PM

The unit test definition you are suggesting has a rather questionable foundation. The definition cherry picks some common input/output services but in effect you are saying "speed" is a definiting characteritic of a unit test.

The execution speed determines whether the unit test is slow or fast, not whether it is a unit test or not. What you seem to be leaning on for your definition is test independence. A test that does database access is not independent. Everything has dependencies at some level so that's not a criterion with much practical value either.

A unit test should not depend on other unit tests. You define what a "unit" is and you may very well have several levels of granularity. Functional tests are tests from the user's perspective, be that a human being or another system:

http://www-128.ibm.com/developerworks/library/j-test.html

Side note: the comment was made that the XP solution to slow tests was to optimize the tests. Think about that. Is it the test or the UUT (unit under test) that is slow? Refactoring is not a unit test speed-up process.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: A Set of Unit Testing Rules

Posted: Apr 13, 2006 7:32 PM

> The unit test definition you are suggesting has a rather
> questionable foundation. The definition cherry picks some
> common input/output services but in effect you are saying
> "speed" is a defining characteristic of a unit test.

Just because people already know many of the others.

> The execution speed determines whether the unit test is
> slow or fast, not whether it is a unit test or not. What
> you seem to be leaning on for your definition is test
> independence. A test that does database access is not
> independent. Everything has dependencies at some level so
> that's not a criterion with much practical value either.
>
> A unit test should not depend on other unit tests. You
> define what a "unit" is and you may very well have several
> levels of granularity. Functional tests are tests from the
> user's perspective, be that a human being or another
> system:
>
> http://www-128.ibm.com/developerworks/library/j-test.html
>
>
> Side note: the comment was made that the XP solution to
> slow tests was to optimize the tests. Think about that. Is
> it the test or the UUT (unit under test) that is slow?
> Refactoring is not a unit test speed-up process.

Yes, refactoring is not optimization. The fact is this "slow test" problem is a big deal with many teams. And I can't remember a case where the speed of the code under test was an issue and the code didn't do something with an external system (network, file system, database). I know it can happen, but I haven't run into a team with the problem of compute-intensive unit tests. External system access, leading to glacially slow unit tests, seems to be a much more popular way to fail.

Rupert Smith

Posts: 1
Nickname: rupertlssm
Registered: May, 2006

Re: A Set of Unit Testing Rules

Posted: May 24, 2006 3:55 PM

I've encountered this a few times. A code base running under cruise control on continuous integration that slowly grinds to a halt as the tests take longer and longer to run. In one situation a once healthy build system started taking 2 hours to run; and the blame could be placed on too much integration style testing (starting J2EE containers, deploying and then testing through cactus).

Generally, I now seperate my unit tests into two levels. Pure unit tests and integration unit tests, just like you describe in your article. In one situation though, some real pure tests that were doing some fairly exhaustive testing on a lot of different combinations of options were taking too long to run, so they got moved into the integration test section. Sometimes things from the integration test section that are quick and easy and considered important to run on every build get moved into the pure section.

I run the pure level on continous build (many times a day). The integration level gets run less frequently (say, once a day). I don't really care about the exactness of what is pure and what is not, the real seperation is more pragmatic; its just what is fast and what is slow. But as you rightfully point out these two definitions of how to split the tests are usually the same thing. Generally this keeps the continous build quick and encourages developers to write real pure unit tests.

A further disadvantage of the integration style tests is that they are, in my experience, more likely to fail. Pure unit tests once passing tend to stay that way (until the code is altered). Integration tests are a bit more risky because maybe some external resource is not working. This is annoying, especially as they get run only once a day because the interval between fixing them and re-running them is longer. Generally, I let my daily build pass even with broken integration style tests but report the errors to the developers but then take a stricter view at certain times. For example, nearing the end of a development cycle I will start to enforce the integration style tests.

Karar gulfesolutions.com

Posts: 1
Nickname: gulfesolut
Registered: Aug, 2006

Re: A Set of Unit Testing Rules

Posted: Aug 1, 2006 8:25 AM

Sir,

An application, which interacts with a database quite often (several times within each method), which unit test program should be used?
If we are required to write our own customised program for unit testing, how are we required to write such program, please suggest.

Regards,

Karar Haider

GulfeSolutions
http://www.gulfesolutions.com

John Doe

Posts: 5
Nickname: mrpantsuit
Registered: Oct, 2006

Re: A Set of Unit Testing Rules

Posted: Oct 30, 2006 8:40 AM

As others have stated in this topic, your DB access should be encapsulated behind some interface, e.g., MyDatabaseService or MyDAO (DAO = Data Access Object). The methods in your app that access the DB will do so through this interface, and as such can be mocked/stubbed (I recommend EasyMock for this). If you DI (dependency injection, a.k.a. IOC, inversion of control), changing your methods to use the mock/stub instead of the real DB-accessing implementation of MyDatabaseService is easy (I recommend Spring for the DI).

When unit testing the implementation of MyDatabaseService itself, use an in-memory DB, e.g., HSQLDB. There is no reason unit tests should ever access a stateful DB; this is what integration tests are for. If you cannot write unit tests without accessing an external resource like a DB, you need to refactor your app code.

DI and the practice of dividing your app up into modules/services that all have generic interfaces is a wonderful enabler of proper "pure" unit testing. Go to Spring's website for examples of these practices.

John Doe

Posts: 5
Nickname: mrpantsuit
Registered: Oct, 2006

Re: A Set of Unit Testing Rules

Posted: Oct 30, 2006 9:01 AM

I beleive concurrency unit testing deserves a special mention here. I have written concurrency unit tests (CUTs) that are "pure" according to your rules, but still require a non-trivial running time.

The unfortunate nature of CUTs is that they are non-deterministic, i.e., one iteration does not prove that the code is correct, as it does for a regular unit test. Uncovering concurrency bugs requires hitting the code with multiple threads for a sufficient length of time. (Even then, correctness can only be stated in terms of probability, e.g., I have hit this code with 100 threads for an hour, so it is highly probable that there are no concurrency bugs.)

Some might argue that CUTs are load tests, but I beleive that the purpose of a unit test is to verify the correctness of a unit of code, and a unit of concurrent code cannot be verified without CUTs.

Perhaps a fast/slow categorization of tests is more useful than a pure/impure one for the purposes of deciding run frequency. Although pure tests are typically fast and impure can often be slow, the inverse is not uncommon.

John Doe

Posts: 5
Nickname: mrpantsuit
Registered: Oct, 2006

Re: A Set of Unit Testing Rules

Posted: Oct 30, 2006 9:11 AM

BTW, I figured out recently that just because you hit code with multiple threads for a significant period of time doesn't mean that there are no concurrency bugs. Under testing conditions, the threads will often always be interrupted at the same points in the code. I found a library by IBM called ConTest (bad name) that instruments code so that the interruption points will vary on each iteration. I added a task to my Ant scripts that instruments my concurrent code before running my unit tests.

Flat View: This topic has 50 replies on 4 pages [ « | 1 2 3 4 | » ]

Previous Topic

Next Topic