Weblogs Forum - A Set of Unit Testing Rules

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Weblogs Forum
A Set of Unit Testing Rules

50 replies on 4 pages. Most recent reply: Jan 21, 2011 2:19 AM by Steve Merrick

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 50 replies on 4 pages [ « | 1 2 3 4 | » ]

Matt Gerrans

Posts: 1153
Nickname: matt
Registered: Feb, 2002

Re: A Set of Unit Testing Rules

Posted: Sep 14, 2005 8:39 PM

> Teams can create whatever rules they want or name things whatever
> way they wish to.

I'm not sure I agree with this. Communication across teams and the industry is important too. Why muddy the waters by everybody defining things as they like? The whole idea of design patterns is to apply a shared vocabulary to common patterns in code. Maybe unit tests are just a design pattern...

I think you should stick to your strict definition, but I think these other kinds of tests need a good name. What if you are developing an OS, a database, etc.? You need to have plenty of tests, whether you call them unit or something else. How would you do TTD development on these things if you were limited to unit tests? Or is TDD not limited to unit tests?

Matt Gerrans

Posts: 1153
Nickname: matt
Registered: Feb, 2002

Re: A Set of Unit Testing Rules

Posted: Sep 14, 2005 8:48 PM

> But in software, we have a wrinkle to contend with:
> What, exactly, is "one part"? Some say it's a single
> method, however, I've seen methods that were pages long.
> This brings us to the concept of Test Driven Design;
> refactor that big method into many small methods that
> each do only one thing, and they become easier to test.

What if you are coding outside the ivory tower? Suppose you are working at Microsoft and your assignment is to fix a bug in CreateFile() (http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createfile.asp), whose declaration looks like this:

HANDLE CreateFile(
  LPCTSTR lpFileName,
  DWORD dwDesiredAccess,
  DWORD dwShareMode,
  LPSECURITY_ATTRIBUTES lpSecurityAttributes,
  DWORD dwCreationDisposition,
  DWORD dwFlagsAndAttributes,
  HANDLE hTemplateFile
);

First, you'd probably like to write some unit tests (well, I guess that would be impossible, since any test of this function would not be a "unit test" by the definition proposed here), to make sure you don't break the existing behavior.

Refactoring to a lot of little functions is not an option here -- that would break thousands and thousands of applications.

James Tikalsky

Posts: 12
Nickname: jt2190
Registered: Dec, 2003

Re: A Set of Unit Testing Rules

Posted: Sep 15, 2005 10:54 AM

> What if you are coding outside the ivory tower? Suppose
> you are working at Microsoft and your assignment is to fix
> a bug in CreateFile()
> (http://msdn.microsoft.com/library/default.asp?url=/library
> /en-us/fileio/fs/createfile.asp), whose declaration looks
> like this:
>

HANDLE CreateFile(
>   LPCTSTR lpFileName,
>   DWORD dwDesiredAccess,
>   DWORD dwShareMode,
>   LPSECURITY_ATTRIBUTES lpSecurityAttributes,
>   DWORD dwCreationDisposition,
>   DWORD dwFlagsAndAttributes,
>   HANDLE hTemplateFile
> );

> First, you'd probably like to write some unit tests (well,
> I guess that would be impossible, since any test of
> this function would not be a "unit test" by the definition
> proposed here), to make sure you don't break the existing
> behavior.
>
> Refactoring to a lot of little functions is not an option
> here -- that would break thousands and thousands of
> applications.

If by "Refactoring to a lot of little functions" you're thinking that we'd change the method signature, then you're right. We'd be breaking the existing API, which is a no-no.

However, as developer/bug-fixer, what is to stop us from extracting the buggy piece of code inside of CreateFile() into it's own function or functions, that we could then unit test more easily? (And if the reason in this example is I/O performance, does that mean that this approach to refactoring/writing tests would NEVER work?)

Gregg Wonderly

Posts: 317
Nickname: greggwon
Registered: Apr, 2003

Re: A Set of Unit Testing Rules

Posted: Sep 17, 2005 4:45 AM

> A test is not a unit test if:
> <li>It talks to the database
> <li>It communicates across the network
> <li>It touches the file system
> <li>It can't run at the same time as any of your other
> unit tests
> <li>You have to do special things to your environment
> (such as editing
> config files) to run it.

If you don't know Peter Deutches 8 fallacies of distributed computing, then you'd probably feel really good about not testing your applications ability to handle partial failure. It may feel like it's slowing you down early on, but at some point, you really need to have real world tests that put your application under really heavy loads to test concurrent data structure access. You also need to have fault injection capabilities to stress the failure handling code to make sure idempotent behaviors are true to form. There's lots of reasons to interface to a real-world environment. It might not be in your unit tests, but you should have a plan, during design and coding on how you are going to instrument and inject for real-world testing.

James Tikalsky

Posts: 12
Nickname: jt2190
Registered: Dec, 2003

Re: A Set of Unit Testing Rules

Posted: Sep 17, 2005 10:02 AM

> ... you'd probably feel really
> good about not testing your applications ability to handle
> partial failure. It may feel like it's slowing you down
> early on, but at some point, you really need to have real
> world tests that put your application under really heavy
> loads to test concurrent data structure access.
> You also
> need to have fault injection capabilities to stress the
> failure handling code to make sure idempotent behaviors
> are true to form. There's lots of reasons to interface to
> a real-world environment. It might not be in your unit
> tests...

You're right (and this is Michael's point) when you say that you don't want load tests or integration tests running with your unit tests. Imaginge that every time you wanted to compile you had to set up a test database, some network services, etc., etc. You'd probably be able to compile once a day!

There is also a philosophical question you're touching on: When is testing a developer function, and when is it a software tester function?

Unit tests are meant to be written by a developer, and run numerous times a day, as the developer writes new code. The tests are a way for the developer to assert that the code they've written works the way they think it should. Obviously, the tests will be incomplete unless the developer thinks of every test case up front, and they often don't. Typically, the developer adds these new tests as he fixes the bugs found by the software tester.

A software tester will devote themselves to the more complex testing tasks, including setting up test environments that mimic real-world conditions. (Note that when I say software tester, I'm talking about someone who is as adept at breaking code as the developer is a writing it. I'm not talking about someone who's never programmed in their life, and who just clicks buttons all day.) Their tests will be far more complete than the developer's unit tests, and will be the true test of the software's quality.

As tools become more powerful, developers will find themselves with more time to do "true" testing, so the line between developer and software tester is blurring. Ultimately, I think it'd be great if more developers really understood the various ways their code could break, and I think that developing unit tests may help them learn this. For now, most programmers don't have the testers mindset and so a good software tester is necessary.

> ... but you should have a plan, during design and
> coding on how you are going to instrument and inject for
> real-world testing.

I think those who advocate TDD would say that code that is easy to unit test is also easier to test in other ways.

Ingo Lundberg

Posts: 2
Nickname: lundbergi
Registered: Jun, 2005

Re: A Set of Unit Testing Rules

Posted: Sep 19, 2005 5:58 AM

Hi Michael.

Just curious. Are you a "mock/interaction UTer" or a "state check UTer"? (Fowler talks about these two at http://www.martinfowler.com/articles/mocksArentStubs.html)

I've been shifting from state-based testing to interaction testing lately. Could it be that interaction based UTing makes it easier to conform to your rules? I'm leaning toward that conclusion.

Regards,
Ingo

Jay Flowers

Posts: 369
Nickname: jflowers
Registered: Sep, 2005

Re: A Set of Unit Testing Rules

Posted: Sep 19, 2005 6:10 AM

I have a related post here http://jflowers.blogspot.com/2005/09/understanding-unit-testing.html.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: A Set of Unit Testing Rules

Posted: Sep 19, 2005 6:19 AM

> Just curious. Are you a "mock/interaction UTer" or a
> "state check UTer"? (Fowler talks about these two at
> http://www.martinfowler.com/articles/mocksArentStubs.html)
>
> I've been shifting from state-based testing to interaction
> testing lately. Could it be that interaction based UTing
> makes it easier to conform to your rules? I'm leaning
> toward that conclusion.

I tend more toward state checking. I've sat with a number of people who do interaction style testing and often the designs go too Demeter-ish for my tastes. But that said, it's something I feel I have to do more of to appreciate. I'm good friends with some of the people who were on the team that triggered the mock objects paper and every time I see them I talk about how I don't see it yet.

With regard to the rules, I think they really push stubs at the system boundaries but allow objects to use other objects internally. I guess I'm a state based tester if I say that I use stubs (mocks) only as a concrete response to pain.

Obie Fernandez

Posts: 608
Nickname: obie
Registered: Aug, 2005

Re: A Set of Unit Testing Rules

Posted: Sep 19, 2005 6:40 AM

After a run-in with a team that egregiously violated some of the unit test guidelines cited in your article, Aslak and I wrote the following library:

http://ashcroft.codehaus.org/

Ashcroft is implemented as a custom Java security manager and causes tests that violate proper unit testing rules to fail. The current (and only) release is being used successfully at various ThoughtWorks client sites.

Erick Reid

Posts: 7
Nickname: erickreid
Registered: May, 2003

Re: A Set of Unit Testing Rules

Posted: Sep 19, 2005 10:08 AM

> causes tests that violate proper unit testing rules to
> fail.

that'll teach 'em! :)

Erick Reid

Posts: 7
Nickname: erickreid
Registered: May, 2003

Re: A Set of Unit Testing Rules

Posted: Sep 19, 2005 10:16 AM

did my best to scan all responses, not wanting to be redundant...

i had trouble with understanding unit v. functional in years past (perhaps still do, but it feels better nowadays).

a wacky coworker put this reminder to us, "if your unit test 'poops in the sandbox' it is not a unit test". meaning, if the unit test altered info in a db, then it is a functional test.

i was initially incredulous/confused, "but, but, but, i've got to see if it works or not!"

in order to keep the sandbox clean, and keep the unit tests true to intent, i've had a good time using mockrunner (http://mockrunner.sourceforge.net/).

Dave Nicolette

Posts: 6
Nickname: dnicolet
Registered: Sep, 2005

Re: A Set of Unit Testing Rules

Posted: Sep 23, 2005 9:00 AM

I'm with Michael Feathers and Ed Gibbs. The 'unit' under test shouldn't have any external dependencies. If it does, the test may be valid and necessary, but it isn't a unit test.

For Java, you have tools available like HSQLDB and DBUnit that will let you set up an in-memory database for a test case and extend JUnit to run tests against it. Works with straight JDBC or with o/r tools like Hibernate. There's no need (excuse?) to access an external database in a unit test. Doing so always seems to cause problems.

Network and filesystem accesses can be mocked or mimicked using stream classes, including stdout/stderr and including any external config files the class under test might access.

You can avoid most other dependencies, too. There are testing tools like smtpUnit and who knows how many others that let you mock various external services your application might be using.

Isolating the class under test isn't that hard to do, yields a lot of value, and keeps your unit tests from becoming fragile due to external dependencies outside your control.

It's worth the effort to get into these habits. Michael's checklist is right on the mark.

Dave Nicolette

Posts: 6
Nickname: dnicolet
Registered: Sep, 2005

Re: A Set of Unit Testing Rules

Posted: Sep 23, 2005 9:11 AM

> There is also a philosophical question you're touching on:
> When is testing a developer function, and when is it a
> software tester function?

It's not just a philosophical question. At our company we're involved right now in hashing out exactly what sort of testing ought to be done on development projects and whose responsibility each piece really is.

We're at a point in integrating Agile methods into the mainstream IT department where there are collisions between different groups and disagreement about fundamentals, and nailing down these issues is important to make sure the ball isn't dropped somewhere along the line.

For instance, developers tend to think they needn't take unit testing as far as UI testing, say with tools like httpUnit or htmlUnit. They consider it functional testing, and in the jurisdiction of the testing group. But from a holistic point of view, code that had been unit tested up to the UI layer would certainly be more solid and functional testing would go more smoothly. We've already had problems with sloppy output from webapps that would have been nipped in the bud had developers taken unit testing just one more layer up.

Unit testing at that level can follow Michael's guidelines quite easily and successfully. There's no need to unit test with a real browser any more than there is with a real file or a real database or a real mail server.

Anyway, I understand Michael's comments as pertaining specifically to unit testing, which is clearly a developer function. Some of the other points people have raise are perfectly valid, but they don't pertain specifically to unit testing.

Rick O'Shay

Posts: 2
Nickname: velocity
Registered: Apr, 2006

Re: A Set of Unit Testing Rules

Posted: Apr 13, 2006 7:01 PM

The unit test definition you are suggesting has a rather questionable foundation. The definition cherry picks some common input/output services but in effect you are saying "speed" is a definiting characteritic of a unit test.

The execution speed determines whether the unit test is slow or fast, not whether it is a unit test or not. What you seem to be leaning on for your definition is test independence. A test that does database access is not independent. Everything has dependencies at some level so that's not a criterion with much practical value either.

A unit test should not depend on other unit tests. You define what a "unit" is and you may very well have several levels of granularity. Functional tests are tests from the user's perspective, be that a human being or another system:

http://www-128.ibm.com/developerworks/library/j-test.html

Side note: the comment was made that the XP solution to slow tests was to optimize the tests. Think about that. Is it the test or the UUT (unit under test) that is slow? Refactoring is not a unit test speed-up process.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: A Set of Unit Testing Rules

Posted: Apr 13, 2006 7:32 PM

> The unit test definition you are suggesting has a rather
> questionable foundation. The definition cherry picks some
> common input/output services but in effect you are saying
> "speed" is a defining characteristic of a unit test.

Just because people already know many of the others.

> The execution speed determines whether the unit test is
> slow or fast, not whether it is a unit test or not. What
> you seem to be leaning on for your definition is test
> independence. A test that does database access is not
> independent. Everything has dependencies at some level so
> that's not a criterion with much practical value either.
>
> A unit test should not depend on other unit tests. You
> define what a "unit" is and you may very well have several
> levels of granularity. Functional tests are tests from the
> user's perspective, be that a human being or another
> system:
>
> http://www-128.ibm.com/developerworks/library/j-test.html
>
>
> Side note: the comment was made that the XP solution to
> slow tests was to optimize the tests. Think about that. Is
> it the test or the UUT (unit under test) that is slow?
> Refactoring is not a unit test speed-up process.

Yes, refactoring is not optimization. The fact is this "slow test" problem is a big deal with many teams. And I can't remember a case where the speed of the code under test was an issue and the code didn't do something with an external system (network, file system, database). I know it can happen, but I haven't run into a team with the problem of compute-intensive unit tests. External system access, leading to glacially slow unit tests, seems to be a much more popular way to fail.

Flat View: This topic has 50 replies on 4 pages [ « | 1 2 3 4 | » ]

Previous Topic

Next Topic