The Artima Developer Community
Sponsored Link

Weblogs Forum
What are Your JUnit Pain Points, Really?

65 replies on 5 pages. Most recent reply: Nov 13, 2009 5:01 PM by Nemanja Trifunovic

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 65 replies on 5 pages [ « | 1 2 3 4 5 | » ]
Michael Hobbs

Posts: 51
Nickname: hobb0001
Registered: Dec, 2004

Re: What are Your JUnit Pain Points, Really? Posted: Oct 28, 2009 11:15 AM
Reply to this message Reply
Advertisement
> I'm curious: How do others doing TDD deal with this?

This is what I was implying with my statement that
> Truly valuable testing, however, often requires a fair amount of integration testing.

I see that Nemanja Trifunovic has also said as much, but I just wanted to reiterate that one of JUnit's biggest pain points is what is being commonly sold as "unit testing".

Frank Sommers

Posts: 2642
Nickname: fsommers
Registered: Jan, 2002

Re: What are Your JUnit Pain Points, Really? Posted: Oct 28, 2009 12:26 PM
Reply to this message Reply
> > I'm curious: How do others doing TDD deal with this?
>
> This is what I was implying with my statement that
> > Truly valuable testing, however, often requires a fair
> amount of integration testing.
>
> I see that Nemanja Trifunovic has also said as much, but I
> just wanted to reiterate that one of JUnit's biggest pain
> points is what is being commonly sold as "unit testing".

That's an interesting point. Are you, in effect, saying that unit testing is overly emphasized, and at the expense of other forms of testing?

Michael Hobbs

Posts: 51
Nickname: hobb0001
Registered: Dec, 2004

Re: What are Your JUnit Pain Points, Really? Posted: Oct 28, 2009 4:14 PM
Reply to this message Reply
> Are you, in effect, saying that unit testing is overly
> emphasized, and at the expense of other forms of testing?

Exactly. Unit testing, in the strictest sense of the term, is rather good at finding bugs in the sort of self-contained code that is typically used to implement algorithms and data structures. I'd wager, however, that very few modern programmers spend much time on that sort of code any more, since most of that functionality is now provided by off-the-shelf components. (e.g., HashMap, ArrayList, RDBMS, etc.)

Speaking for myself, most of my time is spent connecting together multiple non-deterministic and lightly documented systems. (For example, creating a web app that connects humans to a legacy document management system.) Most of the bugs that crop up result from when the systems combine in unique and unexpected ways.

You could try to isolate each individual variable that contributed to the bug and then unit test each variable independently, but it's usually more fruitful to create a regression test reproduces the whole interaction. First, it tends to be 10x less work and just as effective. Second, future development will likely add, change, and remove variables in the interaction. When that happens and you have a bunch of small unit tests, you will need to update and re-validate a bunch of small unit tests, with no guarantee that the interaction as a whole will succeed as a result.

Frank Sommers

Posts: 2642
Nickname: fsommers
Registered: Jan, 2002

Re: What are Your JUnit Pain Points, Really? Posted: Oct 29, 2009 1:51 AM
Reply to this message Reply
> Speaking for myself, most of my time is spent connecting
> together multiple non-deterministic and lightly documented
> systems. (For example, creating a web app that connects
> humans to a legacy document management system.) Most of
> the bugs that crop up result from when the systems combine
> in unique and unexpected ways.
>
> You could try to isolate each individual variable that
> contributed to the bug and then unit test each variable
> independently, but it's usually more fruitful to create a
> regression test reproduces the whole interaction. First,
> it tends to be 10x less work and just as effective.
> Second, future development will likely add, change, and
> remove variables in the interaction. When that happens and
> you have a bunch of small unit tests, you will need to
> update and re-validate a bunch of small unit tests, with
> no guarantee that the interaction as a whole will succeed
> as a result.

I agree with that. My experience has also been that most of problems crop up in the kind of situations you describe.

But, most importantly, I painted my self in the unit test corner several times, where I had so many unit tests that I loathed to make big refactorings to improve design. These days I tend to use ScalaTest to create high-level tests and seldom bother to test low-level methods. Interestingly, though, this still allows me to maintain good test coverage.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: What are Your JUnit Pain Points, Really? Posted: Oct 29, 2009 9:34 AM
Reply to this message Reply
> > Are you, in effect, saying that unit testing is overly
> > emphasized, and at the expense of other forms of
> testing?
>
> Speaking for myself, most of my time is spent connecting
> together multiple non-deterministic and lightly documented
> systems. (For example, creating a web app that connects
> humans to a legacy document management system.) Most of
> the bugs that crop up result from when the systems combine
> in unique and unexpected ways.

Right on.

Do you (or anyone) have any recommendations for tools that assist with this? I've handrolled a couple but I've come up empty-handed when trying to find something better: easier to create new tests, understand existing tests, maintain existing tests. I also want to explore mocking in this context for which I know there are options but any pointers are welcome. I don't want a QA type tool either. I don't need tools that capture mouse clicks on anything.

Cedric Beust

Posts: 140
Nickname: cbeust
Registered: Feb, 2004

Re: What are Your JUnit Pain Points, Really? Posted: Oct 29, 2009 11:03 AM
Reply to this message Reply
Hi Frank,

> That's an interesting point. Are you, in effect, saying
> that unit testing is overly emphasized, and at the expense
> of other forms of testing?

This has also been my experience, although to be honest, I see this problem more in agile/XP literature than in the real world.

This is the reason why I claim that:

- TDD encourages micro-design over macro-design
- TDD generates code churn

If you obsessively do TDD, you write tests for code that you are pretty much guaranteed to throw away. And when you do that, you will have to refactor your tests or rewrite them completely. Whether this refactoring can be done automatically or not is beside the point: you are in effect creating more work for yourself.

When I start solving a problem, I like to iterate two or three times on my code before I'm comfortable enough to write a test.

Another important point is that unit tests are a convenience for *you*, the developer, while functional tests are important for your *users*. When I have limited time, I always give priority to writing functional tests. Your duty is to your users, not to your test coverage tools.

You also bring up another interesting point: overtesting can lead to paralysis. I can imagine reaching a point where you don't want to modify your code because you will have too many tests to update (especially in dynamically typed languages, where you can't use tools that will automate this refactoring for you). The lesson here is to do your best so that your tests don't overlap.

--
Cedric

Mark Thornton

Posts: 275
Nickname: mthornton
Registered: Oct, 2005

Re: What are Your JUnit Pain Points, Really? Posted: Oct 29, 2009 5:09 PM
Reply to this message Reply
>
> Another important point is that unit tests are a
> convenience for *you*, the developer, while functional
> tests are important for your *users*. When I have limited
> time, I always give priority to writing functional tests.
> Your duty is to your users, not to your test coverage
> tools.
>

I often use JUnit to test rather larger assemblies of code --- to big to be considered units --- and closer to functional testing. This sometimes leads to issues with the runtime environment, such as requiring a larger heap than the default.

Anders Grusell

Posts: 4
Nickname: rimfrost
Registered: Oct, 2007

Re: What are Your JUnit Pain Points, Really? Posted: Oct 30, 2009 3:36 AM
Reply to this message Reply
From wikipedia: "Code refactoring is the process of changing a computer program's internal structure without modifying its external functional behavior or existing functionality"
I.e, if you have to rewrite your tests frequently, this might be a sign that you should test functionality/behaviour rather than implementation.

Eirik Maus

Posts: 15
Nickname: eirikma
Registered: Oct, 2005

Re: What are Your JUnit Pain Points, Really? Posted: Oct 30, 2009 6:59 AM
Reply to this message Reply
I second to that.

I try as soon as possible to create a very simple test for the high-level api (externally observed behaviour) that I create, but I usually have to create some of the code first to get a grip on what kind of classes I will have in the solution.

Later I add a few test for a slightly more difficult scenario and leave it to that. I usually only create tests for implementation details when I have problems making the (test for the) high-level api work correctly due to hard-to-find bugs. Particularily tricky implementation details sometimes get a test too, but generally such tests are only added as a means to make these work correctly for the high-level tests.

Also, I have a list if items that I don't write tests for:
* get/set operations. somebody else is in charge for controlling that assignment statements in java works.
* trivial linear logic (for instance some setup object instantiations) that are going to be executed as part of higher-level tests anyway.

Also: It is actually good advice to try and limit interconnections between systems, since it makes them very hard to test and maintain. Don't re-use running instances unless it is absolutely necessary, since that creates transitive maintainance challenges. Don't create program code that only works if the other system is up and running, since that will be notoriously difficult to test without affecting the other system. All kinds of security issues applies, too.

Often it can be simpler to, for instance, import reference data into local temporary tables/materialized views using database links or similar. That way, the program logic behaves as if the data is "magically available" in local database tables. For most testing, they are, too.

It should be possible to test on a local PC with access to nothing (little) else than a database and the local file system.

Michael Hobbs

Posts: 51
Nickname: hobb0001
Registered: Dec, 2004

Re: What are Your JUnit Pain Points, Really? Posted: Oct 30, 2009 11:13 AM
Reply to this message Reply
> Often it can be simpler to, for instance, import reference
> data into local temporary tables/materialized views using
> database links or similar. That way, the program logic
> behaves as if the data is "magically available" in local
> database tables. For most testing, they are, too.

And that is my biggest JUnit pain point. Recreating that magic interaction can be a real PITA. It would be nice if there was a tool that could magically record that data and play it back during testing.

Peter Niederwieser

Posts: 3
Nickname: pniederw
Registered: Dec, 2007

Re: What are Your JUnit Pain Points, Really? Posted: Oct 31, 2009 7:46 PM
Reply to this message Reply
I believe JUnit 4.8 will provides this. They call it "Categories".

Peter Niederwieser

Posts: 3
Nickname: pniederw
Registered: Dec, 2007

Re: What are Your JUnit Pain Points, Really? Posted: Oct 31, 2009 7:53 PM
Reply to this message Reply
My previous post was an answer to the following:

>Just yesterday i saw that the Ruby testing framework Cucumber allows you to add tags to tests. When you run a test suite you can use the tags to filter the tests to be executed. That would be nice to have in junit.

Cedric Beust

Posts: 140
Nickname: cbeust
Registered: Feb, 2004

Re: What are Your JUnit Pain Points, Really? Posted: Oct 31, 2009 11:09 PM
Reply to this message Reply
I'm not aware of any Category feature in JUnit, can you be more specific? (I see this in JUnitExt, though)

I'm surprised that JUnit is still not implementing this despite a few feature requests over the past years (because it's a feature that is useful even if you focus on unit testing specifically).

By the way, TestNG call these "groups", and they are by far the most popular feature of the framework.

Michele Simionato

Posts: 222
Nickname: micheles
Registered: Jun, 2008

Re: What are Your JUnit Pain Points, Really? Posted: Nov 2, 2009 9:01 AM
Reply to this message Reply
> > Speaking for myself, most of my time is spent
> connecting
> > together multiple non-deterministic and lightly
> documented
> > systems. (For example, creating a web app that connects
> > humans to a legacy document management system.) Most of
> > the bugs that crop up result from when the systems
> combine
> > in unique and unexpected ways.
>
> Right on.
>

Interesting that everybody here is saying the same thing
and had the same experience (me too, of course).
I remember 6-7 years ago, when the unit test religion was
at its apex: at that time nobody was talking about these
issues. It seems the religious phase is finally ended.

John Zabroski

Posts: 272
Nickname: zbo
Registered: Jan, 2007

Re: What are Your JUnit Pain Points, Really? Posted: Nov 2, 2009 9:43 PM
Reply to this message Reply
Cedric,

I read your Test-NG a few years ago when it first came out.

If I wanted to read a book on general library-based testing, and come up with a list of features, where would I go?

BTW, I found your book a refreshing read when I read it - I felt the same way, but unlike you didn't do anything about it.

Flat View: This topic has 65 replies on 5 pages [ « | 1  2  3  4  5 | » ]
Topic: We No Longer Need Power Previous Topic   Next Topic Topic: Is Scala really more complicated than Java?


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us