The Artima Developer Community
Sponsored Link

Weblogs Forum
Testivus - Testing For The Rest Of Us

51 replies on 4 pages. Most recent reply: Jul 25, 2007 5:43 PM by Ashwath Akirekadu

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 51 replies on 4 pages [ « | 1 2 3 4 | » ]
James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Testivus - Testing For The Rest Of Us Posted: Feb 7, 2007 6:12 PM
Reply to this message Reply
Advertisement
> One trick there was to
> breakpoint every possible branch in your app, so that the
> debugger would kick in whenever a new branch got hit, and
> you could see which bits hadnt been explored.

If coverage is defined to be executing each branch of the code, then I find it be a very minimal level of testing.

This is one of the things that I am having trouble with where I work now. Testing code is defined to be running through the different parts of it one time. That doesn't prove much other than that the code isn't completely broken. It definitely doesn't guarantee the code is correct or that it won't fall over under other (perhaps normal) circumstances.

Personally, I think unit testing is a good thing but it's much lower on my priority list than making sure things work. A program with broken units can perform flawlessly. A program with tested units can fail to work entirely. At the end of the day, what really matters is whether the application works, not whether the units work. I see far too much focus on units and far too little on holistic testing.

Mike Kaufman

Posts: 123
Nickname: mkaufman
Registered: Jan, 2007

Re: Testivus - Testing For The Rest Of Us Posted: Feb 7, 2007 7:00 PM
Reply to this message Reply
Hi Alberto,

Think you've mixed me up with someone called Morgan Conrad!!!

The quote "we need a 'testing for the rest of us' movement" is from my comment to your article, not from any Morgan Conrad (who made an entirely unrelated comment as the next one after mine). Same goes for the subsequent comments where you propose the "testivus" movement and I say I'm happy for you to take it forward but don't have the time to take part myself, and don't especially like the "testivus" name. For info, it was also me that said I "spend considerably more time writing tests than writing code" (in my case, I estimate it's 2-3 times as long, more in some cases).

Happy for you to take this forward and hope it proves successful, but please attribute the original "inspiration" and quotes correctly (not least so that nobody blames poor "Morgan Conrad" for it - the poor guy is completely innocent on this).

Regards,
Mike Kaufman

Tim Vernum

Posts: 58
Nickname: tpv
Registered: Dec, 2002

Re: Testivus - Testing For The Rest Of Us Posted: Feb 7, 2007 7:08 PM
Reply to this message Reply
> But since I haven't had anyone come to the defense
> of debugging - until now - I hadn't given much thought to
> it.

Well I think we all got sick or arguing the point against Robert Martin.
http://www.artima.com/weblogs/viewpost.jsp?thread=23476

> I don't like this approach because time invested in
> testing results in reusable assets, while time spent
> on the debugger in this way is typically spent chasing
> a particular problem and leaves no reusable artifacts.

No matter which method/tool you choose, you're still tracking down a bug and then you try to fix it. Writing tests doesn't fix bugs. It is one way to track down bugs, but it doesn't fix them. A debugger is another way to track down bugs. Sometimes it's a faster way. Sometimes it's slower.

I think we need to stop calling the tool a "debugger", and start calling it a "runtime analyser".

If someone performs testing on a piece of software and finds something that doesn't work, and then you go and try to make it work, that's debugging.
It doesn't matter whether you're using a testing tool, a "runtime analyser", printf statements, or some other tools, it's still debugging.
If doesn't matter whether the bug was found by an automated unit test, a fucntional test, a regression test, or exploratory tests. It doesn't matter whether you found it straight away after writing the code, or someone else found it 3 years later. The process of removing a defect is called debugging.

If I have a failing test, I quite often fire up the "runtime analyser" to help me work out why the test and the code don't match. When I know that, I can fix whichever one is wrong. Sometimes I will change my tests to catch this scenario a little more explicitly so that next time around (if the bug returns) the root cause will be more obvious and can be fixed more quickly.

The tool isn't the problem.
It doesn't matter [*] what tool people use to work out why the bug (i.e. failing test) is there. The question is what they do once they find it. If they simply fix the bug and then ship it, then they're setting themselves up to have the bug come back again. So they should make sure they have a test to cover that scenario.

[*] Well, some ways are more efficient than others, so it probably does matter to the person paying the bills, but it shouldn't matter in this argument.

I would make this one of the guiding principles of testing. Aim to fix each bug once only.

* Use whatever tool you want to find the bug.
* Use whatever tool you want to fix the bug.
* Use whatever tool you want to keep the bug from coming back.

I agree with Cedric that the suggestion that the debugger is redundant once you start doing automated testing (or TDD, or BDD, or whatever) is a harmful one. It is a useful tool that been specifically designed to allow you to observe the behaviour of a piece of software as it executes.
I think one of the greatest dangers is in people sticking to one tool exclusively (be that their testing library, their debugger, or their logging package) when there are a great number of tools available that can deliver value in different ways.

Bill Burris

Posts: 24
Nickname: billburris
Registered: Aug, 2003

Re: Testivus - Testing For The Rest Of Us Posted: Feb 7, 2007 7:15 PM
Reply to this message Reply
Another aspect of my code which doesn't lend itself to unit testing is the user interface. There is a lot of drawing code which is tested by visually inspecting the results.

Recently the user reported that the scaling on the graph was messed up when he maximized the window. I had tested this repeatedly, as I was tweaking the code to get good visual results when the window was resized. I assumed he broke something when he modified the code. As it turns out he was maximizing by double clicking the title bar and I had always done it with the maximize button. The debugger came in handy for tracking down the problem. I set break points everywhere I could think of where scaling changes were made to the graph. It turns out that the picture box in my program was catching a mouse up event at the tail end of the double click on the title bar. The window resize moved the picture box under the mouse cursor before Windows generated the mouse up message. It was a portion of my zoom to rubber band rectangle code which was catching the stray mouse up event.

My testing is usually done by visually inspecting the results. When things don't look right the debugger is used to track down the problem.

Testivus for me would be writing unit tests for the stuff that fills in between the instrumentation code and user interface code. Thinking about if a particular piece of code is testable or not, would help keep the 3 layers separate from each other.

Pair programming for me is: I work on the code during the day and the user works on the code during the evenings and weekends.

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: Testivus - Testing For The Rest Of Us Posted: Feb 7, 2007 7:46 PM
Reply to this message Reply
>Think you've mixed me up with someone called Morgan Conrad!!!

Hi Mike,

Right you are. How embarassing - I should have tested my post better.

I apologize to you and Morgan. I'll see if I can edit the original post, but you blew your chance for not being associated with Testivus :-)

Alberto

Morgan Conrad

Posts: 307
Nickname: miata71
Registered: Mar, 2006

Re: Testivus - Testing For The Rest Of Us Posted: Feb 7, 2007 10:39 PM
Reply to this message Reply
> >Think you've mixed me up with someone called Morgan
> Conrad!!!

Yes, I will confirm that it was Mike who pushed Alberto into this. :-)

My post in the earlier thread is still relevant. To rehash, for the "type 3" testers, you should make the tedious repetitive tests that they think are stupid dogma easy. (Frankly, I think some of these tests are stupid too)

A colleage wrote test helper code to use reflection to call all the setXXX and then getXXX, testing that get returned what you set. Saved writing tons of tedious tests that a Type 3 tester would probably argue are stupid. But, if you make it trivial to do, it's not so bad to write the test.

Another colleague wrote helper code to pass nulls to various methods that don't accept nulls, to verify that an IllegalArgumentException was thrown. Again, saved tons of tedious tests that a Type 3 tester might argue are stupid anyway.

I wrote code to make it easy do all the proper testing of equals(). It also tested hashCode(), and found several cases where somebody had overridden equals() but not hashCode().

Mike Kaufman

Posts: 123
Nickname: mkaufman
Registered: Jan, 2007

Re: Testivus - Testing For The Rest Of Us Posted: Feb 7, 2007 11:34 PM
Reply to this message Reply
Thanks Alberto (and Morgan),

Guess I'm stuck with it now, you'd better make a success of it :-)

A couple of intial thoughts:

- Not sure where this best fits into the manifesto (or if you think it's already covered), but I'd say a key point is that different situations may need different approaches - maybe the "some people...some time..." should also include "some situations".

- Beyond the manifesto, I'd hope testivus can develop some kind of definitive terminology for different approaches, tools and techniques - even if that includes having to explain any remaining disputes over meaning. For example, there seem to have been various attempts to define stub vs. mock vs. fake vs. dummy etc (and I have one of my own to add - "mimic"), but there still seems a lot of confusion and disagreement. A bigger one is people saying "unit test" and meaning entirely different things. Maybe it's intractible, but if testivus isn't pushing any one particular approach it might be ideally placed to tackle this. A genuinely shared terminology might help reduce some of the more pointless arguments that one always seems to find at the start or end of any discussion on testing. Ok, just a dream...

Mike

Guilherme Chapiewski

Posts: 1
Nickname: gc
Registered: Feb, 2007

Re: Testivus - Testing For The Rest Of Us Posted: Feb 8, 2007 5:25 AM
Reply to this message Reply
Great point of view, Alberto!

This is the best proposal I've seen in the last months :)

Steve Loughran

Posts: 9
Nickname: stevel
Registered: Feb, 2006

Re: Testivus - Testing For The Rest Of Us Posted: Feb 8, 2007 5:59 AM
Reply to this message Reply
> I certainly didn't say I was single stepping through every
> single line of code, but there are often a few critical
> parts of what I submit that I want to double check.

Seems reasonable. We sometimes use a bit of the old formal logic from time to time too, because neither debugging nor testing are adequate to be 100% sure that you don't have concurrency problems, but a bit of maths can give you some confidence.

> This might also be due to the fact that for the past three
> years, every single line of code that I check in gets
> reviewed by one of my peers, a constraint that probably
> very few developers face in real life. This kind of
> scrutiny really makes me want to make sure that my code
> looks *and* works well before I submit, regardless of the
> tests I submit.

well, code review is shown to be the best way to find bugs, and its not the peers, its the original author that find the problems.

-steve

disney

Posts: 35
Nickname: juggler
Registered: Jan, 2003

Re: Testivus - Testing For The Rest Of Us Posted: Feb 8, 2007 6:42 AM
Reply to this message Reply
>> Most developers who practice TDD/Test-First do it in
>> very quick test-code-test-code cycles:

8< (snip)

> OK I can see how that would work but I just really have
> doubts that it tests anything truly meaningful. You can
> only write trivial tests in a matter of minutes.

Yes, TFD is: write one simple test, then code until it passes. Repeat until all features of the method-under-test have been tested.

While it's true "you can only write trivial tests in a matter of minutes", it's sometimes surprising how well you can test your code with quite a few such tests. Just as you can generate flocking/swarming behaviour with two or three simple rules.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Testivus - Testing For The Rest Of Us Posted: Feb 8, 2007 9:08 AM
Reply to this message Reply
> While it's true "you can only write trivial tests in a
> matter of minutes", it's sometimes surprising how well you
> can test your code with quite a few such tests. Just as
> you can generate flocking/swarming behaviour with two or
> three simple rules.

I think a lot of code can be tested this way and that it's a great way to test that kind of code. My point is that this one size-fits-all model for testing doesn't make a lot of sense to me.

Take your flocking-swarming example. How many unit tests do you need to verify that the flocking-swarming works properly under all operating conditions for all numbers of entities from 2 to 2 million?

Again, I'm not arguing that unit-testing isn't helpful, I'm arguing that it isn't sufficient and given that it isn't sufficient, questioning whether putting the effort in to do it for things where it isn't sufficient an efficient use of time?

Ken Koster

Posts: 1
Nickname: kenkoster
Registered: Feb, 2007

Re: Testivus - Testing For The Rest Of Us Posted: Feb 8, 2007 1:07 PM
Reply to this message Reply
I think that what you're trying to say is correct, but is coming out all wrong.

It's true that some code is best tested with unit tests. And that other code is better tested with some other kind of test. This is the true spirit of testivus; no dogma, just test.

But you're mixing up sufficiency and efficiency. No testing method is 100% sufficient. If, for a given problem, technique X is more efficient than unit testing, by all means use X. But don't go thinking that X is going to be 100% sufficient!

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Unit Test and Developer Test Posted: Feb 8, 2007 1:15 PM
Reply to this message Reply
> - Not sure where this best fits into the manifesto (or if
> you think it's already covered), but I'd say a key point
> is that different situations may need different approaches
> - maybe the "some people...some time..." should also
> include "some situations".

Good point.

> - Beyond the manifesto, I'd hope testivus can develop some
> kind of definitive terminology for different approaches,
> tools and techniques - even if that includes having to
> explain any remaining disputes over meaning. For example,
> there seem to have been various attempts to define stub
> vs. mock vs. fake vs. dummy etc (and I have one of my own
> to add - "mimic"), but there still seems a lot of
> confusion and disagreement. A bigger one is people saying
> "unit test" and meaning entirely different things. Maybe
> it's intractible, but if testivus isn't pushing any one
> particular approach it might be ideally placed to tackle
> this. A genuinely shared terminology might help reduce
> some of the more pointless arguments that one always seems
> to find at the start or end of any discussion on testing.
> Ok, just a dream...

Yes, terminology is one of the things that seems to be getting in the way.

I agree that the "unit" in unit testing causes a lot of confusion and creates some unnecessary restrictions. I vaguely remember talking with Kent Beck a while back, and he was also having second thoughts about calling his framework JUnit because different people use, misuse, and abuse the term.

I propose to use (and have been trying to use with some consistency) the term "developer testing", and "developer test" instead of "unit testing" and "unit test".

Here's a draft definition of "developer test". It's just a starting point, so let us know how we can improve upon it:

A developer test is a reusable, automated, self-verifying test, produced and used by a programmer while creating or modifying the code. Developer tests are typically written in the same language as the code they test and take advantage of test frameworks such as JUnit, TestNG, NUnit, etc. Typically, but not always, developer tests focus on a functional, behavioral, or code unit.

Go at it!

Morgan Conrad

Posts: 307
Nickname: miata71
Registered: Mar, 2006

Re: Testivus - Testing For The Rest Of Us Posted: Feb 8, 2007 1:26 PM
Reply to this message Reply
> Take your flocking-swarming example. How many unit tests
> do you need to verify that the flocking-swarming works
> properly under all operating conditions for all numbers of
> entities from 2 to 2 million?


This should be covered in our Manifesto. A TDD guru told me that in general, you test 0, 1, and "many". And that 2 qualifies as "many". Seemed a bit strange to me at first, but it's a decent starting point.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Testivus - Testing For The Rest Of Us Posted: Feb 8, 2007 1:34 PM
Reply to this message Reply
> I think that what you're trying to say is correct, but is
> coming out all wrong.
>
> It's true that some code is best tested with unit tests.
> And that other code is better tested with some other kind
> of test. This is the true spirit of testivus; no dogma,
> just test.
>
> But you're mixing up sufficiency and efficiency. No
> testing method is 100% sufficient. If, for a given
> problem, technique X is more efficient than unit testing,
> by all means use X. But don't go thinking that X is going
> to be 100% sufficient!

You're absolutely right. This is a much better explanation of the point I'm trying to make. Thanks for taking the time to figure it out.

Flat View: This topic has 51 replies on 4 pages [ « | 1  2  3  4 | » ]
Topic: Testivus - Testing For The Rest Of Us Previous Topic   Next Topic Topic: What Will the iPhone Mean for Mobile Development?


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us