The Artima Developer Community
Sponsored Link

Weblogs Forum
Testing Genes, Test-Infection, and the Future of Developer Testing

30 replies on 3 pages. Most recent reply: Feb 4, 2007 11:08 AM by nguyen Phuc Hai

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 30 replies on 3 pages [ « | 1 2 3 | » ]
Erik Engbrecht

Posts: 210
Nickname: eengbrec
Registered: Apr, 2006

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 25, 2007 9:12 AM
Reply to this message Reply
Advertisement
>I think a lot of people are writing 'unit tests' to test
>these higher-level interactions but I think there's a
>difference and the way to test them is different too. What
>I call this is regression testing and personally I think
>that the testing community has put much too little effort
>into making it easy and automated.

+1

...and too much focus on "a class is a unit," or maybe that's just sycophants. I don't know.

I usually think of it as "automated testing," and it is just one part of fail-fast behavior. The fastest code can fail is with a compiler error. The slowest is after it's in production use.

Mike Kaufman

Posts: 125
Nickname: mkaufman
Registered: Jan, 2007

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 25, 2007 10:52 AM
Reply to this message Reply
Fully agree with the last few posts (Ravi Venkataraman, Erik Engbrecht).

I usually write Javadoc then code then tests, but in very small pieces and usually pretty much all in parallel: each makes me adjust the other. I think I spend 2-3 times as long writing the tests than on the writing code, maybe rather more in some cases. I write tests for all of my code, and try to cover all of the code's documented behaviour and all of its statements/branches. In general these are tests of each class/method's specified behaviour, taking into account any internal features of the code but assuming that any used classes/methods are tested separately and are working as per their spec (as per calls to a library or underlying platform). This seems easier and less fragile than "pure" unit testing of each component in complete isolation. By doing code/test in small chunks and having comprehensive automated tests, the theoretical worry that a "ripple" of failures might be hard to debug has never been a real problem. Yes, this is often slow and tedious work, but it gives me strong confidence in the code plus tests that will immediately pick up pretty much any regression.

I refer to these tests as "unit tests" because they are testing a "unit" of some kind, though the current terminology seems to use "unit test" to mean testing a class in complete isolation (whatever that really means in practice). Maybe this is nearer to the "regression" terminology (though I read that as a more general term), or what some people mean by "integration" testing (though that often seems to mean a larger granularity, whereas I'm testing the detail of individual methods).

I suspect that amongst those that do actually do "developer" testing there might be a silent majority that do it something like this (though maybe not as comprehensively), despite the greater publicity of TDD, testing-in-isolation, "expectation"-checking mocks etc (none of which I have anything against).

Maybe we need a "testing for the rest of us" movement. The terminology certainly seems to have become more of a hindrance than a help when discussing this stuff - we have lots of specific terms but everyone means slightly different things by them, so you end up having to explain what you mean by each term and having arguments over it. Wonder if there's any way we could get back to basics, or agree some definitive set of fundamental terms?

Morgan Conrad

Posts: 307
Nickname: miata71
Registered: Mar, 2006

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 25, 2007 11:18 AM
Reply to this message Reply
One way to encourage testing is to write helper tools to ease the trivial, repetitive tasks, and others to ease some trickier tasks.

For an example of the former, a colleague wrote a method that uses reflection to find all the properties of a bean, then calls the setters and getters to verify that they match up. I personally don't think this is very valuable but if it's trivial to do I'll do it. :-)
Similar methods try to set selected properties to null and look for the presence or absence of IllegalArgumentExceptions.

For the latter, I wrote a method to take a 1st Object, a 2nd Object that is suppossed to be equals to that Object, and a 3rd object that isn't. It runs all the trickier stuff: e.g. o1.equals(o2), but also o2.equals(o1), and it also verifies that o1.hashcode() == o2.hashcode(). I found a lot of classes not writing hashCode when they wrote equals()

Andre Mesarovic

Posts: 5
Nickname: andre9
Registered: Oct, 2004

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 25, 2007 11:26 AM
Reply to this message Reply
Hmm... As a developer, if you don't test your code then how do you know it works? Whether it be repeatable fully automated test suites, repeatable manual tests, interactive text or Swing test consoles, every developer should be have a set of these tools in their toolbox. At the very least, a developer should be able articulate their testing methodology. Not to be able to do so reflects poorly. After all we are engineers, and engineers do test their constructs, don't they?

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 25, 2007 11:50 AM
Reply to this message Reply
> I refer to these tests as "unit tests" because they are
> testing a "unit" of some kind, though the current
> terminology seems to use "unit test" to mean testing a
> class in complete isolation (whatever that really means in
> practice). Maybe this is nearer to the "regression"
> terminology (though I read that as a more general term),
> or what some people mean by "integration" testing (though
> that often seems to mean a larger granularity, whereas I'm
> testing the detail of individual methods).

Right, this is what I was trying to get before. I used to also call this unit testing but what I found was that unit testing tools were actually a hindrance and not helpful. Sure, they made it easy to run the tests but I can easily run the tests from a script.

For example, right now I am testing processes that are driven by a queuing service. The code that is not specific to queues is testable independent of the queue and unit tests work well for verifying the behavior of these 'units' but for verifying the how everything works together and how it works on the queuing service, it's pretty much useless.

I played around with using Jython at my last job for unit testing and regression testing and found it to be pretty effective because it's faster and I'm not concerned that my unit test is built like a brick-shit-house. If it falls down, you fix it as needed.

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 25, 2007 2:13 PM
Reply to this message Reply
>What do you mean by this? Do you think that IDEs should >generate tests from the code that you've written? That would >seem to mean that the tests come after the code. Also; being >generated from the code, I would imagine that those tests >would fail to find any bugs since any such bugs would form >part of the definition of the tests.

Hi Vincent.

You make some very good points and I agree with you that, ideally, there should be some form of specs or tests (and I see tests as a form of executable spec).

But there are two situations where tests automatically generated by the code can be useful.

Automatically Generated Characterization Tests
----------------------------------------------

Situation 1: There are no tests and no spec anywhere in sight or, more common, there are very few insufficient tests, and you need to work with that body of code. I know, really bad situation to be in - it should never happen, but unfortunately it does happen all the time.

In this case, an automated tool can generate characterization tests. I.e. tests that characterize the actual and current behavior of the code. Not what the code should do, but what it actually does.

The terms characterization tests was introduced my Michael Feathers in his excellent book "Working Effectively with Legacy Code".

Characterization tests are change detectors, and they are useful because, after you make a change to the code, you rerun the tests and if any of them fail you can look at them to determine if the change in behavior is something you expected/anticipated or if you accidentally "broke" something.

Not as valuable as a test suite driven by an actual specification and encoding desired behavior by quite useful in its absence.

Complementary, Exploratory, Additional Tests
--------------------------------------------

Situation 2: You have a thorough body of tests. Perhaps you have even developed the entire code using TDD. You are in great shape, better than 90% of the projects out there.

However, we know that sometimes it's easy for developers to overlook some corner cases or unanticipated parameter values. In those cases, analyzing the tests that are automatically generated may help you discover some code behaviors that you did not consider while programming:

"Ooops. I forgot to check that the input String should not contain any non-ASCII characters".

...

I have spent the best part of my professional career working on automated test generation and I don't mind admitting that it has been a very long, difficult, and sometimes frustrating road, and that progress has been much slower than those of us in this field would have hoped for.

For a while I was convinced that the best approach would be to have the developers write some machine readable specs that could then be used to help them in testing. The technology worked, but most developers did not want to write specs, the same way most of them did not want to write documentation, or tests.

However, with all the CPU power available today, I am finally starting to see some light at the end of the tunnel. Especially in the area of automated generation of characterization tests. I believe that characterization tests have potential for getting developers test infected because there may not be specs, or docs, or tests, but there will always be code. And if we can use that code as a starting point to populate their project with some tests, we can gently guide them to review and enhance those automatically generated tests. Consider those tests a stepping stone.

What's your reaction to these application of characterization tests and tests generated from code?

I grant you that they are not, and cannot, be a substitute for tests generated with human insight, domain knowledge, and creativity. But do you share my view that they offer some complementary value?

Thank you for the interesting dialog.

Alberto

P.S.

Since reading Michael Feather's book about legacy code and characterization tests, I decided to focus my research at AgitarLabs on them. As part of our effort, we have created a freely available, web-based, characterization test generator for Java and JUnit called JUnitFactory. You can check it out at:

http://www.junitfactory.com

Please keep in mind that it's very news and it's experimental. Also, please don't upload any confidential/proprietary code or IP since JUnitFactory is designed for researchers and academia and the servers are not secure.

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Testivus: Testing for the rest of us Posted: Jan 25, 2007 4:53 PM
Reply to this message Reply
>Maybe we need a "testing for the rest of us" movement.

Hi Mike,

I love the suggestion of a "testing for the rest of us" movement.

Especially because, as most fans of the Seinfeld TV show probably know, they have an episode in which one of the character, fed up with the commercialism, dogma, pressures, etc., of Christmas, creates a new tradition/holiday called Festivus, with the immortal line:

"A Festivus for the rest of us!".

The Festivus concept resonated with large numbers of people. So much so that it went from a fictitious holiday to something that a number of people actually celebrate (Agitar's yearly company pic-nic is called Festivus :-)).

I don't know if you were serious about your "testing for the rest of us" movement, but if you were, how about calling this movement you bring up "Testivus - Testing for the rest of us" and actually do something about it?

I am pretty serious about it. The name, and origin of the movement name may be amusing, but the purpose and mission is a serious one.

>I suspect that amongst those that do actually do >"developer" testing there might be a silent majority that >do it something like this (though maybe not as >comprehensively), despite the greater publicity of TDD, >testing-in-isolation, "expectation"-checking mocks etc >(none of which I have anything against).

That's the perfect Testivus attitude (i.e. practical, open to all approaches, not too dogmatic). The kind of attitude that's going to appeal to the T2 and not scare away the T3.

What do you say, shall we get serious about this movement? Use it to get back to basics, get agreement on a set of fundamental terms, etc.

I think it has enough potential that I have reserved a few testivus.xxx domains which I'd be happy to donate if we start this thing.

Alberto

Todd Blanchard

Posts: 316
Nickname: tblanchard
Registered: May, 2003

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 25, 2007 7:12 PM
Reply to this message Reply
> I think one way to encourage T2 to become T1 would be
> through better tool support in IDEs. As of now IDEs are
> mostly concentrating on making coding easier and very
> little on generating tests.

Depends on your environment. I work in Smalltalk. If you look at most smalltalk code, there is often a comment in many methods that shows what the method should do. Select it and 'print it' and you can run the method right there and see what it does.

This is what most of my testing looks like. Unit tests, sure. But not built into a big unit test running framework. The only time I am likely to bother with that is when I am planning to port this code, publish it as a core library, or anticipating big changes to the system down the road.

Like everything else - you put as much effort into the testing system as the situation demands. If you were to write unit tests for a shell script that was going to be run once, that would be crazy.

Mike Kaufman

Posts: 125
Nickname: mkaufman
Registered: Jan, 2007

Re: Testivus: Testing for the rest of us Posted: Jan 25, 2007 8:20 PM
Reply to this message Reply
Hi Alberto,

The "testing for the rest of us" wasn't meant especially seriously, but wasn't quite a joke either. I'd forgotten about "Festivus", but I can see the comparison.

I'd love to try taking it up, but sure it'd be a much bigger and harder task than I could possibly get into at the moment. I'm finishing development of a product that I intend to set up a micro-ISV around, and unless and until it's done and actually getting sales I've got no income, just a big risk, so the clock is ticking ominously... I'm already involved in too many things both at work and at home. Especially with all these tests that need writing (that bit really is meant just as a joke, by the way).

But I'm completely happy if you want to set this off yourself, or with anyone else, and if you do I'll try and chip in as and when I can. Not sure the name "testivus" sounds too smart, and might mean something rude in some language or other, but if you're willing to do something under that name it's fine by me. Would be most impressive if you get something rolling!

Bill Burris

Posts: 24
Nickname: billburris
Registered: Aug, 2003

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 26, 2007 6:24 PM
Reply to this message Reply
I just haven't been able to figure out how to write useful tests for what I do.

For me testing is writing a little bit of code and running it to see what happens. Did the right status lights on the instrument come one? Did I get the right timing for the signals on the oscilloscope? Did I get data from instruments x, y, & z within the right time window? Did the stepper motor move to the right position? How many times during the day did the GPS receiver fail to send out its timing correction? Was the timing correction values missed because the computer was busy or because the receiver didn't send them out?

Agile programming books rarely focus on what Bruce Powel Douglass refers to as Doing Hard Time.

Bill

Erik Engbrecht

Posts: 210
Nickname: eengbrec
Registered: Apr, 2006

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 26, 2007 6:52 PM
Reply to this message Reply
> I just haven't been able to figure out how to write useful
> tests for what I do.
>
> For me testing is writing a little bit of code and running
> it to see what happens. Did the right status lights on
> the instrument come one? Did I get the right timing for
> the signals on the oscilloscope? Did I get data from
> instruments x, y, & z within the right time window? Did
> the stepper motor move to the right position? How many
> times during the day did the GPS receiver fail to send out
> its timing correction? Was the timing correction values
> missed because the computer was busy or because the
> receiver didn't send them out?
>
> Agile programming books rarely focus on what Bruce Powel
> Douglass refers to as Doing Hard Time.
>
> Bill

So you should have a test bench (or benches) that simulate the behavior of the external components of the system. Your equivalent of a "mock object" is box or separate piece of software the pretends to be a motor or a GPS receiver or whatever you need. You have the simulator in software so you can test your software indepently of the hardware on which it will eventually run, and you run it on the hardware plugged into a box that simulates the external world so you can test the hardware and software together in a predictable fashion.

Yes, this isn't what gets covered in Agile books, and it can be a good deal more expensive. But it can also be more valuable. A information system developer can have an db server and an application server running right there on his desktop, so he can do a fair amount of testing free of mock objects. But proper test benches that simulate the rest of the system and/or the real world are absolutely critical for doing embedded work or work that relies on external sensors, transmitters, etc.

Bill Burris

Posts: 24
Nickname: billburris
Registered: Aug, 2003

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 26, 2007 9:59 PM
Reply to this message Reply
Building test benches takes too long. I usually just develop the software on the system where it will be used. Usually the software is only used on one system. Often code is written when the hardware designer has the scope probe in hand, and asks for me to try something so he can see it on the scope. Over time the code for testing the hardware morphs into the application.

It is difficult to create mock objects when you don't really know how the hardware works until you play with it. The hardware often will change in less time then it would take to create a mock object to simulate it. When using FPGAs, the hardware can change several time a day.

For commercial hardware like the GPS receivers, voltmeters, current sources and stepping motor controllers, it works best to just connect it to the computer for testing the code. Most instrument manuals don't make much sense until after you have worked with the instrument for a while. The examples in the manuals very rarely have anything to do with what I am trying to do and rarely use the same programming language and environment that I am using. Examples written in BASIC in the 80s look nothing like what I need to create in C# today.

If I created mock objects I would be spending a lot of time writing code which be used once or never.

The only way mock objects would be useful is if the instrument manufactures wrote them and posted them on their website. This would help be write test code to help decide which instruments to purchase.

Since I an an electronics tech working in a mad house (I mean University), my comments are probably not relevant to you people out in the Real world.

Ravi Venkataraman

Posts: 80
Nickname: raviv
Registered: Sep, 2004

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 27, 2007 7:09 AM
Reply to this message Reply
Bill wrote:
>
> Since I an an electronics tech working in a mad house (I
> mean University), my comments are probably not relevant to
> you people out in the Real world.

Just a question, why is the world of academics not real? And what is so "real" about the commercial world where the average programmer writes shoddy programs and everybody accepts atrocious software as the standard?

Another question: What is easier? For a person from the "commercial world" to join the field of academics or the other way araound?

I am working in the commercial world but find this constant derision of the academic world to be ridiculous. Especially when the people from the commercial world doing the mocking themselves are average, at best.

Ravi

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 28, 2007 10:08 AM
Reply to this message Reply
I think that a lot of this has to do with culture. When I went to school, there was this unspoken assumption that it was you against the machine, that you literally tested yourself against the computer.

The good programmers were the ones that could keep the "whole system in their heads," and knew code was right before they wrote it. I think that sort of hubris inhibited the idea of developer testing for a long time and echoes of it remain.

David Vydra

Posts: 60
Nickname: dvydra
Registered: Feb, 2004

Re: Testing Genes, Test-Infection, and the Future of Developer Testing Posted: Jan 28, 2007 12:31 PM
Reply to this message Reply
Alberto,

While your theory about testing genes sounds both plausible and amusing, I think that its management that in the end will determine how widely developer testing will be adopted. I think that the adoption of developer testing by management will happen for the same reason Toyota approach to car manufacturing is winning -- its just a better long-term strategy.

I have observed that most folks do not engage in self-actualizing behavior when they show up at work (this point already made by others on this thread). If I tell my team that we have decided to base our productivity and quality strategy in part on developer testing and tell them that we will offer them training and mentoring, most employees will be just as happy working this way as before, as long we offer them fair compensation, a culture of integrity, and state-of-the-art tools. On the tools front, I think that AgitarOne has reached a point where I can recommend it in most situations for all the reasons that you already mentioned.

David Vydra
Developer Productivity Consultant
testdriven.com
Full Disclosure: Ex-Agitar Employee

Flat View: This topic has 30 replies on 3 pages [ « | 1  2  3 | » ]
Topic: Testing Genes, Test-Infection, and the Future of Developer Testing Previous Topic   Next Topic Topic: Interface Implementations Not Just From Factories


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us