Re: Testing Genes, Test-Infection, and the Future of Developer Testing
Posted: Jan 25, 2007 2:13 PM
>What do you mean by this? Do you think that IDEs should >generate tests from the code that you've written? That would >seem to mean that the tests come after the code. Also; being >generated from the code, I would imagine that those tests >would fail to find any bugs since any such bugs would form >part of the definition of the tests.
You make some very good points and I agree with you that, ideally, there should be some form of specs or tests (and I see tests as a form of executable spec).
But there are two situations where tests automatically generated by the code can be useful.
Automatically Generated Characterization Tests
Situation 1: There are no tests and no spec anywhere in sight or, more common, there are very few insufficient tests, and you need to work with that body of code. I know, really bad situation to be in - it should never happen, but unfortunately it does happen all the time.
In this case, an automated tool can generate characterization tests. I.e. tests that characterize the actual and current behavior of the code. Not what the code should do, but what it actually does.
The terms characterization tests was introduced my Michael Feathers in his excellent book "Working Effectively with Legacy Code".
Characterization tests are change detectors, and they are useful because, after you make a change to the code, you rerun the tests and if any of them fail you can look at them to determine if the change in behavior is something you expected/anticipated or if you accidentally "broke" something.
Not as valuable as a test suite driven by an actual specification and encoding desired behavior by quite useful in its absence.
Complementary, Exploratory, Additional Tests
Situation 2: You have a thorough body of tests. Perhaps you have even developed the entire code using TDD. You are in great shape, better than 90% of the projects out there.
However, we know that sometimes it's easy for developers to overlook some corner cases or unanticipated parameter values. In those cases, analyzing the tests that are automatically generated may help you discover some code behaviors that you did not consider while programming:
"Ooops. I forgot to check that the input String should not contain any non-ASCII characters".
I have spent the best part of my professional career working on automated test generation and I don't mind admitting that it has been a very long, difficult, and sometimes frustrating road, and that progress has been much slower than those of us in this field would have hoped for.
For a while I was convinced that the best approach would be to have the developers write some machine readable specs that could then be used to help them in testing. The technology worked, but most developers did not want to write specs, the same way most of them did not want to write documentation, or tests.
However, with all the CPU power available today, I am finally starting to see some light at the end of the tunnel. Especially in the area of automated generation of characterization tests. I believe that characterization tests have potential for getting developers test infected because there may not be specs, or docs, or tests, but there will always be code. And if we can use that code as a starting point to populate their project with some tests, we can gently guide them to review and enhance those automatically generated tests. Consider those tests a stepping stone.
What's your reaction to these application of characterization tests and tests generated from code?
I grant you that they are not, and cannot, be a substitute for tests generated with human insight, domain knowledge, and creativity. But do you share my view that they offer some complementary value?
Thank you for the interesting dialog.
Since reading Michael Feather's book about legacy code and characterization tests, I decided to focus my research at AgitarLabs on them. As part of our effort, we have created a freely available, web-based, characterization test generator for Java and JUnit called JUnitFactory. You can check it out at:
Please keep in mind that it's very news and it's experimental. Also, please don't upload any confidential/proprietary code or IP since JUnitFactory is designed for researchers and academia and the servers are not secure.