
Re: Reviewable, Runnable Specifications Ensure Software Quality

Posted: Jun 11, 2010 4:52 AM


> > But your (prime number) algorithm is wrong. > > Yes. That was my intention. > Ah, yes. The perfect argument. Take the assertion to the ultimate extremity, construct an unassailable case against that, and claim that the entire argument is therefore null and void.
But engineering is about matters of degree. A key element in managing the complexity is to categorize the cases, and the enumerate thosenot the individual data points.
Even with numbers, there are natural breaking points. For example: All positive integers less than 2*8, zero, negative integers, 2**, and 2*8+1. For numeric algorithms used in the vast majority of applications that kind of breakdown is sufficient.
The use of a prime number algorithm was particularly adroit, in that regard. Since it is not possible to enumerate in advance all possible primes, it is not even theoretically possible to prove the algorithm is correct. But as one illuminary is quoted as saying, "We can never prove an algorithm is correct. The best we can do is prove that we couldn't make it fail."
Note, too, that staticallygenerated tests can never prove that an algorithm is robust! Following the testfirst methodology, I can make it work for "1" by returning the expected value. Then I can make it work for "2" by adding another return statement! But of course, the resulting "algorithm" fails for 3 and any number I didn't code for. Far from robust. Still, my tests succeed!
One idea, then, would be categorize the range of inputs for which the software is expected to work, and the test harness would select a random value in that range. That kind of testing would distinguish "vaporware" (hard coded values) from code that was backed by an actual algorithms.
In practice, though, that's overkill. We code algorithms, not hard values, because we know that otherwise, it will fail the minute users work with it. The tests are only there as a "check", not as "proof".
But, and this was the assertion in the article, if I can examine the list of tests you've constructed, I will pretty rapidly be able to determine the robustness I can expect from the algorithm you implemented. If the tests for paths only use forward slashes, for example, I'll expect it to break on systems that use backward slashes in their paths.
In short, I've categorized the kinds of cases the algorithm might possibly handle: forward and backward slashes, absolute and relative paths, dot and dotdot segments, etc. Were I looking for "proof", it would then be necessary to enumerate and test every possible combination. But for a "quality check", I'm only looking for a representative sample.
A representative sample is sufficient, because if your algorithm handles a decent number of "interesting" cases, I have a high level of confidence in it. And that "level of confidence" is another way of saying "quality".

