Re: Reviewable, Runnable Specifications Ensure Software Quality
Posted: Jun 12, 2010 5:33 PM
> I understand where this is coming from but I would prefer
> if you would take me at my word when I say that tests are
> great. I am not in any shape or form suggesting that the
> process you are suggesting should not be done or that it
> doesn't produce value.
Ah, great. We are indeed on the same page. Apologies for having missed that viewpoint as the thread deepened.
> What I am suggesting is that what you are advocating is a
> good idea but not, by itself, good enough. Words matter.
> If you say 'ensures' when you don't really mean
> n 'ensures', people won't necessarily know that.
Fair point. I tend towards maxims. "Time is money!" That's not an assertion of an absolute equality, or an insistence that time is the only thing that matters. It is, instead, a succint and memorable way of stating one aspect of things--one that does matter, so saying it in a memorable way is helpful, if not absolutely accurate.
> Side-bar: isn't using statically defined tests the 'right
> way' to do things?
Definitely. I mispoke when I said "statically-generated". Your choice of "statically-defined" is much more accurate, and unambiguous. Mind if I steal it?
> Again, what you are doing is great but validating that '.'
> and '..' work doesn't mean that './..' will work properly.
> In my experience the really painful bugs are the ones
> that deal with corner cases.
Right, but again, we're dealing with an engineering problem. It is tempting to consider a tool that dynamically constructs hundreds of combinations, and tests them all--but even then, we are dealing with issues that may never arise in practice, or for which there are fairly simple workarounds. So it's possible to spend far too much time on a quality "proof", when a reasonable assurance is all you really need. (I've put together really bulletproof programs that did great stuff--for which the hardware no longer exists to run them, much less the operating system. Talk about over-engineering. "Good enough" would have made a lot more sense.)
> If you start
> testing all the different permutations of inputs, you are
> back in unfeasible territory.
Definitely. Any attempt to make a 100% argument is doomed to failure. We simply need to do the best we can.
> From what I understand, research shows that code-reviews
> and prototyping more effective than unit testing in terms
> of finding bugs. If you are disputing that, I think you
> should give some evidence or at least a rational argument
> for why.
That's an interesting argument. But I dispute that any person in the world reads code the same way the computer executes it! Those are good practices, but early design reviews were found to be a far more significant predictor of success. (If memory serves, it was the /only/ really significant determin-er.) But design reviews only work for the first version. They don't protect against regressions. Spec reviews, like design reviews, ensure quality--but in a way that also prevents future regressions.)