The Artima Developer Community
Sponsored Link

Weblogs Forum
QA and TDD

13 replies on 1 page. Most recent reply: Sep 11, 2005 1:12 AM by Curt Sampson

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 13 replies on 1 page
Christopher Diggins

Posts: 1215
Nickname: cdiggins
Registered: Feb, 2004

QA and TDD (View in Weblogs)
Posted: Sep 3, 2005 12:00 PM
Reply to this message Reply
Summary
My recent blog-post "Post-TDD" has generated some interesting and lively discussion. Here are some more of my thoughts and clarifications.
Advertisement
There have been a lot of insightful and education comments (at least for me) so far on the weblogs forum in response to my blog post Post-TDD for which I am very appreciative. One of my points though I think didn't come across.

My thesis: given two design which accomplish the same thing, the one which requires fewer tests is a better design. This I believe to be true from either a Quality Assurance (QA) point of view, or a TDD point of view. This is why I say that we want to somehow strive towards elimination of tests in our designs. When it is possible to forecast the number of tests required for a design decision, in theory it should make it easier to make design choices. Perhaps this could lead to software which makes design decisions, though this is probably pretty far out there.

My previous conflation of TDD and QA was somewhat intentional, because I think that QA is automatically a goal of any programmer, and any methodology. I assert that the biggest advantage of TDD by itself is QA through incidental correctness testing.

A development methodology in of itself is useless, unless it helps us achieve our goal as programmers:

Writing correct software in the shortest amount of time.


John D. Mitchell

Posts: 244
Nickname: johnm
Registered: Apr, 2003

Parsimony and Context Posted: Sep 3, 2005 2:10 PM
Reply to this message Reply
> My thesis: given two design which accomplish the same
> thing, the one which requires fewer tests is a better
> design.

That's an arguably laudable goal but:

* It's a post hoc measurement. I.e., as I noted in the previous thread, if you're actually developing software (TDD-wise) how do you know what test is "necessary" and which one isn't until after you've developed the software? If you're doing post-coding testing then hopefully it's already obvious that that's post-hoc. But note that even in post-hoc testing, you don't a priori know the minimal set of tests to perfectly cover a system (think: halting problem).

* It presupposes a static view of the system. I.e., systems are dynamic and evolving. So there's always going to be a rhythm between necessary and sufficient code/tests/etc. That's a fundamental point of the value of "refactoring" as a key building block of TDD. The "designer" mindset is heavily biased towards these sorts of static views but hasn't the success of the agile movement made it popularly clear that dynamic, adaptive evolution
is the fundamental reality of software systems?

> This I believe to be true from either a Quality
> Assurance (QA) point of view, or a TDD point of view. This
> is why I say that we want to somehow strive towards
> elimination of tests in our designs. When it is possible
> to forecast the number of tests required for a design
> decision, in theory it should make it easier to make
> design choices. Perhaps this could lead to software which
> makes design decisions, though this is probably pretty far
> out there.

I totally concur that refactoring the tests is important -- almost as important as refactoring the code, IMHO.

Also, as I noted in the previous thread, if you care about precision in correctness then using assertive checking (e.g., contracts (ala Design by Contract)) is a *necessary* adjunct to the various levels and types of use-tests.

> My previous conflation of TDD and QA was somewhat
> intentional, because I think that QA is automatically a
> goal of any programmer, and any methodology. I assert that
> the biggest advantage of TDD by itself is QA through
> incidental correctness testing.

Bah, again. :-)

IME, the biggest value of TDD has been getting developers to shift their focus to a much finer iterative level and a more actual-use feed-forward/-back mindset. Quality is just one of the by-products.

> A development methodology in of itself is useless, unless
> it helps us achieve our goal as programmers:
>
> Writing correct software in the shortest amount of time.

Is that really the job of programmers?

Hmm... Perhaps you're implying a lot into the word "correct"...

Strategically, what's the "correct" things to do for the needs of the e.g., business? I.e., "Doing the right things." Often, that's *not* doing that project (as specified).

Once there's a strategic need then the focus is tactically on "doing things right." That's my impression of what you're talking about.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: Parsimony and Context Posted: Sep 3, 2005 3:26 PM
Reply to this message Reply
> > Writing correct software in the shortest amount of
> time.

>
> Is that really the job of programmers?
>
> Hmm... Perhaps you're implying a lot into the word
> "correct"...
>
> Strategically, what's the "correct" things to do for the
> needs of the e.g., business? I.e., "Doing the right
> things." Often, that's *not* doing that project (as
> specified).


Agreed.

There's also the issue of "shortest amount of time." Many people can run a three minute mile if they are allowed to die at the end. To me sustainable development speed should be the goal, and you don't know whether you have it or not until you start confronting your earlier choices.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: QA and TDD Posted: Sep 4, 2005 10:18 AM
Reply to this message Reply
> There have been a lot of insightful and education comments
> (at least for me) so far on the <a
&gt; href="http://www.artima.com/forums/flat.jsp?forum=106&threa
&gt; d=125723">weblogs forum</a> in response to my blog post <a
&gt; href="http://www.artima.com/weblogs/viewpost.jsp?thread=125
&gt; 723">Post-TDD</a> for which I am very appreciative. One of
> my points though I think didn't come across.
> <p>
> My thesis: given two design which accomplish the same
> thing, the one which requires fewer tests is a better
> design. This I believe to be true from either a Quality
> Assurance (QA) point of view, or a TDD point of view. This
> is why I say that we want to somehow strive towards
> elimination of tests in our designs.

Chris, can you explain your rationale? Why do you think that fewer tests means better design?

Christopher Diggins

Posts: 1215
Nickname: cdiggins
Registered: Feb, 2004

Re: QA and TDD Posted: Sep 4, 2005 10:45 AM
Reply to this message Reply
> Chris, can you explain your rationale? Why do you think
> that fewer tests means better design?

To be precise I think that better designs require fewer tests. Here is a simple example:

class string {
  public {
    concat(string s) : string {
      // postconditions
      after {
        ?(result.count() == count() + s.count());
        for (int i=0; i < count(); ++i) {
          ?(result[i] == this[i]);
        }
        for (int i=0; i < s.count(); ++i) {
          ?(result[i + result.count()] == s[i]);
        }
      }
      // implementation
      ....
    }
    reverse(string s) : string {
      // postconditions
      after {
        ?(result.count() == count());
        for (int i=0; i < count(); ++i) {
          ?(result[i] == this[(count() - 1) - i]);
        }
      }
      // implementation
      ...
    }
    trivial_concat_reverse(string s) : string {
      // testing not-needed
      // trivial implementation
      return concat(reverse(s));
    }
    optimized_concat_reverse(string s) : string {
      after {
        ...
      }      
      // non-trivial implementation
      ...
    }
  }
}


Does this make sense? The ?() statement is an assertion statement and the after block is a block of code executed after the function is done, which is usually used to check postconditions.

In the example I argue that there is no need to test trivial_concat_reverse, and that using it in a design makes for a less complex and easier to maintain design.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: QA and TDD Posted: Sep 4, 2005 10:53 AM
Reply to this message Reply
That's fine, but where are the tests you used to write it?

Christopher Diggins

Posts: 1215
Nickname: cdiggins
Registered: Feb, 2004

Re: QA and TDD Posted: Sep 4, 2005 11:01 AM
Reply to this message Reply
> That's fine, but where are the tests you used to write it?

The postcondition check (after block) is the test which I use to guide development.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: QA and TDD Posted: Sep 4, 2005 11:11 AM
Reply to this message Reply
> > That's fine, but where are the tests you used to write
> it?
>
> The postcondition check (after block) is the test which I
> use to guide development.


Okay, I get it. So, you're saying that because trivial_concat_reverse is built up from two tested pieces, it doesn't need a test. Where's your limit? Suppose it was three?

Christopher Diggins

Posts: 1215
Nickname: cdiggins
Registered: Feb, 2004

Re: QA and TDD Posted: Sep 4, 2005 11:20 AM
Reply to this message Reply
> > > That's fine, but where are the tests you used to
> write
> > it?
> >
> > The postcondition check (after block) is the test which
> I
> > use to guide development.
>
>
> Okay, I get it. So, you're saying that because
> trivial_concat_reverse is built up from two tested pieces,
> it doesn't need a test. Where's your limit? Suppose it
> was three?

Good point, I don't know what the limit. But I believe this is a similar problem that is encountered in any testing scenario: to what grain of resolution do we test? Clearly testing every line of code is silly, but I don't know that any rules could devised to express, what needs to be tested and what doesn't.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: QA and TDD Posted: Sep 4, 2005 11:29 AM
Reply to this message Reply
> Good point, I don't know what the limit. But I believe
> this is a similar problem that is encountered in any
> testing scenario: to what grain of resolution do we test?
> Clearly testing every line of code is silly, but I don't
> know that any rules could devised to express, what needs
> to be tested and what doesn't.

The XP thing is to skip things that can't fail.. usually getters and setters. The trivial_concat seems to be in that category too, but I admit that out of habit, I'd write a test for it. I could skip it, but I don't think I'd be saving much.

The conventional wisdom is that the bottleneck in programming is 'thinking time' not 'typing time.' So, yes, it's hard to see what we'd save.

That said, I agree that simpler designs are better and to the degree that we need fewer tests to create them, we're better off.

Kristian Dupont

Posts: 22
Nickname: chryler
Registered: Dec, 2003

Re: QA and TDD Posted: Sep 5, 2005 3:06 AM
Reply to this message Reply
<blockquote>
Writing correct software in the shortest amount of time.
</blockquote>

I the additional requirement that my code must be easy to change. Loose coupling and lots of tests help me with this.

Curt Sampson

Posts: 21
Nickname: cjs
Registered: Mar, 2004

Re: QA and TDD Posted: Sep 5, 2005 4:33 AM
Reply to this message Reply
> The XP thing is to skip things that can't fail.. usually
> getters and setters. The trivial_concat seems to be in
> that category too...

So it sounds like what we really need are better languages. More declarative languages often tend to let you more easily produce trivial code that's obviously correct, especially pattern-matching languages such as ML.

Tanton Gibbs

Posts: 20
Nickname: tanton
Registered: Aug, 2005

Re: QA and TDD Posted: Sep 10, 2005 3:16 PM
Reply to this message Reply
In all honesty, I've screwed up things as simple as your trivial_concat_reverse, so I would definitely write a test for it.

Also, what if you want to change the implementation later? Perhaps the two functions are not as efficient as one function would be and this function is called quite frequently. Do you write a test then? In your example, the tests are inline so you know that it doesn't have a test and can write one; however, in a language such as Java, the programmer may not know that a test doesn't exist. They may run the unit tests expecting everything to be great, but in reality their new code is not being tested.

Finally, TDD is more than QA. It is also about creating a specification for your code. It is informing other users how your code will react to various input and output. This is a vital part of self-documenting code. I think a great post on this was made by Dave Astells. You can find it here: http://daveastels.com/index.php?p=5

Curt Sampson

Posts: 21
Nickname: cjs
Registered: Mar, 2004

Re: QA and TDD Posted: Sep 11, 2005 1:12 AM
Reply to this message Reply
Let's take your comments point by point:

1. If you need to write a test to satisfy your particular comfort level of coding, by all means do so. Everybody's different, and everybody makes different kinds of mistakes.

2. If you want to change the implementation later, write the tests later. You write code for now, not later, so write your tests for now, not later. When you go back to code to change something, check the tests first and make sure that the tests cover potential problems introduced by your changes. If they don't, write new tests first, just as you would if you were writing new code.

3. When writing code that should be covered by existing tests, you do need to check that the tests are actually testing that code. A good strategy in such a circumstance is to write the code with bugs in it that the tests should catch, run the tests, and make sure that they indicate that the code is broken. Then fix the code and watch the tests pass.

4. If you need to write a test for the purposes of specification, do so.

All I'm saying here is that you shouldn't write unnecessary tests any more than you should write unnecessary code. Tests are good, but tests also cost money, and when you're spending money working on a test, you're making a decision not to spend that money elsewhere, which means that you think you can't get better value for your customer than writing that test.

Flat View: This topic has 13 replies on 1 page
Topic: Looking at Heron Primitives Previous Topic   Next Topic Topic: Heron Operator Precedence


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us