The Artima Developer Community
Sponsored Link

Weblogs Forum
Working Effectively With Characterization Tests - Part 2

3 replies on 1 page. Most recent reply: Jun 14, 2007 8:15 AM by Ged Byrne

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 3 replies on 1 page
Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Working Effectively With Characterization Tests - Part 2 (View in Weblogs)
Posted: Mar 13, 2007 1:04 PM
Reply to this message Reply
Summary
In part 2 of the series we look under the hood of legacy code we inherited and experience a moment of panic when we see how cryptic the code is. Then present a simple strategy that does not require us to fully understand the code in order to write good characterization tests.
Advertisement

In Part 1, we introduced the concept of characterization tests, pretended to have been given responsibility for some legacy sales management software, and wrote our first characterization test. We finished by asking ourselves: “What’s next?” and “How many more characterization tests do we need to write before we can safely start modifying the code?”. Let’s start answering these questions.

The first heuristic Michael Feathers provides for writing characterization tests is:

1. Write tests for the area where you will make your changes. Write as many test cases as you feel you need to understand the behavior of the code.

That sounds like a reasonable goal. I will create as many tests as I need to feel that I have understood – and captured – the behavior of the code. This would be a very tall order if I had to write black-box tests. Without being able to look at the implementation and without a specification it’s hard to know what’s required to get adequate test coverage and capture all behaviors. Fortunately, when writing characterization tests we are not only allowed, but encouraged, to look at the code.

I know what some of you are thinking: Trying to understand a system’s behavior by looking at a pile of code that you did not write is like trying to understand a cow’s behavior by looking at a butcher’s shelf. It’s hard to fully understand what’s actually going to happen by just looking at the code; you’d have to simulate the various paths of the code’s execution in your head, keep track of variable values, etc. – tough things to do for anything but the most trivial code. Fortunately, as you will see, with characterization testing we don’t have to do that. We don’t look at the code to gain complete understanding of what it does; we look at the code for clues and suggestions on what to test. Let’s continue with our example so you can see exactly what I mean.

The sample code we are working with is quite simple: a single method that calculates a sales commission by taking a numeric value as input and returning a numeric value as output. How hard can that be to understand? If we get lucky, the original developer will have been considerate enough to write some nice, clean, self-documenting (or at least well commented ) code. But in many cases, we get … this:

public class SalesUtil {
final static double BQ = 10000.0;
final static double BCR = 0.20;
final static double OQM1 = 1.5;
final static double OQM2 = OQM1 * 2;
      public static double calculateCommissionDue(double totSales) {
              if (totSales <= BQ) {
                      return totSales * BCR;
              } else if (totSales <= BQ * 2){
                      return
                              (BQ) * BCR +
                              (totSales - BQ) * BCR * OQM1;
              } else {
                      return (BQ) * BCR +
                      (totSales - BQ) * BCR * OQM1 +
                      (totSales - BQ * 2) * BCR * OQM2;
              }
      }
}

Take a minute to see if you can understand the higher level specification, or description, of what this code is trying to accomplish by simply looking at it. If you had to write the documentation for this method in English, what would you write?

What I am hoping to get across with this example is that, even for a simple, fully self contained, method that uses only basic arithmetic, understanding its higher-level behaviors by just looking at the code is non-trivial. You can imagine how much more difficult this would be if, instead of dealing with simple addition and multiplication, you had to do the same thing for a more complicated method involving several other classes and method invocations whose behavior you don’t know.

Fortunately, our job at this moment is not to understand what every single variable and operation does, let alone to decide if it’s correct or not. Our job right now is to write some characterization tests that will capture and embody the end-result of all those operations. We can worry about making the code cleaner and more readable after we have tests that will ensure we have preserved the current behavior.

Here’s how I’d go about it.

I notice that the conditional expressions involve a comparison between the parameter totSales and the constant BQ. Since this code calculates sales commissions, I guess that the Q in BQ probably stands for quota. I have no idea what the B stands for, so I assume that it stands for basic. But none of this really makes much of a difference at this point. For all we know, BQ might stand for Banana Quarks. All we care about is that this particular code will exhibit three distinct behaviors based on the relative values of totSales and BQ. Since totSales is set to 10000, we have the following three behaviors to characterize:

Behavior 1: if totSales <= 10000

Behavior 2: if BQ < totSales <= 10000 * 2

Behavior 3: if totSales > 10000 * 2

We have already written a test that satisfies the condition for the first behavior in Part 1 (not intentionally, we simply picked a random value of 1000 and it just happened to be a value that would satisfy the first condition). We can satisfy the two remaining conditions and capture all three behaviors by invoking the method with the values of, say, 20000, and 30000.

We write the second test as follows (remember at this point we are trying to guess a return value that will cause a failure so we can see from the error message what the actual return value is):

public void testCalculateCommissionDue2() {
        assertEquals(-1, SalesUtil.calculateCommissionDue(20000.0));
}

When we execute the test, we get the following error message:

junit.framework.AssertionFailedError: expected:<-1> but was:<5000.0>

The current behavior of the code is such that the calculated commission on $20,000 of sales is $5,000. Again, I don’t know if that’s wrong or right, but it’s the actual behavior of the code, so I am going to characterize that behavior by editing the assertion so that the test will pass:

public void testCalculateCommissionDue2() {
        assertEquals(5000.0, SalesUtil.calculateCommissionDue(20000.0));
}

I repeat the steps using a value of $30,000 for totSales and come up with the last characterization test:

public void testCalculateCommissionDue3() {
        assertEquals(14000.0, SalesUtil.calculateCommissionDue(30000.0));
}

If you ask me, a $14,000 commission on $30,000 in sales seems awfully high. This might be a bug, so I make a note of it – just because we are writing characterization tests, does not mean that we should not keep an eye out for possible existing bugs.

I now have the luxury of three characterization tests for calculateCommissionDue code:

public void testCalculateCommissionDue1() {
        assertEquals(200.0, SalesUtil.calculateCommissionDue(1000.0));
}

public void testCalculateCommissionDue2() {
        assertEquals(5000.0, SalesUtil.calculateCommissionDue(20000.0));
}

public void testCalculateCommissionDue3() {
        assertEquals(14000.0, SalesUtil.calculateCommissionDue(30000.0));
}

I run them with a code coverage analyzer and, as expected, they all pass and I get 100% statement and condition coverage. Great. I can now proceed to modify the code with much more confidence than I had before – even though I still don’t know exactly what the code is supposed to do, or what the identifiers BCR, OQM1, and OQM2 could possibly mean.

Time to wrap-up part 2. We have seen that, when it comes to characterization testing, looking at the code is very helpful even if you don’t understand everything that’s going on. At the very least, you can get some clues that will help you come up with relevant test data to improve your code coverage and get more behaviors to catch.

In Part 3, we are going to start taking advantage of the characterization tests we have written so far. Time for some payoff. Make sure you check it out.


John Zabroski

Posts: 272
Nickname: zbo
Registered: Jan, 2007

Re: Working Effectively With Characterization Tests - Part 2 Posted: Mar 17, 2007 3:12 PM
Reply to this message Reply
@Alberto Savoia
@I know what some of you are thinking: Trying to understand a system’s behavior by looking at a pile of code that you did not write is like trying to understand a cow’s behavior by looking at a butcher’s shelf.

I don't want to sound rude, but... what on earth does that mean?

Alberto, I usually love your analogies and think you are a phenomenal communicator, but this one time I did not understand you at all. A cow and a butcher was not at all what I was thinking when I was reading this. I was focused on learning the benefits of characterization tests.

A much better way to explain this to someone would to work the reader backwards. A system is a complete program. In order to get a complete program, we use unit testing, then component testing, then integration testing, and finally system testing. At each stage, we move in increasing essential complexity: From linear complexity, to dependent quadratic complexity, to quadratic complexity, to exponential complexity or greater. Once the complexity is added, we cannot easily take it away.

I submit that "trying to understand a system's behavior by looking at a pile of code that you did not write" is difficult because it is harder to work from exponential complexity back to linear complexity. If it were so easy, then there would be a lot more people "cheating" and looking at competitors' code in IDA and other reverse engineering software utilities. The actions taken by a capable reverser are very much like a good characterization test. Experts look for certain characteristics of code when reversing it. For instance, they might look for a binary command that adjusts process token privileges, since this changes the security descriptor. Of course, when a reverser does this, they are usually only after a certain *aspect* of the code: exploits. Similarly, in part one of your series on characterization tests, you mention that first step given by Michael Feathers' heuristic process is to "write tests for the area where you will make your changes." In addition, in part two of your series you state, "We don’t look at the code to gain complete understanding of what it does; we look at the code for clues and suggestions on what to test." Reversers do the same exact thing.

Therefore, I submit that a more appropriate way to communicate characterization tests is to defer to the complexity developers must battle, perhaps by drawing a tangent to reverse engineering. Looking at code with no clue what it does or how it works is very much "reverse engineering", only perhaps with more helpful variable names.

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: Working Effectively With Characterization Tests - Part 2 Posted: Mar 19, 2007 2:23 PM
Reply to this message Reply
> @Alberto Savoia
> @I know what some of you are thinking: Trying to
> understand a system’s behavior by looking at a pile of
> code that you did not write is like trying to understand a
> cow’s behavior by looking at a butcher’s shelf.
>
> I don't want to sound rude, but... what on earth does that
> mean?
>
> Alberto, I usually love your analogies and think you are a
> phenomenal communicator, but this one time I did not
> understand you at all. A cow and a butcher was not at all
> what I was thinking when I was reading this. I was
> focused on learning the benefits of characterization
> tests.

Hi John,

Thank you for your feedback. In retrospective, I guess that my cow/code analogy is not a very good one.

What I was trying to get across is the idea that trying to understand how a system behaves by looking at its components in isolation is possible, but very difficult (and you do a very good job of explaining why it is so difficult), and that you can short-cut a lot of that complexity simply by exercising/executing the code and look at what actually happens.

At some point I am planning to combine these blogs in a single mini-tutorial and I'll see how I can improve on that analogy.

Thanks again for taking the time to give feedback and for your complexity-based explanation of why reverse engineering is such a challenge.

Alberto

Ged Byrne

Posts: 22
Nickname: gedb
Registered: Nov, 2003

Re: Working Effectively With Characterization Tests - Part 2 Posted: Jun 14, 2007 8:15 AM
Reply to this message Reply
I would just like to say that I like the key Analogy.

Also, when doing this type of thing you may find Feather 'Vice' tool useful:

"Essentially, Vise is a tool that helps you record a set of values that occur in your code and then use those values as a behavioral invariant. Once the values are recorded, if they change the next time the code executes, Vise throws an exception."
http://www.artima.com/forums/flat.jsp?forum=106&thread=171323

Flat View: This topic has 3 replies on 1 page
Topic: Working Effectively With Characterization Tests - Part 2 Previous Topic   Next Topic Topic: An exercise in compromise in class interface refinement.


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us