What can you learn from testing? When you look beyond the red and the green, the fail and the pass, you can learn a lot more about the nature of the code and the nature of the problem domain. And there is a lot to learn — software development is called knowledge work for a reason.
Software development is often described as knowledge work. This label is invariably used as a shorthand for "work that doesn't involve getting your hands dirty", "jobs your parents never had and you struggle to explain to them" or, without any apparent irony over the use of keyboards, "the digital economy". But step back from these simple substitutions and an obvious yet deeper truth becomes visible: knowledge work is about knowledge. It's about knowing stuff and, most often, also about how you deal with stuff you don't know.
By far and away the most popular response to ignorance is to ignore it. Sound good? Feel free to stop reading! You must have stumbled across this article by mistake.
OK, you're still here. In that case I'll assume curiosity on your part. Which is a good thing: curiosity is key to how we address a lack of knowledge. Questions are the agents of curiosity. Even without answers, questions can help us to increase and refine our knowledge, to learn.
Software testing is a form of learning. A set of tests can be considered a set of questions. The most obvious question a test poses is "Does the code pass?" to which there are two simple answers: yes or no. A test allows us to move from belief to knowledge — for example, to move from merely believing something works to knowing that, in a particular context, it does. Even limiting the scope of testing to just this question and these two answers reveals a rich little set of possible outcomes:
It passes, which is what we expected.
It passes and we are surprised, as this is not what we expected.
It fails, which is what we expected.
It fails and we are surprised and disappointed, as this is not what we had hoped or expected.
The reasons for failure (or success) run even deeper: a test might fail because the test is at fault, not the code; a test might pass because both the test and the code are at fault and in sympathy; and so on.
And that's just what we can learn from a simple red or green! Of course can is not will — having the opportunity is not the same as taking it — but the first part of any feedback-based process is generating the feedback; what you do with it becomes the next challenge.
But more learning opportunities are available from testing: in the formulation rather than the execution of tests.
What does it mean if a test is hard to write? In particular, a unit test. Difficulty in writing unit tests is a principal demotivator among programmers trying to get into unit testing, causing many to get out of it. "We tried unit testing, but it took too much effort" is a common lament. The more experienced practitioner will typically rejoin with "if your code is difficult to test, it means your code is messy", a comeback that is not necessarily unreasonable but also not necessarily correct. At best it is a provocative oversimplification intended to make you reflect. At worst it may prevent you from learning about your assumptions and the nature of your work.
The difficulty in being able to write a test can be boiled down to the two broad themes of complexity and ignorance, each manifested in a couple of different ways:
The essential complexity of the problem being solved. Such complexity is intrinsic to the problem domain and is not the fault of the software architecture, the code quality or, indeed, any of those who have worked on the code, past or present. If you're looking for someone to blame try the problem domain or the customer. Perhaps the customer would prefer a "Hello, World" app instead of software to control the rail network? That would be one way to eliminate the essential complexity, but probably not what the customer needed. The reality is that some things just are harder to code and test no matter how you write the code or the tests. When complexity was being handed out, not all applications and domains of interest were created equal.
The accidental complexity of the problem being solved. Such complexity is an artefact of the way the software has been developed, external to the nature of the problem. It is non-essential and, in principle, avoidable. This is the realm of technical debt, baroque frameworks and rococo architectures. It's where speculative generality has come home to roost, copy-and-paste code has blossomed and coupling has weighed anchor. This is where the observation "if your code is difficult to test, it means your code is messy" may often apply.
Uncertainty over what the code should actually do. One of the most common responses to the question of why one should test one's code is "to show that it works". This offers all the illusion of being a satisfactory answer without any of the substance. It leaves us with a lingering, killer question: What exactly do we mean by it works? What is it actually supposed to do? In short, what's the spec? Tests may be difficult to write because we don't actually know exactly what we want. We may understand the gist, but not the detail. We may apparently be able to write the code — having followed the gist we may have elaborated something in code that followed and meandered along that gist — but writing a test throws our ignorance into the sharp relief of cold light. Or perhaps not: being unaware of our own ignorance is a common unawareness and ignorance, a difficult-to-see blindspot. This kind of difficulty in testing is an open invitation to talk to someone else, to clarify and find out more.
Lack of testing know-how. Programmers may be unaware of or unfamiliar with the techniques necessary to make a particular test easier, or even possible. If programmers don't know about test doubles (mocks, stubs, etc.), what are the odds they will reinvent and rediscover this approach? Without help, would a typical programmer know how to test outputs whose sequence is non-deterministic or collections whose ordering is unspecified? What about testing the output of an image processing algorithm, such as lossy compression or edge detection? The difficulty of testing may reflect testing skills we need to acquire or techniques we need to improvise.
Of course, each of these aspects is not entirely independent from the others: a challenging domain may often require novel techniques for testing; uncertainty over the functionality may reflect a complex or, by being ill-defined, seemingly complex domain; and so on. Nonetheless, by recognising these four aspects we can learn more about our assumptions, our knowledge, our ignorance and the nature of our work than either writing off tests as too hard to be worthwhile or assuming that messy code is necessarily the root cause.
I agree with the writer that testing software is very important. But I would not say that testing moves us to knowledge. That's not true. The test itself can be wrong. Thus we can never know (in the strong sense) that our software is correct just because it passed all tests. But our confidence in our implementation may grow.
I think what makes tests invaluable is that testing is indeed a scientific method. It is the like the experiments of the physicist. We ask questions to the software and get our feedback. As a result the software developer may change the software and repeat the same experiment (the physicist, as she cannot change nature, may change the experiment or his hypothesis). The software developer possesses already his hypothesis (this is the software specification), and checks if it coincides with reality (implementation). In case of the physicist it is just the other way round: The reality is already there and unchangeable and what is to be adapted is the hypothesis. Both advance by doing experiments.
> I agree with the writer that testing software is very > important. But I would not say that testing moves us to > knowledge. That's not true. The test itself can be wrong. > Thus we can never know (in the strong sense) that our > software is correct just because it passed all tests. But > our confidence in our implementation may grow. > I agree with this. When I describe the value of testing, I always emphasize that it helps understand the solution space, not the problem space. And, since I work commercially, rarely is the solution space going to add to overall knowledge; pretty much everything is a repeat there.
> I agree with the writer that testing software is very > important. But I would not say that testing moves us to > knowledge. That's not true. The test itself can be wrong.
You may find the article deserves another reading as the observation concerning faulty tests and the relationship to knowledge was mentioned early on. Knowledge is never complete or perfect, but it can be expanded and improved.
> I think what makes tests invaluable is that testing is > indeed a scientific method.
Your observation concerning hypothesis forming and the scientific method is spot on.
> When I describe the value of testing, I > always emphasize that it helps understand the > solution space, not the problem space. And, since I > work commercially, rarely is the solution space going to > add to overall knowledge; pretty much everything is a > repeat there.
If you reread the article you may discover that you have been missing out by confining yourself to considering testing only in the context of learning about the solution space. The point about testing (and the article) is that testing can reveal assumptions and questions in both the problem and solution spaces.