This post originated from an RSS feed registered with Python Buzz
by Andrew Dalke.
Original Post: Is TDD better than Apps Hungarian?
Feed Title: Andrew Dalke's writings
Feed URL: http://www.dalkescientific.com/writings/diary/diary-rss.xml
Feed Description: Writings from the software side of bioinformatics and chemical informatics, with a heaping of Python thrown in for good measure.
Greg Wilson and Jorge Aranda's essay Two Solitudes
describes the different viewpoints and additudes between software
engineering researchers and practicioners. Here is one of the
complaints:
At industry-oriented gatherings, it seems that a strong opinion, a
loud voice, and a couple of pints of beer constitute "proof" of almost
any claim. Take test-driven development, for example. If you ask its
advocates for evidence, they'll tell you why it has to be true; if you
press them, you'll be given anecdotes, and if you press harder, people
will be either puzzled or hostile.
Most working programmers simply don't know that scientists have been
doing empirical studies of TDD, and of software development in
general, for almost forty years. It's as if family doctors didn't know
that the medical research community existed, much less what they had
discovered. Once again, the first step toward bridging this gulf
seemed to be to get each group to tell the other what they knew.
Greg worked with Andy Oram to collect and edit the essays in Making
Software: What Really Works, and Why We Believe It, as a way to
present some of that research evidence to software practicioners. The
Two Solitudes essay refers to that book, saying its "topics range from
the impact of test-driven development on productivity (probably small,
if there is one at all) to whether machine learning techniques can
predict fault rates in software modules (yes, with the right
training)."
Why is there a debate over TDD?
Did you catch the "probably small, if there is one at all"
part? Why then is there a continued debate over the effectiveness of
TDD when the conclusion from reseach is that there is at best only a
small improvement in productivity? The Two Solitudes
essay goes into some of the possible reasons. You should read it.
That got me to thinking about other well-accepted or well-rejected
practices: version control software and Apps Hungarian notation. We as
practicioners generally accept that version control software is a Good
Thing and generally view Apps Hungarian as a negative. If we "know"
that version control systems are useful, then can software engineering
researchers find experimental evidence to support or reject the claim?
If they have, then it might be an example of what a successful
research experiment might look like. If it can't, then perhaps it says
something about the weakness of the current experimental approaches
compared to personal experience.
We can ask the same about other development practices. Can software
testing show if Apps Hungarian improves productivity over other
variable naming systems?
With those experiments in place we could start asking questions like
"are the gains from introducing TDD to the process more like the gains
in introducing Apps Hungarian, or more like the gains in introducing
Mercurial?" Or more crudely, "is TDD more effective than Apps
Hungarian?"
(For "TDD" put pair programming, unit tests, coverage analysis,
version control systems, or most any other practice.)
So let's ask the software engineering researchers if they can provide
evidence for those beliefs of the practicioners. Perhaps they've done
it already?
Do version control systems make people more productive?
As far as I can see (which isn't very far), no one has done
experimental research to tell if a version control system makes people
more productive. I asked Greg, and he was a bit chagrined to reply
that he didn't know of any published research either. Yes, he promotes
that everyone use a version control system, despite not knowing of
research evidence to back that belief.
Curious, isn't it?
My challenge to researchers is, can you carry out experiments to test
the effectiveness of version control software for different types of
projects?
This research, like TDD, is complicated by many real-world
factors. There are alternatives to using version control software:
Don't use any version control software, just one directory,
Depend on system backups (including Time Machine for the Mac),
Make manual backups of the working code from time, as a
tar/zip file, or saved to another directory,
Use a versioning filesystem, which might be a distributed
file system;
different development scenarios:
Single developer,
Pair-programming using the same account,
Multiple developers all using the same file system,
Multiple developers across multiple sites;
different types of developers:
Trained as software developers,
... electrical engineers,
... computational scientists,
Undergraduates in a first semester programming course;
and even different needs:
Research spikes where the code is thrown away at the end,
Scientific research where one person works on a new approach
for several months, publishes a paper, and never works on it again,
Traditional main-branch-development-only coding,
Multi-branch development and multiple supported releases.
It sounds like a lot of work to test these possibilities when we
already know the answer, but that's my point. If research methods
can't address this question then how do we trust those methods on
other topics, like TDD or pair-programming?
Perhaps the research will point out something interesting. Perhaps it
will find that research projects which take less than three months and
where there is only one developer are 10% more effective using
automatic versioning file system over a manual version control
system. If that were true, perhaps we should recommend that most
scientists use a versioning file system instead of version control.
Or perhaps we should develop better connections between the various
file system monitoring tools (like inotify) and the
version control systems, to give an automatic/no-touch interface to
the version control system.
For that matter, some people start continuous integration tests based
on checkins. Why not do CI based on filesystem changes? Obviously some
will commit less often, but perhaps productivity will improve.
I can continue to conjecture, but of course, without evidence it's
little better than blather. The world is full of blather. That's why
we need research.
Does Apps Hungarian make people more productive?
Hungarian Notation is a classic topic. Quoting from Michael Feathers
on the c2.com wiki
page:
Hungarian notation inspires some of the most bitter religious wars
among programmers. Detractors claim that it is indecipherable and that
no real standards exist for its usage. Adherents claim that it becomes
more likeable with use, and the instant mnemonic recognition of type
that one gains with familiarity aids program comprehension
immensely.
Hungarian notation seems to be in decline. Microsoft's .Net "General
Naming Conventions" document says outright "Do not use
Hungarian notation." Nor is Hungarian the de facto style in any of
the few dozen languages that I'm familiar with, though some libraries,
especially those with a Microsoft heritage, do use it.
This despite the efforts of people like Joel Spolsky to promote a
distinction between Apps
Hungarian and System Hungarian. The former was is "extremely
valuable" while the latter is from "the dark side.")
The Wikipedia page lists other "notable opinions" both for and against.
Some people say it's useful in untyped languages. Others say it
Hungarian Notation just gets in the way. Is there any research to back
up any of these opinions, or are variable naming schemes yet another
unexplored field?
Does TDD make people more productive?
I'm picking on TDD, but this section can be applied to many other
topics.
In any TDD debate you'll hear from people who find it useful, and
those who don't. You'll hear about people who tried TDD but didn't
find it useful, and you'll hear from others who want to help the first
group do TDD correctly. And you'll hear from people who tried git and
found it confusing ... though they often move to other version
control systems instead.
Look at the old Hungarian discussions and you'll see the same
patterns, including people that tried it, liked it, won't move away,
and believe that others just don't understand Hungarian correctly.
It looks like the effectiveness of TDD has had many more studies then
the effectiveness of either version control systems or Hungarian
Notation, so let's use that as the reference. Is the difference in
effectiveness between using PEP-8 style names and Apps Hungarian more
or less than the difference between TDD and coverage-based testing?
Is the difference in effectiveness between using version control
instead of manual backups more or less than the difference between TDD
and test-last?
And more importantly, are the answers to those questions opinions
based on experience or are they backed by evidence? Can we use these
experiments to calibrate our understanding of what software
engineering researchers can identify?
Feedback
Do you know of any research which tests the effectiveness of version
control systems over other approaches? What about of App Hungarian
over other approaches? Perhaps you have a "strong opinion,", "loud
voice," or have had a "couple of pints of beer" and want to leave a comment?
I sent the seed of this essay to Greg, and he posted it on "It will never work in
theory." I think that's the best place to continue this dicussion,
so I hope to see you
there.
I'll end with a quote from Peter Medawar's "Advice to a Young Scientist":
I cannot give any scientist of any age better advice than this: the
intensity of a conviction that a hypothesis is true has no bearing
over whether it is true or not. The importance of the strength of our
conviction is only to provide a proportionately strong incentive to
find out of the hypothesis will stand up to critical evaluation.
Postscript
The above essay places the entire burden on software engineering
researchers to do new experiments. I don't mean for things to be so
one sided. This was also meant for software practicioners.
Software practioners claim a lot of things. Some, like version control
systems make sense. Hungarian notation, not so much. I don't think TDD
makes much sense, and the research seems to back me up.
If you are software practioner then take that final quote from Medawar
to heart. You can make claims about your method X or approach Y, but
in the back of your head, remember that I'll be asking you "how do you
know that your approach is any better at improving productivity than
switching to Apps Hungarian?" You won't be able to answer that
question ... unless X or Y is "Apps Hungarian," of course.
At best you'll point to success cases, and then I'll point to the
success of Microsoft Word and Excel in the 1990s, and to shops like
Fog Creek.