Summary
As long as we believe that it's possible to deterministically control the development of software, deterministic solutions will be the only ones that we can entertain. Only by letting go of the idea of determinism can we open up to other, less perfect ways of influencing the process.

A previous essay elicited this response:

Commenter:The problem is not that determinism is an illusion...it is not. It is just us that we are not able to calculate the outcome. For example, take a French billiard table with two balls only: we can easily predict their trajectories, when they will collide etc. Add another ball, and the problem becomes very difficult, if not impossible, to predict. But the outcome is deterministic nevertheless because it is defined by physical laws.

This comment describes is the approximation that falls within the realm of Newtonian mechanics. These are the "physical laws" that people believed governed everything in the universe, until they discovered through experimentation and observation of things that are very big, very small, very fast, very cold, or very hot, that the window where Newtonian mechanics worked was the fairly narrow one of our direct experience. It only works that way because some of the factors, like the gravitational effects between small bodies like billiard balls, or the Brownian motion at typical billiard-table temperatures, are negligible. And probability can even come into play -- the nap of the felt could influence an outcome that's on the edge.

Also, I think the commenter is trying to describe the 3-body problem with the pool table example. However, the 3-body problem involves gravity, and that's why it gets so complicated -- so the bodies need to be big enough for gravitation to have an effect in order for it to be a true 3-body problem.

I think this commenter is making my point for me. Physics always involves approximations, and discovering which factors, in that particular situation, are important. The problem occurs when we start ignoring those factors altogether, and pretending that they don't exist and that our model applies everywhere. A useful model is only an approximation that works under the current circumstances. I think a previous commenter already mentioned Goedel; he and others after him have proven, in essence, that all models are incomplete.

What I've been trying to say in these recent essays is that we have been attempting to apply the same logic in all situations, just as this commenter has asserted that the pool table example is "defined by physical laws" (where probability doesn't come into play). We produce success with a certain team of people, and we draw conclusions such as "it was because I used this methodology," or "it was because the team was less than 8 people and we worked in this kind of environment." And those things may actually have a predominant effect ... in some cases. But those things may also be overwhelmed in other cases; a good example is if you have a particularly strong personality, positive or negative, on the team.

The problem is that science in general and physics in particular is all about looking at the situation and understanding what the dominant factors are, and discarding the factors that, in the current situation, do not influence the outcome enough to care about it for whatever we are doing at the moment (the factors are still there -- they may simply not be measurable, or we may not care about their impact). But in physics, these are repeatable phenomenon, so we can run the experiment again and again and duplicate the results.

With software development, you can't run the same experiment twice, because all the factors are always different. Even if you use the same team, that team will change significantly from one experiment to the next. If we were to use the same approximations that physicists do, one argument would be that all the factors can be discarded except for the people factors, which tend to overwhelm everything else (and this argument appears to be true in a large number of cases). Now we're left with a group of individuals who are all changing. And it's impossible to model their behavior.

Maybe there are cases where the people factors can ignored because something more powerful is involved. I think that's the scenario that many managers are pursuing, because those more powerful factors might be controllable and thus the outcome is more predictable. But I don't think this has ever happened on a software project. I think people have believed that they are in control and have tried to steer things based on what they thought controlled the outcome.

In all the cases I've ever seen, however, the people factors overwhelm everything else, and determinism is, in fact, an illusion. That doesn't mean that you have to give up trying to influence the outcome, but just that looking at the problem in a deterministic fashion is a dead end. This is why complexity theory is so interesting, because it looks at problems that are far too complex to model using some kind of equation, and yet it says "let's see what we can do with this anyway."

But deterministic thinking won't allow you to go there, and that's the problem. As long as you're seeking a solution that promises a predictable outcome for a set of inputs, you'll fail to come up with a useful approach, and that's what we've been seeing for many years in the software industry.

But if we can let go of determinism, this allows us to say instead, "The problem is far too complex to solve, but might there be ways we could improve our chances of success?" I think we then open ourselves up to new and potentially much better ways of doing things. That's the step I think the agilists have taken. I think all the agile methods have in common that they shorten the feedback loop and create lots of little experiments with evaluations and course corrections in between. I suspect that a lot of the resistance to this is because it doesn't promise a deterministic solution like methodologies of the past have promised, and if you are very attached to determinism, you can't be satisfied with that.

Talk Back!

Have an opinion? Readers have already posted 45 comments about this weblog entry. Why not add yours?

RSS Feed

If you'd like to be notified whenever Bruce Eckel adds a new entry to his weblog, subscribe to his RSS feed.

Digg | del.icio.us | Reddit

About the Blogger

Bruce Eckel (www.BruceEckel.com) provides development assistance in Python with user interfaces in Flex. He is the author of Thinking in Java (Prentice-Hall, 1998, 2nd Edition, 2000, 3rd Edition, 2003, 4th Edition, 2005), the Hands-On Java Seminar CD ROM (available on the Web site), Thinking in C++ (PH 1995; 2nd edition 2000, Volume 2 with Chuck Allison, 2003), C++ Inside & Out (Osborne/McGraw-Hill 1993), among others. He's given hundreds of presentations throughout the world, published over 150 articles in numerous magazines, was a founding member of the ANSI/ISO C++ committee and speaks regularly at conferences.