This post originated from an RSS feed registered with Agile Buzz by Laurent Bossavit. Original Post: Why good ideas fail (and how they might succeed) Feed Title: Incipient(thoughts) Feed URL: http://bossavit.com/thoughts/index.rdf Feed Description: You're in a maze of twisty little decisions, all alike. You're in a maze of twisty little decisions, all different. Latest Agile Buzz Posts Latest Agile Buzz Posts by Laurent Bossavit Latest Posts From Incipient(thoughts)

Suppose you hear about a Better Way of doing business. An interesting example might be Extreme Programming, which suggests that software development contracts might be handled differently than they usually are, to significant benefits. I'm only using that as an example - there's a very general question you might ask about such ideas, especially ideas which have been around for a little while.

The question is: "If this is such a hot idea, then organizations using it should outperform others. Organizations are subject to a sort of economic Darwinism - the better performing ones stick around, the others die. So we should be seeing more organizations using this Better Way. Why hasn't that happened yet?"

You might be tempted to answer, "because it wasn't such a hot idea after all; looks good but doesn't work in the real world, back to the drawing board".

That's a premature conclusion, and you could be missing out on what's really a good idea - if you knew how to cash in. To understand why, all you need is a thin slice of Game Theory.

First, we need to discuss the Prisoner's Dilemma. In the simplest form, this is a two-player game with two possible moves, "Cooperate" or "Compete".

The game is about winning as many points for yourself as you can. Suppose the two of us are playing - if you choose Cooperate and I choose Compete, I win five points - and you win nothing ! Frustrating ? But on the other hand, if you choose to Compete while I Cooperate, you win the five points and I get nothing. If we both Cooperate, each of us wins three points. Finally, if we both Compete, each of us wins a measly one point.

The names of the strategies are merely suggestive - they have nothing to do with how best to play the game. What matters is the "payoff matrix", as below:

You'll want to spend some time thinking about the game, if you have never had an occasion to do so before. (Play it online at the link provided above, or play with friends and colleagues, read about it...)

The Prisoner's Dilemma yields a good model, though hugely simplified, of a situation in which good ideas fail. Imagine, if you will, an economy based on playing that game. Imagine further that everyone - all the businesses in that economy - typically play the Compete move. With any interaction, any two businesses make some little profit - one point.

In that context, Cooperation is demonstrably a Better Way: it would be possible to get at three points rather than one out of every interaction if everyone was using it. Why doesn't everyone start doing it ? Because it makes no sense for any one business to switch their strategy to Cooperation, unilaterally. They would be "rewarded" by going up against Competitors and gaining no points at all from any business they did. It would take a coordinated decision for everyone to move to Cooperation, but in a "free" economy there's no opportunity for such coordination. (The simplest game theoretical model of a "free" economy assumes that businesses don't communicate about their strategies - so you couldn't have a conditional strategy of Cooperating only with Cooperators. More realistic, and more complex models exist - but then I'm writing this precisely to motivate you to take an interest in them.)

This type of reasoning is representative of one common objection raised when discussing, e.g. Extreme Programming: "What you suggest works in theory, but in the real world customers don't trust software contractors and so they will prefer up-front specification to ongoing customer involvement; that being the case, you just have to go along if you want to do business." And this line of reasoning is quite valid, considered in the light of game theory.

This seemingly desperate situation is called a Nash equilibrium, after John Nash, the mathematician who studied them. They're an important result in game theory because they allow you to ignore most of the myriad combinations of M strategies in a population of N players: you can expect that any given population will stabilize at a mix of strategies that form an equilibrium. They help explain why populations of people or businesses may converge on strategies that are not necessarily Better Ways.

Robert Axelrod's book, The Evolution of Cooperation, tells the fascinating story of a series of experiment involving games of Prisoner's Dilemma conducted as computer simulations and "tournaments". I strongly recommend the book as your first stop if you're interested in learning more about the game, or about game theory. It contains little that looks like "theory" - equations or long logical arguments - but mostly focuses on the results of simulated games, pitting against each other strategies submitted by some of the best mathematical and scientific minds in Axelrod's distinguished entourage. The results have practical relevance to anybody interested in how Better Ways spread... or don't.

In particular, Axelrod shows how Cooperation can in fact - almost against all hope - win over Competition: our models (and Axelrod's simulations) need to introduce the notion of territoriality - i.e. the idea that anyone's interactions in the game will mostly be with "neighbours". (They're not necessarily neighbours in physical space - "idea space" could be a good way to think of it.) In Axelrod's simulations, it turns out that small clusters of Cooperating neighbours can eventually "invade" a population stuck at a Nash equilibrium.

Based on this, for instance, you might expect that the full benefits of Extreme Programming would arise if and when a "community" of businesses interacting mostly with each other in that particular way. That is, assuming that this particular way is indeed a Better Way; but that's a different question. At any rate, it is one that can't be settled by saying "if it really worked, everybody would be doing it by now".