A short response to Tim Bray's recent blog post, Doing it Wrong.
Tim Bray's recent blog post, Doing it Wrong has received a fair amount of attention in recent days. In the post, Bray compares the supposedly light-weight agile development approaches used at Web-facing software startups with more formal and heavy-weight processes employed by enterprises, and includes the following observation:
The community of developers whose work you see on the Web, who probably don’t know what ADO or UML or JPA even stand for, deploy better systems at less cost in less time at lower risk than we see in the Enterprise. This is true even when you factor in the greater flexibility and velocity of startups...
This is unacceptable. The Fortune 1,000 are bleeding money and missing huge opportunities to excel and compete. I’m not going to say that these are low-hanging fruit, because if it were easy to bridge this gap, it’d have been bridged. But the gap is so big, the rewards are so huge, that it’s time for some serious bridge-building investment... This seems by far the most important thing for my profession to be working on.
Bray then attempts to dissect the reasons for the great discrepancy, and concludes that technology and culture both play a role in the increased development cost at enterprises.
Readers of Doing it Wrong have already commented that Bray's analysis falls short of mentioning the large number of startup failures. Others have noted that the venture-sponsored development style of startups holds scant relevance for enterprises that must carefully plan the use of limited resources: Managers really must know up-front the cost and delivery times for a software project, and developers, in turn, must be able to estimate those factors accurately at a project's inception.
Owners vs Caretakers
Yet, Bray's analysis as well as the comments to it neglect an important factor that, I think, accounts for a large portion of the discrepancy in the cost and quality of software projects. The difference is due not so much to that between enterprises and Web-facing start-up companies, but to the difference between two types of managers and developers: caretakers and owners.
From a business perspective—Bray's main viewpoint—caretakers use other people's money, whereas owners use their own money, or are at least very significantly and personally exposed to monies invested in a project. More generally, owners of a project or a codebase have very strong personal incentives to make their users happy, be those incentives financial or purely professional. As a result, for owners efficiency and reduced costs are musts, not simply desirable attributes. Caretakers generally have less urgent objectives.
Owners and caretakers are present in enterprises and startups. I've worked with owners of very large enterprises—companies owned by a few individuals as opposed to many shareholders—that insisted on very lean development practices; and also with enterprises where decisions were made by caretakers who had no personal financial or urgent professional stake in the process. But I've seen startups, too, where, although employees had a heavy incentive to make the business successful, the ready availability of venture money made frugality a somewhat remote concept.
Owners and Agility
I've found that working with owner-types forces agile practices on a project: For one, those folks have absolute control over the purse, and you, as a vendor or consultant, simply won't get paid unless you show significant and regular progress. The more frequent the iterations, the better. All the agile practices work to a developer's advantage in such a setting. Also, owners are generally very hesitant to throw (their own) money at enterprise software that can't show short-term return on investment. That, in turn, determines to some extent the technologies used in the sense Bray describes. Finally, owners of successful enterprises are interested in the long-term maintainability and cost of applications—again, agile practices better able to manage change are more suited for this sort of an environment.
Caretaker types, by contrast, are seldom incentivized in the sense owners are: From a business perspective, caretakers may receive a bonus or promotion based on quarterly or annual performance, whereas owners typically have a much longer view. Caretakers also have less riding on a project's success professionally: If a project doesn't pan out, they would just look for another project or job. Owners must please their customers, but caretakers must please their boss above all. As a result, caretakers are more interested in making safe choices: They have to be in a position to defend themselves in case something goes wrong with a project. Caretakers may also be less incentivized to follow iterative practices and deliver regular, measurable results, in part because they typically receive a fixed salary at regular time intervals. For owners, however, nothing is ever guaranteed: they must earn their pay each time they sit in front of a customer.
Bray mentions Basecamp as one of his examples; in that case, all the developers really were owners of a relatively small company. They had clear incentives to make the application lean and clean, and easy to maintain, which then impacted their technology choices. At the other end of the spectrum are large-scale government-sponsored projects, where the owner is the amorphous tax payer and where everyone on the project is a caretaker. Bray mentions some of the more spectacular project failures in that sector.
In-between those extremes are employees of firms that are somehow incentivized to make the best decisions for their companies. The "best" decision may not always be to save the most amount of money or to use the most capable technologies. "Best" requires context, and people are able to appreciate different contexts defined by different environments: I've known folks who recommended and used one set of technologies in their day jobs, and used something completely different in a startup they worked on at night. In one context they were caretakers, in the other, owners, leading to different motivations and decisions.
Should We All Be Owners?
Looking at this problem from the perspective of owners vs caretakers suggests that Bray's blog post is really about an age-old business problem: As soon as a business starts to have employees, or caretakers, an entirely different set of incentives and motivations enters the firm. While an owner-only company may be very efficient at every turn, you can hardly grow a business without employees.
The same is true of a software project, even when it is an open-source project. It may be ideal to work alone or with just a special cadre of developers with deep and personal stakes in the project's success, that exclusiveness restricts growths if the project becomes widely used. While I don't have personal experience with this, I imagine that large open-source projects, such as the Linux kernel or the Eclipse IDE, require more resources than do small projects with a handful of owner-developers.
You need to accept somewhat reduced efficiency in return for the ability to grow and serve more customers or users. That has been the case with many technology companies, from Microsoft to Google: Employee no. 20,000 at Google likely has different motivations from employee no. 5 when making technology decisions.
We can't all be owners, but a company can take steps to motivate employees to think more like owners. Agile practices already promote the notion of "product owner" or even "business owner," and smart firms can make such phrases a fact in business terms, too.
It must also be said that while owners have the right incentives, they don't always make the right decisions. A sizable bad decision can put a company out of business, losing the owners' principal and dashing their hopes.
Do you agree that Bray's argument can be interpreted in terms of how "owners" vs "caretakers" make technology decisions? What do you think of Bray's blog post?
I agree, there is a motivation shift. As a consequence it get's more and more important for a growing company to employ self-motivated people, people who are interested in the technology and love their job. Otherwise the quality of the products will decrease and customers will go somewhere else.
I think in cases like this people use big words to make their own personal biases sound "right". As an example, you talk about owners
"owners are generally very hesitant to throw (their own) money at enterprise software that can't show short-term return on investment."
"caretakers may receive a bonus or promotion based on quarterly or annual performance, whereas owners typically have a much longer view."
So, owners have a much longer view, but they are very much wanting to see short term ROI? While not technically contradictory, this is a very hard line to walk, wanting your cake and eating it too.
I'm not saying anybody is right or wrong in this instance. I think the discussion has just gotten boring. For many years now we've been hearing about how great agile is. And new and exciting arguments come up to support it. And depending on who you talk to it is good for big projects, small projects, short term projects, long term maintaintance and it can, in fact, be all things to all people and make everybody happy, managers, stake holders, customers and developers.
Meanwhile, places like NASA and the military and big companies are slow and plodding and use outdated technology and old practices and things that might resemble what could be referred to as engineering.
This just seems like another place where people are saying "use agile, it's great" and if enough reasons are given, and it is said often enough, then it will resonate with somebody. Just ignore the fact that if you look at all of the reasons agile is great you will see a lot of contradiction. It's all ok, really. Just use agile.
Also, I have to disagree completely with this statement "for owners efficiency and reduced costs are musts, not simply desirable attributes."
In every case I know of, efficiency and reduced costs are musts for caretakers, and desireable for owners. If you look at owner in the startup sense of the word, and Bray's post talks about companies that started small and became huge, then they either fail completely or become hugely successful. They took a relatively small investment and turned it into something orders of magnitude bigger. If you grow something by a factor of 10,000, what does it matter if something costs 2X as much as you originally guessed?
I also find the reporting HORRIBLY one sided when people talk about agile successes. Everybody points out the same few companies/projects like Tim does in his post and, well, if an agile project fails, it's because they were doing agile wrong. It couldn't be that agile doesn't fit the type of project, or the project would have failed no matter what methodology was used or any of 100 different reasons a project could fail. They simply did agile wrong. Fix that, and *poof*, everything will be a smashing success.
Lastly, I think the owner/caretaker and agile/non-agile arguments are orthogonal. I don't think one has much, if anything, to do with one another. When it comes down to it, to answer the second to last question in your post, "Do you agree that Bray's argument can be interpreted in terms of how "owners" vs "caretakers" make technology decisions?" which to me is the only interesting question in there, owners use what will work for them (for whatever definition of "work" is important to the owner) and caretakers use whatever they have to. As a caretaker, you don't have the ability to rewrite something if you believe there is a 'better way to do it'. You are, in most cases, stuck with what you are given. You have to take the tools you are given and make them work as best you can.
I don't think you can frame Bray's argument that way.
Finally, to conclude this long winded ramble, I believe that Tim's last paragraph, his Plan B, is completely illogical. Ask a blue suit provider to build twitter or basecamp? Why on earth would they? As he notes, it would be very, very hard to put a number on the costs, and you don't know if they are worth building until they are already successful. If you have 5 guys in a room that are willing to risk it and have a devil may care attitude, you can do that sort of thing. If you are a company that employs thousands and have a board of directors and shareholders to answer to, you simply can't afford that type of risk. They just don't work that way for a variety of reasons.
Hell, even Google doesn't work the way they used to much anymore. They just buy stuff now, for the most part. They aren't the company they were 10 years ago. Nor should they be. They have a different set of problems and a much different set of responsibilities than they had when they were just a small company offering a superior search engine. They cannot possibly do things the way they did then and meet their current obligations.
> "owners are generally very hesitant to throw (their own) > money at enterprise software that can't show short-term > return on investment." > > "caretakers may receive a bonus or promotion based on > quarterly or annual performance, whereas owners typically > have a much longer view." > > So, owners have a much longer view, but they are very much > wanting to see short term ROI? While not technically > contradictory, this is a very hard line to walk, wanting > your cake and eating it too.
There's no getting anything by you is there?
> Meanwhile, places like NASA and the military and big > companies are slow and plodding and use outdated > technology and old practices and things that might > resemble what could be referred to as engineering.
This prompted a funny thought. Raise your hand if you'd choose to fly on an airplane with a computer system designed and developed the way Twitter is? Not that Twitter is bad, it just doesn't really matter that much whether it works or not. If you go to an ATM and it empties you bank account without producing any bills, you could have some serious problems and would likely be very upset (and so will the government, BTW.)
> This just seems like another place where people are saying > "use agile, it's great" and if enough reasons are given, > and it is said often enough, then it will resonate with > somebody. Just ignore the fact that if you look at all of > the reasons agile is great you will see a lot of > contradiction. It's all ok, really. Just use agile.
It occurs to me that the problem I have with Agile is similar to the problem I have with SOA. If you look at the source of these ideas and what they had in mind, they are good ideas (IMO). But I think there's a fatal flaw. These technologies (in the general sense) create new technical definitions using existing general terms. And as much as these new terms may have formal definitions, a lot of people don't adopt them or even bother to read them. Everyone knows what a 'story' is. Everyone knows what a 'service' is. Of course, how each person relates what they know that word to mean to software isn't the same. Eventually you have a lot of people doing things they way they like to do things and using the terminology of Agile or Service Orientation to describe it. You can have two groups doing wildly different things but using the exact same descriptions for what they do.
I actually had an argument on a forum similar to this in which the person I was conversing with was unwilling or unable to understand why I refused to declare my "own" definition for 'SOA'. In my mind, that's like coming up for my own definition of 'water' but that's how most people are using these concepts. They have become meaningless.
> I actually had an argument on a forum similar to this in > which the person I was conversing with was unwilling or > unable to understand why I refused to declare my "own" > definition for 'SOA'. In my mind, that's like coming up > for my own definition of 'water' but that's how most > people are using these concepts. They have become > meaningless.
For as much as we've butted heads on these forums, I knew there was a reason I liked reading what you wrote, regardless. This is priceless :-)
With regard to the original blog, my biggest point of contention is the repetition of the old it's better to buy than build assumption. He redeems himself in the next two sections but even so, I think it's too strongly put.
It's definitely better to buy than build: IFF you can get something that meets all your critical needs and IFF it works properly and performs well etc. and IFF you can't better build what you need for less money.
That's three big ifs. The last will depend on the quality of the developers in the department which is typically abysmal (especially if you contract it out)and so can often be ignored. But buying piece of shit that doesn't meet your needs will never be better than building a piece of shit that does meet your needs. A lot of the problems in IT departments (I want to say most but won't) come from what purchased and have nothing to do with developing software in-house.
> > For as much as we've butted heads on these forums, I > knew > > there was a reason I liked reading what you wrote, > > regardless. This is priceless :-) > > What do we but heads about? C++? That's just geek > boxing. Not a 'real' fight.
That's about it, but for some people that's a big deal. I tend to be pretty thick skinned and don't take too much on these (or any) boards personally, so I don't much care about the sparring. I just know some people sometimes get REALLY bent outta shape. Even (or maybe especially?) when they're wrong ;-)
I'm just more interested in learning and for me that usually involves asking a lot of questions about things I don't understand (for which I've been called dumb, among other things. whatever) or calling people out on things that I know/think they are wrong about. I like being corrected when I'm wrong and I enjoy having things pointed out to me that can help me be better and sometimes I do the same for other people. In an overzealous manner I've been told on several occasions.
Anyway, I'm probably going to be stealing that quote from you in the future.
> > > For as much as we've butted heads on these forums, I > > knew > > > there was a reason I liked reading what you wrote, > > > regardless. This is priceless :-) > > > > What do we but heads about? C++? That's just geek > > boxing. Not a 'real' fight. > > That's about it, but for some people that's a big deal. I > tend to be pretty thick skinned and don't take too much on > these (or any) boards personally, so I don't much care > about the sparring. I just know some people sometimes get > REALLY bent outta shape. Even (or maybe especially?) when > they're wrong ;-)
To totally honest, I have on the past picked fights on forums to blow off steam. I like to think I've outgrown it. It really wasn't personal but you're right, dissing someone's programming language can be like dissing their football team. They take it very personally. Which is why I've decided to stop doing that. OK, well, I'm also bored with arguing about languages (I'm no saint.)
My principle complaint with Tim's piece is that he continues with the shibboleth that the point is to make software that works the first time, is easy to maintain, uses appropriate technology, etc. That's never been true. The points are to: 1) get there first with something that sort of works 2) lock-in as many clients to your software as possible 3) maintain bureaucratic control of development (I am God, and you drudges will do as I say), once 1) is accomplished
In the Enterprise world, most applications that are needed were created 30 years ago, so that world lives at point 3) forevermore. Hell, there are "legacy" spreadsheet macros that can't be improved because "they work OK".
Webbies are approaching point 3) in far more cases than is widely understood. Parts of ESPN remain tied Tea Servlet. The God Guy there is personally invested. Each of you, Dear Readers, knows of a similar situation.
> This prompted a funny thought. Raise your hand if you'd > choose to fly on an airplane with a computer system > designed and developed the way Twitter is?
Maybe one should try it out? This is usually the idea of doing design studies and research projects. At least we live in a culture that formally values inventiveness and curiosity higher than dogmatism. Not all is lost.
> > This prompted a funny thought. Raise your hand if > you'd > > choose to fly on an airplane with a computer system > > designed and developed the way Twitter is? > > Maybe one should try it out? This is usually the idea of > doing design studies and research projects. At least we > live in a culture that formally values inventiveness and > curiosity higher than dogmatism. Not all is lost.
You can be the first. Let us know how that goes.
I love that sound principles are referred to as dogmatism. You're obviously very enlightened. That's just great. Personally, in cases like this, I'm a big fan of "dogmatism". I hope they continue to dogmatically build bridges, roads, vehicles, radar systems, medical instruments, nuclear devices, and so on and so forth....
> > This prompted a funny thought. Raise your hand if > you'd > > choose to fly on an airplane with a computer system > > designed and developed the way Twitter is? > > Maybe one should try it out? This is usually the idea of > doing design studies and research projects.
There was a time that airplanes were developed this way and it was a very dangerous time to fly. When a plane fails, it doesn't just throw up a fail-whale. People usually die. Most other cases aren't that dire but the key is that when you develop something that not only appears to work but needs to work reliably day-in and day-out, there's a lot more effort.
Having said all that, I think IT is plagued by a bunch of bad ideas that come from (most) IT management lacking a deep understanding of technology.
@Merriodoc "I love that sound principles are referred to as dogmatism."
Sound principles become dogmatism when they are applied to situations they are not meant for. A startup company in the "Ramen Noodle" phase could not possibly take a slow, disciplined approach to its software: the company would be out of business within a year. (This is of course an extreme case, but it's not hard to point out whole industrial segments that benefit by rapid development.)
I don't know which methodology aviation software companies use but it's probably Waterfall, Spiral, or some variation of them. Intuitively it seems that these approaches would work well for life-threatening situations. However, to Kay's point, it would be interesting to find out if other methodologies have been tried - as well as analysis of errors not handled properly in software designed with the current methodology. It seems that the critical factor is whether or not aviation software meets a stringent set of tests put forth by an independent body such as the FAA.
One last point is that no methodology will save hastily conceived software. Executives have been slow to accept that good products take time to develop. Ironically, the Waterfall method's greatest contribution is that it forces a project to move slowly. Dr. Royce made mention of how the cost (in time and dollars) made the method not feasible in many organizations.
@James "Having said all that, I think IT is plagued by a bunch of bad ideas that come from (most) IT management lacking a deep understanding of technology."
I'm in full agreement that IT management deserves plenty of blame. However, during the hiring spree in the mid to late 90's, too many people were hired for developer positions who had no experience. The situation could have worked if the same people, who were only apprentice level, were not also put into critical roles. The documented failures since then made people who decide on strategy look for "Silver Bullets" in the tool and process areas to mitigate the lack of skill in the people area.
Flat View: This topic has 39 replies
on 3 pages