"If you don't create the replacement for your current product, your competition will do it for you."
-- Accepted product wisdom

The Second Time Around

Small companies struggle to get their first product out the door. That's probably no surprise. But, the next hurdle may be even more difficult: getting the second product, the next generation product, out the door. The problems are not necessarily technical -- but they can be deadly for a young company nonetheless.

I believe part of software engineering is the management of risk: risk assessment and risk abatement. There are risks associated with second generation projects in start-ups and small companies that are not present in first projects. These new risks are often a result of organizational changes made since the first product's development. In order to manage a risk, you must first understand it. It also helps to have a name for it.

In the book AntiPatterns: Refactoring Software, Architectures, and Projects in Crisis the authors discuss things done wrong over and over. Providing a name for a pattern of idiocy can be useful. I'd like to borrow the term "antipattern" for this installment of my blog to highlight three particular kinds of idiocy associated with second generation project development and the risks that they bring. They are:

• Waiting too long to start,
• Ancestor worship, and
• Elitism and Jealousy

There are probably many more but these three should provide a good starting point for discussions.

Waiting Too Long to Start

R&D costs, especially in technology companies, can be a daunting, up-front expense. Once "out of that phase", the company's management is measured on their ability to get the company to profitability. This means, of course, increasing revenue and, most likely, reducing expenses.

Technology products typically have a limited useful life in the marketplace. Their lifetime varies, depending on what kind of technology we're discussing. A particular model of computer hardware may have a useful lifetime in the marketplace of 12 months. A particular version of software may have a lifetime of 18 months. A video game might have a useful lifetime of 4 to 6 months. Devices and software outside of the consumer channel last longer. It is this type of market, where the product churn is slower, where I believe most problems occur.

In the early 1990's I worked for a small company that made restaurant management systems. They had a product that they had sold for years... many more years than its useful market life. They could see that the sales of their existing product were falling. Yet, for a very long time, they did nothing. Finally, they hired a fellow to lead a team create a new product to replace their old product. I was hired into the team later.

The first problem encountered by the new product team was discussed in an earlier blog: no product management leadership. Everybody knew that the old product was no longer viable and that a new product was necessary if the company was to survive. Unfortunately, nobody could agree on what form this new product should take. What if you made a decision or expressed an opinion and it was wrong? Now that revenue was dropping precipitously on the old product, the new product had to be a success. Nobody wanted to be blamed for its failure.

There was a concern that the new product wouldn't sell. Note how this is different from the typical mood around a first product of a start-up! The prevailing notion there is, "if we can just get it built, we'll get rich." There is rarely the fear that "nobody is going to buy this pig."

There was also a sense of betrayal among the staff that had worked in product support for years. The original product had become quite familiar to them. It was a comfortable relationship. Now, there was going to be this new product, new problems, and new technology with which they were untrained. For a time, they would need to support two products: the old product and the new one. There was a sense of resentment that their world would be turned up-side-down. That sentiment was probably universal throughout the company.

As money began getting tight, the pressure to ship the new product was intense. Finally, with the company on the edge of bankruptcy, the product was shipped -- at least 6 months before it was really ready. It was a disaster.

Then things made a turn for the worse. The cadre from Marketing and Sales now descended into Engineering blaming the developers for the poor quality and unhappy customers. The software was buggy and unreliable. It was also unfinished (a point we made repeatedly and fell on deaf ears at the time). Instead of working with Engineering to develop a plan to get the software finished, however, the Marketing and Sales forces leaned on Engineering for more features. Presumably, these were the features that the latest round of sales prospects mentioned.

I'm sure you've heard this at some point: "If we don't have this feature, we can't make the sale." If Engineering refuses, Sales tells management they couldn't close the deal because of Engineering's recalcitrance. If Engineering tries to put in the feature and it doesn't work right (and how could it?), then Marketing and Sales tell management that they can't sell a product that is so buggy.

At some point, one of the Marketing bozos came to talk to me directly. I was just an engineer at the time. (I would later run the department after my manager's departure.) Like each of the many days before, he had some feature he wanted built immediately. I pointed out that the checks printed by the restaurant system weren't always right and that the accounting system for the restaurant itself didn't always balance. Didn't he want us to fix those things first?!

"No", he replied. He wanted his features because it provided him political insulation. There was no entrepreneurial spirit here. So, I created a small sign that hung outside my cube. It was a modest protest, but it was the only thing I could manage.

Buggy features made while you wait.
Buggy features fixed in about 6 months.

Can an entrepreneurial company that converts to a revenue producing company return to its entrepreneurial roots? Or, is the management structure of an R&D early start-up company and the management structure of a steady-state adolescent company so different that one cannot emulate the other -- even temporarily? Is such a state-change inside a company irreversible?

The story does have a happy ending. The company went bankrupt. But, in a package deal, the assets were sold to another group of investors. With all of the morons from Marketing and Sales gone, Engineering was able to quietly finish the product, fix the bugs, and resolve all of the outstanding customer issues.

As it turns out, all those features which were critical to the next sale actually weren't necessary at all. Most all of our sales prospects were happy with the product the way it was. Although they sometimes listed a couple of these features as nice to have, their absence didn't stop a single sale that I know of.

Engineering had built the right product; they just hadn't had a chance to finish it. But, once left alone, completing the project (and making customers happy) was just a question of hard work and solid engineering practices. That, I believe, is the most unsurprising part of all of this.

Ancestor Worship

I worked for an equipment manufacturer that was in need of their next generation product. The founders of the company created the first product and it held a special place in their hearts. Unfortunately, their romantic memories of the project were not always accurate.

I was recruited by the company to run the software department for the next generation machine project. When I arrived, the effort to scope the project and produce a proposal for a DARPA grant was in full swing. I spent my first few weeks talking with the engineers, gathering their input, and creating a master list of tasks that I captured on a set of white boards in a conference room adjacent to Engineering. I dubbed this place "the little nasty room" and the engineers quickly agreed that it was a room that was not much fun to be in.

Software scheduling is always a difficult activity. It is far too easy to miss tasks, underestimate the difficulty of things, and, in general, be too optimistic. This project had some extra pressures associated with it. The first problem was obvious: this was our second attempt with DARPA to get the project done. Our first attempt had been so horribly underestimated that it had no chance of success. Certainly, there was a strong desired to "get it right" this time.

The second problem was more subtle and yet more dangerous: nobody believed my estimates. They appeared to upper management, the ones who created the first machine, to be far, far too high.

The problem was this: they didn't remember how long it took to create the first machine, nor did they remember how many people worked on it.

Here's how I goofed up: I thought this was a technical problem. I thought that the problem could be solved with a simple memory jogging memo that reviewed the previous machine's construction and highlighted the task ahead for the new machine. I also threw in some other familiar data to help make my case. I came at the problem in a couple of ways:

• Accounting for the previous effort
  I created a spreadsheet with the names of the engineers along the vertical axis and the quarters and years along the top horizontal axis. I then filled in the matrix. Johnny worked here 2 years, from here-to-here. Sally worked on the project 3 years, from here-to-here, etc. When I'd finished, I tallied the amount of time spent by the engineers (in total). It was something like 60-80 engineering-years. This was a number close to my estimate for the new project.


• Typing fecundity
  We had software for the current machine. I did some measurements. If all we had to do was "type in the answer", and if the new project was only as hard as the original problem, it would take between 6 and 9 months of the proposed engineering team's time just to type in the programs! Forget about design time, test time, etc. It was going to take 6 to 9 months just to get the typing done! Certainly, this should help set a lower bound on any discussions.


• Data from another, similar project
  As it happens, the management team (who had created the first generation product) had also overseen the development of another machine at another company. A couple of the engineers that had worked on that machine years before also worked on my project now. So, I did a spreadsheet to tally up the effort for that machine. Again, the estimate compared nicely to mine.

I put the finishing touches on my report, had a few of the engineers review it, and finally gave it to my VP in hopes of winning a little respect and support. Guess what happened...

... he buried it. He didn't show it to a soul. The bidding process for the contract continued and the project manager decided, unilaterally and at the last moment, that my submitted estimate was too high. In the end, my bid was cut in half. I was told to consider it a 'management challenge'. My VP (the one who buried my report) and the project manager (the one that cut my estimate in half) both left the company within a month after submitting the bid to DARPA, leaving me with a fresh death march.

The bidding process was only one example of the effects of ancestor worship. Throughout the ensuing death march there was a prevailing attitude from the first generation team that the second generation project team wasn't up to the task. Some of this might be explained by the antipattern below (Elitism and Jealousy) but the bulk of this attitude came from the notion that the new team, the rookies, couldn't hold a candle to the original team.

This project failed. In fact, the company failed as well. It has been over twenty five years since The Mythical Man Month was published providing Fred Brook's answer to the question "why does software cost so much?" and yet we are still dealing with management unwilling to live with the answer: it always cost this much, you've just forgotten.

Elitism and Jealousy

The last antipattern I'll discuss is Elitism and Jealousy. Creating this pattern is easy: split Engineering into two teams: one on the existing product and one on the new product. Or, just create an advanced development function and give new product development responsibilities to them. Finally, remove any Engineering management leadership, sit back, and wait. The fur will be flying in no time.

I was working for a small company in the mid-1980's that needed a next generation of small data terminal products. The original products, used primarily as credit card processing equipment in gas stations, was expensive to produce and unable to compete with some of the new offerings brought forth by our competitors. A fellow was hired as the Director of Advanced Development and charged with building a team to create the next generation of products. I was hired into that team.

The members of the original Engineering team were nonplussed by this sudden appearance of these upstarts. Why did they have to do all the "grunt work" supporting the old product while we got to "play" and create all the new, cool stuff? Hadn't they earned that chance?

Our (the Advanced Development team's) assessment of their work wasn't always complimentary. One of the reasons why the original product line didn't age well is because the Engineering team had been short on theory and long on "hack and wack". Their designs were old. Even their new efforts had too many parts and were difficult to assemble. The pin counts and part counts were too high and reliability suffered. They also didn't take advantage of some of the more interesting, and economical, technology to appear and, in the words of the company President, "they were putting a $20 bill in every box they shipped." The old designs in the existing products had problems and we called them on it. The war was on.

The Advanced Development group suggested moving to a new processor family to cut manufacturing costs, lower software development costs, reduce pin count, increase reliability, and reduce circuit board footprint size. The suggestion was dismissed out-of-hand by Engineering. We made other suggestions and other offers to help with the existing product line but these, too, were rebuffed. I don't blame them. A few ill-timed and obnoxious comments by a few Advanced Development team members towards the existing Engineering team had started a feud that would last long after my departure from the company years later.

The company's problems worsened. Sales now slumped badly from a crop of new competitors. Our costs continued to climb. Even the revenue we had was costing us money since we had negative margins on those products we could sell. Finally, convinced something had to be done, the President, several VPs (including the Engineering VP that had allowed the feud to develop and fester), my manager (the Director of Advanced Development), and a few others -- including me -- were summoned to a conference room for a "what are we going to do about this?" meeting.

The first question the President had was a good one: "what are our competitors doing that they can build these things cheaper, faster, and better than us?" Bored, and a bit uncomfortable in this meeting, I had been absentmindedly playing with my Swiss Army knife and the collection of competitors products arrayed across the conference room table. Finally, while the discussion continued, I took one of the boxes and began disassembling it. When I had finished taking the thing apart, I dropped it on the table with a thud. That got everybody's attention.

Inside this box was a design remarkably similar to the one the Advanced Development team had suggested. The Hitachi 64180 processor was there (the processor we had suggested) along with some of the other design changes we insisted would both reduce costs and increase reliability. Our competitors had done the same analysis the Advanced Development team had done and arrived at the same conclusions. The competitors had acted; we had not. As the quote at the top stated, "If you don't create the replacement for your current product, your competition will do it for you."

Again, showing my grotesque naivete, I thought this might settle it. Obviously we had the right answer now. We should do the right engineering thing, right? Of course not. This was never a technical problem; this was a people problem.

That company was eventually purchased by another company from out-of-state. The Advanced Development team quit one-by-one over the course of a few weeks soon thereafter. I was last to go. None of our designs, none of our products, ever saw the light of day.

This wasn't a technical problem. None of the previous problems were technical problems. We can say that these were management failures or a failure of leadership but that doesn't explain why they happened or hint how these problems might have been handled differently.

Managing Risks

As I stated in the beginning, there are risks associated with second generation projects. I've listed a few of them above but I'm certain there are more of these "antipatterns" out there. What other risks do these "second time around" projects have? How would you handle them? I look forward to your ideas.