In his speech to the 2006 MIT Emerging Technologies Conference, Amazon.com founder Jeff Bezos noted that about 70% of a project's time and effort is spent on tasks that, while important, do not differentiate a product or a business in any way from its competitors. Bezos suggested that such features, or "muck," may be outsourced to third parties.
An often-used criteria in measuring a developer's effectiveness is how quickly a developer can complete a given task. That's because most organizations are able to break down a bigger project, such as a new product release, into smaller tasks, and to the extent those tasks involve writing code, prefer to hire developers who can execute those tasks fast.
Thus, from a developer's point of view, getting something done quickly is a positive. While most developers would want to proceed with a task as quickly possible, many things slow a developer down on the way to the finish line. Apart from considerations of quality, and human factors—which are not the subject of this post—less glamorous forces can loosely be grouped under the term "infrastructure."
By infrastructure, I mean those aspects of a project that do not directly define a product or a service, but are nevertheless necessary ingredients of the solution, in the way a road is a necessary, but not sufficient, element in reaching a terrestrial destination.
While infrastructure elements are seldom defining components of a product, they have a big impact on the productivity of each developer working on that product or project. And that's not just because developers have to work with such infrastructure elements, but also, and mainly, because developers are often tasked with creating that infrastructure in the first place.
According to Amazon.com founder Jeff Bezos, on average we spend about 70% of our time on dealing with such tasks—or "muck," as he calls those chores—instead of developing the aspects of our products that really matter. In addressing the MIT Emerging Technologies Conference, Bezos noted that,
Everybody who ever started a successful company wants to ... go directly from [an] idea to a successful product. The reality is that there are a lot of obstacles between your idea and a successful product...
Your major bottleneck in developing your ideas and your successful products and service offerings turns out to be something that's actually not unique to your business at all. It's completely undifferentiated. This is certainly true at Amazon.com, and it's certainly been true from the very beginning from one person to twelve thousand people: that at least seventy percent of your time, energy and dollars, go into this back-end heavy-lifting infrastructure. I call this heavy-lifting, muck.
The thing about this muck is, you have to do it, unfortunately, at the highest quality level. Because if you don't do all of these things that have nothing to do with your business at superb quality levels, your idea will never succeed...
So you have your idea, and between you and your idea is a bunch of undifferentiated heavy lifting. And you have to cycle through that loop, and the winners are the ones who can cycle through that loop faster than others [can]. So you end up being slowed down.
I was not completely surprised to hear that 70% of the effort of a business is spent on non-differentiating infrastructure components. If you ever worked on user management, authentication, discussion forums, inventory management, and so on, many of those features might very well all belong to the 70% of necessary, but un-differentiating, product features. Many projects repeat core infrastructure-level features over and over, slowing that business down in delivering features that differentiate the business from its competitors.
It is inevitable that products and innovations will help reduce the time developers spend on non-differentiating tasks. The question is what form those services and tools will take.
In the past, such progress occurred by raising the level of abstraction developers work with. Economies of scale have allowed operating system vendors to completely automate tasks such as saving data to disk or shuffling bits between network nodes. While current tools attempt to make developers faster in developing code, if the past is any indication of things to come, products and tools will help entirely eliminate the need to develop large areas of a project's code, perhaps, again, by raising the abstraction level within the context of an application domain.
Spending some time learning Rails lately, I saw some of that already in the domain of Web applications. Many developers like Rails because it reduces the chunk of time spent on "muck," allowing a developer to shift that time to producing features that can differentiate a product or a business. Figuring out how to display error messages, or perform input validation, or even to sign up and manage users, are hardly factors that make or break or business. As Bezos said, those things also have to be done very well, but are alone insufficient in creating a compelling application.
While toolkits, such as Rails, reduce "muck" to a great extent, in the future network-based services may speed up a developer's work even more. Instead of creating a user administration framework, we might be able to just invoke a network service that helps manage users. Or, instead of writing code to save images to a file system, we might just invoke a network storage service, such as Amazon's S3 Web service.
Many companies and development teams have worked so closely with their applications, and for so long, that they have over time come to believe that almost every aspect of their application is unique, and that most every feature helps differentiate the application from the competition. In reality, only a few features or areas of functionality are truly crucial for a business' success. Because development resources and time are finite, the more honestly a business or development team can asses what those critical, differentiating feature are, the more likely that team will succeed. And, equally important, the more productive and speedy developers will feel.
But what to do with the rest of the application, with the necessary but non-differentiating "muck"? As countless hours are spent re-inventing the wheel and developing non-differentiating features, the pressure will increase to outsource such features to network-based services. That will especially be the case as such services evolve into viable alternatives in the coming years.
If you took an honest look at your application, what percentage of your work would you say is dedicated to truly differentiating features? And to what extent would you trust third-parties to handle your application's "muck"? Would you trust Amazon.com to, say, manage your application's users or your users' files? How much more productive do you think you would be if you could outsource those essential but non-differentiating functions?
I totally agree with the philosophy of outsourcing muck. Web services (with hopefully an easier way of using them) I am hoping will actually help us achieve this goal well.
Regarding Ruby being faster than Java, even though I cannot comment on Ruby, since I have never worked on it, I am increasingly getting a frustrating feeling working with Java. I don't think it's because of the hype created by other languages, but more because there are just way too many things a Java developer has to know and keep up with in order to do work. Even for building simple web based applications developers must be familiar with:
The Java language The core Java API JDBC Servlets JSP + Tags, EL (and now JSF) Struts (or another MVC framework) Hibernate (or another OR tool or EJB's)
Learning these technologies takes a long time, but after learning one also has to keep himself updated on these technologies, and we know the rapid rate at which stuff changes in the Java world.
When we make a simple web based application, beyond the source code we deal at least with 3 configuration files; web.xml, struts-config.xml, and hbm.xml, not to mention the developer needs to know about packaging a J2EE application using jar, war, ear, ... files.
All this seems like way too much work to make small to mid sized applications. Sure the technologies have value, but the effort is disproportionate to the value recieved in terms of 'programmer productivity'.
Maybe we do need all these formalisms and specifications for large distributed applications, or maybe not. I am not sure, but I am sure that we do not need them for small to mid sized applications.
If anyone has had experience developing large distributed applications in a language other than Java, like PHP, Python, or Ruby, then please share your experience with us.
This topic is a continuation of 'what limits software growth'. The current software technology is more limiting than enabling, requiring programmers years of experience for relatively simple tasks like an online shop.
The solution is to change the way operating systems work: instead of filesystems/processes/drivers/libraries we need data/functions/types/automatic persistence/network transparency.
Of course the old way shall not go away either because some tasks really need low level access...but the majority of work does not.
> In any product development, the unique aspects will always > be a small percentage of the overall effort. > > That fact doesn't really say anything in general about the > desirability of outsourcing or the availability of > appropriate generic building blocks. > > That deterimination must made on a project-by-project > basis. > Buy versus build decisions are difficult and important. It isn't as simple as outsourcing anything that isn't a "unique aspect" of your business.
First of all, one problem is that when you outsource, you have to spend time and money incorporating the outsourced system into yours, so you have to try and estimate those costs, and compare them against the cost of building it yourself. Those costs are not always so easy to judge beforehand.
Second, unless you don't outsource to some implementation of a standard, which theoretically would still leave you the opportunity later to switch providers, you will often be marrying your project to that provider. There's risk there that must be weighed against the risks of trying to build your own.
Third, if you're doing something interesting, many of these seemingly infrastructure pieces may actually be unique in your case. Frank mentioned user management as a potentially undifferentiating feature in his blog. Often it may be, but in our case, Frank and I spent about a week discussing all the different ways we could do user management. Do we use a nickname, or an email as the login? If a nickname, can a user account have multiple email addresses? Also, if an email is an option for login, does the user need to select a nickname up front, or could that be optional? What can they do if they forget their password but no longer have access to the email address? What if they give an invalid email address upon registering before realizing they need to visit a URL they receive an email to complete the registration? If they specified their favorite nickname, have they lost the opportunity to use it? What happens if they log into one site in our network, and then hop over to a different one? Do they need to sign in again? Do they need to supply their password again before viewing or editing personal data? Can a user subscribe to a mailing list without registering? If so, what happens to those subscriptions when that email is given in the context of a subsequent user account? There were probably hundreds of such questions and scenarios to consider.
It really took us a week to hash through all the myriad possibilities, and make decisions as to what we were going to do in Artima's new architecture. Sure, most websites have a requirement to allow users to create accounts, login, get reminders of passwords, etc. But if you look closely, you'll often see slightly different requirements. That is one reason the cost of incorporating an outside framework is difficult to predict up front--because it is hard to predict when your own requirements will become slightly unsupported by the framework. When that happens the cost of outsourcing goes up.
I think that being able to make prudent buy versus build decisions is one characteristic of a good architect. It takes experience and insight, and perhaps a bit of luck.
i think you touched on the hardest part bill, it's data. data data data! data drives it all; how do i move it, how do i store it, how do i protect it, where do i get from, is it clean, where should i put it, will i be allowed to access it etc.
i think modern programming completely misses this as a major major point. there should be better tools than databases and flimsy abstractions like orm to make bulk data management easier.
if i had one thing that i could point at and say, make this easier, i would point it at the data.
My experience is that it is usually cheaper to build than buy, even when the cost of buying is zero dollars. "All" components aquired from elsewhere are effectively crap, because (A) they've generally got low code quality, and (B) nobody here knows them.
It's not 100% true - using Apache, perl/Ruby, a C compiler, and an operating system written by others has been a good choice. There's some other components that has been OK. Most components we've used has been a clear loss - it would have taken less time to write something really appropriate than it has taken in extra debugging and adaption work from using that component. This is especially true of binary components.
> i think you touched on the hardest part bill, it's data. > data data data! data drives it all; how do i move it, > , how do i store it, how do i protect it, where do i get > from, is it clean, where should i put it, will i be > allowed to access it etc. > > i think modern programming completely misses this as a > major major point. there should be better tools than > databases and flimsy abstractions like orm to make bulk > data management easier. > > if i had one thing that i could point at and say, make > this easier, i would point it at the data.
You hit the nail right on the head.
For example, logging in a web site should be as simple as this:
type User = [string username, string password] persistent var users = set of User var username = '' var password = '' var loginForm = doLogin(username, password) var ok = do(loginForm) if users.contains([username, password]) then ... else ... end if
Instead of being so easy, one has to write many pages of code, too large to post it here.
Registering a user should be as simple as:
var username, password var ok = doNewUser(username, password) if users.contains([username, password]) then doUserExists(username) else users.add([username, password]) doUserRegistrationSuccessful(username) end if
Instead of the above, we currently have to write lots of lines of code for entering a new user into the database (and without taking into account failure).
And the simplicity comes from data being declared persistent within the language...
"Bezos suggested that such features, or "muck," may be outsourced to third parties."
If you bring this all together there is an unstated (at least in this article) assumption: outsourcing the muck saves time.
From what I have seen with outsourcing, this is not always true. I'm being generous here because my experience is not only that it isn't true but that it actually can slow things down. Aside from that, the quality of outsourced work I have seen is almost always extremely poor in terms of reliability, correctness, and ease of maintenance.
Outsourcing often greatly increases management overhead and steals time from senior team members as they have to teach a new group of developers the same lessons on every project.
Outsourcing is a useful tool for temporarily increasing resources for unusually large projects but outsourced work cannot be 'thrown over the wall'. Outsourced resources must be integrated with the in-house teams.
On the other hand, using improved tools, libraries, or techniques to reduce 'muck' is something that I believe all shops should be doing in order to improve productivity and quality. If a reoccuring task can be automated, it usually should be.
Great post. I work in a large IT shop (at least for our area) and it constantly amazes me how long it takes to get even simple tasks accomplished. Alot of the overhead we face is self-imposed. We have created groups that are so specialized that to even add a field to a database table on a development server can take coordination across four groups. This often takes several days to accomplish.
What I think is interesting is that this is not a new problem. Lean manufacturing is always looking at how long it takes for a product to go through an assembly line. Lean manufacturing folks are especially keen on wait vs. work time. How long does a task take because of waiting (decisions from other groups, delays in setting up meetings) vs. work time (i.e. the time to actually do the work).
What many people do not really think about is that if you can gather these numbers (i.e. wait vs. work time) you can very quickly pinpoint many of the inefficiencies within your software development processes and take steps to correct them.
One advantage of outsourcing the dull stuff is that your developers wont reinvent the flat tyre in 15 interesting flavours.
I am constantly surprised by how many different ways we have found at the company I work at to do each and every task. None of them could be described as best so why do we have nearly a dozen places that we store our website configuration data!
Anything to stop the developers yak shaving has got to be a good thing.
This is an unfair question, so don't take it to heart.
Peter Hickman wrote > I am constantly surprised by how many different ways we > have found at the company I work at to do each and every > task. None of them could be described as best so > why do we have nearly a dozen places that we store our > website configuration data!
> This is an unfair question, so don't take it to heart. > > Peter Hickman wrote > > I am constantly surprised by how many different ways we > > have found at the company I work at to do each and > every > > task. None of them could be described as best so > > why do we have nearly a dozen places that we store our > > website configuration data! > > What are you doing to fix that?
Its not a bad question. I struggle with it myself. It always easy to identify that there are problems. What am I going to do fix the problem? In my organization, I lead several different technical development teams. I always hear people say, "There needs to be some leadership here...." One thing I always do (and I have to follow my own advice sometimes) is: "In the absence of leadership maybe you are the one that needs to lead...."
I think the biggest challenge is being able to provide hard numbers around the problem and then offer solutions. In my group we are trying to collect wait vs. work time. We want to then look at how we can break down the walls between the different groups (or at the very least eliminate them).
Outsourcing can solve some problems, but there if often overhead associated with outsourcing that most people do not think of. The management team only sees the low cost of the outsourced work and not the communication or infrastructure overhead needed to make outsourcing work.
> One advantage of outsourcing the dull stuff is that your > developers wont reinvent the flat tyre in 15 interesting > flavours.
Do you feel it is better when the contractors reinvent the wheel?
Here's an anecdote from one of my experiences with a (large) outsourcing company:
They were showing us their code. I noticed that they were explicitly checking for errors in every step of their processes except for the logging calls. Given that the tool they were using had a very useable exception handling built-in, I asked why they were doing this manually (many dozens of times in scores of processes) instead of using the built in exception handling features.
The contractor said that they didn't want to catch errors from the logging calls and attempted to move on. Stopping him, I pointed out that a failure from the (custom) logging call would then result in an unhandled (silent) process failure. He paused, said "yes" and then continued with the presentation.
As it turns out, this exact methodoloy was the cause of quite a few serious bugs in the system for the reason I pointed out and other reasons that would be hard to ignore with proper exception capturing.
> I am constantly surprised by how many different ways we > have found at the company I work at to do each and every > task. None of them could be described as best so > why do we have nearly a dozen places that we store our > website configuration data!
And I have been surprised how contractors will redevelop solutions to problems that have already been solved by the full-time team of developers much more effectively.
I'm not really sure why people assume that outsourcing means you will get better developers. Maybe if you pay a lot for premium outsourcing but then one of the primary reasons for outsourcing comes into question i.e. savings. My experience is that a lot of contractors are fresh out of school with zero production system experience. They are learning (and making mistakes) on the client's dime.
> Outsourcing can solve some problems, but there if often > overhead associated with outsourcing that most people do > not think of. The management team only sees the low cost > of the outsourced work and not the communication or > infrastructure overhead needed to make outsourcing work.
The question I have here is that if your full-time developers can't find these existing solutions, what chance does a developer with absolutely no familiarity with your system have? Even if the contractors take the time to find out what is there, they are doing that instead of developing solutions. And if you don't get documentation on what they found, you'll be paying for that again when the next contractor comes through, assuming they bother to do it.
Flat View: This topic has 21 replies
on 2 pages