The Artima Developer Community
Sponsored Link

This is Rediculous
Continuous Integration Hell
by Rick Kitts
January 22, 2004
It's not always as easy as it sounds


As I understand it the canonical form of XP coding is something like this:

  1. Sync
  2. Write Tests
  3. Code
  4. Build cleanly
  5. Test cleanly
  6. Merge
  7. Build cleanly
  8. Test cleanly
  9. Commit
Often times a restructuring of code results in the modification of a large number of files. For example let's say a goodly portion of your code implements an interface or subclasses something called ReallyPervasiveAndUsefulSomethingOrOther and you've decided to rename it to the more finger friendly UseItOrDie. With a reasonable tool this is more or less at mechanical operation requiring little thinking. Being a good XP citizen you run the unit tests and, since this logically single operation has touched lots and lots of files you need to run the entire suite. Let's say this suite takes some real time because your application isn't some out of a book trivial something or other, it's a real product with lots of bits and pieces. Maybe you start the suite and go to lunch. Come back from lunch and, hey, everything passed. No surprise you didn't really change anything. Push the commit button.

Suddenly the lights dim. It's hot. Smokey. The smell of brimstone assaults your nose and your eyes are watering. You hear the unmistakable sounds of people, some cursing under their breaths, some quietly weeping. The frantic, panicy click-clack slapping of keys. Welcome to Continuos Integration Hell. If you haven't been here before you might be wondering how you got here. I mean, honestly, you followed the true path and doesn't that path lead to coding nirvana? Evidently that's only in the story books.

If the time it takes to run a set of tests exceeds the average time it takes for a commit to the tree to occur by someone not yourself, your commit can fail because, surprise, your large and pervasive change has touched a file or files that someone else was working on and has now checked in. This is the basis for Continuous Integration Hell. You merge/build/test/commit/fail again and again and again, trying to keep up with everyone on your team. There is no tasteful exit from this loop.

You can lock the tree. Doing this often is an excellent way of discovering who your real friends are. It is also an excellent way of discovering that you have fewer friends than you thought you did.

You can try to address the fundamental problem and make the tests run faster. There is a point where even this won't help. If you have 2000 tests with an average time of just 1 second it's still 1/2 an hour. And 2000 tests doesn't seem to be a particularly large test suite.

Partitioning the problem space is an excellent idea but has profound implications for the XP process. I won't address those here. Suffice to say that XP in the large looks to be a really dubious proposition.

You can not run all, or any, of the tests. This is what seems to happen most frequently and seems to be the most human path. Inevitably you break things. Eventually everyone breaks things. So when you break something nobody is too terribly bothered. It's unfortunate because, of course, sometimes some poor soul has pulled your changes into their tree. If they see the broken bits you just put in they may (usually will) spend some time determining it's not them. Best in this case to hope they broke things at some point in the past also, lest they decide to cause you harm. That's human too.

You can schedule your change for a time when nobody will be working on the tree. This is semantically equivalent to locking the tree with out the impact on those you formerly called your friends. Sleep or the illusion of friendship is often a hard choice to make.

For myself continuous integration is, far and away, better than isolated branches and the big bang merge towards the end of a project. You know more sooner, and your schedule is outrageously easier to manage. But sometimes it's just hell.

Talk Back!

Have an opinion? Readers have already posted 40 comments about this weblog entry. Why not add yours?

RSS Feed

If you'd like to be notified whenever Rick Kitts adds a new entry to his weblog, subscribe to his RSS feed.

About the Blogger

Rick Kitts has been making a living writing software for a little while. He's started a company or two, worked at bigger companies, but mostly at startups. Constantly on the look out for things to help him build better systems he's a bit of a tool and process slut, though he can't bring himself to try C# or get serious about UML. Go figure. He's convinced being invited to have a weblog on Artima is the result of some glitch in the matrix. He's keeping quiet about it though.

This weblog entry is Copyright © 2004 Rick Kitts. All rights reserved.

Sponsored Links


Copyright © 1996-2018 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use