The Artima Developer Community
Sponsored Link

Weblogs Forum
Unit Testing Frameworks - When to Abandon or Migrate?

6 replies on 1 page. Most recent reply: Oct 28, 2008 8:56 AM by Michael Feathers

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 6 replies on 1 page
Andy Dent

Posts: 165
Nickname: andydent
Registered: Nov, 2005

Unit Testing Frameworks - When to Abandon or Migrate? (View in Weblogs)
Posted: Oct 11, 2008 12:36 AM
Reply to this message Reply
Summary
A large body of unit tests as a regression test system is important to many organizations. What do you do when your framework of choice dies or is neglected? When is it time to move on and how do you budget for abandoning or rewriting tests? Have you ever thought about being caught by this situation?
Advertisement

After several years in other languages, I've currently got two projects in C or C++ with a unit test requirement. One code migration and major refactoring presents an opportunity to add tests. The other project needs updating from Visual Studio.Net 2003 and already has a couple of hundred tests which I expect to keep running.

This is the kind of situation many people will encounter if there is a need to maintain a legacy project, especially a mature one which has worked with little maintenance for several years.

My first step was to go and get the current version of cppunit but with some dismay found that the projects included were last used with Visual Studio v6, don't build in VS.Net 2008 and the project admin has gone AWOL - emails bounce (Update: this bad impression is courtesy of a bug with sourceforge mail handling - I've since been contacted and told that webmail through the contact form works and the project is, just barely, alive).

Whilst there is a range of alternatives and debate over which C++ framework to use, due to my familiarity with its patterns and amount of legacy code with tests I am hesitant to migrate.

Thinking about the issue of migration made me wonder - do people ever plan for their testing framework going away?

Hopefully, the bulk of the test bodies would remain the same if you had to migrate, possibly with changes to the assert macros. However, I expect that the registration of tests, establishment of fixtures and combining tests into suite would all change significantly. These seem to be the areas of most dispute in how to design test frameworks :-)

Would you migrate to a new framework, or would you assume the burden of maintaining the old test framework as well as your code?


Henry Ludemann

Posts: 3
Nickname: asdf1011
Registered: May, 2006

Re: Unit Testing Frameworks - When to Abandon or Migrate? Posted: Oct 17, 2008 1:50 AM
Reply to this message Reply
Our system also has many tests written using cppunit, but also many other types of tests (for example, regression tests, domain specific tests...). I don't see a problem with using multiple testing frameworks; the existing tests aren't broken, aren't inhibiting the development and refactoring of the main source code, but still provide value.

If cppunit no longer does what you need, and another framework does, just use both; by their very nature unit tests don't need to work together. So long as your build script is capable of launching both types (and interpreting the results), that's good enough.

Andy Dent

Posts: 165
Nickname: andydent
Registered: Nov, 2005

Re: Unit Testing Frameworks - When to Abandon or Migrate? Posted: Oct 17, 2008 6:09 AM
Reply to this message Reply
In the specific case of cppunit, we want to move on from Visual Studio.Net 2003 and the original version no longer compiles.

This raises an interesting issue about the robustness of a framework over time. If you have a GUI, that's probably the fastest-decaying and most vulnerable part of a test framework. So, whilst I value being able to pick and choose subsets of tests when working in a Test-Driven style, for robust testing it makes sense to have the simplest possible non-GUI framework.

Bringing things back to a Unix-style pipeline architecture allows a build script to run tests and interpret results and mix tests written in a variety of languages, provided they can be executed and generate textual output. Whether you then wrap that in a GUI or use a simple build script can be made a separate decision. One corollary for cppunit is that it makes sense to have a simple command-line build even for Windows with the GUI test-runners decoupled so they can evolve separately.

Matt Doar

Posts: 9
Nickname: mdoar
Registered: Feb, 2004

Re: Unit Testing Frameworks - When to Abandon or Migrate? Posted: Oct 17, 2008 1:10 PM
Reply to this message Reply
I see projects get bitten by this occasionally. Their unit tests are tightly connected to the particular framework, their code expects a certain build tool or even version control tool. It's one of the arguments against building using an IDE, since not everyone uses the same one.

I just bear it in mind when using *any* tool these days and check both import and export options when evaluating.

Henry Ludemann

Posts: 3
Nickname: asdf1011
Registered: May, 2006

Re: Unit Testing Frameworks - When to Abandon or Migrate? Posted: Oct 19, 2008 12:18 AM
Reply to this message Reply
Cppunit is already quite portable (windows / posix, with the core functionality available in command line programs), in part due to its simplicity (which, as you say, increases robustness). When weighing the likely low cost of fixing / porting cppunit against the known large cost of porting to a new framework, or the loss in 'features' by cutting the unit tests, I fall on the side of maintaining the existing test framework.

This is a specific instance that we've dealt with recently, made easier for us by the fact that we've already made some custom modifications to cppunit. It is unfortunate however that the upstream project seems to be unmaintained.

I agree in the general sense though, as even a compiler version change can be seen as a change in platform; sometimes incompatible features have to be cut. But planning for the testing framework itself going away; no, not a situation we've dealt with.

Bob Lauer

Posts: 5
Nickname: rfl4711
Registered: Jul, 2008

Same problem going backwards Posted: Oct 21, 2008 5:19 AM
Reply to this message Reply
I had to change to Visual Studio 6 for one project. The prototype had been written in Visual Studio 7, using UnitTest++ (excellent framework btw).

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: Same problem going backwards Posted: Oct 28, 2008 8:56 AM
Reply to this message Reply
If it's open source it never goes away.

I think that for many organizations, the best thing to do with something as simple as a UT framework is to fork internally. Sure, you don't get updates from the outside, but at least you've taken responsibility and you aren't held hostage by fear of the original developers abandoning it.

Unit testing frameworks are really simple.

Flat View: This topic has 6 replies on 1 page
Topic: Continuous Integration Hell Previous Topic   Next Topic Topic: Goodbye Artima


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us