The Artima Developer Community
Sponsored Link

Weblogs Forum
Continuous Integration Hell

40 replies on 3 pages. Most recent reply: Oct 28, 2008 12:14 PM by tracy ragan

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 40 replies on 3 pages [ « | 1 2 3 | » ]
Peter Hickman

Posts: 41
Nickname: peterhi
Registered: Mar, 2003

Re: Continuous Integration Hell Posted: Jan 23, 2004 5:08 PM
Reply to this message Reply
Advertisement
If you are renaming a class the simple solution would be to create a new class which is a copy of the old class and then commit. Now change a few of the references to the old class to the new, retest and commit. Repeat until all references to the old class are replaced and then delete the old class.

Then you can go behind the scenes in CVS (for example) and merge the old classes history into the new class.

Making broard changes across the whole system and saving the commits up for one big bang doesn't sound XP to me, or even TDD. Lots of small changes is what you want to be looking for, commits that do only one, almost trivial, thing.

The problem is not 'Continuous Integration' but failing to commit changes when they are complete.

todd hoff

Posts: 22
Nickname: thoff
Registered: Jan, 2004

Re: Continuous Integration Hell Posted: Jan 25, 2004 1:14 PM
Reply to this message Reply
If interested i put some thoughts on the TDD part of this
at http://radio.weblogs.com/0103955/categories/stupidHumanProgramming/2004/01/24.html#a111.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: Continuous Integration Hell Posted: Jan 25, 2004 5:02 PM
Reply to this message Reply
> If interested i put some thoughts on the TDD part of this
> at
> http://radio.weblogs.com/0103955/categories/stupidHumanProg
> ramming/2004/01/24.html#a111.

Todd, I hear what you are saying, but I think there is something special about the case that you present that is atypical. If I'm writing a piece of communications software and it can fail due to all sorts of communication related failures, then yes, it makes sense to exercise through the real communication path and identify the possible failures. But, when you do that you are doing something almost akin to requirements gathering, you are figuring out how things you depend upon fail and using that to drive change to your software.

On the other hand, when you really have a good idea of the results you need for a piece of software you are going to write, you can mock things out and create "known pieces", things that you can tie together to satisfy higher level tests.

I've seen some communications software that looks like it is nearly all error checking, and when you look at what you would unit test independently, computationally there isn't much there. But, not all software is like that (he said with incredible understatement).

Keith Ray

Posts: 658
Nickname: keithray
Registered: May, 2003

Re: Continuous Integration Hell Posted: Jan 25, 2004 9:41 PM
Reply to this message Reply
"Fast Unit Tests" is not a rule to be followed because it's a rule, it's a guideline to be followed because it helps people integrate more often.

Slow tests makes frequent integration painful, which means integration occurs less often, which makes integration more painful because more changes have piled up, etc.

The team has to have an intelligent balance between thorough verification and speed of development -- because while your tests are running, you pretty much not likely to be doing something else.

todd hoff

Posts: 22
Nickname: thoff
Registered: Jan, 2004

Re: Continuous Integration Hell Posted: Jan 25, 2004 10:50 PM
Reply to this message Reply
>But, when you do that you are doing something almost akin
>to requirements gathering, you are figuring out how
>things you depend upon fail and using that to drive
>change to your software.

Why yes, of course. For me that is the big T in testing.
I don't test to find out stuff i already know.
The little t is to drive the design.

If on win32 the directory bits don't work
when archive is set, or some other whacky thing,
then i need to adjust my code. I need to know
that before i release my code into the world.
Having working code in the deployed
environment is the point. If your code breaks
but you say it worked with the mocks, then you
aren't doing your job. I can't get behind
a process that doesn't go to extremes to
eliminate bugs.

todd hoff

Posts: 22
Nickname: thoff
Registered: Jan, 2004

Re: Continuous Integration Hell Posted: Jan 25, 2004 10:54 PM
Reply to this message Reply
> it's a guideline to be followed because it helps
> people integrate more often.

That's not the goal though. Truly working code is the goal.
Frequent integrations can't be used as a reason
not to meet this goal.

>because while your tests are running, you pretty much not >likely to be doing something else.

Why not? I do this all the time.

Keith Ray

Posts: 658
Nickname: keithray
Registered: May, 2003

Re: Continuous Integration Hell Posted: Jan 26, 2004 9:15 AM
Reply to this message Reply
>If on win32 the directory bits don't work
>when archive is set, or some other whacky thing,
>then i need to adjust my code. I need to know

Are functions involving win32 directory bits very slow to call? If not, then don't mock them.

Michael Feathers

Posts: 448
Nickname: mfeathers
Registered: Jul, 2003

Re: Continuous Integration Hell Posted: Jan 26, 2004 10:01 AM
Reply to this message Reply
> >But, when you do that you are doing something almost akin
>
> >to requirements gathering, you are figuring out how
> >things you depend upon fail and using that to drive
> >change to your software.
>
> Why yes, of course. For me that is the big T in testing.
> I don't test to find out stuff i already know.
> The little t is to drive the design.

Then there's the case of things that we think we know are true, but aren't. Or, once were true but because of a bad change, aren't any longer.

> If on win32 the directory bits don't work
> when archive is set, or some other whacky thing,
> then i need to adjust my code. I need to know
> that before i release my code into the world.
> Having working code in the deployed
> environment is the point. If your code breaks
> but you say it worked with the mocks, then you
> aren't doing your job. I can't get behind
> a process that doesn't go to extremes to
> eliminate bugs.

I don't think anyone advocated that. Everyone is responsible for their work, and people need whatever number of tests it takes to be confident in their code. When you can't feel confident in your code regardless of the number of tests, often it's a serious design problem.

There is a reason to have tests even for things that you are confident in (and I think that's the difference between our positions) and that's to prevent bugs. People make mistakes when they change old code at times, that's just life. Tests for "known" bits of code keep them known and make change easier. People don't have to fall into the that conservative mind-set that often causes design to degrade: "let's change this code as little as possible."

Going to the extreme to eliminate bugs? I'm all for it, as long as people don't forget that you can prevent bugs with tests also.

Keith Ray

Posts: 658
Nickname: keithray
Registered: May, 2003

Re: Continuous Integration Hell Posted: Jan 26, 2004 10:49 AM
Reply to this message Reply
>If on win32 the directory bits don't work
>when archive is set, or some other whacky thing,
>then i need to adjust my code. I need to know

And of course, you have localised dealing with this problem into a class or function, and have unit tests for it. All other code goes through this class or function, which has an intention-revealing revealing name. Right?

todd hoff

Posts: 22
Nickname: thoff
Registered: Jan, 2004

Re: Continuous Integration Hell Posted: Jan 26, 2004 10:58 AM
Reply to this message Reply
>And of course, you have localised dealing with this problem
> into a class or function, and have unit tests for it.
>All other code goes through this class or function,
>which has an intention-revealing revealing name. Right?

I do not have a special test because it comes out in
the original test. What is special is making sure you
have example file system data of every type and i do
have that.

We have several ways of dealing with product variants.
In this case it is handled by an ifdef.

todd hoff

Posts: 22
Nickname: thoff
Registered: Jan, 2004

Re: Continuous Integration Hell Posted: Jan 26, 2004 11:14 AM
Reply to this message Reply
> There is a reason to have tests even for things that you
> are confident in (and I think that's the difference
> between our positions) and that's to prevent bugs.

I know that i don't know. And this has been proven
so many times with such disasterous consequences that
i believe firmly that every assumption you are making
must be tested. The degree of discipline to which
i follow this rule i notice the fewer bugs i have in the
field. And this is the real measure. In my area errors
usually have multi-million dollar penalty clauses
attached, so it matters.

> Going to the extreme to eliminate bugs? I'm all for it,
> as long as people don't forget that you can prevent bugs
> with tests also.

Mocks prevent one class of bugs. Another class of
bugs are dependent on your assumptions and more glaringly,
the assumptions behind the assumptions. And in a complex
system there are even more assumptions so that you are
dealing with a dependency tree 20 deep. Interfaces
are useful abstractions. Like all abstractions
they lie, usually unselfconsciously.

If you are not testing your assumptions then you are
not really testing, imho. Some assumptions like getters/setters you may be able ignore. Where we
differ is you seem to move most everything not directly
related to a class into the ignorable realm for
reasons i find similar to reasons people say unit
tests are necessary in the first place.

Robert C. Martin

Posts: 111
Nickname: unclebob
Registered: Apr, 2003

Re: Continuous Integration Hell Posted: Jan 26, 2004 10:49 PM
Reply to this message Reply
> It seems in XP everything is black or white.

No, just in discussions about XP. Real teams doing XP are actually allowed to use their brains.

Robert C. Martin

Posts: 111
Nickname: unclebob
Registered: Apr, 2003

Re: Continuous Integration Hell Posted: Jan 26, 2004 10:57 PM
Reply to this message Reply
> As I understand it the canonical form of XP coding is
> something like this:

Does this mean that you don't have any direct experience using XP?

> <ol>
> <li>Sync</li>
> <li>Write Tests</li>
> <li>Code</li>
> <li>Build cleanly</li>
> <li>Test cleanly</li>
> <li>Merge</li>
> <li>Build cleanly</li>
> <li>Test cleanly</li>
> <li>Commit</li>
> </ol>

Not quite. The idea is to write unit tests and code concurrently for a short period. Get all the tests to pass. Merge with the checked-in system, keeping all the tests passing, and then check in.

> Often times a restructuring of code results in the
> modification of a large number of files.

Yes, this happens from time to time.

> If the time it takes to run a set of tests exceeds the
> average time it takes for a commit to the tree to occur by
> someone not yourself, your commit can fail because,
> surprise, your large and pervasive change has touched a
> file or files that someone else was working on and has now
> checked in. This is the basis for Continuous Integration
> Hell. You merge/build/test/commit/fail again and again and
> again, trying to keep up with everyone on your team. There
> is no tasteful exit from this loop.

If you ever get caught in a situation like this, then you tell everyone that you are integrating, and would everyone please refrain from checking anything in until you are done.

Next you find a way to make your tests run faster. I currently work on a system that has about 600 tests that execute in about 20 seconds.

todd hoff

Posts: 22
Nickname: thoff
Registered: Jan, 2004

Re: Continuous Integration Hell Posted: Jan 27, 2004 12:47 PM
Reply to this message Reply
> > It seems in XP everything is black or white.
>
> No, just in discussions about XP. Real teams doing XP are
> actually allowed to use their brains.

Then why don't they do the same in discussions?

Jon Strayer

Posts: 10
Nickname: jstrayer
Registered: Sep, 2003

Re: Continuous Integration Hell Posted: Jan 29, 2004 1:59 PM
Reply to this message Reply
Here are some real numbers. According to Junit I have 420 tests. They executed in 5.88 seconds. That is, instead of the one second per test that was assumed I measured 14 milliseconds.

2,000 tests would take 28 seconds.

Flat View: This topic has 40 replies on 3 pages [ « | 1  2  3 | » ]
Topic: The Adventures of a Pythonista in Rubyland/1 Previous Topic   Next Topic Topic: Unit Testing Frameworks - When to Abandon or Migrate?


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us