The Artima Developer Community
Sponsored Link

Weblogs Forum
Software Metrics Don't Kill Projects, Moronic Managers Kill Projects

50 replies on 4 pages. Most recent reply: May 16, 2008 1:38 PM by Thomas Cagley

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 50 replies on 4 pages [ « | 1 2 3 4 | » ]
Cameron Purdy

Posts: 186
Nickname: cpurdy
Registered: Dec, 2004

Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 1, 2007 7:42 AM
Reply to this message Reply
Advertisement
New Study Finds that Projects Don't Kill Moronic Managers, Blunt Traumas Kill Moronic Managers.

Full Story at 11.

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 1, 2007 10:33 AM
Reply to this message Reply
> > Guns don't kill people, people
> > kill people
>
> And frankly that sums it up, IMHO. Avoiding a potentially
> useful tool because it can be abused/misused is not a sane
> strategy.

Perhaps we can require a 14-days waiting period and a permit to use some metrics :-).

Alberto

Mike Petry

Posts: 34
Nickname: mikepetry
Registered: Apr, 2005

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 1, 2007 11:27 AM
Reply to this message Reply
Metrics can be hard for software folks because we are so binary. Either the finished product is cool or it sucks.

I personally believe in using proven methodologies and insist on delivering well designed, cleanly written, maintainable software. However, I have seen cases in which the most ugly software practices have delivered products that users loved. Conversely I have seen disciplined teams deliver high quality steaming piles of poo (from the users perspective).

What bothers me about metrics is that they can make things more complicated than simpler.

Do is suck or is it cool?

Vincent O'Sullivan

Posts: 724
Nickname: vincent
Registered: Nov, 2002

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 2, 2007 6:15 AM
Reply to this message Reply
The purpose of a software metric is to make a complex thing simple so that it can be evaluated quickly. It's a compression process that reduces a piece of code to a single integer. In the same way that if a picture was reduced to a single pixel, all detail is lost in the process.

The average picture, reduced to a single pixel, would produce a pixel that is grey with a brightness of about 18%. If you take a picture and process it in that way and your result is slightly green with a brightness of 25% then what does that tell you about the quality of the original picture? Nothing.

If I run a metric against my day's work and the result is "17" and a co-worker's day's work can be summed up as "12", what does that tell anyone about the work that we've done today? How good is our code? How much of it is there? How complex was the task being coded? How old is the code? Etc., etc. All that information is not only lost but it was never embodied in the code in the first place.

The fact is that the metric - being a mechanical derivative of the code - is of no real value except to the individual developer and then only as a blunt pointer to potentially problematic code.

Manager are rarely interested in implementation details, so there little or no reason for them to be interested in a single numbers derivated from those details. Fortunately, my experience is exactly that.

Vince.

Ed Kirwan

Posts: 3
Nickname: kirwan
Registered: Nov, 2007

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 2, 2007 8:25 AM
Reply to this message Reply
Metrics will never be 100% agreed-upon while disagreement is possible.

In other words, the only metrics that cannot be argued are those that are mathematically, provably true; software and proof, however, seldom go hand-in-hand.

You often have to jump through hoops just to prove the most basic of commonly-held beliefs:
http://www.edmundkirwan.com/encap/intro.html

Amen to Halonen's, "The lack of metrics and the ability to equate them to our business value is our own fault."

.ed

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 2, 2007 3:08 PM
Reply to this message Reply
> The purpose of a software metric is to make a complex
> thing simple so that it can be evaluated quickly. It's a
> compression process that reduces a piece of code to a
> single integer.
> ...
> The fact is that the metric - being a mechanical
> derivative of the code - is of no real value except to the
> individual developer and then only as a blunt pointer to
> potentially problematic code.

Hi Vincent,

I believe that with this line of thinking we are throwing away the proverbial baby with the bathwater.

Of course metrics simplify and reduce, that's what they are meant to do. It's a strength, not a weakness. Metrics can contain much more information that you give them credit for. By your line of reasoning, we should do away with most metrics and indices:

The Dow Jones Industrial Index, unemployment rate, MPG, cholesterol levels, ppm of X in Y (parts per million of X in Y - e.g. ppm of lead in kid toys), etc.

I believe that most people understand that a single number cannot tell the whole story. But a single number can be an extremely useful indicator and sometimes it's a necessity. Other than averaging stocks into some kind of index, or up/down ratio, or something else, for example, how are you going to communicate at a high-level what went on in the stock market?

Most things have properties and qualities that are of potential interest to somone. In grapes, let's say, dieters may care about calories, environmentalists may care about ppm of pesticide residue, vinemakers may care about % of sugar, and so on.

Similarly, if one believes that some code patterns are potentially risky (e.g. very complex code without any tests) and others are benefical, to me it makes a lot of sense to identify, measure, and study those those patterns.

I continue to be very surprised by some people's aversion to any and all software metrics. I love working with software. I think it's one of the best jobs in the world and I know that it involves a lot of creativity and unique problem solving. But I am not so enamored of it, or think it's such a special thing, that there are no aspects of it that can or should be measured and compared. On the contrary, I believe that our complete lack of any agreed-upon measurements and/or standards has hurt the software industry and will continue to do so.

Alberto

Peter Doubleday

Posts: 2
Nickname: aardvark
Registered: Nov, 2007

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 2, 2007 5:49 PM
Reply to this message Reply
I look forward to a post that actually suggests some value, either empirical or philosophical, inherent in the use of software metrics. This one isn't it.

No, software metrics are not evil. Many things are evil. Software metrics are more on the lines of "pathetic." They are definitely useless -- although software planning metrics on a historical and statistical basis, a la Spolsky, may well be valuable.

We can only judge by results, because the IT industry presently has no real concept of blind tests. I would refer your readers to www.hacknot.info for an in-depth discussion on this issue. One of the points that "Ed" makes on that site, and I think it's important to the way we approach these nebulous measurements, is that we all (and I've become convinced that this applies to me, too) need to apply critical thinking in a logical way. Unfortunately, when you say:

"Better yet, the programmers would have great evidence to have the moronic manager fired."

... this is actually meaningless. First of all, you're proposing the lemma that there is, indeed, some sort of meaningful software metric, or collection thereof. (This being a lemma, it's not possible without further proof to say whether you are wrong in proposing it or the software manager is wrong in misusing it.)

Secondly, you define this as "great evidence." Even if you went forward from the lemma to the proof (QED), you'd still need quantifiable evidence based against, presumably, some sort of algorithmic expression of that proof. Which would probably fall within the domain of statistics. And would probably be questionable on the basis of methodology, even given a universal acceptance of the proof.

Thirdly, you assume that evidence of 100% failure to do one's job, as defined in the tedious legal crap that we all sign in order to do our job, is sufficient cause to be fired.

It's not.

There's no such thing as a silver bullet.

But if there was, I wouldn't reserve it for vampires -- because I <i>know</i> they don't exist. And I wouldn't reserve it for bosses, because I <i>know</i> my only control over them is to leave.

I'd reserve it for people who argue that software metrics have any conceivable value at all, without significant comparative work.

Morgan Conrad

Posts: 307
Nickname: miata71
Registered: Mar, 2006

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 3, 2007 10:40 PM
Reply to this message Reply
Whoa Peter,

While I'm with you that today metrics may not mean much, IMO they are worth collecting as data, to determine if any are significant. We need to correlate various metrics with "did this project go well". Maybe, maybe, there is some metric or combination thereof that, for a given type of project, indicates that you are on the right track. Maybe there isn't.

But without collecting the metrics we have no chance.

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 4, 2007 7:50 AM
Reply to this message Reply
Peter,

Frankly, I don't see the need for such a vitriolic, and rather insulting, post. I noticed it's your first post on Artima, but still ...

We are trying to have a conversation here. If you read the original post more carefully, you'll see that it's trying solicit feedback and experiences from both sides of the metric camp so that we can learn from other people's experiences. We want to gain a better understanding of other people's position so we can revisit and adjust ours - not sling mud at each other.

> No, software metrics are not evil. Many things are evil.
> Software metrics are more on the lines of "pathetic."
> " They are definitely useless

Those are very strong words. I can see someone saying things like: "I have tried several metrics and found them not useful", or "software metrics don't really apply in my situation"; but I can't see how anyone can categorically label all metrics as 'pathetic' and 'definitively useless'.

Unless, that is, you consider yourself above all other people in the industry in terms of intelligence, experience and insight, and consider everyone who uses metrics as 'pathetic' and deluded. I, and many others, have found software metrics - when used judiciously and in the right context - to be incredibly useful; if not downright necessary in some cases.

> I'd reserve it [a silver bullet] for people who argue
> that software metrics have any conceivable value at
> all, without significant comparative work.

Thanks for the warning - I'll stay out of firing range :-).

I don't know what you would consider 'significant comparable' work; but I would argue that there has been a lot of significant and valuable research on software metrics at pretty much every major academic and industrial institution involved in software that you can name. You could probably spend the rest of your days catching up with all the work in this area.

If you took the time to check the body of research and experimentation in software metrics, you'll see that most researchers and practitioners approach the subject with a great deal of skepticism and respect. They know that metrics must be used with care. They know that, in the wrong hands, they can be OVER-used, ABused, and MISused. But they also know that properly used, software metrics can provide uniquely valuable insights.

IMO, to label the work of these people as 'pathetic' and 'definitely useless' shows not only a lack of respect, but a profound lack of understanding.

Vincent O'Sullivan

Posts: 724
Nickname: vincent
Registered: Nov, 2002

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 5, 2007 3:26 AM
Reply to this message Reply
My argument is not that metrics are good or bad (they're just data, after all) but that they are gross simplification of the thing that they are measuring.

The question is always; does the metric provide useful information? I had to replace the fuel tank on my car recently. Knowing its volume was useful information but knowing its colour wasn't. (On the other hand; if I rode a motorbike then maybe the colour would be important, too.)

In the case of metrics that are compiled from the code; I think that a manager who has no interest in the code implementation details (because it's not something that they need to know) equally should have no interest in a metric generated from that code - whether it be line count, crappiness, complexity or whatever.

I said in my original statement that they are a blunt tool of interest only to the actual developer. I think that the 'crap' metric illustrates that. The 'crap' metric is a meta-metric in that it takes two metrics (each of which measured independent properties of the code) and combined then to produce a third figure. In doing so knowledge, of what the actual problem in the code was that was being flagged, was lost.

I have applied it to some code I wrote recently and it highlighted a number of methods that potentially warranted further attention. What it did not indicate is why those methods were 'crap'. To do that I needed to look at the two figures that were combined to created that metric. Until then I did not know if the code was too complex or too untested (or both). Fortunately, the tool you provided (for the eclipse IDE) makes that information readily available but the fact is that the original metric is just a flag that carries no information than to say "look here".

I therefore stand by my assertion that code metrics, that are mechanically generated from the code, are only of interest to the code developer and do not provide any useful information to a project manager.

David Halonen

Posts: 5
Nickname: dhalonen1
Registered: Oct, 2007

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 5, 2007 4:36 AM
Reply to this message Reply
Vincent, your arguments are decent, but the application of the arguments to the population in general is not useful.

Look at it this way, most of use think "common sense" is a shared thought by the population in general. In statistical terms, this would mean your thoughts are very close to the mean - and in the worse case, no further out than 1 std dev.

A more rigorous approach to this would be to assume one's thoughts are 5 or 6 std dev's from the mean and then look for data that indicates this thought is indeed, "common sense".

Your arguments have a built-in assumption that the developers really, really care about doing good work. Whatever that means. If mgm't doesn't define what really good work is, then folks are left to their own devices. Consequently, everyone does good work, regardless of the "real" quality.

Therefore, it's incumbent upon mgm't to define good work. Right now, all they know (and all we are willing to help them with) is: schedule, bug count, cost & features. After that, it doesn't matter - we all do good work! My self esteem is growing by the minute!

Is it possible for Cadillac to build a quality car w/ inferior parts? Does each component manager ignore the internals of their parts and rely on the good graces of the engineers to "do the right thing"? Would you like medical software or your IRA to be build in this manner?

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 5, 2007 8:07 AM
Reply to this message Reply
Hi Vincent,

First of all let me thank you for keeping this conversation/debate open and useful by demonstrating good faith by actually experimenting with one of the metrics in questions before expressing an opinion. What a concept :-).

I will try to follow your good example and REALLY try to understand your position better. To keep things simple, I am going to focus on a specific metric (I will use CRAP since we recently discussed it and we've both using it on our code) but the concepts should apply to most other software metrics.

Your argument - and please correct me if I paraphrase it incorrectly - is that metrics such as CRAP oversimplify to such an extent that the metric number by itself carries no actionable information at the 'manager level'. Since it takes a developer to go and look at the details before deciding if any justifiable action should be taken, the metric should not be reported above the developer level.

If that's the case, I believe we are more in agreement than you think. I agree that a metric such as CRAP is much more useful to, and actionable by, developers than managers. CRAP is actually meant to be used primarily by developers (hence the IDE plug-in). I also agree that, in mapping the code into a single digit there is an unavoidable loss of information. But I don't agree that such a number does not provide any useful information to a manager. As a matter of fact, I don't think you believe something that black-and-white (i.e. that it's completely useless) either, because you say:

>...but the fact is that the original metric
>is just a flag...


I think I see an opportunity for reaching some sort of agreement because a 'flag' is not completely useless. I need to do some a lot more thinking about it, but I am willing to explore the possibility that metric may be too strong a word for most things we call metrics. Perhaps they are just flags; warning signs, binary indicators that something is potentially wrong and that someone should go take a closer look.

Are we moving closer Vincent?

Alberto

Morgan Conrad

Posts: 307
Nickname: miata71
Registered: Mar, 2006

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 5, 2007 12:20 PM
Reply to this message Reply
Maybe these metrics will end up being used more like medical tests. For example, if someone has high cholesterol, it doesn't mean that they are "bad code" and will die of a heart attack. But it means to watch out for other things. For example, if your code had a bad "cholesterol metric", maybe it works fine. It sells great, acceptable number of bugs, customers are happy.

But when marketing wants to add a "sit around on your butt all day" feature and integrate into the new Web 3.0 "smoke and eat donuts" XML standard, it will be more dangerous to port that code, and he should expect it to take more time, require more testers, etc...


Another, more relevant example. I frankly never worry about cyclic coupling. Cause, in my experience, it's best to bundle everything into a very small number of big .jar files anyway. (see Note a below) So, other than a few basic divisions, there's really no "value" in working very hard to layerize the code. So, I'm guessing that my code has a high cyclic coupling. IF management wants or needs to split it up into many smaller jars, that's a red flag.


(Note a)

One project at our company had ~150 distributable jars. I guess the idea was to have simple updates, versions, etc. Also, not all were required for all applications.

But, it's literally impossible to manage this. For example, if there are 3 versions of each, plus a "null version" (not having that jar at all) you have 4^150 possibilities. Which I think is more than the estimated number of particles in the universe.

Vincent O'Sullivan

Posts: 724
Nickname: vincent
Registered: Nov, 2002

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 6, 2007 2:53 AM
Reply to this message Reply
> Your argument - and please correct me if I paraphrase it
> incorrectly - is that metrics such as CRAP oversimplify to
> such an extent that the metric number by itself carries
> no actionable information at the 'manager level'.
> Since it takes a developer to go and look at the details
> s before deciding if any justifiable action should be
> taken, the metric should not be reported above the
> developer level.

Yes indeed. That's a very good summation of my argument.

> If that's the case, I believe we are more in agreement
> than you think. I agree that a metric such as CRAP is
> much more useful to, and actionable by, developers
> than managers. CRAP is actually meant to be used
> primarily by developers (hence the IDE plug-in). I also
> agree that, in mapping the code into a single digit there
> is an unavoidable loss of information.

You're right. We broadly agree about the tool, what it does and how it can be used by a developer.

> But I don't agree
> that such a number does not provide any useful
> information to a manager. As a matter of fact, I don't
> think you believe something that black-and-white (i.e.
> that it's completely useless) either, because you say:
>
> >...but the fact is that the original metric
> >is just a flag...

Yes, here is the nub of our disagreement. I've just delivered a small application as part of a two man development team, reporting to a team leader and thence - as required - to various managers (the development/delivery manager, the manager of the maintenance team that will be looking after the code into the future and, not neast, the manager of the people that will be using the code (oh and the intranet manager that will be hosting the code)).

In all these cases, none of these managers have any interest in the source code nor any particular property of that code. They do, however, want to know that the code works, is compatable with existing systems, is written in a company approved language/framework that the maintenance team are familiar with, is in the company CVS repository.

To be assured on these fronts, all the managers delegate getting assurance on all these questions to their teams and those team members come (at various stages of the project) and eyeball what we're delivering (UI, system interfaces or code, as appropriate). Those people may (potentially) look also at metrics, but - to date - none has. They will then report back to their managers on whether or not our code is fit for purpose but it's unlikely that the report would carry specific metrics on the source code. (Well that's not entirely true. They do seem to have an unnatural preoccupation with code 'volume' here, e.g. "Application X takes up 3.56Mb of disk space on the server", etc.)

> I think I see an opportunity for reaching some sort of
> agreement because a 'flag' is not completely useless.

OK, agreed that it's "not completely useless" for managers(provided you agree that that's a synonym of "not very useful"). :)

> I
> need to do some a lot more thinking about it, but I am
> willing to explore the possibility that metric may
> be too strong a word for most things we call metrics.
> Perhaps they are just flags; warning signs, binary
> indicators that something is potentially wrong and
> d that someone should go take a closer look.
>
> Are we moving closer Vincent?

In all likelihood. I think the problem is not so much in the mechanics of generating a metric (where you appear to be well grounded) but in determining just what exactly is the best way to "summarise" a block of code, what it is that needs to be said about the code (and to whom) and how much or how little informationneeds to be provided.

Eivind Eklund

Posts: 49
Nickname: eeklund2
Registered: Jan, 2006

Re: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Posted: Nov 8, 2007 1:13 AM
Reply to this message Reply
> I am not arguing that all software should be held to the
> same standard wrt any given metric. For example, I expect
> that medical applications to be more thoroughly tested
> than, say, a video game.

Actually, in practice, it would surprise me if this were generally true. At least console video games get very hard testing for their domain; the general rule for allowing a release used to be 100 hours of testing without finding any flaw, *after* all flaws found in previous testing had been fixed. I suspect that medical software that isn't used in life-and-death situations see significantly less quality assurance.

This illustrates a point: It is very hard to accurately define what domains it would be reasonable to use what metrics over. "Medical software" isn't a single domain - there's a large difference between a reporting app in psychiatry, where a crash is just an annoyance wasting a bit of a doctor's time, and the embedded software running on a pacemaker, where a crash kill the user.

It is even different in different parts of the same app. In the reporting app, recording data may be critical - it's non-fixable - so that part of the code may need be simple, well tested, well documented, etc. On the other hand, a graphical version of textual report generated by the tool may be called up once a forthnight and save a user 10 minutes. Here, the code is non-critical - it can be complex and not have automatic tests. This may be OK, in the same way that lack of automatic tests for video games is OK, because the code end up with very little maintenance done, and manual inspection of the output as OK and manual testing finding that the code run without crashes/memory leaks can be enough.

So, I don't think we need standard metrics. What I think we would probably benefit from is good metric tools and practitioners with knowledge of metrics, so the developers themselves can use metrics to find out what to clean up - possibly most when picking up a new codebase.

Eivind.

Flat View: This topic has 50 replies on 4 pages [ « | 1  2  3  4 | » ]
Topic: Software Metrics Don't Kill Projects, Moronic Managers Kill Projects Previous Topic   Next Topic Topic: Thinking in Java 4e

Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2019 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use