The Artima Developer Community
Sponsored Link

Weblogs Forum
The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-in

38 replies on 3 pages. Most recent reply: Jan 29, 2008 11:12 AM by Alberto Savoia

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 38 replies on 3 pages [ 1 2 3 | » ]
Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-in (View in Weblogs)
Posted: Oct 1, 2007 8:12 PM
Reply to this message Reply
Summary
After much talking, experimenting, and blogging, we implemented a prototype version of crap4j - a code Change Risk Analyzer and Predictor (i.e. CRAP) for Java. A tool to help you defend yourself against overly complex and untested code. Read all about it and find out how to download and install the crap4j Eclipse plug-in.
Advertisement

[Cue in sound of a giant can of worms being opened]

[Edit 10/18/07 - Given the warm reception and interest in Crap4j, we created a dedicated Crap4J website. Please use this website for getting plug-in updates, reporting bugs, and feature requests. Thanks, Alberto]

Disclaimer

The CRAP metric and the crap4j plug-in are, at this point, highly experimental in nature. The CRAP formula is based on what we believe are sound principles and ideas; but at this time the actual numerical values used in its calculation and interpretation should only be considered a starting point which we plan to refine as we gain experience. Of course, it's also possible that we'll "scrap the CRAP" if we determine that it's not useful.

If you are adventurous and care enough about the topic, read on, download the plug-in, run it on some code, and work with us to improve it (or at least give us feedback, good or bad). Otherwise, I suggest you check back in a few months - after we've gained some experience with this early prototype and did the first round of tuning and cleaning up.

Introduction

There is no fool-proof, 100% objective and accurate way to determine if a particular piece of code is crappy or not. However, our intuition – backed by research and empirical evidence – is that unnecessarily complex and convoluted code, written by someone else, is the code most likely to elicit a “This is crap!” response. If the person looking at the code is also responsible for maintaining it going forward, the response typically changes into “Oh crap!”

Since writing automated tests (e.g., using JUnit) for complex code is particularly hard to do, crappy code usually comes with few, if any, automated tests. The presence of automated tests implies not only some degree of testability (which in turn seems to be associated with better, or more thoughtful, design), but it also means that the developers cared enough and had enough time to write tests – which is a good sign for the people inheriting the code.

Because the combination of complexity and lack of tests appear to be good indicators of code that is potentially crappy – and a maintenance challenge – my Agitar Labs colleague Bob Evans and I have been experimenting with a metric based on those two measurements. The Change Risk Analysis and Prediction (CRAP) score uses cyclomatic complexity and code coverage from automated tests to help estimate the effort and risk associated with maintaining legacy code. We started working on an open-source experimental tool called “crap4j” that calculates the CRAP score for Java code. We need more experience and time to fine tune it, but the initial results are encouraging and we have started to experiment with it in-house.

Crap4J is currently a prototype and it’s implemented as an Eclipse (3.2.1 or later) plug-in which finds and runs any JUnit tests in the project to calculate the coverage component. If you are interested in contributing to crap4j’s open-source effort to support other environments (e.g. NetBeans) and test frameworks (e.g. TestNG) and/or coverage tools (e.g. Emma) please let us know. Instructions for installing the crap4j plug-in are below, but first let’s re-introduce our first pass for the CRAP formula.

The CRAP Formula Version 0.1

Given a Java method m, CRAP for m is calculated as follows:

CRAP(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m)

Where comp(m) is the cyclomatic complexity of method m, and cov(m) is the test code coverage provided by automated tests (e.g. JUnit tests, not manual QA). Cyclomatic complexity is a well-known and widely used metric and it’s calculated as one plus the number of unique decisions in the method. For code coverage we use basis path coverage. Low CRAP numbers indicate code with relatively low change and maintenance risk – because it’s not too complex and/or it’s well-protected by automated and repeatable tests. High CRAP numbers indicate code that’s risky to change because of a hazardous combination of high complexity and low, or no, automated test coverage.

Generally speaking, you can lower your CRAP score either by adding automated tests or by refactoring to reduce complexity. Preferably both; and it’s a good idea to write the tests firsts so you can refactor more safely.

Like all software metrics, CRAP is neither perfect nor complete. We know very well, for example, that you can have great code coverage and lousy tests. In addition, sometimes complex code is either unavoidable or preferable; there might be instances where a single higher complexity method might be easier to understand than three simpler ones. We are also aware that the CRAP formula doesn’t currently take into account higher-order, more design-oriented metrics that are relevant to maintainability (such as cohesion and coupling). Since the perfect software metric does not exist and, regardless of design issues, overly complex methods and lack of tests are usually bad things, we decided that – even in its current state – the crap4j metric provides useful information that we should start experimenting with. This way we have something concrete to give us experience and data for further refinement.

You can get more details on the hows and whys of the current CRAP formula, along with other people’s opinions on it, by checking out the original blog:

http://www.artima.com/weblogs/viewpost.jsp?thread=210575

Interpreting Crap4j Results

For a given method, the CRAP number ranges from 1 (for a method of complexity 1 and 100% code coverage) to a very large number (e.g. a method of complexity 100 with 0% code coverage – we have seen such beasts – would score 10,100).

Individual Method Interpretation:

Bob Evans and I have looked at a lot of examples (using our code and many open source projects) and listened to a LOT of opinions. After much debate, we decided to INITIALLY use a CRAP score of 30 as the threshold for crappiness. Below is a table that shows the amount of test coverage required to stay below the CRAP threshold based on the complexity of a method:

Method’s Cyclomatic Complexity        % of coverage required to be
                                      below CRAPpy threshold
------------------------------        --------------------------------
0 – 5                                   0%
10                                     42%
15                                     57%
20                                     71%
25                                     80%
30                                    100%
31+                                   No amount of testing will keep methods
                                      this complex out of CRAP territory.

In other words, you can have high-complexity methods BUT you better have a lot of tests for them. Please note that we don’t recommend having zero tests – even for simple methods of complexity 5 and below. Every piece of code that does something non-trivial should have some tests but – after many discussions – we believe that the CRAP metric will be more useful if it highlights the truly crappy and risky code, not just the code that could be made better by refactoring or adding some tests.

Aggregate (Project-Level) Interpretation:

At the project level, we report the percentage of methods above the CRAP threshold (i.e., all methods with a CRAP score of 30 or higher). We understand that nobody is perfect and that in some cases people have good excuses for having a few methods that are more complex, or have fewer tests, than the ideal. Project wide, we allow up to 5% crappy methods before labeling the entire project as crappy. Some people will think that this is too generous (Bob and I think it is), while others will thing that it’s too draconian – as we gain experience we’ll adjust accordingly or let people set their own thresholds.

CRAP Load:

Crap4j also reports CRAP load, this is an estimate of the amount of work required to address crappy methods. It takes into account the amount of testing (with a small refactoring component) required for bringing a crappy method back into non-CRAP territory. Broadly speaking, a CRAP load of N indicates that you have to write N tests to bring the project below the acceptable CRAP threshold. The CRAP load metric is even more experimental than the rest, so we will not spend too much time on it at this point. I’ll blog more about it in the future – after getting some experience with it.

That’s it for now. If this is of interest to you, it’s time to download it and start experimenting. Let us know what you think, how you’d change the metrics, improve the plug-in, etc.

Download and Installation Instructions

If you already have Eclipse 3.2.1 or later (right now we don't support earlier versions), you can install the plug-ins from our update site at

http://www.junitfactory.com/crap4j/update/.

Important Note: This prototype version of Crap4j uses JUnit Factory’s test runner because the default JUnit runner does not have the built-in code coverage information that are needed to calculate the CRAP score. JUnit Factory (www.junitfactory.com) is Agitar Labs’ free experimental web-based JUnit generator (originally meant for open-source projects, students, and test automation researchers, but open to anyone who does not mind sending their bytecode over the Internet). JUnit Factory’s test generation services are not downloaded, and are not needed to run crap4j. If you want to try JUnit Factory use the update site: http://www.junitfactory.com/update/. Going forward, we will consider supporting other open-source coverage tools such as Emma – if you want to help with that and contribute to crap4j’s open-source effort let us know.

Follow the steps below to install crap4j and the JUnit Factory runner with built-in code coverage.

  1. In Eclipse, select Help > Software Updates > Find and Install
  2. Choose Search For New Features to Install and select Next
  3. Select New Remote Site
  4. Enter a name for this server: Crap4J
  5. Enter (or copy/paste) this url: http://www.junitfactory.com/crap4j/update/
  6. Install all plug-ins in the Crap4j category and restart Eclipse

Usage Instructions:

Once installed you should see a distinctive – if not exactly tasteful – toilet-paper crap4J icon in the Eclipse toolbar. Select an open Eclipse project (i.e. click on the top-level project icon) and click on the crap4j icon. Crap4j will automatically identify and run all the JUnit tests in the project, record the coverage information, and calculate the cyclomatic complexity for each method. After it’s done (it may take a while if you have a lot of tests to run), it will display the results in a new window. The results page show high-level information and has links to more detailed pages (e.g. all methods sorted by complexity, coverage, CRAP, or CRAP load).

Remember that this is a prototype implementation intended primarily for our research. As such, we've only done limited testing using Eclipse 3.2.

Summary

Crap4j is the prototype for a planned open-source implementation of the CRAP metric.

Crap4j is currently implemented as an Eclipse plug-in downloadable from:

http://www.junitfactory.com/crap4j/update/.

We plan to evolve and refine both the CRAP metric and Crap4j based on user feedback and experience using an open-source model both for the formula and the various implementations. We are getting things started with an implementation for Java/JUnit (done as an Eclipse plug-in), but it would be great if people are willing to port it to other languages, IDEs, and testing frameworks.


david david

Posts: 16
Nickname: david21001
Registered: Mar, 2007

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-in Posted: Oct 3, 2007 9:19 AM
Reply to this message Reply
OUTSTANDING! and funny -- I wonder what the current trend is, i.e. if folks were uninhibited enough to post results, it would be interesting to see what the overall average is...

Maybe you should put a crap meter on your blog, a bar graph, and break it out several ways...total lines of code, number of programmers working on the baseline, etc

Let's see where the the software starts to break down.

Kind regards,

P.S. It was a funny read too.

Tim Mooney

Posts: 1
Nickname: leftmoon
Registered: Oct, 2007

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-in Posted: Oct 3, 2007 9:35 AM
Reply to this message Reply
Pretty good stuff. I ran it against of a couple of projects and it gave me about what I'd expect from them, so that's great. I'll probably disable it though for wasting toolbar space when it should be under the project menu or avaiable via a right-click. Nothing against the tool or the funny approach.

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-i Posted: Oct 3, 2007 10:14 AM
Reply to this message Reply
> Pretty good stuff. I ran it against of a couple of
> projects and it gave me about what I'd expect from them,
> so that's great. I'll probably disable it though for
> wasting toolbar space when it should be under the project
> menu or avaiable via a right-click. Nothing against the
> tool or the funny approach.

What? You are not going to keep the toilet paper icon in your tool-bar? Think how much fun you'll have explaining it to people :-).

Seriously, we'll take the suggestion to make crap4j available as a project menu or right-click item.

Thanks a bunch for trying it!

Alberto

Vincent O'Sullivan

Posts: 724
Nickname: vincent
Registered: Nov, 2002

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-in Posted: Oct 3, 2007 12:06 PM
Reply to this message Reply
Followed the instructions (several times)... ...got the following (each time):

Network connection problems encountered during search.
Unable to access "http://www.junitfactory.com/crap4j/update/".
Error parsing site stream. [Premature end of file.]
Premature end of file.
Error parsing site stream. [Premature end of file.]
Premature end of file.

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-i Posted: Oct 3, 2007 12:43 PM
Reply to this message Reply
Hi Vincent,

I just tried a fresh install (from a guest wireless network) and it worked fine.

I know other people have downloaded and used the plug-in in the last few hours without problems (and our server has been up continuously) so you must be experiencing some kind of problems with your proxy/firewall. Eclipse's update manager does not support proxy server authentication, so one solution is for us to provide a "zip" version of the project to enable a local install - we'll get on it as soon as we can.

Sorry you had a crappy experience...

Alberto

Kerim Mansour

Posts: 4
Nickname: todi
Registered: Oct, 2006

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-i Posted: Oct 4, 2007 3:05 AM
Reply to this message Reply
I just tried it.
The positive:
easy to use, good overall information (you know right away where the problems in the project are or if it is ok (overall)

The negative:
I have several projects for clients where the tests are in seperated project spaces. So it would be fine if selecting a (source-) project would also allow for selecting the corresponding (test-)project provided there are no tests found in the first one.

Vincent O'Sullivan

Posts: 724
Nickname: vincent
Registered: Nov, 2002

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-i Posted: Oct 4, 2007 7:29 AM
Reply to this message Reply
> ...you must be experiencing some
> kind of problems with your proxy/firewall. Eclipse's
> update manager does not support proxy server
> authentication...

That is probably the problem. I've only tried it from work, so far. I'll try it at home some time soon.

V.

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-i Posted: Oct 4, 2007 9:44 AM
Reply to this message Reply
Hi Kerim,

Thank you for trying it and your request for supporting multiple projects has been duly noted.

We had some requests and suggestions for leveraging existing code coverage tools and frameworks (e.g. Cobertura and Maven) so we can collect whatever coverage you get through your regular build and test cycles.

Alberto

Venkat Reddy Chintalapudi

Posts: 1
Nickname: cvenkat
Registered: Jan, 2003

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-i Posted: Oct 4, 2007 10:18 AM
Reply to this message Reply
This is interesting & we must wait for some time to see the same in the full action.

Your formula combines the Cyclomatic Complexity along with the code coverage.

Some metrics already exists around CC & It looks like an extension to the same.

Check out the following links
1. http://www.sei.cmu.edu/str/descriptions/cyclomatic_body.html
2. http://venkatreddyc.wordpress.com/2007/07/19/use-cyclomatic-complexity-to-determine-the-risk-and-test-scenarios/

Capturing these metrics at an application level might be worth exploring
1. http://venkatreddyc.wordpress.com/2007/10/04/non-linear-paths-from-application-code/

Ivan Lazarte

Posts: 91
Nickname: ilazarte
Registered: Mar, 2006

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-in Posted: Oct 4, 2007 11:25 AM
Reply to this message Reply
lol

Alberto Savoia

Posts: 95
Nickname: agitator
Registered: Aug, 2004

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-i Posted: Oct 4, 2007 11:59 AM
Reply to this message Reply
Hi Venkat,

> This is interesting & we must wait for some time to see
> the same in the full action.

Indeed, but you can start experimenting with it now. All you need is some Java code and Eclipse.

> Your formula combines the Cyclomatic Complexity along with
> the code coverage.
>
> Some metrics already exists around CC & It looks like an
> extension to the same.
>...

Absolutely. We are not re-inventing the wheel. As I mentioned in previous posts, we are building upon existing metrics and research.

Alberto

Carfield Yim

Posts: 32
Nickname: carfield
Registered: Sep, 2002

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-in Posted: Oct 4, 2007 1:40 PM
Reply to this message Reply
May I post bug report here? I get following exception when crap4j execute my test : java.lang.NoClassDefFoundError
at hk.com.carfield.core.CacheManager.<init>(CacheManager.java:35)
at hk.com.carfield.core.Context.<init>(Context.java:26)
at hk.com.carfield.core.Context.getTestContext(Context.java:56)
at test.unit.core.CacheManagerTest.testGetContent(CacheManagerTest.java:16)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja va:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3Test Reference.java:130)
at com.agitar.eclipse.junit.runner.AgitarJUnitTestReference.run(AgitarJUnitTestRef erence.java:40)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at com.agitar.eclipse.junit.runner.batch.LiteJUnitRunnerEx.executeAndNotifyTests(L iteJUnitRunnerEx.java:25)
at com.agitar.literunner.LiteJUnitRunner.b(LiteJUnitRunner.java:123)
at com.agitar.literunner.LiteJUnitRunner.main(LiteJUnitRunner.java:58)

However, my classpath should setup correctly at eclipse.... how can I get more information to fix this?

Robert Evans

Posts: 11
Nickname: bobevans
Registered: Jun, 2003

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-in Posted: Oct 4, 2007 2:35 PM
Reply to this message Reply
Hi Carfield,

Do the tests run with the JUnit test runner in Eclipse?
If so, then it sounds like you have found a bug in crap4j.

We basically tell the test runner to do it's thing for the project, so it should use all the same run config as a normal test run.

Thanks,
Bob Evans
Agitar Labs

Vincent O'Sullivan

Posts: 724
Nickname: vincent
Registered: Nov, 2002

Re: The Code C.R.A.P. Metric Hits the Fan - Introducing the crap4j Plug-i Posted: Oct 5, 2007 3:05 AM
Reply to this message Reply
Downloads fine at home.

It produces some curious results though. On some occasions it even seems to imply my code could be improved...

Flat View: This topic has 38 replies on 3 pages [ 1  2  3 | » ]
Topic: The Code C.R.A.P. Metric Hits the Fan -   Introducing the crap4j Plug-in Previous Topic   Next Topic Topic: ScalaTest 0.9.1 Released


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us