The Artima Developer Community
Sponsored Link

Weblogs Forum
Comparative Studies – the rods we hit each other with?

0 replies on 1 page.

Welcome Guest
  Sign In

Go back to the topic listing  Back to Topic List Click to reply to this topic  Reply to this Topic Click to search messages in this forum  Search Forum Click for a threaded view of the topic  Threaded View   
Previous Topic   Next Topic
Flat View: This topic has 0 replies on 1 page
Calum Shaw-Mackay

Posts: 58
Nickname: calum
Registered: Mar, 2004

Comparative Studies – the rods we hit each other with? (View in Weblogs)
Posted: Oct 28, 2004 5:51 PM
Reply to this message Reply
Summary
Sounds a bit extreme? Some companies seem to live and breathe benchmarks, responses to studies, and responses to responses. Isn’t it time we realised what benchmarks are really about and stopped it all getting out of hand.
Advertisement
Firstly, I’ll point out here and now that this blog is based on this thread on TSS about Websphere Vs .Net.

This is the first thing that always gets on my nerves - the term ‘Vs.’ when in the context of benchmarking. Why is it there? It’s not some fight at Caesar’s Palace

Anyway, in all of the opinions and flame bait, there was one golden, gleaming treasure of a comment - ‘Every good programmer knows that the only benchmark that holds water is his own.’. We all have to have our own benchmarks, the one that says, “Yeah I’m happy with choosing X over Y”, or whatever, but it wasn’t always like this; we didn’t have to be always self-reliant when comparing products, did we?

When I first started out on this journey into programming, years ago, benchmarks actually meant something; they may have been small in scope but there were plenty of parties involved and they were fair enough comparisons. But nowadays, benchmarks are considered in these huge ‘studies’ that compare only two or three products. And the scope of these studies is often vast, concerning products with huge number of features and configuration options.And most of the objectivity and rational critique gets lost in the sheer size of the thing.

Now, every other week we have someone publish a benchmark about something, and it all turns into a war, people quoting benchmarks like they are sticks to hit each other with. You can always tell when an analyst company does it’s job, and it’s benchmarks are good – people accept the findings, and get on with making their products better – that is most people, bar the vocal zealot few.

One thing I realised is that everyone selectively uses benchmarks to prove their own point of view. Benchmarks are after all statistics, and, you know, you can prove anything with statistics. Oh .Net is better than J2EE and it’s more productive, so why are so many Java community tools being ported to .Net? You see what I did there – it’s called the apples-to-oranges effect. Produce a claim and defend your position with a counter-claim that has nothing to do with the claim you’re disputing. Now if you ask me do I know Java is better than .Net? Overall, I don’t know. There are areas that are better for Java as I’m sure there are for .Net. But this comes back to a scoping issue; trying to say is J2EE better than .Net is such a big question, it’s like saying is day better than night?

I stopped even contemplating Java and .Net benchmarks and studies, the day I, by chance, picked up a copy of MicroMart and I actually got an unbiased, critique of a product; the same morning I had been reading the first TMC comparison for J2EE and .NET. The product was only a PC, and did I have to pay for the study, yes it was about £1.80.

If you’re going to provide studies and benchmarks, make them much more focused and relevant, and broaden the participation. So instead of looking at J2EE Vs. .Net, lets look at Java RMI/Jini ERI as opposed .Net Remoting; JavaMail Vs. Exchange; JMS Vs. MSMQ, etc, etc.. Make them relevant, not this vendor paid-for marketing gimmick. Benchmarks are for techies, marketing is for Business Decision Makers - let’s not try to turn one into the other

Topic: Primitive Types ... In a Library?! Previous Topic   Next Topic Topic: On Promoting and Designing a Programming Language


Sponsored Links



Google
  Web Artima.com   

Copyright © 1996-2014 Artima, Inc. All Rights Reserved. - Privacy Policy - Terms of Use - Advertise with Us