Time to review the rating System?

[admin_exported]

A recent Eurostar trip to the UK gave me time to peruse the latest What Hi-Fi, and, once I'd finished it (Lille, I'm a fast reader), think about the star rating system.

Having noted that the one star award is redundant, since only two products merit this lowest of the low of which one such rating was clearly an error, I was struck by the overwhelming number of four and five star reviews in each price band. In this lies a problem, since it doesn't allow for products whose performance transcends their price band. For example, the CA Dacmagic and the Musical Fidelity V-DAC are products which rate extremely highly in German Magazines where they are considered amongst the best available, yet What Hi-Fi bands them in budget products. This means if you're looking for the best DAC and have 500 pounds to spare, you'll miss out on two of the best available.

Also, in a category not price-banded, where there are few products such as turntables, the casual reader could be fooled into thinking that a budget model is just as good as one ten times the price. Sure, the one sentence summary does indicate level, but there's no indication why an LP12 is so much better than a Pro-ject Debut, assuming it is.

What I'm suggesting is that products are rated against each other irrespective of price band, and then given a rating according to their class. This does mean, of course, that there is a clearly-defined test rating system overall, which in turn means that 5 stars wouldn't be adequate. Clearly it could be misleading to give a middle range product in a budget category two stars whereas, in fact, it is one of the best at its price level. Points out of 50 or a hundred would fill the bill. These points in turn would be awarded to a particular test within you suit of tests. I'm not sure what your pre-defined areas are, but these might be:

Dynamics 10

Design 10

Stereo imagery 10

etc etc.

An example of a rating summary using this system would be:

Pro-ject Debut 3 25 pts Price band: *****

Rega P3 30 pts Price band ****

Anyway, it's just a thought, and it passed the time quite pleasantly!

 

[Anonymous]

I agree somewhat.

For example, is a 5 star product at a certain price point only half as good as a 5 star product at twice the price point?

 

[plastic penguin]

I see your point - don't get too hung up on the stars. If you take them too literally you'll end up with more queations than answers. Sure, use them as a guide, but an all 5 star doesn't always equal a great sound.

The best guide is your ears. . . .

 

[Anonymous]

Exactly, and a little more. They don't mean anything any more, so either drop them entirely or change the system so they do have meaning.

 

[Clare Newsome]

Not going to rush to change a globally recognised ratings system that's been going for 33 years just yet, i'm afraid (and I could find plenty of people to whom the ratings system means a hell of a lot!)

However, all feedback taken on board - we know it's not a perfect system. However, the basic five-star system is taken further with Best Buys and Awards, where we aim to make it clear where we feel the natural progressions are, in terms of the best-value items at each price point/of each specific product type.

 

[plastic penguin]

Exactly, Clare. The star system is very good, your mag always emphasises the need to listen before buying. But I don't believe there is a perfect guide. Sadly you can't satisfy everyone.

 

[Clare Newsome]

I'd also add that we're readily available here online to answer any questions people have about any of our star ratings, verdicts, Awards, Best Buys or just about anything else!

 

[Anonymous]

All reviews should be taken in context of their price, I think.

But if it were exceptional a note can be made within remarks, even beats twice the price'

I think the easiest improvement would be to double the resolution by having half a star .

Is there already 5 red and 5 gold stars given?, sorry can remember?

 

[Clare Newsome]

We canned the gold stars in the Buyer's Guide as they just confused people.

Instead we flag up all Award winners in the text, plus have introduced longer, Best Buy panels in the Buyer's Guide to reflect the Best Buys listing available here online.

That's in addition, of course, to the Best Buys spread included in the front half of every issue of WHF, which gives an at-a-glance look at what we feel are the class leaders in a wide range of categories.

 

[Anonymous]

Clare Newsome:
We canned the gold stars in the Buyer's Guide as they just confused people.

Instead we flag up all Award winners in the text, plus have introduced longer, Best Buy panels in the Buyer's Guide to reflect the Best Buys listing available here online.

That's in addition, of course, to the Best Buys spread included in the front half of every issue of WHF, which gives an at-a-glance look at what we feel are the class leaders in a wide range of categories.

Yes, but that doesn't help: you still have no idea how globally good or indifferent a product is when compared to others. Since your Best Buys are also price-banded, that really doesn't help, either.

 

[Clare Newsome]

I beg to differ - if it doesn't deserve a best buy mention or Award, it doesn't get one, regardless of whether that leaves us a hole in that price/just one winner in a category.

Hence - just to give one example - the DACmagic being the only DAC to scoop an Award last year, as it represents such exceptional value.

 

[Anonymous]

We're never going to agree! Yes, but is the Dacmagic better than the Chord DACs, for example? Is the best in category turntable better than the one two categories above? Could be, price may not be the sole determinant of quality.

Actually, I agree with your points about the forum and the website, which I use when I'm researching stuff for an upgrade. But I do first check Audio and Stereoplay first just to see if there are affordable products up with the best of them, and it would be nice to be able to do that in What Hi-Fi.

 

[Anonymous]

This all assumes too that price is an indicator of performance, whereas the reality is that it is not.

A global rating system where the absolote best gains 100% would allow relative comparisons, irrespective of cost.

For example, the best CD player ever heard might score 95% or above. The entry level CD player in their range may score 30% but it wouldn't look like a low score in context because other CDP in its price bracket will score approximately similar.

 

[Andrew Everard]

Yes, but how would you feel if you had only £200 to spend on a component and the best you could get was something rated at 21%. And how often would one have to move the goalposts because something even better and more exotic had been tested?

So the 21% might become a 15%...

Much better to show people what's the best they can get for their money, rather than depress them with how little their chosen product scores. If that means we're anti-elitist, and far from purist, a) so be it and b) excellent.

 

[Anonymous]

Andrew Everard:
Yes, but how would you feel if you had only £200 to spend on a component and the best you could get was something rated at 21%. And how often would one have to move the goalposts because something even better and more exotic had been tested?

So the 21% might become a 15%...

Much better to show people what's the best they can get for their money, rather than depress them with how little their chosen product scores. If that means we're anti-elitist, and far from purist, a) so be it and b) excellent.

Agreed Mr.E,very well said.

 

[Clare Newsome]

Indeed, heaven forfend we ever get to that 'Top Gear' mentality of 'if you can'tÿafford a sports/classic/monstrously huge/vintage car, you're some sort of subhuman to be mocked

 

[professorhat]

Tarquinh:but is the Dacmagic better than the Chord DACs, for example?

Well, surely if the Chord Chordette Gem (for example) receives a 5 star rating at £400, it must be better than the DacMagic since the DacMagic only costs £200. If the DacMagic was better at half the price, then how could the Chordette Gem possibly get a 5 star rating?

Seems fairly simple to me but maybe I'm missing something...

 

[PJPro]

I feel that five star ratings are too common place. It's like O Levels.....everyone's getting A*. I feel that the full range of scores seem to be generally under utilised.

I would support a move to make it a lot harder for a piece of kit to get five stars. For me, this would serve to differentiate betwen the very, very good and the truly excellent.

 

[Alec]

Andrew Everard:

Yes, but how would you feel if you had only £200 to spend on a component and the best you could get was something rated at 21%. And how often would one have to move the goalposts because something even better and more exotic had been tested?

So the 21% might become a 15%...

Much better to show people what's the best they can get for their money, rather than depress them with how little their chosen product scores. If that means we're anti-elitist, and far from purist, a) so be it and b) excellent.

Bravo!

Ditto Clare's Top Gear point. but then im very poor. and have been called subhuman, but thats another matter.

 

[Anonymous]

PJPro:I feel that five star ratings are too common place. It's like O Levels.....everyone's getting A*. I feel that the full range of scores seem to be generally under utilised.

I would support a move to make it a lot harder for a piece of kit to get five stars. For me, this would serve to differentiate betwen the very, very good and the truly excellent.

I totally agree here, spot on...

 

[Anonymous]

"Indeed, heaven forfend we ever get to that 'Top Gear' mentality of
'if you can't afford a sports/classic/monstrously huge/vintage car,
you're some sort of subhuman to be mocked

Clare Newsome"

Far better to say:

Indeed, heaven forfend we ever get to the elitest mentality of "if you can't afford an expensive turntable/amplifier/loudspeaker, you're some sort of subhuman who shouldn't be allowed to know about them because they'll only make you unhappy," which is what some of your staff members seem to be saying.

Lord Andrew, thank you for putting me in my place. I promise now to stick to my budget price band and never to aspire to anything better for the rest of my life. I know my place, a humble peasant. You're right, ignorance is bliss. Long live the feudal system!

 

[margetti]

In general, I find that with the more in-depth reviews (super tests et al), the write-up does a very good job of clarifying/explaining the star rating given, often with direct comparisons between products in a supertest saying why Product A got 5 stars and Product B got 4.

It is in the shorter reviews that I can relate the the point being made by the OP. Take for example the anaglogue interconnect test in the current issue. The Atlas Equator gets 5 stars, and the one (I forget which brand it is) at £140 gets 4 stars. I understand perfectly the 'sound for pound' thing, but what this review didn't tell me was that if I did have £140 to spend an an anologue interconnect, which was the better of the two in terms of audio performance, and by how much.

Yes, if I was in that position I could, and would, follow it through with a post here, and I guess that answers the point: the star system gives you a starting point from which to research further, and ideally whittle down your choices to 2 or 3 and then listen for yourself.

 

[Clare Newsome]

margetti:
the star system gives you a starting point from which to research further, and ideally whittle down your choices to 2 or 3 and then listen for yourself.

Bingo! And as said, we're here online to help further - whatever your intended budget or aspirations....

 

[Andrew Everard]

Tarquinh:Lord Andrew, thank you for putting me in my place. I promise now to stick to my budget price band and never to aspire to anything better for the rest of my life. I know my place, a humble peasant. You're right, ignorance is bliss. Long live the feudal system!

Which, of course, is exactly the opposite of what I was saying...

The clue was sort of in "If that means we're anti-elitist, and far from purist, a) so be it and b) excellent."

 

[Anonymous]

My point was that you were inadvertently being elitist by effectively assuming you know what's best for the likes of us, but, hey, it really isn't important.

Hadn't thought about the demotion part of your current rating system, by the way. My initial feeling is that, in such a narrow band, it can mislead, because after all time hasn't remove the qualities you found in the product to begin with, its just that new features or cheaper products have come along. If the global points scale went to fifty, removal of a star wouldn't seem as catastrophic.

Oh, and please note that moving to global comparison doesn't remove the star system, as I've illustrated. Also, many manufacturers and users might be chuffed to see their budget product being ranked up there with the best.