Why are blind equipment tests bad?

[aliEnRIK]

Mr Modesty:
It doesn't matter if it's conducted properly.

So youd happily use medicines that havnt been properly tested???

Further to that, even if you would others wouldnt (Including me)

And if it wasnt properly conducted and it was anything hifi related then it wouldnt mean anything either

 

[Anonymous]

Lets not get over complicated, the same panel of judges who review hi-fi normally just need to be used again but in a blind test situation, then these reviewed products would be more accurate than what we get at the moment when they are viewed according to price or by make, the results would certainly be very interesting

 

[idc]

garyjos:....... the results would certainly be very interesting

True, but I don't think that the results would be any more than interesting. People buy hifi with their eyes open. There have been many posts where people have listed different hifi, none of which have been heard, but there appears to be one clear favourite and the poster is just looking to have their choice validated. There have also been open admissions that kit has been bought because of its looks.

A blind test that I think would be interesting would be a pair of weird looking designer speakers against a more normal stylish design. I would like to know how much looks affect perceived sound.

 

[Anonymous]

Mind you, I'd concede blind tests of cars or motorbikes might not be a good idea.

 

[Anonymous]

>So youd happily use medicines that havnt been properly tested???

No, but I would use hifi gear that hadn't been reviewed.

 

[aliEnRIK]

Mr Modesty:

>So youd happily use medicines that havnt been properly tested???

No, but I would use hifi gear that hadn't been reviewed.

Your aware that blind tests have indicated that ALL amps over 200 quid sound the same?

 

[chebby]

aliEnRIK:Your aware that blind tests have indicated that ALL amps over 200 quid sound the same?

Please can you provide references or links to those tests?

I have access to the local University library and I am even prepared to order and buy the publication the test is reported in if necessary.

People are quoting tests they have heard of. I would like to see them.

Thanks.

 

[Anonymous]

aliEnRIK:

Your aware that blind tests have indicated that ALL amps over 200 quid sound the same?

There has never been a test of any sort that has covered all amplifiers over £200.

 

[Anonymous]

The amplifier blind test that indicated they all sounded the same. Richard Clark amplifier challenge? $10,000 prize

Due to the restrictions on the comparison, particularly in my opinion the ones in bold I am not suprised no one won.

Cost to take the test is $200.00. $500.00 for people representing companies. Payable in advance

All amps must be brand name, standard production, linear voltage amplifiers. The amplifiers in the test must be operated within their linear power capacity. Power capacity is defined as clipping or 2% THD 20Hz to 10kHz, whichever is less.

The levels of both left and right channels will be adjusted to match to within .05 dB. Polarity of connections must be maintained so that the signal is not inverted. Left and Right cannot be reversed. Neither amplifier can exhibit excessive noise. Channel separation of the amps must be at least 30 dB from 20Hz to 20kHz.

All signal processing circuitry (e.g. bass boost, filters) must be turned off, and if the amplifier still exhibits nonlinear frequency response, an equalizer will be set by Richard Clark and inserted inline with one of the amps so that they both exhibit identical frequency response. The listener can choose which amplifier gets the equalizer. The EQ is most likely to be used when comparing a tube amplifier to a solid state amplifier

If our speakers are not acceptable, the listener can provide any commercially available speaker system as long as it uses dynamic drivers. The actual measured impedance cannot exceed the rated load impedance of the amplifiers tested.

No test signals can be used - only commercially available music.

A test session will consist of 12 A/B sequences. Passing the test will require a positive identification of each amp for all 12 sequences. The listener must pass two complete sessions of 12 comparisons. Passing the test means 24 correct responses.*

One of the reasons I believe blind testing and reporting the results will never be popular with magazines is advertising revenue. I would not advertise in a magazine if they reviewed my expensive product and did not compare it favourably. A reviewer maybe impartial in what they do say, but this does not neccessarily cover what they do not say. If they do not have anything good to say about an advertisers product, better to say nothing at all. If a reviewer consistantly lost a magazine advertising contracts, they would quickly be out of a job. They would also be out of a job if they could not tell the difference between products, as they would have little to write. The blind tests would also have to be repeated many times to remove blind luck, and confirm the differences heard are real. So the job of hifi reviewer gets to be alot more stressfull.

 

[the record spot]

Tarquinh:
Oh dear, I thought this had gone away

No-one gets upset about blind wine tasting. There's room for both blind and sighted reviews, surely.

Big thumbs up here Tarquinh, big thumbs up here...the whole argument's like two bald men fighting over a comb if you ask me.

 

[the record spot]

Andrew Everard:
shado:...and I am sure you are proud of your japanese connections with relation to Hifi.

Bit confused again - what 'japanese connections with relation to Hifi'? Though admittedly I do have Japanese relatives...

No more trips to hi fi shows in the far east for you then Mr. E!

And you can forget flying All Nippon Airways when you're at it!

 

[chebby]

I would still like the references for (or links to) the independent blind test(s) that proved all hifi amplifiers over £200 sound the same.

It has been asserted so many times on this forum but I have never seen anyone back it up with a reference to a full report in a publication (or provide a link).

I have to assume such a test is hearsay or a hifi 'urban myth' otherwise.

 

[Andrew Everard]

the record spot:No more trips to hi fi shows in the far east for you then Mr. E!

And you can forget flying All Nippon Airways when you're at it!

I'm sure there's a point here somewhere, but I'm afraid it eludes me...

 

[Clare Newsome]

knightout:


One of the reasons I believe blind testing and reporting the results will never be popular with magazines is advertising revenue. I would not advertise in a magazine if they reviewed my expensive product and did not compare it favourably. A reviewer maybe impartial in what they do say, but this does not neccessarily cover what they do not say. If they do not have anything good to say about an advertisers product, better to say nothing at all. If a reviewer consistantly lost a magazine advertising contracts, they would quickly be out of a job.




Interesting speculation, but certainly not the case here on What Hi-fi? Sound and Vision.

Just this year, for instance, two major advertisers have pulled campaigns from the magazine (and indeed from other titles across the group) because they haven't liked our reviews of their products. It happens sometimes, but certainly doesn't lead to those reviewers being out of a job - we pride ourselves on our editorial independence.

 

[the record spot]

Andrew Everard:
the record spot:No more trips to hi fi shows in the far east for you then Mr. E!

And you can forget flying All Nippon Airways when you're at it!

I'm sure there's a point here somewhere, but I'm afraid it eludes me...

Actually, it was a joke - I think the other poster who implied you and the Japanese hi-fi industry are in each other's back pockets made the point that eluded most of us!

Anyways, stick to British hi-fi that's built in garden sheds, then all you'll have to put up with are people saying the mag doesn't support the wider industry...

 

[Andrew Everard]

Oh, I see... Not.

Mind you, may be better to avoid JAL rather than ANA at the moment. JAL sucks...

 

[Andrew Everard]

knightout:One of the reasons I believe blind testing and reporting the results will never be popular with magazines is advertising revenue. I would not advertise in a magazine if they reviewed my expensive product and did not compare it favourably. A reviewer maybe impartial in what they do say, but this does not neccessarily cover what they do not say. If they do not have anything good to say about an advertisers product, better to say nothing at all. If a reviewer consistantly lost a magazine advertising contracts, they would quickly be out of a job. They would also be out of a job if they could not tell the difference between products, as they would have little to write. The blind tests would also have to be repeated many times to remove blind luck, and confirm the differences heard are real. So the job of hifi reviewer gets to be alot more stressfull.

So now the problem is what isn't written in the magazine? Please be assured that everything not written is not written to the same levels of propriety as that which is published. Or something.

And as for the link between reviews, reviewers and advertising, two points:

One, there isn't any such link, at least on WHFSV.

Two, all our reviewers are in-house staff, not freelance, one reason for which being that it's much harder to see what infleunces and pressures reviewers are subjected to when they are off-site and not directly controlled.

On a more general level, what a strange, twisted view of the world you have...

 

[Anonymous]

"Just this year, for instance, two major advertisers have pulled campaigns from the magazine (and indeed from other titles across the group) because they haven't liked our reviews of their products"

Glad to hear you have some integrity. Be carefull biting the hand that feeds you.

Care to name the advertisers and give ballpark figures as to the amount of revenue lost. With some magazines it strikes me as a bidding war for good reviews / favourable comparisions and of course awards and in any war there can be unhappy losers. After all in theory it all translates into extra sales.

Another major problem with blind testing I can see is reviewers heads exploding due to confusion over what they can hear. Afterall they can hear things most people can not. Like the difference between optical cables in this months what hifi.

 

[Andrew Everard]

knightout:Glad to hear you have some integrity. Be carefull biting the hand that feeds you.

What a stupid remark. Advertising considerations, as we have now said many times, have no influence on reviews.

knightout:Care to name the advertisers and give ballpark figures as to the
amount of revenue lost.

No. You wouldn't expect any company to reveal such sensitive information on a public forum.

knightout:With some magazines it strikes me as a
bidding war for good reviews / favourable comparisions and of course
awards and in any war there can be unhappy losers. After all in theory
it all translates into extra sales.

And again your shoddy insinuations...

knightout:I can see is reviewers
heads exploding due to confusion over what they can hear. Afterall they
can hear things most people can not. Like the difference between
optical cables in this months what hifi.

If our reviewers weren't confident in their assessment of the products, the results wouldn't be published in the magazine.

You can keep making these ridiculous, unfounded and - to date - unsupported accusations as long as you like. But it is getting a bit dull, isn't it?

 

[Clare Newsome]

'The hand that feeds us' is our readership - we're here to serve your interests first and foremost!

I know it's a significant sum lost, but to specify numbers or names would be inappropriate. Talking of which, less of the sniping agenda, please. Smacks of

 

[John Duncan]

(sigh)

Does anybody actually know how advertising sales works?!?!?!?

 

[professorhat]

It's not based on magazine circulation is it?

No, sorry, ridiculous idea.ÿ

ÿ

 

[Andrew Everard]

professorhat:
It's not based on magazine circulation is it?

No, sorry, ridiculous idea.

Give that man a

 

[the record spot]

Andrew Everard:
Oh, I see... Not.

Mind you, may be better to avoid JAL rather than ANA at the moment. JAL sucks...

Yoiks! I've heard of a bird strike, but "big metal container strike"...???

 

[Anonymous]

Dear all,

This is a fascinating thread, and as someone with plenty to be ignorant about, I only add to it with some trepidation. Of course, I could be wrong about this but for many the sound of the product is not the only consideration. A more expensive product may be indistiguishable in a double blind A/B/A test from a cheaper one, but it may be better made. If a customer plans to use the equipment for a long time, or particularly intensively, then the higher price may be justifiable. Similarly, if a company has a better reputation for customer support, or providing spares for many years after production, then a higher price may be worth paying.

In addition, the look of a product may be important to some. If your speakers sit in a rather lovely room with expensive decor, perhaps the more expensive pair with better veneer are a wise investment, even if they sound exactly the same? Maybe you have another item of equipment from the same manufacturer and you value the convenience of a single remote? I know for the pure audiophile such things are unimportant, but in the real world they matter. My wife, for example, has considerable say in speaker choice for aesthetic reasons, and so she should, she has to look at them too!