Blind Tests

[admin_exported]

It is laudable that WHF is one of the few, HiFi mags that perform blind tests on audio kit. I do have some questions about the protocols. It seems to me that there are 5 big problems with the way that blind test are performed by mags.

1. They are most often single blind, this allows for the Clever Hans effect where cues are inadvertently passed on to the listeners.

2. They are seldom properly randomised, listeners generally know that there is a change between one condition and the next, a random set of "no change" conditions would add power.

3. The number of iterations is low, most often one setup is tested then some component is swapped and it is tested again, then that is it, there is no going back and repeating the tests in a different order.

4. The number of partcipants is low, often as few as 3 or 4, bigger numbers = more power

5. Related to 2 , listeners are not required to prove that thay can really distinguish between two items by identifying whether X is A or B. The ABX box allows that direct comparison.

Just my few reflections...

 

[Anonymous]

I thought WHF didn't do blind testing????

If so, then they do not fall foul of any of your 5 big problems!

 

[idc]

They do with many of The Big Question features where forum members go to the listening rooms and are subjected to blind tests on matters from cables to bit rates. The last issue had a comparison of downloaded high quality files vs CDs.

Welcome to the forum Nick_Charles.

 

[drummerman]

Who cares anyhow, it's entertaining reading, whichever publication and if I want to buy, the only opinion/ears that matter are mine.

 

[carter]

idc:

They do with many of The Big Question features where forum members go to the listening rooms and are subjected to blind tests on matters from cables to bit rates. The last issue had a comparison of downloaded high quality files vs CDs.

Welcome to the forum Nick_Charles.
i could sign your copy if you like.no charge

 

[Anonymous]

you certainly have a way with words chebby

have you ever considered a career in a.v. journalism

 

[the record spot]

I think Miller Research undertakes some testing for Hi Fi news (at least that's a link to the site from the HFN page)m while Hi Fi World has it's own test facilities and prints measurements for each test it undertakes. Noel Keywood on that title used to design a load of World Audio kit, so knows his stuff from the sharp end.

 

[Andrew Everard]

the record spot:I think Miller Research undertakes some
testing for Hi Fi news (at least that's a link to the site from the HFN
page)

Given that Miller Audio Research was set up by the editor and supplies QC equipment to a number of manufacturers, I think you're on fairly safe ground with that assumption, trs.

 

[crusaderlord]

drummerman:Who cares anyhow, it's entertaining reading, whichever publication and if I want to buy, the only opinion/ears that matter are mine.

definately agree - i enjoy reading them - it gives me a steer thats all, the rest is down to me to listen for myself and decide

anyway despite all these great listening tests etc a good deal of hi-fi decision making still has an element of what it looks like in mind - maybe not a perfect decision but if your system is on show then many of us will pick certain brands to try because we like the design

 

[the record spot]

Andrew Everard:
the record spot:I think Miller Research undertakes some
testing for Hi Fi news (at least that's a link to the site from the HFN
page)

Given that Miller Audio Research was set up by the editor and supplies QC equipment to a number of manufacturers, I think you're on fairly safe ground with that assumption, trs.

Ah, right, that explains that one off then! In terms of the facilities, do you know are they shared by any other UK titles Andrew (I'm assuming not WHFS&V who have their own)?

 

[idc]

I think it is great WHF are doing the Big Question and are doing their bit to tackle criticism of us audiophiles believing too many alleged myths.

Is there someone who runs the Big Question available to answer the OPs original question?

Could those who have been there give us more details on what happened and the integrity of the testing?

Is there room for improvement in the way the Big Question is run?

 

[the record spot]

Well, the BQ relies on a subjective response and although based on fairly random sample, that element can't be ignored. Equally, it's an interesting read!

 

[Andrew Everard]

the record spot:Ah, right, that explains that one off then! In terms of the facilities, do you know are they shared by any other UK titles Andrew (I'm assuming not WHFS&V who have their own)?

As far as I know only HFN uses HFN's facilities.

 

[idc]

But what about WHFs facilities and how the Big Question is run?

 

[Andrew Everard]

Clickety

 

[chebby]

..