The Blind (Mis) Leading the Blind

[aliEnRIK]

Having recently read an article pretty much confirming what ive always believed anyways, I thought id point everyone to this article on blind tests and the theory that they are NOT the 'be all and end all' of testing

HERE

I would like to point out some facts in brief ~

"under double-blind test conditions, 60 expert listeners over 554 trials
couldn’t hear any differences between CD, SACD, and 96/24"

"blind listening test conducted by Stereo Review that concluded that a
pair of Mark Levinson monoblocks, an output-transformerless tubed
amplifier, and a $220 Pioneer receiver were all sonically identical"

"The test involved 60 “expert” listeners spanning 20,000 evaluations over a period of two years."........."his decision was based on data gathered during the 20,000 “double-blind, triple-stimulus, hidden-reference” listening trials"........."Swedish Radio sent a tape of music processed by the selected codec to the late Bart Locanthi,"......"After Locanthi informed Swedish Radio of the artifact (an idle tone at
1.5kHz), listeners at Swedish Radio also instantly heard the distortion"........."How is it possible that a single listener, using non-blind
observational listening techniques, was able to discover—in less than
ten minutes—a distortion that escaped the scrutiny of 60 expert
listeners, 20,000 trials conducted over a two-year period, and
elaborate “double-blind, triple-stimulus, hidden-reference”
methodology, and sophisticated statistical analysis? "

Taken from the same source but NOT in that link ~

"The answer is that blind listening tests fundamentally distort the listening process and are worthless in determing the audiobility of a certain phenomenon"

Ive never believed in blind tests myself for a many number of reasons, but Robert has certainly put forward many points about just HOW worthless they really are

 

[RodhasGibson]

Yep

and as weve said before. Get yourselves a recommended shortlist, trust your own eyes and or ears, then decide its right for you. Then for gods sake get on and enjoy it.

 

[Anonymous]

Be careful with emotive articles such as this. Some questions have to be asked before jumping to any conclusions.

Is the writer unbiased in his views? Does he have vested interests one way or another? Is his evidence conclusive? How many examples does he cite to prove his case? What is his premise?

Now, I'm neutral either way, and I agree, it's you ears that count in the end. This article I found unconvincing, and far from giving a definitive answer if one exists.

 

[Anonymous]

aliEnRIK: How is it possible that a single listener, using non-blind observational listening techniques, was able to discover-in less than ten minutes-a distortion that escaped the scrutiny of 60 expert listeners, 20,000 trials conducted over a two-year period, and elaborate "double-blind, triple-stimulus, hidden-reference" methodology, and sophisticated statistical analysis? "

One person's "distortion" is another person's "idiosyncracy in the recording/music, and part of its charm."

Previous posters are spot on, trust your own ears.

 

[aliEnRIK]

Tarquinh:
Be careful with emotive articles such as this. Some questions have to be asked before jumping to any conclusions.

Is the writer unbiased in his views? Does he have vested interests one way or another? Is his evidence conclusive? How many examples does he cite to prove his case? What is his premise?

Now, I'm neutral either way, and I agree, it's you ears that count in the end. This article I found unconvincing, and far from giving a definitive answer if one exists.

What do you see as a 'worst case scenario'?

Rob writes for a magazine called "The Absolute Sound" and on this website ~

AVGUIDE

BLOG

Far as I can tell hes quite genuine and no real motive for lying. He simply loves his music (Like we all do) and wants to dispell some of the awful myths surrounding hifi (For example ~ all amps sound the same

)

In particular 'blind tests' As all the none believers stick RIGIDLY to blind test results even though by their very nature theyre not particularly fair

For example ~

Lets say 100 people test some interconnects. The test is simply designed to tell IF there is ANY audible difference when used in a true ABX Blind test.

Lets say the results were pretty much average. Some could some couldnt which doesnt prove a thing.

Lets take the top 5 people of THAT test (The ones that COULD seemingly tell a difference) and test them again

Id expect the test results to be WAY above average then as people are being used who CAN percept these differences 'on the fly'

This is just one of many examples of why ABX blind testing is flawed as this has (To my knowledge) never been done in ANY blind test

I myself have tested cables (Not blind, just switching) and I find it very difficult to tell if I like one interconnect over another straight away. It takes TIME to decide wether its too 'bright' sounding or lets more detail through or the bass is 'wooly' or whatever the case may be (ANOTHER major flaw in ABX testing)

 

[VoodooDoctor]

Whilst an interesting article, he says nothing that convinces me that blind tests are useless. Many of the subsequent comments pick valid holes in his statements. Without reading the actual papers that he is criticising to look for potential sources of error, it would be unwise to accept his sweeping conclusions.

 

[Anonymous]

I remain unconvinced , too many variables to come to your conclusions .

Double Blind testing provides accurate scientific data on the differences between components to peoples hearing.

Blind testing by its very nature is unbiased .

 

[aliEnRIK]

drumlins4ever:
I remain unconvinced , too many variables to come to your conclusions .

Double Blind testing provides accurate scientific data on the differences between components to peoples hearing.

Blind testing by its very nature is unbiased .

Then your clearly one of those people that never WILL be convinced otherwise

 

[Anonymous]

Erik,

I'm certainly not saying he is lying, but that he is biased. His intention is to disprove blind testing because he does not believe in it. Therefore, all his arguments are designed to prove blind testing is fundamentally flawed.

As he puts it "I contend that such tests [the study in question] are an indictment of blind listening tests in
general because of the patently absurd conclusions to which they lead." Further he says "Most such tests, including this new CD vs. high-res comparison, are
performed not by disinterested experimenters on a quest for the truth
but by partisan hacks on a mission to discredit audiophiles." My conclusion from these and the other things he says is that he is heavily biaised against blind testing, otherwise why use such emotive language? Note that he also gives no evidence to support either of his statements.

If you're still not convinced, then look at the title of the paper he gave to the Audio Engineering Society which was "The Role of Critical Listening in Evaluating Audio Equipment Quality."

Accepting his biais, and his use of emotive language, there's also the small matter of the number of examples he uses to back up his arguments. These are few.

Also, he cites as proof of the absurdity of blind testing a test that was done in 1987. His argument for this is that its absurd to think a cheap amplifier sounds as good as one costing many times more. But, if that's what people thought, then how is their conclusion invalid simply because it doesn't agree with his beliefs?

A better article would have spent less time on emotive language and simple gainsaying, and more on argument backed up by examples.

That's all I'm trying to say. As I said, I'm strictly neutral on this.

 

[Anonymous]

Erik , No i wont be convinced otherwise ,,,

But using yourÿreasoningÿi guess you must believe in Homeopathy, physic healingÿand ghosts tooÿ.

 

[aliEnRIK]

drumlins4ever:
Erik , No i wont be convinced otherwise ,,,

But using your reasoning i guess you must believe in Homeopathy, physic healing and ghosts too .

Can you please define for me what YOU believe MY reasoning is on THIS subject?

 

[Anonymous]

aliEnRIK:drumlins4ever:
Erik , No i wont be convinced otherwise ,,,

But using yourÿreasoningÿi guess you must believe in Homeopathy, physic healingÿand ghosts tooÿ.

ÿ

Can you please define for me what YOU believe MY reasoning is on THIS subject?

ÿ

simply put , That blind testing is worthless ÿ

 

[aliEnRIK]

Tarquinh:
Erik,

I'm certainly not saying he is lying, but that he is biased. His intention is to disprove blind testing because he does not believe in it. Therefore, all his arguments are designed to prove blind testing is fundamentally flawed.

As he puts it "I contend that such tests [the study in question] are an indictment of blind listening tests in
general because of the patently absurd conclusions to which they lead." Further he says "Most such tests, including this new CD vs. high-res comparison, are
performed not by disinterested experimenters on a quest for the truth
but by partisan hacks on a mission to discredit audiophiles." My conclusion from these and the other things he says is that he is heavily biaised against blind testing, otherwise why use such emotive language? Note that he also gives no evidence to support either of his statements.

If you're still not convinced, then look at the title of the paper he gave to the Audio Engineering Society which was "The Role of Critical Listening in Evaluating Audio Equipment Quality."

Accepting his biais, and his use of emotive language, there's also the small matter of the number of examples he uses to back up his arguments. These are few.

Also, he cites as proof of the absurdity of blind testing a test that was done in 1987. His argument for this is that its absurd to think a cheap amplifier sounds as good as one costing many times more. But, if that's what people thought, then how is their conclusion invalid simply because it doesn't agree with his beliefs?

A better article would have spent less time on emotive language and simple gainsaying, and more on argument backed up by examples.

That's all I'm trying to say. As I said, I'm strictly neutral on this.

Tar' ~ I cant disagree much of what you have said there. Certainly his small article could have been a lot better, but then there is a LOT more to his article (Unfortunately whilst it WAS free to see (Which I missed) you now need to pay)

"His argument for this is that its absurd to think a
cheap amplifier sounds as good as one costing many times more. But, if
that's what people thought, then how is their conclusion invalid simply
because it doesn't agree with his beliefs?"

Just out of sheer intruige, are you saying all amps DO sound the same? Because if they dont (And we all KNOW they dont), then that brings 'that' particular blind test into question straight away. Hes also only quoting blind tests that can be easily found in the relevent books to anyone that cares to go research more which I see no problem with. If he was quoting them from very hard to find papers id certainly question it a lot more.

In answer to your question there. Im pretty sure that taken as an average, people DID believe they sounded the same (Thats NOT to say all of them, just that the blind tests indicated that there was no positive proof one way or the other). These same people would probably EASILY tell the differences given a few days to test out various tracks etc, which again brings blind testing under scrutiny (I say this because ive lost count of the number of people that have been a part of blind tests and have struggled to tell any difference but when at home and properly tested 'over time' could tell the differences quite easily)

 

[aliEnRIK]

drumlins4ever:

simply put , That blind testing is worthless

No ~ I said ive never believed in them. The writer of the article said theyre WORTHLESS. Ive already given 2 examples myself as to why I dont trust blind tests.

Properly constructed blind tests can INDICATE to a conclusion. Nothing more.

All
ive done is pointed people towards an article which backs me up a
little further. If you have evidence to the contrary then please share.

 

[Craig M.]

i would say that if blind listening sounded different to sighted listening, it was because somehow what you see alters your perception of sound. therefore its all in your head.

 

[Anonymous]

aliEnRIK:drumlins4ever:
ÿ

simply put , That blind testing is worthless ÿ

ÿ

ÿNo ~ I said ive never believed in them. The writer of the article said theyre WORTHLESS. Ive already given 2 examples myself as to why I dont trust blind tests.

Properly constructed blind tests can INDICATE to a conclusion. Nothing more.

Allive done is pointed people towards an article which backs me up alittle further. If you have evidence to the contrary then please share.

ÿ

Well if you don't believe in Blind testing then your a quack , sorry man but in medicine for example blind testing has been invaluable in gathering data that shows what works, what dosent and what's in peoples heads.

Blind testing has helped to show the important difference between Voodoo andÿPenicillinÿ.

ÿ

ÿ

 

[Anonymous]

Craig M.:i would say that if blind listening sounded different to sighted listening, it was because somehow what you see alters your perception of sound.ÿ therefore its all in your head.

ÿ

Exactly , well said.ÿ

 

[Tear Drop]

drumlins4ever:Well if you don't believe in Blind testing then your a quack , sorry man but in medicine for example blind testing has been invaluable in gathering data that shows what works, what dosent and what's in peoples heads.Blind testing has helped to show the important difference between Voodoo andÿPenicillin

Blind testing for audio/hifi barely resembles the type of blind tests done in medicine so there really is no comparison I'm afraid. Audio blind tests are something of a shambles - if they were done exactly like those done in the medical world ( if that is at all possible) then all these forum arguments would happily stop (I wish...).

 

[Anonymous]

Trouble is with blind tests in that situation is that most hifi sceptics are just going to say everything sounds the same.....I was reading an article a while ago run by a rival forum where four forum members did some interconnect blind tests and 3 of them found it hard to pick which was the £500 or the £5 interconnect but the fourth got it right everytime or near as dam it! Some of us have better ears than others.

 

[idc]

Blind tests are valid, but not totally conclusive. Examples of blind tests are commonly found in the world of wine and cosmetics. Experts put to such tests often get it wrong, see Oz Clarke in Oz and James' wine adventures, he regularly gets it wrong. With regards to the use of blind tests in medicine such tests are only a part of the whole testing process and that is because they are not wholly reliable, but they do give good indications as to how effective a medicine is.

 

[Anonymous]

Craig M.:i would say that if blind listening sounded different to sighted listening, it was because somehow what you see alters your perception of sound. therefore its all in your head.

Nail hit on head.

Once visual clues are removed so are preconceived notions about the equipment. It's hard to understand how anyone can criticise a test that relies purely on the perceived quality of sound. Isn't sound quality what hifi is supposed to be about?

 

[Anonymous]

It is one thing to argue that the results of a blind/double blind test may not necessarily prove anything. However, what can surely not be disputed is that blind (or ideally double blind) testing ought to reveal a heck of a lot more about the respective merits of different products than a test in which the subject (or reviewer) knows exactly what the product is and how much it costs.

To take a simple example, if a reviewer is going to compare the stereo music performance of a multichannel AV amp with that of a dedicated 2 channel amp at the same price point, then it is safe to say there will be a strong expectation of a particular outcome. Same with an expensive cable versus a cheap cable.

Imagine that you are planning to choose between two pieces of equipment that happen to have been featured in a product comparison review. Is there anyone that can honestly say that they would not find it more helpful knowing that the comparison had been done under blind conditions?

 

[Anonymous]

Are we sure that any of the blind testers dont have an agenda? poor hearing? Im only concerned with what I myself can hear to be honest.

 

[aliEnRIK]

drumlins4ever:aliEnRIK:drumlins4ever:

simply put , That blind testing is worthless

No ~ I said ive never believed in them. The writer of the article said theyre WORTHLESS. Ive already given 2 examples myself as to why I dont trust blind tests.

Properly constructed blind tests can INDICATE to a conclusion. Nothing more.

Allive done is pointed people towards an article which backs me up alittle further. If you have evidence to the contrary then please share.

Well if you don't believe in Blind testing then your a quack , sorry man but in medicine for example blind testing has been invaluable in gathering data that shows what works, what dosent and what's in peoples heads.

Blind testing has helped to show the important difference between Voodoo and Penicillin .

Pedantic much?

I was talking about HIFI terms. Not sorting a medicinal placebo from what you appear to be smoking now

 

[Andrew Everard]

Back when I used to do a lot more testing and review-writing than I do now, I always found I got the most interesting results from blind tests when the 'victims' had no interest in hi-fi whatsoever, and had no idea what kind of products they were comparing.