The Blind (Mis) Leading the Blind

[aliEnRIK]

Cyril Mason:
Craig M.:i would say that if blind listening sounded different to sighted listening, it was because somehow what you see alters your perception of sound. therefore its all in your head.

Nail hit on head.

Once visual clues are removed so are preconceived notions about the equipment. It's hard to understand how anyone can criticise a test that relies purely on the perceived quality of sound. Isn't sound quality what hifi is supposed to be about?

What im trying to get out there (Which a lot of people seem to be failing to realise) is that these 'blind tests' are VERY poorly done.

And if you believe ALL blind tests 100% im assuming your in agreement then? That ALL amps sound the same and theres NO audible difference between CD, SACD and 96/24?

 

[Andrew Everard]

OK, drumlins and Erik - stop the insults before they go any further.

 

[aliEnRIK]

garethwd:Trouble is with blind tests in that situation is that most hifi sceptics are just going to say everything sounds the same.....I was reading an article a while ago run by a rival forum where four forum members did some interconnect blind tests and 3 of them found it hard to pick which was the £500 or the £5 interconnect but the fourth got it right everytime or near as dam it! Some of us have better ears than others.

I saw one last year where there were 2 none believers of interconnects and 2 believers (Hadly large numbers I admit but I do have a point to make)

The 2 none believers failed to tell any difference at all between the interconnects. The 2 believers did so nearly 100%

Although this is obviously a very poor example due to the low numbers involved it does indicate that some people have better perception than others within the time constraints placed upon blind tests (Much like some people can run faster etc)

Ive found that the majority (Maybe ALL?) of the none believers of mains cables etc have a very poor perception of sound

 

[aliEnRIK]

EDITED by Mods - do not discuss moderation, and re attacking with equal strength, not on here you won't...

 

[chebby]

As far as auditioning equipment I may want to own - then I cannot really make out how any of this will help me.

The dealer is not going to set up elaborate blind testing sessions for me and I don't want them too.

Say I am thinking of upgrading my 'speakers...

I will decide a shortlist based on budget, size, efficiency/amp matching, magazine reviews, user feedback, reputation of company, finish & looks (domestic acceptability), dealer and manufacturer support etc and then get to hear as many as I (practically) can - and with my own amp and source equipment wherever that is possible - given a local dealer who can oblige.

The only part of that (imperfect) process that might be subject to the rigour of blind testing is the magazine reviews. Some do it, some don't. Either way it is out of my control.

Maybe some manufacturers rigorously do blind tests/comparisons when developing a new product. Good for them if they do but they are probably not doing so based on any criteria that I would understand. Again, out of my control.

Until such time as all the manufacturers deliver samples of product to my door for me to do blind comparisons in my living-room for all my family (and presumably a random sample of the public plucked from the street), then it is purely academic.

 

[Alec]

chebby:

As far as auditioning equipment I may want to own - then I cannot really make out how any of this will help me.

The dealer is not going to set up elaborate blind testing sessions for me and I don't want them too.

Say I am thinking of upgrading my 'speakers...

I will decide a shortlist based on budget, size, efficiency/amp matching, magazine reviews, user feedback, reputation of company, finish & looks (domestic acceptability), dealer and manufacturer support etc and then get to hear as many as I (practically) can - and with my own amp and source equipment wherever that is possible - given a local dealer who can oblige.

The only part of that (imperfect) process that might be subject to the rigour of blind testing is the magazine reviews. Some do it, some don't. Either way it is out of my control.

Maybe some manufacturers rigorously do blind tests/comparisons when developing a new product. Good for them if they do but they are probably not doing so based on any criteria that I would understand. Again, out of my control.

Until such time as all the manufacturers deliver samples of product to my door for me to do blind comparisons in my living-room for all my family (and presumably a random sample of the public plucked from the street), then it is purely academic.

Nice

 

[the record spot]

I do a bit of both; blind tests, A/B, the works if I can. Usually with used kit, harder to do with brand new stuff from a dealer, but some will (hello Loud and Clear in Edinburgh!). I took a gamble with my CD player, but that paid off, no testing just bought on strength of review and price. Blind tested an Audio Analogue Paganini which had floor-shaking bass, but Mrs. R_S was less convinced.

I find it very useful, but not the sole element of working out how to get kit. Sometimes it just isn't practical, or possible to do this. I rely on a mix - reviews, both magazine and existing user feedback where obtainable, spec, price; from this I can get my shortlist. From that, I make my choice. That's the ideal world, but sometimes that option. Even with my last CD player, although I hadn't heard it, nor was able to hear it locally, I could at least look up a handle of reviews and decide from there.

Is blind listening perfect? Probably not, but in line with other means available to try equipment out, it can be helpful in coming to a decision where the option to do this exists.

 

[Anonymous]

I have performed blind tests and sighted tests, using AB and ABX test protocols. I have been both a tester and a participant in these tests. So, I have some experience of both conducting and processing them.ÿ

I am entirely unsurprised that under blind conditions there is no statistically significant difference between CD, SACD and DVD-Audio. Under blind conditions, I doubt whether the difference between two versions of Beethoven's Fifth would be noticed, no matter how significant the differences between the two interpretations. Despite this, I am not 'anti' blind testing. I just think it needs to be used in the right context.

Blind testing (carefully conducted) eliminates bias. It can also be used to determine whether two things are essentially identical.ÿABX testing has no subjective content. An ABX test determines whether subjects can accurately differentiate A from B, not whether A is 'better' or 'worse' than B. There are variants of the ABX test that include a subjective component, but these are infinitely harder to implement, especially in a large sample.

It should also be noted that strictly speaking, a test that shows A and B cannot be differentiated is still subject to strict scientific falsification - that scientific statements can only be 'held' to be true, with the proviso that they can later be demonstrated to be false as our understanding of the subject deepens.ÿ

There is also the problem of listener training. This is not about 'golden eared' reviewers or gifted individuals who claim to be able to hear better than a fruit bat. Large sample ABX tests (using random-sample members of the public) often give 'no difference' results. On the other hand, there are numerous well-researched long-standing ABX test schema that use a team of trained listeners, and these deliver more discerning results (loudspeaker companies with designers and engineers who were a part of the on-going Canadian National Research Council evaluation programme are a perfect example - companies like Revel and Paradigm both run blind tests with trained listeners to 'voice' their speakers).

There are other forms of testing that are just as demanding as ABX, but are ignored for no explicable reason. A longitudinal study could be used to evaluate performance, for example by timing the visitors in a room in a show, or comparing the number of visitors inside the room (relative to the total number of show attendees in the corridor outside the room). If you 'improve' a parameter in the room, then people should stay longer and there should be more people inside the room (relatively). As you would effectively be creating a cohort of the show attendees, this is a commonplace test run in medicine and should be scientifically valid.

I suppose it all swings on whether you take ABX to be the definitive evaluation methodology, or merely a tool in the arsenal of evaluation. More specifically, it's whether you think absence of evidence is - or is not - evidence of absence.

ÿÿ

 

[aliEnRIK]

Alan Sircom:
I have performed blind tests and sighted tests, using AB and ABX test protocols. I have been both a tester and a participant in these tests. So, I have some experience of both conducting and processing them.

I am entirely unsurprised that under blind conditions there is no statistically significant difference between CD, SACD and DVD-Audio. Under blind conditions, I doubt whether the difference between two versions of Beethoven's Fifth would be noticed, no matter how significant the differences between the two interpretations. Despite this, I am not 'anti' blind testing. I just think it needs to be used in the right context.

Blind testing (carefully conducted) eliminates bias. It can also be used to determine whether two things are essentially identical. ABX testing has no subjective content. An ABX test determines whether subjects can accurately differentiate A from B, not whether A is 'better' or 'worse' than B. There are variants of the ABX test that include a subjective component, but these are infinitely harder to implement, especially in a large sample.

It should also be noted that strictly speaking, a test that shows A and B cannot be differentiated is still subject to strict scientific falsification - that scientific statements can only be 'held' to be true, with the proviso that they can later be demonstrated to be false as our understanding of the subject deepens.

There is also the problem of listener training. This is not about 'golden eared' reviewers or gifted individuals who claim to be able to hear better than a fruit bat. Large sample ABX tests (using random-sample members of the public) often give 'no difference' results. On the other hand, there are numerous well-researched long-standing ABX test schema that use a team of trained listeners, and these deliver more discerning results (loudspeaker companies with designers and engineers who were a part of the on-going Canadian National Research Council evaluation programme are a perfect example - companies like Revel and Paradigm both run blind tests with trained listeners to 'voice' their speakers).

There are other forms of testing that are just as demanding as ABX, but are ignored for no explicable reason. A longitudinal study could be used to evaluate performance, for example by timing the visitors in a room in a show, or comparing the number of visitors inside the room (relative to the total number of show attendees in the corridor outside the room). If you 'improve' a parameter in the room, then people should stay longer and there should be more people inside the room (relatively). As you would effectively be creating a cohort of the show attendees, this is a commonplace test run in medicine and should be scientifically valid.

I suppose it all swings on whether you take ABX to be the definitive evaluation methodology, or merely a tool in the arsenal of evaluation. More specifically, it's whether you think absence of evidence is - or is not - evidence of absence.

 

[Alec]

Alan Sircom: If you 'improve' a parameter in the room, then people should stay longer and there should be more people inside the room (relatively). As you would effectively be creating a cohort of the show attendees, this is a commonplace test run in medicine and should be scientifically valid.

So...patients who don't like the needle that has jsut been revealed leave the room?

Sorry, in all seriousness i don't understand how that works in the medical world nor how it would work with hifi?

 

[Anonymous]

al7478:
Alan Sircom:ÿÿIf you 'improve' a parameter in the room, then people should stay longer and there should be more people inside the room (relatively). As you would effectively be creating a cohort of the show attendees, this is a commonplace test run in medicine and should be scientifically valid.

So...patients who don't like the needle that has jsut been revealed leave the room?

Sorry, in all seriousness i don't understand how that works in the medical world nor how it would work with hifi?

ÿ

Longitudinal studies are used in epidemiology, not in drug testing per se. The Framingham Heart Study is a perfect example of this; more than 5,000 people studied across 50 years. The subjects were not necessarily heart disease patients or those more predisposed to heart disease, but were monitored through their lives to see whether they developed heart disease and what commonalities can be determined from this data. ÿ

How this applies to hi-fi is as follows: Take a room in a hi-fi show. Control all other parameters apart from one component change and the flow of people in and out of the room (same volume level, music, lighting, temperature etc). Change the component regularlyÿthroughout the duration of the showÿand observe the changes in the flow of people in and out of that room. Do not mention that any change has taken place to the show goers. You would need to observe the number of people walking past the room as control, and to determine how many people are in the room relative to the number of people in the entire show.

If the parameter being tested has no influence over the sound, then the number of people going in and out of the room - and the average time they stay in the room - should remain constant. If the parameter influences the sound, there should be a change in both visitor frequency and time in room.

ÿ

ÿ

ÿ

 

[Anonymous]

I'm just picturing the scene... Mr HiFi Reviewer has just finished writing up the findings of his latest blind test.

Mr HFR - "Here's the completed article Mr Editor. As you can see, I found product A to be an insightful performer whose main strengths included timing and rhythmic groove, combined with great bass agility and a sparkling top end. Product B produced a richer, more refined sound with greater bass extension although the upper midrange suffered from a slight lack of clarity and punchiness."

Mr E - "Errr....we used the same product for both trials."

Mr HFR - (in the style of 'Unlucky Alf' from the Fast Show)

"Ooooh, b****r!"

 

[idc]

Just so I have this absolutely clear;

a blind test is where you know what the products are but not which one is being demonstrated at the time,

a double blind test is one where you do not know what the products are, but you know you are being tested

and a longitudinal test is where you do not even realise you are being tested?

 

[Anonymous]

idc:

Just so I have this absolutely clear;

a blind test is where you know what the products are but not which one is being demonstrated at the time,

a double blind test is one where you do not know what the products are, but you know you are being tested

and a longitudinal test is where you do not even realise you are being tested?

The difference between blind and double blind is that in the latter case the person administering the test (or, in a hifi setting, switching between one component to the next) does not know the identity of the object being tested. This eliminates any possibility of them inadvertently giving off any subtle cues that could reveal anything to the test participant.

 

[aliEnRIK]

A lot of hifi is tested using ABX BLIND TESTS

But theyre VERY difficult to comply with

 

[Andrew Everard]

Wild and crazy idea: listen. If you enjoy it, buy it. If you don't, don't.

 

[Alec]

Andrew Everard:Wild and crazy idea: listen. If you enjoy it, buy it. If you don't, don't.

Are you completely off your rocker?!

 

[Andrew Everard]

al7478:Are you completely off your rocker?!

Probably, yes.

 

[chebby]

Blind drunk testing:- You love all the products. They are all your best friends. Hic!

Double blind drunk testing:- There are two of everything tested and they are all fascinating the way they keep spinning around the room and changing colour.

Longitudinal testing:- Everything sounds better lying down because.. "The bass is coming from the very heart of the earth man!"

 

[John Duncan]

I find Cyrus does better in blind testing than sighted...

 

[the record spot]

Andrew Everard:Wild and crazy idea: listen. If you enjoy it, buy it. If you don't, don't.

Radical, man!

 

[Anonymous]

idc:
Just so I have this absolutely clear;

a blind test is where you know what the products are but not which one is being demonstrated at the time,

a double blind test is one where you do not know what the products are, but you know you are being tested

and a longitudinal test is where you do not even realise you are being tested?

ÿ

Sort of, no and sort of.

A blind test is pretty much as you describe. There are those who argue that the subjects of the test should not know the specifics of that test until after the test is over (in ABX testing, this would mean you listen to two things - A and B - and try to determine whether the third listening - X- is A or B, but at no time would you know what A and B are).ÿ

A double-blind test follows exactly the same process as the blind test, with one crucial difference. The person conducting the test cannot know what is being tested either. This means there has to be someone (or something) else 'pulling the levers' who cannot tell the person conducting the test what X is at that time. This eliminates a second level of bias in the system.

Subjects in a longitudinal test are informed they are part of a test group, but not of the protocols of that test. ÿ

ÿ

 

[idc]

This post previously moderated. Please do not repost - Mods

 

[aliEnRIK]

Alan ~

Im kinda intrigued and wondered if you could answer a question for me please?

On page 1 I told my own MAIN reason for not being happy with blind testing (Any type)

Talking hifi only now ~ lets say 100 people do a blind test. And the results dont prove a thing (As usual

)

But then the top 5 people who seemingly COULD tell a difference were then tested again and got much higher results (As they were more perceptive say?)

The problem being that true blind test results need RANDOM people to be scientifically valid. But if thats true then results will always be poor as poor old 65 yr old grandma is never going to tell a difference is she

Anyways ~ have you EVER known ANY blind test to be run as I described? a group of people then the TOP group tested again?

 

[Andrew Everard]

aliEnRIK:results will always be poor as poor old 65 yr old grandma is never going to tell a difference is she

What on earth makes you think that? (Apart, of course, from ageist prejudice...)