Why are blind equipment tests bad?

[admin_exported]

Hi there,

I am a little confused. I've been reading in a few well regarded high-end audio magazines that blind A-B (or A-B-A) testing of equipment is flawed. Try as I might, I cannot understand the logic to this. Surely blind tests help to reduce or minimize bias. If I knew what a specific piece of equipment is supposed to sound like, wouldn't I be inclined to hear what I think it should sound like? Can someone explain this to me in simple English? Thanks.

Rydal

 

[Anonymous]

Because you have to train your guide dog to operate your system.

Thankyou and goodnight...

 

[Anonymous]

rydalc:ÿI am a little confused. I've been reading in a few well regarded high-end audio magazines that blind A-B (or A-B-A) testing of equipment is flawed. Try as I might, I cannot understand the logic to this. Surely blind tests help to reduce or minimizeÿbias. If I knew what a specific piece of equipment is supposed to sound like, wouldn't I be inclined to hear what I think it should sound like? Can someone explain this to me in simple English? Thanks.

Blind testing of equipment is dangerous. It might show some components are not very good. Budget components may also do better than expensive ones. There's an entire fringe industry based on being able to spot differences that would be threatened.

There is a case against blind testing, in that often these tests are of short duration which doesn't allow enough time for a realistic assessment of the equipment being tested, which is at least not without foundation. Often the virtues of an amplifier, for example, may not show though strongly in the short time of a test whose aim is to compare rather than listen. Also, blind tests are usually conducted with limited test material.

 

[Anonymous]

Some good reasons why A/B may ot be ideal were already given above. Also, one might be choose a punchier sound that later turns out to be more tiring. Or listen to the same material over and over again, that may even be a non ideal recording that is covered up more effectively by some setups.

But.... Given that you (and the shop owner) have enough time there is nothing against it. Also one can compare systems at low nd high volumes and see how they compare. Moreover, what it will do is let you judge whether there is any noticable difference at all between components, exclude placebo effects and save some money. No need to start another cable discussion here. I've never done extensive comparisons it in private, listened just in shops and at demo's, but often enough I just could not hear any difference, between some amplifiers and cd players (usually within the same price range). Never so for speakers, there is always a difference, often quite big, even for family members in a series. Once you hear a difference it is time to decide what you prefer. In listening sessions it often helps to be with a few people to discuss, and listen to various types of music.

 

[idc]

Hi rydak. Here is a very involved thread on the pros and cons of blind testing;

http://community.whathifi.com/forums/1/231445/ShowThread.aspx

My own view is that blind testing has its place, but it is often used to as a means to try and ridicule audiophiles who state that they can hear differences between kit and cables, whereas others say there is no difference.

 

[eternaloptimist]

I always have a chuckle at the arguments of the "blind tests are a waste of time" proponents.

Sounds very similar to the arguments against evidence-based medicine which were used (and unfortunately in some circumstances continue to be used) to support interventions / drugs etc. which have subsequently been shown to be useless. "In my experience... expensive new drug X is better than cheaper drug Y..." - yet when blinded there is no difference or... drug Y is better! As a paediatrician with 20 years in medicine it is common for instinct to be wrong as there are so many confounders. Hifi seems to be the same....

How hard is it to see that if you cannot tell the difference when the kit / cables etc. are "blinded" that there is no difference?

If the trial is well setup, taking into account the length of time to audition gear, music samples etc. then where is the argument against this as a basis to compare kit separated from knowledge of the brand / cost / visual appeal etc ?

A whole industry of cable manufacturers could suddenly be in trouble if (heaven forbid) audiophiles (who claim to want to get as close to pure reproduction of the source as possible) actually used blinded tests to get closer to this goal rather than defending snake oil. Who knows, maybe a good calibre, well constructed cable is equivalent to the "oxygen free" blah blah blah "high end" model!

HiFi assessment is not exempt from psychology - the knowledge of the brand, the visual appeal, the cost.... affects subjective assessment as it does with human assessment of anything from cola to cars.

Yes, I am happy to ridicule the ridiculous.

Incidentally, I did audition my gear as blinded as I could. The ICs and cables were based on both visual appeal and that they are solidly constructed with good amounts of copper etc.

 

[jaxwired]

First let me say that I think equipment should be blind tested. Period, end of story. This especially needed in those areas that are suspect (like say the sound improvement from an audio rack). I also think that blind testing does not replace sighted testing. Both are good.

With that said, there is a very famous case of blind testing producing bad results. In the USA in the 1980's the two biggest cola brands were at war, coke and pepsi. Pepsi did an AD campaign where they went around the country giving blind taste tests to random people. This was called "The Pepsi Challenge". Even though coke was by far the market leader, these test consistenly proved that people preferred the taste of Pepsi. When coke saw this, they did their own blind testing and confirmed the pepsi results. So coke changed their formual and called it "New Coke". Now known as one of the biggest marketing blunders in American commercial history. People hated "New Coke" and eventually coke went back to their old formula and dropped "New Coke". This was a very expensive mistake for the coke company.

What went wrong? It turns out that the blind taste tests were all conducted with little tiny 2 ounce samples of the product. In that small quantity people did indeed prefer the slightly sweeter taste of pepsi, but in large full glasses, they preferrred coke. Oops.

-Jax

 

[Anonymous]

idc:
My own view is that blind testing has its place, but it is often used to as a means to try and ridicule audiophiles who state that they can hear differences between kit and cables, whereas others say there is no difference.

Don't know if that's entirely true. HiFi Choice used to test all their review equipment blind, followed by a period sighted, and I don't think they were trying to ridicule anyone. I know that forums tend to use results of blind test to argue against the faithful, but I doubt whether test are set up deliberately to ridicule, that's a bit different. ÿÿ

 

[the record spot]

One or two might even have us believe that a quick glance over a white paper and the performance measurements are you need. Maybe I just need to look at a tracklisting and the songwriters to know how good an album is.

Bunkum. Blind testing's valid and an integral part of the whole process. It helps you to come to a conclusion about what you might be buying; it lets the paper claims stand up against another piece of kit for a direct head to head comparison. All other elements of the listening/buying process taken into account.

 

[Anonymous]

The reason blind tests are deconstructed and argued against is becuase they tend to show that expensive upgrades to interconnects, power leads, mains conditioners, stands and the like are a waste of money. I am not saying a good interconnect or stand is a waste of money. Just that I doubt people would consecutively pick out a £500 interconnect from a £50 one and likewise with a hfii stand.

Its a very sane idea that listening blind to an upgrade in comparison to a cheaper component over a reasonable amount of time is a good way of judging whether its worth your money or not.

When you have spent £200 on a power cord you will hear the difference whether there is one or not.

 

[Anonymous]

Blind testing to be viewed as credible by those that want it needs to meet alot of strict criteria. It is way too much time and trouble to do regularly. To prove or disprove a point I can see the sense, but for everyday product evaluation too much trouble.

For example with cd players I have read of.

The equipment must not be faulty, it must be tested to make sure it meets the redbook spec, so you are not identifying it due to it distorting the sound. So you are taking measurements with test equipment anyway.

The output voltage must be the same, the spec is 2.5v. Players range from 2-3v with some higher than 3v which technically means they have a design fault as they fail to meet the redbook cd specification. A higher voltage will lead to a higher volume, at low levels this would not sound like an increase in volume but an increase in clarity. If too high a voltage for the amplifier used, it could cause distortion, this would not be readily distinguishable as distortion, with my Sony dvd player it made it sound more dynamic, crisp, detailed particularly in the bass, but made vocals slightly less natural and music occasionally wearing, the sound was not as smooth as it should have been.

Level matching to less than 1/10th of a dB when you can only easily distinguish 1dB difference as an increase in volume. The 1/10th of a dB will make it sound different.

Time matching players which unless you are listening to a test tone or pink noise can mean opening up the cd player, as they have to be time matched to the point no subconcious timing clues remain to enable you to pick out which player is which.

Double blind with controls so no clues from those organizing the test, due to the skeptics this means having independent observers or no one will believe the results are not due to poor test controls.

Alot of testers and repeat tests to make the results statistically relevent rather than just dumb luck.

For stereo there is a sweetspot for listening and with lots of listeners not everyone maybe in that spot, the people will also effect sound quality by their presense, acting as sound absorbers of high and mid tones and objects in the room causing sound reflections making the listening environment less ideal and sound quality harder to distinguish.

Audio short term memory for easy comparison, is very short so you are listening to a succession of short identical clips. Lots of different frequencies and harmonics would need to be tried. It would be tiring and if people believe they have golden ears and are trying to be critical listeners this could be very frustrating.

This is probably just the tip of the iceberg. In short it is alot of time and trouble to do right, and if you do it wrong no one believes your results.

You also have to ask why bother, audio fidelity can I believe be measured with equipment, so what is the point in blind listening tests (except in arguements over what is and is not audible to humans). I want to know if the cd player distorts the sound and if its output voltage is correct, if the amplifier distorts sound, how it handles clipping and if its quoted power rattings are accurate. What the speakers frequency range is, how flat their response, if they introduce any distortions, what their ideal placement would be, what their impediance is like across their frequency range. There are lots more things that can be measured and have a known effect on sound quality. You just have to educate the reader.

I would argue more for informative articles, educating the readers to enable them to understand what makes one thing sound better than another rather than just someones opinion from blind or sighted testing saying it does. I would like to see measurements using test equipment and protocols of competing products so they can be objectively compared. This would inspire me to trust the experts who are advising me know what they are talking about and measurments would help to illustrate and validate their subjective opinions more than flowery words.

For example the following site tests dvd players using measurement and seeks to educate it readers, I find it more usefull than a subjective opinion on what dvd players looks best. http://www.hometheaterhifi.com/volu...uide-to-progressive-scan-shootout-1-2003.html

EDITED BY MODS for house rules violation

 

[Anonymous]

rydalc:
Hi there,

I am a little confused. I've been reading in a few well regarded high-end audio magazines that blind A-B (or A-B-A) testing of equipment is flawed. Try as I might, I cannot understand the logic to this. Surely blind tests help to reduce or minimize bias. If I knew what a specific piece of equipment is supposed to sound like, wouldn't I be inclined to hear what I think it should sound like? Can someone explain this to me in simple English? Thanks.

Rydal

Imagine an audio magazine reporting that nobody can hear differences in mains stuff or equipment racks, or revealing that a cheap interconnect is as good a performer as a hugely expensive one. Imagine all those audiophiles learning that their £1000 amplifiers sound the same as a £100 one.

One day, one day.

 

[chebby]

There is a BBC R&D report from 2004 entitled "The Selection of loudspeakers for BBC Radio & Music" (R.Walker) that seems to nail the procedure for blind testing.

However it is too long to paste here (I have it in Word format in my documents) and a BBC link is impossible (yet) because their online publications only go up to the 1990s as yet.

The last people I would want involved in any DBT are Hifi hobbyists/enthusiasts or journalists (sorry WHF I don't mean just yours, I mean all of them from any publication in any geography).

I think DBT panels should comprise 'lay' people (with no hifi agenda or axe to grind), musicians from different disciplines (jazz, rock, folk, classical etc) and people who produce/record music or who broadcast it for a living. A mixture of younger and older people and equally representing both sexes.

Dealers, manufacturers, journalists and hifi enthusiasts would make the worst panelists in my opinion, and I speak as someone I would ban from such a panel!

 

[basshound]

Pmaninit:
Because you have to train your guide dog to operate your system.

Thankyou and goodnight...

Lol,nice one

 

[Alec]

The differnece between Pepsi and Cocal Cola (who im reading a book on just now) is plain for all to see. Anyone who disagrees, or finds pepsi superior, needs their ears testing.

im not sure i agree witht the idea that if you cant hear a diference in a blind test, then there isnt one. im also not sure that testing for long periods is necasarily helpful.

Im also deeply unsure of any comaprison between medical tests and blind - still subjective - tests of hifi kit.

 

[aliEnRIK]

"Why are blind equipment tests bad?"

THE BLIND LEADING THE BLIND

 

[Anonymous]

I think a proper blind test where you listen to a variety of different set music on cd and listen to different radio stations over a reasonable amount of time both through the speakers and via the headphone socket would tell you which systems are the best, but they obviously don't want to do this because cheaper systems would probably sound alot better than most dearer one's, and these other companies might then go bust, especially as already said in a previous post here, system accessories (speaker cable etc)

 

[Anonymous]

aliEnRIK:
"Why are blind equipment tests bad?"

THE BLIND LEADING THE BLIND

Be careful with emotive articles such as this. Some questions have to be asked before jumping to any conclusions.

Is
the writer unbiased in his views? Does he have vested interests one way
or another? Is his evidence conclusive? How many examples does he cite
to prove his case? What is his premise?

 

[TKratz]

Come on folks. Let's skip the conspiracy theories! I see a lot of contradicting comments in this thread.

Yes, there are pitfalls in blind testing as mentioned a lot of time, but if set-up appropriately, I simply don't see why this should be any worse than doing the same tests unblinded? You minimize or eradicate the bias by removing the placebo effect, and that can only be a good thing.

And for those of you quetioning the relevans of blind testing of drugs, please let's not go there. How else would you test new medicine? It don't think you have in-depth knowledge on how to run clinical trials, so please do not make this kind of comparison.

 

[Anonymous]

I would have thought blind testing would be the best and most accurate way to assess the relative merits of equipment and accessories. I suspect though that there are far too many vested interests for it to happen, especially as there are so many entrenched positions and good livings to be made. There's an amusing story of a couple of leading US hi fi journalists who did some blind cable testing, and picked out one particular cable as being very good. The usual descriptions ensued, it provided "more spaciousness", a more clearly defined and punchier bass", all the usual. The cable was subsequently revealed to be a thick bit of flex from the American equivalent of B&Q. What would happen to people like this if blind testing became common?

If you do some googling, you will find websites for professional bodies such as electrical engineers. Take some time to look at their fora and prepare to be shocked at their attitude towards audiophiles, and especially some of their suppliers.

I would certainly see nothing wrong with combining a subjective review with a scientific review, However, it would be difficult to promote the claims of a massively expensive copper cable over a cheaper one if the blind testing revealed no statistically significant preferences, and the science revealed little difference in physical and sonic characteristics. However, it would do the audiophile community a huge favour if this were to be the case.

 

[Anonymous]

The arguments that religious folk use against the scientific method are very similar to those expressed against blind tests on here.

Maybe they are afraid of the truth.

 

[aliEnRIK]

Mr Modesty:
The arguments that religious folk use against the scientific method are very similar to those expressed against blind tests on here.

Maybe they are afraid of the truth.

If the truth WAS that all amps sounded the same then I might have some faith in blind tests. As it is................

 

[Andrew Everard]

Mr Modesty:The arguments that religious folk use against the scientific method are very similar to those expressed against blind tests on here.
Maybe they are afraid of the truth.

This line of argument has a familiar ring about it...

 

[Frank Harvey]

They'd only be afraid of the truth if they could know for 100% certain that the blind test could be trusted and relied upon in the first place......

 

[Anonymous]

Rik, I don't think many if any are saying that amps, CD players, speakers etc. sound the same. They patently don't, however, from a personal point of view I would only be interested in looking at kit that had completely neutral reproduction as its goal. I don't want components that colour the sound. Your goals may be different of course, and there is plenty of kit out there to accommodate your tastes. If your sound isn't what you want though, you've probably chosen components you aren't happy with, which is why you're tweaking.

Accessories seem to be the thing that gets everyone talking. Cables, stands and so on. For example, there is a relatively "cheap" mains cable around that GUARANTEES to improve your sound by orders of magnitude. But what does this guarantee mean in real terms? I get to keep the cable for free if it doesn't work? My postage is refunded both ways? I'm told to get lost because my ears aren't up to it? Can you not see why certain aspects of audiophilia are ridiculous? We're talking about a cable that simply acts as a copper conductor, but which has a huge mark up from cost. Unless the physical properties of the cable have been manipulated, so that it no longer acts like copper, then there CANNOT be a difference between a £3.99 kettle lead from B&Q and something costing very much more. If the cable values have been manipulated, then your system no longer sounds as the designer intended, which in turn may have an effect on the rest of your system - thus leading to the need for more tweaking.....