Why are blind equipment tests bad?

[chebby]

Andrew Everard:Mr Modesty:The arguments that religious folk use against the scientific method are very similar to those expressed against blind tests on here.
Maybe they are afraid of the truth.

This line of argument has a familiar ring about it...

I am hearing 'bells' actively pounding out all over the place in the last day or two.

Maybe it is this though.

 

[professorhat]

Septicman:Rik, I don't think many if any are saying that amps, CD players, speakers etc. sound the same.

Mr Modesty:Imagine all those audiophiles learning that their £1000 amplifiers sound the same as a £100 one.

One day, one day.

ÿ

 

[Anonymous]

chebby:Andrew Everard:Mr Modesty:The arguments that religious folk use against the scientific method are very similar to those expressed against blind tests on here.
Maybe they are afraid of the truth.

This line of argument has a familiar ring about it...

I am hearing 'bells' actively pounding out all over the place in the last day or two.

You should try Sunday in Germany. It starts at eight in the morning and finishes at 10 in the evening, and not even decent soundproofing can cut out the cacophonous clamour of a German bell well struck.ÿ

 

[Anonymous]

professorhat:

Septicman:Rik, I don't think many if any are saying that amps, CD players, speakers etc. sound the same.

Mr Modesty:Imagine all those audiophiles learning that their £1000 amplifiers sound the same as a £100 one.

One day, one day.

You are right - in this case. Perhaps I should have been more pedantic. I am sure there is both underpriced and overpriced kit that sounds similar despite being priced very differently. That isn't my point though, as I'm sure you know.

 

[Anonymous]

Imagine this. The difference between a piece of equipment or an interconnect is so subtle that all kinds of arguments rage about how best to experience that difference. In the meantime the money that your are considering spending on said upgrade could be spent on an original Blue Note jazz recording or an original pressing of the Moscow State Symphony Orchestra playing Shostakovich. I would suggest having a good long think about why you wanted a good hi fi in the first place.

 

[Alec]

Whats wrong with individuals just doing whatever testing they comsider necassary to mak etheir minds up, and being satisfied in their own minds that there is or isnt a difference, regardless of anyone else?

Or have i gone mad?

One thing i will say tho is that if people are going to start talking religious philosophy on a hifi site, they may just come across people with some idea what theyre on about, rather than the lushes with whom they hold court down t' pub.

 

[TKratz]

If you by individuals mean 'buyers' then fine by me. Everyone should do as they please when buying equipment.

But if you refer to hifi reviewers I disagree. Even as unbiased they might seem, the only certain way to avoid bias in a review is blind testing.

 

[Alec]

TKratz:

If you by individuals mean 'buyers' then fine by me. Everyone should do as they please when buying equipment.

But if you refer to hifi reviewers I disagree. Even as unbiased they might seem, the only certain way to avoid bias in a review is blind testing.

Yes, i just meant Jo Public, and i agree...to a point...

...except...when i used to read certain music mags i would learn, over time, the preferences of individual reviewers. I would therefore have a fair idea of whether i would like an album based on who they compared it to and, mainly, whether they liked it. Id often buy things the idiot reviewer didnt like, as i knew they would be wrong...

If i knew i had historically liked a certain brand, and i read a negative review of their latest speakers in a mag with inferior testing methods to WHFS&V?s, by a reviewer who i knew liked entirely different speakers, this would still be very helpful in making my shortlist...

I acknowledge, of course, that this would be an example of basing a shortlist on a biased review. Your point about avoiding bias, i quite agree with, i just wonder whther it is entirely necassary...

 

[aliEnRIK]

Septicman:
Rik, I don't think many if any are saying that amps, CD players, speakers etc. sound the same. They patently don't, however, from a personal point of view I would only be interested in looking at kit that had completely neutral reproduction as its goal. I don't want components that colour the sound. Your goals may be different of course, and there is plenty of kit out there to accommodate your tastes. If your sound isn't what you want though, you've probably chosen components you aren't happy with, which is why you're tweaking.

Accessories seem to be the thing that gets everyone talking. Cables, stands and so on. For example, there is a relatively "cheap" mains cable around that GUARANTEES to improve your sound by orders of magnitude. But what does this guarantee mean in real terms? I get to keep the cable for free if it doesn't work? My postage is refunded both ways? I'm told to get lost because my ears aren't up to it? Can you not see why certain aspects of audiophilia are ridiculous? We're talking about a cable that simply acts as a copper conductor, but which has a huge mark up from cost. Unless the physical properties of the cable have been manipulated, so that it no longer acts like copper, then there CANNOT be a difference between a £3.99 kettle lead from B&Q and something costing very much more. If the cable values have been manipulated, then your system no longer sounds as the designer intended, which in turn may have an effect on the rest of your system - thus leading to the need for more tweaking.....

The fact that 'some' blind tests have tried to make it out that all amps above a couple of hundred quid are the same REALLY dents my faith in blind tests for hifi at least. The MAJOR problem I have with blind tests is the rule that you must take RANDOM people. The fact is that just as some people can run faster than others, so too people can HEAR better than others. So why not use people who CAN hear better? But if we broke the 'random' rule the blind test fanatics would cry foul. Its a bit of a vicious circle.

As for the cable phenomenom. Your sort of half right except cables also make a difference if they lessen EMI interference and lessen RFI interference. If 2 cables are EXACTLY the same then im with you all the way regardless if one costs a quid and the other costs 1k.

I also understand you wanting no 'colouration' with different cables and equipment. I too want as close a sound to 'natural' as a I can possibly get but I appreciate other people might want a 'brighter' or 'warmer' sound or whatever

 

[Anonymous]

I have never understood the idea of 'transparency' in hi fi. Do these people go to every concert or studio recording of everything they listen to so they know that the sound they're getting is transparent to the source?

Take the example of a classical music concert. Depending on how and where a recording of the music is made will have quite a large effect on the way it sounds. When listening to this recording, unless someone was there at the live performance, how do they know what they're listening to is transparent, warm, or clinical sounding in comparison to the performance itself?

 

[chebby]

FolsomBlues:Take the example of a classical music concert. Depending on how and where a recording of the music is made will have quite a large effect on the way it sounds. When listening to this recording, unless someone was there at the live performance, how do they know what they're listening to is transparent, warm, or clinical sounding in comparison to the performance itself?

I listen to a lot of sacred music/organ music/choral classical music etc. and I am used to what it sounds like live in churches and cathedrals.

I do not have an aural equivalent of a 'photographic memory' when it comes to all the nuances of the different acoustics of all these various spaces, but I would know if my system was getting it wrong in any significant way regarding the depth being reproduced or the reverberations typical of churches and cathedrals. It would be obvious if the system was making the performance sound like it was coming from a scout hut or underground station!

Obviously 'all bets are off' when it comes to reproducing a cathedral organ at realistic levels with a domestic hifi in a living room(!) but the mind is a wonderful tool for scaling down expectations of such big-scale music in a home environment at normal volume levels.

 

[Alec]

chebby:

FolsomBlues:Take the example of a classical music concert. Depending on how and where a recording of the music is made will have quite a large effect on the way it sounds. When listening to this recording, unless someone was there at the live performance, how do they know what they're listening to is transparent, warm, or clinical sounding in comparison to the performance itself?

I listen to a lot of sacred music/organ music/choral classical music etc. and I am used to what it sounds like live in churches and cathedrals.

I do not have an aural equivalent of a 'photographic memory' when it comes to all the nuances of the different acoustics of all these various spaces, but I would know if my system was getting it wrong in any significant way regarding the depth being reproduced or the reverberations typical of churches and cathedrals. It would be obvious if the system was making the performance sound like it was coming from a scout hut or underground station!

Obviously 'all bets are off' when it comes to reproducing a cathedral organ at realistic levels with a domestic hifi in a living room(!) but the mind is a wonderful tool for scaling down expectations of such big-scale music in a home environment at normal volume levels.

When i saw folsomblues' comment, i thought "hip-hip...", and i still do, but yours makes sense too, Chebby. Maybe its all in what you said about scaling down expectations and the phrase "...in any significant way". Whereas people often use very all or nothing language which leeds the likes o me to scratch our heads thinking "eh? ...b...b...but, it cant be done...".

 

[Anonymous]

aliEnRIK:
The MAJOR problem I have with blind tests is the rule that you must take RANDOM people.

I think that's a rule you just made up.

 

[idc]

There was a link to an article on another thread about blind testing (which I cannot find) where four speakers were tested. In the end all that test proved to me was that you hear differently blind folded that sighted. Some blind tests will be conducted blind folded, but others will be conducted with changes happening out of sight. In a round-a-bout way I am asking probably a dumb question. Do you think of blind testing as just out of sight (which can be easily achieved with everything except speakers) or blind folded. I suspect that two blind tests, one out of sight, the other blindfolded would result in different results.

 

[Anonymous]

Whilst I'm generally all in favour of the scientific method and blind testing in principle I think there are issues with hi-fi that run counter to this principle. The issues are with how different people perceive what they hear and the consequent preferences people have with different types of sound. I can well imagine a group of people coming up with recommendations for equipment from blind tests that might not suit me at all. Didn't I read an article recently about young people actually preferring the sound of highly-compressed MP3 downloads for example?

I think there's also the fact that our perceptions of sound can change over time - it can become more 'atuned'. I have released this over the past few years as I have been studying classical guitar as an adult. A part of my learning has been about learning to listen and differentiate the effects of playing techniques and interpretational elements.

Hi-fi is probably the same - hi-fi enthusiasts and musicians probably have a more atuned ear to the types of sonic quality that we seek. I've no doubt that many people who listen to compressed MP3s alone are perfectly happy with the sound quality, but I am not. How many times have you sat in someone else's car when they've had the stereo on and thought "God I wish they'd turn that treble/bass down a bit - it sounds awful"?

At they end of the day there is no substitute for auditioning kit yourself so that you can let your ears decide. But if I want to work out a shortlist of items to listen to, I'd rather they were 'blind-tested' by people who knew what to listen for at least.

Sorry, does that make me sound like an audio-snob?

 

[Anonymous]

I don't think I disagree with anything you say Chebby. I have a system I think reproduces reasonably well the sound of classical music live. I think my point would be that, given the accoustic problems you mention, if by a system being 'warm' or 'bright' it better reproduces the live experience in my living room then I'm fine with it. It just comes down to buying a system you enjoy.

 

[shado]

I have an opinion where blind testing works with leads and other peripherals as a slight improvement makes a difference, whereas a total upgrade of equipment is easily apparent without having to be blindfolded. I could tell a difference between a Marantz/Sansui/Pioneer Amps however speakers Wharfedale Diamond (TOS) against Sony APM 22ES apart from bass was not a random decision and I would argue that the final link in the chain deserves considerable respect. In my humble opinion the KEF iq30 spkr has the correct balance of bass and character to compliment modern Arcam & Cyrus based system within my budget constraints whereas before, the speaker was the last consideration in the chain. Stands and cables should be honoured with the same respect and this is how my decision has been redifined. You have to enjoy/appreciate the change. I will have from a 80's perspective a hifi system complemented by a sonos media to satisfy the ludidtes (the younger gen of my family). Sorry I have listened to the best buy B&W but I still prefer the Wharfedale Diamond 9.1 and KEF and I find the similar designed Mezzos polite but not exciting. But I found this out without being blindfolded. I believe this may work with computer based media against CD based systems to convince certain sceptics but a reasonable adult can tell the difference because they listen.

 

[Andrew Everard]

The Samuel Beckett lengthy pause at the end of a post had long been underused...

 

[idc]

jgw911:

Whilst I'm generally all in favour of the scientific method and blind testing in principle I think there are issues with hi-fi that run counter to this principle. The issues are with how different people perceive what they hear and the consequent preferences people have with different types of sound. I can well imagine a group of people coming up with recommendations for equipment from blind tests that might not suit me at all. Didn't I read an article recently about young people actually preferring the sound of highly-compressed MP3 downloads for example?

I think there's also the fact that our perceptions of sound can change over time - it can become more 'atuned'. I have released this over the past few years as I have been studying classical guitar as an adult. A part of my learning has been about learning to listen and differentiate the effects of playing techniques and interpretational elements.

Hi-fi is probably the same - hi-fi enthusiasts and musicians probably have a more atuned ear to the types of sonic quality that we seek. I've no doubt that many people who listen to compressed MP3s alone are perfectly happy with the sound quality, but I am not. How many times have you sat in someone else's car when they've had the stereo on and thought "God I wish they'd turn that treble/bass down a bit - it sounds awful"?

At they end of the day there is no substitute for auditioning kit yourself so that you can let your ears decide. But if I want to work out a shortlist of items to listen to, I'd rather they were 'blind-tested' by people who knew what to listen for at least.

Sorry, does that make me sound like an audio-snob?

No, I think that you have made a very good point

Even if a blind test produced a definitive result people listen with their eyes open and may just not like that sound compared to another.

 

[Anonymous]

The perfectly transparent system and Godot - we're all waiting and they're never coming...

 

[shado]

Andrew Everard:The Samuel Beckett lengthy pause at the end of a post had long been underused...
Sorry Andrew not my intention just bandwith problems ;-)

Apart from the continuous slanging match that goes on, I have experienced a rewardable system and I am sure you are proud of your japanese connections with relation to Hifi. At the end of the day it is down to enjoyment and you are fortunate that your work supplements your hobby and to be a journalist in layman terms must be reward enough.

My enjoyment is trying out/listening to new equipment at either Sevenoakes/Richer Sounds and I will reiterate the budget end is the true battle ground for future products - the affordable end!!

 

[Andrew Everard]

shado:...and I am sure you are proud of your japanese connections with relation to Hifi.

Bit confused again - what 'japanese connections with relation to Hifi'? Though admittedly I do have Japanese relatives...

 

[aliEnRIK]

Mr Modesty:aliEnRIK:
The MAJOR problem I have with blind tests is the rule that you must take RANDOM people.

I think that's a rule you just made up.

Not at all. Before a 'blind test' is truly valid (especially in the medicinal world), you must be able to recreate the same results time and time again using random people from anywhere in the world. If it cant be recreated then the test has failed

You COULD do a blind test using just audiophiles with very good hearing but everyone would say its a fix as it cant be recreated elsewhere which defeats the whole point of using them in the 1st place.

Still, if YOU know of any such blind tests im all...ears

 

[Anonymous]

Oh dear, I thought this had gone away

No-one gets upset about blind wine tasting. There's room for both blind and sighted reviews, surely.

 

[Anonymous]

aliEnRIK:Mr Modesty:aliEnRIK:
The MAJOR problem I have with blind tests is the rule that you must take RANDOM people.

I think that's a rule you just made up.

Not at all. Before a 'blind test' is truly valid (especially in the medicinal world), you must be able to recreate the same results time and time again using random people from anywhere in the world. If it cant be recreated then the test has failed

You COULD do a blind test using just audiophiles with very good hearing but everyone would say its a fix as it cant be recreated elsewhere which defeats the whole point of using them in the 1st place.

Still, if YOU know of any such blind tests im all...ears

It doesn't matter if it's conducted properly.