Blind Listening Tests are Flawed

[ifor]

I expect many have read this before, but it's new to me.

http://www.avguide.com/forums/blind-listening-tests-are-flawed-editorial?page=1

I'm open minded when it comes to the interconnect and speaker cable debates. I've never bothered, yet, to make comparisons, either blind or otherwise, but I do get exceedingly irritated by fundamentalist non-believers. What won't they just relax back to a position of extreme scepticism?

I do believe that double blind ABX is probably flawed, but that doesn't make me a believer. Curiosity got me googling and I ended up at the page for which I've given the link.

 

[professorhat]

Oh goody!

:cheer:

 

[cheeseboy]

quite frankly that article is just a crock of balls.

He goes to say that blind listening tests are flawed because he doesn't believe in them (referencing the amplifier test) then uses one single test to try an debunk all the other tests. Was he at the amplifier test - no, but he still is talking as though what he believes is fact with absolutely nothing but his own opinion to back him up. He sets out his stall staight from the off using terms like "absurd conclusion" which instanly makes it more like an ill informed rant than somebody trying to disprove something, and more and more reading through the article, he sounds like a bitter person who refuses to believe that there may or may not be minor differences with things hifi as otherwise his repuation may be at stake.

I don't think anybody wouldn't say blind tests are not flawed, but they are generally a lot less flawed than sighted tests, which is where his argument falls completley flat on his face.

 

[manicm]

Thank you for the link. It puts paid to zealots who espouse blind-testing as the only valid method. Linn have their tune-dem method which is not necessarily blind. And of-course there are other sighted/blind methods too.

 

[cheeseboy]

Thank you for the link. It puts paid to zealots who espouse blind-testing as the only valid method.

but that arrticle doesn't do that at all. It's no different than any random post on here....

 

[manicm]

I don't think anybody wouldn't say blind tests are not flawed, but they are generally a lot less flawed than sighted tests,

Please substantiate this self-admitted generalisation.

 

[cheeseboy]

I don't think anybody wouldn't say blind tests are not flawed, but they are generally a lot less flawed than sighted tests,

Please substantiate this self-admitted generalisation.

hows about I do that after you substantiate why you think one mans rant puts pay to all those "zealots" ?

 

[fr0g]

Thank you for the link. It puts paid to zealots who espouse blind-testing as the only valid method. Linn have their tune-dem method which is not necessarily blind. And of-course there are other sighted/blind methods too.

His reason for damning blind testing is because he believes the results are "absurd".

However, if I take his first example, that HD audio sounds identical to CD quality audio, it isn't absurd at all. the science behind the creation of the CD standard was used to create an "exact" copy of the original analogue sound within the boundaries of human hearing, which it does.

A 16 bit by 44.1 KHz signal is "able" to perfectly construct any analogue sound, within the 0-22 KHz range. The fact that some HD versions of albums sound better is irrelevant, as they have been mastered differently.

So his "absurd conclusion" is in fact the scientfic, yet to be disproved conclusion.

No testing system for sound is perfect. Blind testing is better than sighted as it removes some of the unnecessary stimuli that can and do change our perception...the ONLY thing you need to test something for sound quality is your ears. Any extra inputs from the senses will affect what we *think* we hear. That's a fact, an indisputable fact (look up the oft quoted McGurk affect for the best example).

the article is idiotic in the extreme and proves nothing other than the guy is a moron.

 

[BigH]

He would say that of course.

Of course blind tests don't work because you can't tell whats being tested, so how can you tell if its a $1,000 cable or a $1 one? As these guys found out: http://consumerist.com/2008/03/03/do-coat-hangers-sound-as-good-monster-cables/

Someone who does not agree: http://seanolive.blogspot.co.uk/2009/04/dishonesty-of-sighted-audio-product.html

And someone else: http://www.bostonaudiosociety.org/bas_speaker/abx_testing.htm

 

[TannyToft]

The objective of bind trials is to remove sources of potential bias. Clearly, not every trial is perfect and much psycology comes in to play and confounding factors need to be recognised. However, evidence is evidence and as such can be helpful in making decisions. I suspect that the peson who authored the avguide article will beleive in homeopathic treatments for HIV and cancer.

 

[floyd droid]

quite frankly that article is just a crock of balls.

You owe me one monitor and a gobfull of coffee .

 

[BenLaw]

quite frankly that article is just a crock of balls.

You owe me one monitor and a gobfull of coffee .

Was just going to post my chuckling at the same line. Nicely summarised.

 

[BenLaw]

The author described his piece as 'a fairly reasoned and in-depth analysis of why blind listening tests are flawed'. I missed that bit. Would the OP care to point to the reasoning and analysis?

 

[manicm]

hows about I do that after you substantiate why you think one mans rant puts pay to all those "zealots" ?

I never said blind-testing has no merit. But you've failed to substantiate your generalisation that sighted testing is inferior. To date I've seen no scientific evidence of this.

 

[Overdose]

Blind testing, with all of its variables and flaws still manages to produce repeatable results and evidence.

This article seems to provide no evidence at all other than one mans opinion.

 

[manicm]

the science behind the creation of the CD standard was used to create an "exact" copy of the original analogue sound within the boundaries of human hearing, which it does.

What you expediently and conveniently, and perhaps deliberately leave out is the fact that late into CD development it didn't have the capacity to hold Beethoven's 9th symphony. Herbert Von Karajan almost single-handedly made Sony/Philips fix this.

I have no truck with CDs, I buy them to this day, but your assertion that 'the CD standard was used to create an '"exact" copy of the original analogue sound within the boundaries of human hearing' is neither a fact nor accurate - it may have been good enough but was limited by the technology of the day.

 

[Overdose]

the science behind the creation of the CD standard was used to create an "exact" copy of the original analogue sound within the boundaries of human hearing, which it does.

What you expediently and conveniently, and perhaps deliberately leave out is the fact that late into CD development it didn't have the capacity to hold Beethoven's 9th symphony. Herbert Von Karajan almost single-handedly made Sony/Philips fix this.

I have no truck with CDs, I buy them to this day, but your assertion that 'the CD standard was used to create an '"exact" copy of the original analogue sound within the boundaries of human hearing' is neither a fact nor accurate - it may have been good enough but was limited by the technology of the day.

I think you are confusing the amount of data that can fit on a CD to its dynamic range.

 

[manicm]

Blind testing, with all of its variables and flaws still manages to produce repeatable results and evidence. This article seems to provide no evidence at all other than one mans opinion.

But it's more likely long-term listening, be it sighted/blind, that will reveal real flaws. Earlier this year I've been to audition speakers where setups in dealer showrooms were sub-par, where I could immediately hear where something was wrong e.g. by the dealer's own admission that a cheap CA CD player wasn't doing anything justice - I was hearing sibilance. And this was all sighted.

I repeat, I won't discount blind testing, but to assert that sighted testing absolutely cannot prove anything conclusive is false too.

 

[BigH]

I repeat, I won't discount blind testing, but to assert that sighted testing absolutely cannot prove anything conclusive is false too.

Who said that?

 

[Covenanter]

I expect many have read this before, but it's new to me.

http://www.avguide.com/forums/blind-listening-tests-are-flawed-editorial?page=1

I'm open minded when it comes to the interconnect and speaker cable debates. I've never bothered, yet, to make comparisons, either blind or otherwise, but I do get exceedingly irritated by fundamentalist non-believers. What won't they just relax back to a position of extreme scepticism?

I do believe that double blind ABX is probably flawed, but that doesn't make me a believer. Curiosity got me googling and I ended up at the page for which I've given the link.

I'm deeply concerned that you might think that the article makes any sense.

Chris

 

[manicm]

I think you are confusing the amount of data that can fit on a CD to its dynamic range.

One of the reasons they limited the resolution to 16/44 was because of capacity. If they could have gone higher they almost certainly would have.

I very recently bought Bee Gees The Record compilation, and both CDs are packed with over 78 minutes of audio. My CD player was definitely noiser than usual. This had to be down to the longer length. Also, Too Much Heaven, while not harsh, was not sounding as smooth as I expected. I'm not saying this is absolutely because of the longer length, but I'm suspecting so.

The CD medium was in no way perfect.

 

[hammill]

I expect many have read this before, but it's new to me.

http://www.avguide.com/forums/blind-listening-tests-are-flawed-editorial?page=1

I'm open minded when it comes to the interconnect and speaker cable debates. I've never bothered, yet, to make comparisons, either blind or otherwise, but I do get exceedingly irritated by fundamentalist non-believers. What won't they just relax back to a position of extreme scepticism?

I do believe that double blind ABX is probably flawed, but that doesn't make me a believer. Curiosity got me googling and I ended up at the page for which I've given the link.

An interesting piece, I have just ploughed through the comments and the original article is subjected to a good deal of criticism (as well as praise) and I would suggest that the comments should also be read to find a balanced view. I can't help but think double blind ABX if properly applied should be the way to answer the " are all certified HDMI cables the same" question, but it may be that some of the tests are carried out in such a way that real differences are not identified - this is more a question of test design rather than a problem with the methodology. I certainly think that any test where you know what you are hearing cannot be trusted - there is too much evidence on expectation bias to think otherwise.

 

[manicm]

The author may have had a subtle (or not so subtle) agenda against blind testing, which I would not agree with. But in my experience, when going to dealers, blind testing would be extremely difficult. As I said, sub-par setups, acoustic compromises (one dealer recently moved premises, had massive lengths of speaker cables attached, the room was surrounded by glass) etc etc, would probably yield no benefits of blind over sighted testing.

 

[BigH]

I very recently bought Bee Gees The Record compilation, and both CDs are packed with over 78 minutes of audio. My CD player was definitely noiser than usual. This had to be down to the longer length. Also, Too Much Heaven, while not harsh, was not sounding as smooth as I expected. I'm not saying this is absolutely because of the longer length, but I'm suspecting so.

The CD medium was in no way perfect.

Probably down to different mastering, if its recent no doubt been mastered for loudness, just check the DR database, there a USA Polydor Greatest Hits thats pretty good, others not very.

 

[hammill]

I think you are confusing the amount of data that can fit on a CD to its dynamic range.

One of the reasons they limited the resolution to 16/44 was because of capacity. If they could have gone higher they almost certainly would have.

I very recently bought Bee Gees The Record compilation, and both CDs are packed with over 78 minutes of audio. My CD player was definitely noiser than usual. This had to be down to the longer length. Also, Too Much Heaven, while not harsh, was not sounding as smooth as I expected. I'm not saying this is absolutely because of the longer length, but I'm suspecting so.

The CD medium was in no way perfect.

It does matter how much music is squeezed on to an LP for perfectly good technical reasons. It makes no difference to a CD (apart from the fact that 78 minutes is outside redbook spec and your player may not play it) and you have no reason to suspect it would do. If your Bee Gees compilation sounds awful, blame the recording,production or mastering - it is not the fault of the medium.