Still no blinded tests

[woodbino]

When will we ever get tests of hi-fi systems blinded?

I bet the reviewers Whathifi wouldn't be able to tell the difference between a 500 quid unit and a 15000 quid unit.

Really, don't we want to know what sounds best, regardless of the price?

 

[Frank Harvey]

Even with the law of diminishing returns, if someone can't tell the difference between a £500 unit and a £15,000 unit blind, they shouldn't be reviewers...

 

[woodbino]

True, but we will never know u til it's tried. As far as I can see it's never been readily tried.

 

[Frank Harvey]

I'm hoping to offer blind auditions in one way or another for those that request it. Just need to work out the best way to do it.

 

[insider9]

For me the bigger issue is the fact all products are awarded star ratings based on price categories that fluctuate year to year. That's what makes it difficult for consumers to compare products. I'm only saying this in relation to preparing a shortlist as opposed to making a purchase.

How do you compare a 5* award winner amp up to £250 with a 3* amp sold for £1,200? What happens if the £1,200 unit is suddenly discounted to £900... does it make it 4*? Of course almost extreme example but proving a point nevertheless.

On top of that the categories are so inconsistent year to year. No doubt, so that certain products can be classified and rated more favourably.

 

[Gazzip]

For me the fundamental flaw with blind testing is that you are removing a sense (sight) with which hearing has an interdependency. There is plenty of evidence out there that what you see will influence what you hear. Those obsessed with real world measurement as the arbiter of audio truth would call this phenomenon expectation bias. I think this is an oversimplification and that this collaboration of the senses is much more meaningful and deeply primal in respect of how we perceive the world.

During speech perception for example, our brain integrates information from our ears with that from our eyes. Because this integration happens early in the perceptual process, visual cues influence what we think we are hearing. That is, what we see can actually shape what we "hear." This visual-auditory crosstalk, which happens every time we perceive speech, becomes obvious in a phenomenon called the McGurk Effect. Watch this video...

https://www.youtube.com/watch?v=G-lN8vWm3m0

In this case, despite the fact that you are listening to the same sound (the word "bah"), what you hear depends on which face you are looking at. However do this blind and you will hear the same sound, "bah".

As with the McGurk Effect the perception of hifi sound is for me about the combination of sight and sound. It is this which creates our individual perceptions and what we know as our hearing. So yes an expensive amplifier one can see, can sound "better" to an individual than a cheaper one, and no amount of evidence to the contrary is going to change that for him/her. As hearing is a personal response does not an individuals perception therefore actually make it better, regardless of what might happen in a blind test?

 

[Gray]

I'm sure others will remember that, years ago, this was the USP of Hi-Fi Choice.

They made a big thing of using a blinded panels of listeners for group tests after carefully level matching and randomly rotating products during tests - then comparing listeners notes.

They seem to do only single-reviewer, sighted group tests now though. (They probably explained their reasoning for the change but I missed that).

 

[shadders]

For me the fundamental flaw with blind testing is that you are removing a sense (sight) with which hearing has an interdependency. There is plenty of evidence out there that what you see will influence what you hear. Those obsessed with real world measurement as the arbiter of audio truth would call this phenomenon expectation bias. I think this is an oversimplification and that this collaboration of the senses is much more meaningful and deeply primal in respect of how we perceive the world.

During speech perception for example, our brain integrates information from our ears with that from our eyes. Because this integration happens early in the perceptual process, visual cues influence what we think we are hearing. That is, what we see can actually shape what we "hear." This visual-auditory crosstalk, which happens every time we perceive speech, becomes obvious in a phenomenon called the McGurk Effect. Watch this video...

https://www.youtube.com/watch?v=G-lN8vWm3m0

In this case, despite the fact that you are listening to the same sound (the word "bah"), what you hear depends on which face you are looking at. However do this blind and you will hear the same sound, "bah".

As with the McGurk Effect the perception of hifi sound is for me about the combination of sight and sound. It is this which creates our individual perceptions and what we know as our hearing. So yes an expensive amplifier one can see, can sound "better" to an individual than a cheaper one, and no amount of evidence to the contrary is going to change that for him/her. As hearing is a personal response does not an individuals perception therefore actually make it better, regardless of what might happen in a blind test?

Hi,

What you seem to be inferring here, is that a person who purchases a £10,000 amplifier with an exotic, expensive looking enclosure, will perceive a better sound than if the amplifier was a bargain basement product. Such that, even if the amplifier internals are the same, ie, bargain basement, someone who pays £10,000 will be a lot happier.

Does this mean we or the hifi press, should never use blind testing, and let people pay a lot of money for extremely poor value for money products?

My interpretation is that, the hifi press must inform people of the true sound quality, rather than let the Mcgurk effect, or expectation bias, influence their results.

Contrary to this, many hifi magazines do perform a technical analysis, and this will provide the reader some alternative evidence on whether the product is value for money, or performs sufficiently to warrant the price requested by the manufacturer.

Regards,

Shadders.

 

[Gazzip]

For me the fundamental flaw with blind testing is that you are removing a sense (sight) with which hearing has an interdependency. There is plenty of evidence out there that what you see will influence what you hear. Those obsessed with real world measurement as the arbiter of audio truth would call this phenomenon expectation bias. I think this is an oversimplification and that this collaboration of the senses is much more meaningful and deeply primal in respect of how we perceive the world.

During speech perception for example, our brain integrates information from our ears with that from our eyes. Because this integration happens early in the perceptual process, visual cues influence what we think we are hearing. That is, what we see can actually shape what we "hear." This visual-auditory crosstalk, which happens every time we perceive speech, becomes obvious in a phenomenon called the McGurk Effect. Watch this video...

https://www.youtube.com/watch?v=G-lN8vWm3m0

In this case, despite the fact that you are listening to the same sound (the word "bah"), what you hear depends on which face you are looking at. However do this blind and you will hear the same sound, "bah".

As with the McGurk Effect the perception of hifi sound is for me about the combination of sight and sound. It is this which creates our individual perceptions and what we know as our hearing. So yes an expensive amplifier one can see, can sound "better" to an individual than a cheaper one, and no amount of evidence to the contrary is going to change that for him/her. As hearing is a personal response does not an individuals perception therefore actually make it better, regardless of what might happen in a blind test?

Hi,

What you seem to be inferring here, is that a person who purchases a £10,000 amplifier with an exotic, expensive looking enclosure, will perceive a better sound than if the amplifier was a bargain basement product. Such that, even if the amplifier internals are the same, ie, bargain basement, someone who pays £10,000 will be a lot happier.

Does this mean we or the hifi press, should never use blind testing, and let people pay a lot of money for extremely poor value for money products?

My interpretation is that, the hifi press must inform people of the true sound quality, rather than let the Mcgurk effect, or expectation bias, influence their results.

Contrary to this, many hifi magazines do perform a technical analysis, and this will provide the reader some alternative evidence on whether the product is value for money, or performs sufficiently to warrant the price requested by the manufacturer.

Regards,

Shadders.

I do not think the hifi press should use blind testing. I do however think that they should continue to perform technical analysis of the products they test and publish the results. This is a hobbyist's pursuit Shadders, not the AES, and any hifi publications need to respond to the hobbiest nature of it.

 

[Oldphrt]

Even with the law of diminishing returns, if someone can't tell the difference between a £500 unit and a £15,000 unit blind, they shouldn't be reviewers...

What if there are no audible differences, like in an amplifier operating within spec?

http://matrixhifi.com/ENG_contenedor_ppec.htm

 

[shadders]

For me the fundamental flaw with blind testing is that you are removing a sense (sight) with which hearing has an interdependency. There is plenty of evidence out there that what you see will influence what you hear. Those obsessed with real world measurement as the arbiter of audio truth would call this phenomenon expectation bias. I think this is an oversimplification and that this collaboration of the senses is much more meaningful and deeply primal in respect of how we perceive the world.

During speech perception for example, our brain integrates information from our ears with that from our eyes. Because this integration happens early in the perceptual process, visual cues influence what we think we are hearing. That is, what we see can actually shape what we "hear." This visual-auditory crosstalk, which happens every time we perceive speech, becomes obvious in a phenomenon called the McGurk Effect. Watch this video...

https://www.youtube.com/watch?v=G-lN8vWm3m0

In this case, despite the fact that you are listening to the same sound (the word "bah"), what you hear depends on which face you are looking at. However do this blind and you will hear the same sound, "bah".

As with the McGurk Effect the perception of hifi sound is for me about the combination of sight and sound. It is this which creates our individual perceptions and what we know as our hearing. So yes an expensive amplifier one can see, can sound "better" to an individual than a cheaper one, and no amount of evidence to the contrary is going to change that for him/her. As hearing is a personal response does not an individuals perception therefore actually make it better, regardless of what might happen in a blind test?

Hi,

What you seem to be inferring here, is that a person who purchases a £10,000 amplifier with an exotic, expensive looking enclosure, will perceive a better sound than if the amplifier was a bargain basement product. Such that, even if the amplifier internals are the same, ie, bargain basement, someone who pays £10,000 will be a lot happier.

Does this mean we or the hifi press, should never use blind testing, and let people pay a lot of money for extremely poor value for money products?

My interpretation is that, the hifi press must inform people of the true sound quality, rather than let the Mcgurk effect, or expectation bias, influence their results.

Contrary to this, many hifi magazines do perform a technical analysis, and this will provide the reader some alternative evidence on whether the product is value for money, or performs sufficiently to warrant the price requested by the manufacturer.

Regards,

Shadders.

I do not think the hifi press should use blind testing. I do however think that they should continue to perform technical analysis of the products they test and publish the results. This is a hobbyist's pursuit Shadders, not the AES, and any hifi publications need to respond to the hobbiest nature of it.

Hi,

I don't understand your comment regarding AES and hobbyist pursuit. You raised the scientific issue of Mcgurk and presented your preference, and I have provided my preference.

Many people are interested in Hifi not only for music pleasure, but also the technical and engineering aspects too. A hifi publication can cater for both, technical and non technical, and blind testing can assure the reader that the conclusions of a product are fair and accurate.

Regards,

Shadders.

 

[Blacksabbath25]

How meany people go into richer sounds

and say can I have a blind test done on that marantz PM6005 against the roskin k3 please they would just smile and say we got a right one here today .

and if you can not tell the differences between a marantz PM6005 and a top of the line reference marantz then there is something wrong with your hearing .

blind tests are ok in a hifi club or a bunch of mates were your got the time to mess around like that and even then it's still your ears telling you that sounds different .

 

[shadders]

How meany people go into richer sounds

and say can I have a blind test done on that marantz PM6005 against the roskin k3 please they would just smile and say we got a right one here today .

and if you can not tell the differences between a marantz PM6005 and a top of the line reference marantz then there is something wrong with your hearing .

blind tests are ok in a hifi club or a bunch of mates were your got the time to mess around like that and even then it's still your ears telling you that sounds different .

Hi,

If you live near an audio dealer, then you could ask for a demo, and take along those masks for ensuring a dark environment that people use to sleep. Not sure what they are called.

Then, most people demo products which are similar in price. Probably is unrealistic to expect a hifi dealer to provide a £10k amplifier, and a £400 one as a comparison.

A magazine could do this though, they have access to the relevant rooms and equipment.

Didn't What Hifi implement these type of tests with some volunteer readers?

Regards,

Shadders.

 

[Frank Harvey]

Even with the law of diminishing returns, if someone can't tell the difference between a £500 unit and a £15,000 unit blind, they shouldn't be reviewers...

What if there are no audible differences, like in an amplifier operating within spec?

http://matrixhifi.com/ENG_contenedor_ppec.htm

Oh, I haven't seen that test for a little while. I remember the first time I clicked on a link to that, and I found it quite laughable. For me, there were a number of issues, but ANY blind test should have a clear space between the speakers and between the speakers and the listener, rather than piling it all up in between the speakers, and with half of it encroaching on the space IN FRONT of the speakers.

 

[nopiano]

I'm sure others will remember that, years ago, this was the USP of Hi-Fi Choice.

They made a big thing of using a blinded panels of listeners for group tests after carefully level matching and randomly rotating products during tests - then comparing listeners notes.

They seem to do only single-reviewer, sighted group tests now though. (They probably explained their reasoning for the change but I missed that).

Yes, back in the days when Choice was a compact format released every quarter or so, that was often their approach. Davedotco, who I've not seen here for a little while, writes eloquently on the experience of being a blind panellist.

It isn't realistic for most journals since, as was well said above, seeing is all part of the experience. It's been noted in similar fashion that a performer who swoons and looks overcome by their own playing is "better" than similar playing without the emoting.

Having seen and heard the previous WHF listening suite, and met the main contributors, I don't think there is much doubt about their ability to hear the differences - and all reports are written as consensus views - whether or not an individual listener (like one of us) might disagree.

 

[bigboss]

True, but we will never know u til it's tried. As far as I can see it's never been readily tried.

There used to be a "Big Question" feature in the magazine where participants used to undertake blind tests to compare. I participated in one. Hasn't been held for a couple of years now.

 

[jjbomber]

meany people go into richer sounds

The best typo ever!

 

[Muddywaterstones]

When I turn off the lights and close my eyes (not really necessary but that's what I do), the music inevitably sounds better, fuller, more 3 dimensional. Is blind testing hi-fi a form of cheating*diablo*

 

[chebby]

When I turn off the lights and close my eyes (not really necessary but that's what I do), the music inevitably sounds better, fuller, more 3 dimensional. Is blind testing hi-fi a form of cheating

No. It's a form of looking like an idiot and tripping over s***

 

[drummerman]

I'm sure others will remember that, years ago, this was the USP of Hi-Fi Choice.

They made a big thing of using a blinded panels of listeners for group tests after carefully level matching and randomly rotating products during tests - then comparing listeners notes.

They seem to do only single-reviewer, sighted group tests now though. (They probably explained their reasoning for the change but I missed that).

... and mostly done by David Price. Does he ever get a day off?

Also, Paul Millers measurements are missed.

I rarely buy Choice these days but appreciate its still Hifi rather than AV in which I have little interest (but the majority of punters probably do).

I stick the HN&RR mostly.

 

[drummerman]

meany people go into richer sounds

The best typo ever!

 

[Frank Harvey]

When I turn off the lights and close my eyes (not really necessary but that's what I do), the music inevitably sounds better, fuller, more 3 dimensional. Is blind testing hi-fi a form of cheating*diablo*

Just hearing what is really there. With your eyes open, you're aware of your room boundaries, so you automatically confine what you're hearing to within those walls. With your eyes closed and/or light off, you're no longer aware of those boundaries, so you're able to hear the real capabilities of our system. I have mentioned before that closing your eyes and being able to forget about the room boundaries and the speakers, allowing you to appreciate more of what the system is actually doing.

 

[manicm]

True, but we will never know u til it's tried. As far as I can see it's never been readily tried.

Like someone else said here, they should have 2 systems properly and simultaneously setup, with no stacking of equipment, and speakers properly sited etc. And my guess may be off the mark, but not many dealers would do this.

However I have grave doubts about abx testing - this can seriously test one's patience, and I do not believe testing 30 second snippets of music is adequate to properly come to a conclusion.

 

[Leeps]

For the record, the most recent Hifi Choice (Feb 2017, issue 420) conducts blind listening tests for their review of six amplifiers.

Stopped buying WHF ages ago. They only seem to review soundbars, bluetooth speakers and phones these days. Every time I pick up a WHF, I do ask myself, "What hifi????" They should consider changing their title to reflect the contents of the magazine. How about 'Stuff'?

 

[kukulec]

"I bet the reviewers Whathifi wouldn't be able to tell the difference between a 500 quid unit and a 15000 quid unit." Maybe you have ear problems, but even an avarage listener should be able to hear the difference immediately. And you can do this too with cables. *diablo*