Interesting blog here: http://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-product.html . The links in the article are worth a look, too.
-
Do you stream hi-res music or download hi-res tracks? We want your opinions!
Loudspeaker double-blind listening tests better than sighted? So it seems...
- Thread starter admin_exported
- Start date
chebby
Well-known member
- Jun 2, 2008
- 1,257
- 34
- 19,220
What would you say is the message for the average customer?
Would you recommend we try to get our dealers to set up double-blind testing when purchasing loudspeakers?
It is unlikely that is possible in most cases and - given how profound room effects are - meaningless to the end result in our own home.
Getting dealers to set up double blind testing in our own homes is stretching things even further
As far as magazine reviews are concerned then I am pretty sure Clare has mentioned in the past that blind testing in group tests is standard WHF procedure.
Would you recommend we try to get our dealers to set up double-blind testing when purchasing loudspeakers?
It is unlikely that is possible in most cases and - given how profound room effects are - meaningless to the end result in our own home.
Getting dealers to set up double blind testing in our own homes is stretching things even further
As far as magazine reviews are concerned then I am pretty sure Clare has mentioned in the past that blind testing in group tests is standard WHF procedure.
I've read that article before and it is very interesting. As with most things, common sense prevails. Of course blind testing is better than sighted testing. It's so obvious as to be ridiculous to even ask the question. Sound has no visual component so how could anyone argue for sighted testing.
The arguments against blind testing are generally specious. Yes, blind testing is also flawed, but it's still obviously much better than sighted testing. For professional reviewers, there's really no reasonable excuse for not blind testing to achieve product rankings.
While it might seem that blind testing removes all bias, it's not completely as simple as that. Due to the way the brain processes music, you impressions of speaker B can be impacted by speaker A that preceeded it. A speaker will generally be rated higher when directly preceeded by a bad speaker and vice versa. Another problem is the music. The first time you hear a piece of new music, your brain has to work harder to process it than the second time you hear it. This affects your evaluation. Another problem is that nuances revealed by a good speaker make those nuances more apparant on a bad speaker. What I mean by this is that if speaker A allows you to hear a subtle detail, you will then hear that same detail on the next speaker tested even if you would not normally have been able to. Your brain knows that the subtle detail is now in the music and you will hear it from that point on.
The arguments against blind testing are generally specious. Yes, blind testing is also flawed, but it's still obviously much better than sighted testing. For professional reviewers, there's really no reasonable excuse for not blind testing to achieve product rankings.
While it might seem that blind testing removes all bias, it's not completely as simple as that. Due to the way the brain processes music, you impressions of speaker B can be impacted by speaker A that preceeded it. A speaker will generally be rated higher when directly preceeded by a bad speaker and vice versa. Another problem is the music. The first time you hear a piece of new music, your brain has to work harder to process it than the second time you hear it. This affects your evaluation. Another problem is that nuances revealed by a good speaker make those nuances more apparant on a bad speaker. What I mean by this is that if speaker A allows you to hear a subtle detail, you will then hear that same detail on the next speaker tested even if you would not normally have been able to. Your brain knows that the subtle detail is now in the music and you will hear it from that point on.
Frank Harvey
Well-known member
- Jun 27, 2008
- 567
- 1
- 18,890
jaxwired: While it might seem that blind testing removes all bias, it's not completely as simple as that. Due to the way the brain processes music, you impressions of speaker B can be impacted by speaker A that preceeded it. A speaker will generally be rated higher when directly preceeded by a bad speaker and vice versa.I totally agree with this. I'm not against blind testing, but I've heard many people try a pair of speakers for half an hour, swap to another pair, only to say within 10 seconds "I don't like those". Huge mistake! What they're hearing is completely different to what they've just settled into, and are judging them before they've even had a chance to adjust to them.
Another problem is that nuances revealed by a good speaker make those nuances more apparant on a bad speaker. What I mean by this is that if speaker A allows you to hear a subtle detail, you will then hear that same detail on the next speaker tested even if you would not normally have been able to. Your brain knows that the subtle detail is now in the music and you will hear it from that point on.Again, agreed. Most people think that because they heard the subtle nuance in the cheaper speaker as well, then they must be as good. To me, this says that the speaker that revealed that nuance in the first place is more communicative - if the listener had bought the lesser speakers, would that nuance have ever been brought to their attention?
Another thing about group blind testing, is that I think inexperienced test subjects just tend to pick the speakers on tonal attributes - they'll choose what's easier on the ear, which much of the time is the more budget models due to their softer, warmer, less offensive sound.
Another problem is that nuances revealed by a good speaker make those nuances more apparant on a bad speaker. What I mean by this is that if speaker A allows you to hear a subtle detail, you will then hear that same detail on the next speaker tested even if you would not normally have been able to. Your brain knows that the subtle detail is now in the music and you will hear it from that point on.Again, agreed. Most people think that because they heard the subtle nuance in the cheaper speaker as well, then they must be as good. To me, this says that the speaker that revealed that nuance in the first place is more communicative - if the listener had bought the lesser speakers, would that nuance have ever been brought to their attention?
Another thing about group blind testing, is that I think inexperienced test subjects just tend to pick the speakers on tonal attributes - they'll choose what's easier on the ear, which much of the time is the more budget models due to their softer, warmer, less offensive sound.
That's an interesting one. In another paper he points out that it is becoming increasingly hard to listen to a wide range of speakers at dealers, and that more and more people are buying speakers off the internet. He therefore argues that speaker manufacturers should move away from the meaningless specs they publish, and derive new ones which explain to the consumer how a speaker sounds.
The paper about trained versus untrained listeners draws some interesting conclusions, too, not least of which is that the hihfi dealer test group proved performed better in the tests than the others.
Posted this more because Mr Olive makes good, thought-provoking reading. I've always thought there's a place for both sighted and unsighted listening.
Not sure about WHF - I know part of their cable tests involve blind listening, but not whether this is the case across the board.
The paper about trained versus untrained listeners draws some interesting conclusions, too, not least of which is that the hihfi dealer test group proved performed better in the tests than the others.
Posted this more because Mr Olive makes good, thought-provoking reading. I've always thought there's a place for both sighted and unsighted listening.
Not sure about WHF - I know part of their cable tests involve blind listening, but not whether this is the case across the board.
idc
Well-known member
- Jan 2, 2008
- 1,143
- 117
- 19,370
That article has been linked to on the forum before. I have read through it again and I'm not convinced that blind testing is better than sighted. Look at the conclusions reached. Blind were able to tell more difference in room placement than sighted. Conclusion blind is better. Wrong, the concludsion is that your hear differently with your eyes shut than open. Your eyes tell you where the speakers are, so the brain does not bother processing as much spacial information. There is no connection between that and which way of listening is better.
As for the overall conclusion, sighted were able to place the identical speakers, but with different crossovers as more or less the same, but they did hear a difference. Blind had them as the same, even though they have different crossovers. Sighted rated the cheapest speaker as lower than the most expensive. Blind had it the other way round. So again, where is the link between those results and which way of listening is better?
Here is another article about the problems of blind testing. It goes some way to explaining the flaws in the Steve Olive reported test, and shows that he ignores the flaws in blind testing whilst criticising the flaws in sighted testing.............
http://www.bostonaudiosociety.org/bas_speaker/abx_testing.htm
....so the real conclusion is that both forms are flawed, and both are valid, and they should be run in partnership, and not be considered to be conflicting.
As for the overall conclusion, sighted were able to place the identical speakers, but with different crossovers as more or less the same, but they did hear a difference. Blind had them as the same, even though they have different crossovers. Sighted rated the cheapest speaker as lower than the most expensive. Blind had it the other way round. So again, where is the link between those results and which way of listening is better?
Here is another article about the problems of blind testing. It goes some way to explaining the flaws in the Steve Olive reported test, and shows that he ignores the flaws in blind testing whilst criticising the flaws in sighted testing.............
http://www.bostonaudiosociety.org/bas_speaker/abx_testing.htm
....so the real conclusion is that both forms are flawed, and both are valid, and they should be run in partnership, and not be considered to be conflicting.
First of all, why are you assuming that blind testing is done with the listeners eyes shut? Acoustically transparent but visually opaque material is readily available. I would expect this to be the norm for blind testing. This would invalidate your argument.
The real problem with all listening tests is the inherent inaccuracy of the measuring device, human beings. Recently we debated with the WHF staff about their reviews of audio racks. They ascribed various sound characteristics to them.
I'd be willing to bet a months pay that in a blind listening test of audio racks if I were to trick the listeners and actually use the exact same rack for all 4 samples, the WHF staff would happily rank them 1 through 4 attributing various sound qualities and sound flaws to the various samples.
I how ever would rank them correctly as all equal since I would expect audio racks to all sound the same regardless of structure.
I'd be willing to bet a months pay that in a blind listening test of audio racks if I were to trick the listeners and actually use the exact same rack for all 4 samples, the WHF staff would happily rank them 1 through 4 attributing various sound qualities and sound flaws to the various samples.
I how ever would rank them correctly as all equal since I would expect audio racks to all sound the same regardless of structure.
idc
Well-known member
- Jan 2, 2008
- 1,143
- 117
- 19,370
jaxwired:
First of all, why are you assuming that blind testing is done with the listeners eyes shut? Acoustically transparent but visually opaque material is readily available. I would expect this to be the norm for blind testing. This would invalidate your argument.
This is the reason why I assumed that blind did mean blind (by whatever means). Clearly, since you take a different interpretation of what blind means in the article's context, the article misses out one very important factor, what does it mean by blind?
'where the visual biases and cognitive factors dominated listeners' judgment of the auditory stimuli.'
First of all, why are you assuming that blind testing is done with the listeners eyes shut? Acoustically transparent but visually opaque material is readily available. I would expect this to be the norm for blind testing. This would invalidate your argument.
This is the reason why I assumed that blind did mean blind (by whatever means). Clearly, since you take a different interpretation of what blind means in the article's context, the article misses out one very important factor, what does it mean by blind?
'where the visual biases and cognitive factors dominated listeners' judgment of the auditory stimuli.'
jaxwired:
I'd be willing to bet a months pay that in a blind listening test of audio racks if I were to trick the listeners and actually use the exact same rack for all 4 samples, the WHF staff would happily rank them 1 through 4 attributing various sound qualities and sound flaws to the various samples.
I how ever would rank them correctly as all equal since I would expect audio racks to all sound the same regardless of structure.
So how much do you earn in a month? Just want to know how much to plan spending after your 'cunning' test
But seriously - just because you have a set stance on something, please don't project that onto us. We go into each and every test just looking to report what we experience - whether that's something or nothing different!
I'd be willing to bet a months pay that in a blind listening test of audio racks if I were to trick the listeners and actually use the exact same rack for all 4 samples, the WHF staff would happily rank them 1 through 4 attributing various sound qualities and sound flaws to the various samples.
I how ever would rank them correctly as all equal since I would expect audio racks to all sound the same regardless of structure.
So how much do you earn in a month? Just want to know how much to plan spending after your 'cunning' test
But seriously - just because you have a set stance on something, please don't project that onto us. We go into each and every test just looking to report what we experience - whether that's something or nothing different!
Clare Newsome:
But seriously - just because you have a set stance on something, please don't project that onto us. We go into each and every test just looking to report what we experience - whether that's something or nothing different!
I didn't mean to impune your integrity. I am confident that WHF is honestly assessing products in their reviews.
But seriously - just because you have a set stance on something, please don't project that onto us. We go into each and every test just looking to report what we experience - whether that's something or nothing different!
I didn't mean to impune your integrity. I am confident that WHF is honestly assessing products in their reviews.
Tarquinh:Interesting blog here: http://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-product.html . The links in the article are worth a look, too.
I've debated this article and blind listening in general a few times: my stance is that I encourage manufacturers to use blind testing as part of their creative process, as it helps them to isolate factors that really improve sonic performance... Reviewers (if economically feasible) can use a combination of blind and sighted listening to review products... as for us consumers: well I don't think blind listening is realistically going to happen for us (due to cost and just being a general p.i.t.a. to setup correctly) + visual bias is important to many of us... I don't care how sweet a speaker sounds, if it's going to take up half of my room (wall to wall and floor to ceiling), then I just can't use it (unless I have a dedicated listening room for ugly audio equipment)... WAF is a big deal to many consumers and will continue to be that way, so doing a blind test and discovering that a cheap, nasty looking bit of kit sounds nicer than the gear you plan to buy is not very helpful...
I've debated this article and blind listening in general a few times: my stance is that I encourage manufacturers to use blind testing as part of their creative process, as it helps them to isolate factors that really improve sonic performance... Reviewers (if economically feasible) can use a combination of blind and sighted listening to review products... as for us consumers: well I don't think blind listening is realistically going to happen for us (due to cost and just being a general p.i.t.a. to setup correctly) + visual bias is important to many of us... I don't care how sweet a speaker sounds, if it's going to take up half of my room (wall to wall and floor to ceiling), then I just can't use it (unless I have a dedicated listening room for ugly audio equipment)... WAF is a big deal to many consumers and will continue to be that way, so doing a blind test and discovering that a cheap, nasty looking bit of kit sounds nicer than the gear you plan to buy is not very helpful...
chebby
Well-known member
- Jun 2, 2008
- 1,257
- 34
- 19,220
The best procedure I have ever read for 'blind' testing of speakers was the paper...
"The selection of loudspeakers for BBC Radio & Music" by R. Walker, BBC Research and Development Department. 2004.
It is a very lengthy text and there is no longer a direct link (sorry). I saved it to my documents as a PDF a couple of years ago before the BBC pulled back the dates of their accessible research papers to 1996. Grrrr
You may have some luck finding someone with the whole text on their site.
I dare not paste it here as text because it would blow the server or exceed max post size or have the wrong format. Besides no-one would read it
"The selection of loudspeakers for BBC Radio & Music" by R. Walker, BBC Research and Development Department. 2004.
It is a very lengthy text and there is no longer a direct link (sorry). I saved it to my documents as a PDF a couple of years ago before the BBC pulled back the dates of their accessible research papers to 1996. Grrrr
You may have some luck finding someone with the whole text on their site.
I dare not paste it here as text because it would blow the server or exceed max post size or have the wrong format. Besides no-one would read it
jaxwired:
I'd be willing to bet a months pay that in a blind listening test of audio racks if I were to trick the listeners and actually use the exact same rack for all 4 samples, the WHF staff would happily rank them 1 through 4 attributing various sound qualities and sound flaws to the various samples.
I how ever would rank them correctly as all equal since I would expect audio racks to all sound the same regardless of structure.
I expect the same would be true for all types of connecting leads too!
I'd be willing to bet a months pay that in a blind listening test of audio racks if I were to trick the listeners and actually use the exact same rack for all 4 samples, the WHF staff would happily rank them 1 through 4 attributing various sound qualities and sound flaws to the various samples.
I how ever would rank them correctly as all equal since I would expect audio racks to all sound the same regardless of structure.
I expect the same would be true for all types of connecting leads too!
plastic penguin
Well-known member
- Apr 28, 2008
- 6,589
- 3,195
- 36,570
FrankHarveyHiFi:
jaxwired: While it might seem that blind testing removes all bias, it's not completely as simple as that. Due to the way the brain processes music, you impressions of speaker B can be impacted by speaker A that preceeded it. A speaker will generally be rated higher when directly preceeded by a bad speaker and vice versa.I totally agree with this. I'm not against blind testing, but I've heard many people try a pair of speakers for half an hour, swap to another pair, only to say within 10 seconds "I don't like those". Huge mistake! What they're hearing is completely different to what they've just settled into, and are judging them before they've even had a chance to adjust to them.
Another problem is that nuances revealed by a good speaker make those nuances more apparant on a bad speaker. What I mean by this is that if speaker A allows you to hear a subtle detail, you will then hear that same detail on the next speaker tested even if you would not normally have been able to. Your brain knows that the subtle detail is now in the music and you will hear it from that point on.Again, agreed. Most people think that because they heard the subtle nuance in the cheaper speaker as well, then they must be as good. To me, this says that the speaker that revealed that nuance in the first place is more communicative - if the listener had bought the lesser speakers, would that nuance have ever been brought to their attention?
Another thing about group blind testing, is that I think inexperienced test subjects just tend to pick the speakers on tonal attributes - they'll choose what's easier on the ear, which much of the time is the more budget models due to their softer, warmer, less offensive sound.
I am totally the opposite - I bore the brains out of retailers, because usually take twice as long than intended. That's probably why i keep my set-up for so many years
.
Yes, I think there is credance in blind testing - or double blind testing - when you turn the lights off or if you close your eyes it always sounds different (not by much, but enough).
jaxwired: While it might seem that blind testing removes all bias, it's not completely as simple as that. Due to the way the brain processes music, you impressions of speaker B can be impacted by speaker A that preceeded it. A speaker will generally be rated higher when directly preceeded by a bad speaker and vice versa.I totally agree with this. I'm not against blind testing, but I've heard many people try a pair of speakers for half an hour, swap to another pair, only to say within 10 seconds "I don't like those". Huge mistake! What they're hearing is completely different to what they've just settled into, and are judging them before they've even had a chance to adjust to them.
Another problem is that nuances revealed by a good speaker make those nuances more apparant on a bad speaker. What I mean by this is that if speaker A allows you to hear a subtle detail, you will then hear that same detail on the next speaker tested even if you would not normally have been able to. Your brain knows that the subtle detail is now in the music and you will hear it from that point on.Again, agreed. Most people think that because they heard the subtle nuance in the cheaper speaker as well, then they must be as good. To me, this says that the speaker that revealed that nuance in the first place is more communicative - if the listener had bought the lesser speakers, would that nuance have ever been brought to their attention?
Another thing about group blind testing, is that I think inexperienced test subjects just tend to pick the speakers on tonal attributes - they'll choose what's easier on the ear, which much of the time is the more budget models due to their softer, warmer, less offensive sound.
I am totally the opposite - I bore the brains out of retailers, because usually take twice as long than intended. That's probably why i keep my set-up for so many years
Yes, I think there is credance in blind testing - or double blind testing - when you turn the lights off or if you close your eyes it always sounds different (not by much, but enough).
Frank Harvey
Well-known member
- Jun 27, 2008
- 567
- 1
- 18,890
jaxwired: First of all, why are you assuming that blind testing is done with the listeners eyes shut? Acoustically transparent but visually opaque material is readily available. I would expect this to be the norm for blind testing. This would invalidate your argument. This is an interesting point though.
When you sit in front of a pair of speakers with your eyes open, you can see the speakers, you can see the room. You know where the sound is coming from, and you know the confines of the room - therefore the sound stays within the room.
Try this: Listen to a track on your system as you normally would. Then, close the curtains and any doors into the room turn off the lights, put on some music on and start listening. Close your eyes and focus on nothing except the music. The idea is to focus only on the sound your hearing. If you close your eyes and imagine you're in the room you're in, it'll sound the same. Forget your speakers. forget your room, forget your walls, imagine you're boundaries are infinite: there are no walls, you're not in your living room, you're not in your house. If you can focus only on the music and get lost in it, you will hear the sort of soundstage your system is really capable of, and you'll probably be surprised at just how big it is, and the real depth on offer. I've had speakers where I do this, and it sounds like the speakers are about 2 feet in front of me, which tells me they're very good at projecting their image towards you without holding back. This might sound weird, but if you can do it properly, it's an eye opener (but don't open your eyes, it'll spoil it!). If at any point you become aware of your surroundings again, you'll lose it.
Some people may already do this, but if you don't, try it - you'll need to be alone though and without the possibility of being disturbed.
When you sit in front of a pair of speakers with your eyes open, you can see the speakers, you can see the room. You know where the sound is coming from, and you know the confines of the room - therefore the sound stays within the room.
Try this: Listen to a track on your system as you normally would. Then, close the curtains and any doors into the room turn off the lights, put on some music on and start listening. Close your eyes and focus on nothing except the music. The idea is to focus only on the sound your hearing. If you close your eyes and imagine you're in the room you're in, it'll sound the same. Forget your speakers. forget your room, forget your walls, imagine you're boundaries are infinite: there are no walls, you're not in your living room, you're not in your house. If you can focus only on the music and get lost in it, you will hear the sort of soundstage your system is really capable of, and you'll probably be surprised at just how big it is, and the real depth on offer. I've had speakers where I do this, and it sounds like the speakers are about 2 feet in front of me, which tells me they're very good at projecting their image towards you without holding back. This might sound weird, but if you can do it properly, it's an eye opener (but don't open your eyes, it'll spoil it!). If at any point you become aware of your surroundings again, you'll lose it.
Some people may already do this, but if you don't, try it - you'll need to be alone though and without the possibility of being disturbed.
The work of Sean and Floyd Toole with the MLL (Multi-channel Listening Laboratory) facility at Harman is very interesting.
To participate in the program as a listener you have to undertake a training program. The program is designed to enable the listener to provide useful development feedback to the engineering team. An engineer tends to have a little difficulty dealing with terms such as 'chocolaty bass', so it's better if you can communicate your feedback in a more precise manner - a lift in the response between 80Hz and 150Hz, for example.
No listener knows what speakers they are hearing, or what the group is made up of. Listeners are chosen at random from the pool and never include the engineer on the project. Listener feedback is then run through statistical analysis.
But, it is not used in isolation. Development work includes anechoic measurements, 360ø spin measurements and real-world listening.
To participate in the program as a listener you have to undertake a training program. The program is designed to enable the listener to provide useful development feedback to the engineering team. An engineer tends to have a little difficulty dealing with terms such as 'chocolaty bass', so it's better if you can communicate your feedback in a more precise manner - a lift in the response between 80Hz and 150Hz, for example.
No listener knows what speakers they are hearing, or what the group is made up of. Listeners are chosen at random from the pool and never include the engineer on the project. Listener feedback is then run through statistical analysis.
But, it is not used in isolation. Development work includes anechoic measurements, 360ø spin measurements and real-world listening.
Similarly Bang & Olufsen train staff from across the business - not just the development team - to join a listening panel, which works in blind conditions, with a curtain.
The panel members are also funded to attend live music events, to keep them attuned to how real instruments and voices sound.
The panel members are also funded to attend live music events, to keep them attuned to how real instruments and voices sound.
John Duncan
Well-known member
- Jan 8, 2008
- 2,034
- 30
- 19,720
Do you mean you'd like to mail him? I can swap email addresses for you if he gives his permission.
chebby said:
I'd say that as an average customer, the best thing to do is to go and have a look and listen to the equipment that you will be living with for potentially some time. Find what you like and maybe get a home demo.
Unless you are buying for purely audio qualities, many other factors will come into play, not least of which is aesthetics.
Blind testing of varying types really only has a place for reviews and development, in my opinion. It would be interesting to see the results of a blind test of equipment with a wide range in prices though, say amplifiers in the £500 to £5000 bracket.
John Duncan said:
'Communicate back channel' sounds so much more Jason Bourne, don't you think? 8)
I don't believe that the influence on your eyes always over rules what your ears tell you - on a couple of occasions I have preferred the sound of a pair of speakers that I assumed beforehand I wouldn't like, and therefore wasn't particlarly keen on them. This wasn't blind testing, but my ears weren't fooled by what I wanted to hear.
Similar threads
Facebook
Twitter
Instagram