Subjective/objective testing /AB / AB/X, thoughts.

[TrevC]

Concentrate on speakers and possibly cartridges and the differences are enormous, so once you are using well made and appropriately gauged wire and interconnects and you have a powerful enough amp that is where you will get maximum value from any upgrade. ABX is something hifi magazines should use. It would get rid of the ridiculous mains lead "reviews" for starters, and I'll bet interconnects are not far behind.

 

[busb]

OK, to put busb's remark somewhat sharper (may I, busb?): what if the participants all suffered neurosensory deafness ànd were really honest? Not a single positive result would be reported, so there is no difference, right? Utter :hs:, of course! I'm aware this is a daft example; it's just to make the point clear.

This is a very good point that you and busb make. ABX testing by definition has one very big limitation - it can only prove a positive result but it can't prove a negative one.

Basically if you really can hear a difference then you will get a positive result which categorically proves that you heard a difference. That's all fine and dandy and provides good quality useful data. However the problem with ABX testing is that you can't prove a negative result. If you can't hear any difference it doesn't prove that a difference can't be heard. It just shows that those people on that particular occasion didn't hear a difference. If they tried it again another time they might be able to hear a difference.

I fully agree with you that ABX testing isn't perfect. But it's still the most reliable comparison method available for checking whether or not you can hear any differences between two things. Don't forget that it's not just ABX which can't prove a negative result but all other listening comparison methods can't either.

I'd say ABX as a methodology is so flawed - it's almost pointless. Some say it's the best we got - maybe but I say the best we've got ain't good enough or fit for purpose! In other words it's only going to convince those that have made up their minds that no differences exist. They are not the ones that need convincing!

Any method must be able to prove a negative as well as it can prove a positive in equal measure! There are ways to improve the reliability of ABX though. Scattered throughout the test sequences, their needs to be checks that all but the deaf will hear. These checks could be added harmonic distortion or channel imbalance for example. Any subjects failing to notice these checks could be discounted. The problem is that it doesn't take very long before subjects get very confused as to whether they can hear differences or not - tests need to be short with plenty of breaks to keep subjects awake.

My suspicion is that many people have been so convinced that ABX testing is flawless that they discount their own hearing prowess a little too quickly.

 

[CnoEvil]

My suspicion is that many people have been so convinced that ABX testing is flawless that they discount their own hearing prowess a little too quickly.

Same with Expectation Bias (imo).

 

[busb]

My point being they only seem to work when the differences are extremely obvious.

That's the whole point though - some people were claiming there are extremely obvious differences but another group of people disagreed and sought to challenge those claims.

People talk about night and day differences all the time which is at odds with all the test results. Actual test results are showing that any differences are very minor.

I have always thought that differences between stuff like amplifiers to be fairly subtle. It then becomes a personal choice whether or not the improvements are enought to justify changing something or not. IMO, the difference, albeit it subtle must reach a threshhold beyond doubt.

 

[Overdose]

One can argue the efficacy of ABX or other blind tests, but the results of such tests always seem to contradict the claims of significant differences existing.

If obvious purported differences between equipment on test existed, they would show on any form of blind testing, but they never seem to.

In lieu of any other form of testing, meeting whatever criteria is deemed sufficient to prove some level of accuracy approaching conclusiveness, I'll go with what is currently available and appears to be 99% accurate.

One could always take the openminded standpoint until conclusive proof becomes available, but sometimes you need to act on the balance of probability and one persons opemindedness, is anothers naievity or gullibility.

 

[CnoEvil]

What I find sad, is when all this obsession with testing stops people just going out and LISTENING. Other than a minute percentage of "objective" audiophiles, (made up almost entirely of blokes), this is enough for most normal folk.

I've been accused of "scaremongering unnecessarily" over encouraging people to "try" cables; I find this ironic in the light of where people are being "told", that little of what they hear is real, unless of course, if it conforms to a certain version of "reality".

 

[steve_1979]

What I find sad, is when all this obsession with testing stops people just going out and LISTENING. Other than a minute percentage of "objective" audiophiles, (made up almost entirely of blokes), this is enough for most normal folk.

I've been accused of "scaremongering unnecessarily"" over encouraging people to "try" cables; I find this ironic in the light of where people are being "told", that litlle of what they hear is real, unless of course, if it conforms to a certain version of "reality".

I totally agree that people should listen to as much different HiFi equipment a possible to see what they like the sound of best. There is no substitute to actually listening with your own ears. This is just common sense.

But when it comes to things like replacement mains cables where in 99.9% of cases it will make no difference people should be aware of their own limitations when 'just listening'. In long term tests your auditory memory can easily play tricks on you especially in sighted comparisons where expectation bias can also come into play.

I agree that listening to equipment is of the upmost importance but with things like mains cables this is where methods such as instantaneous A/B switching come in useful during the selection process. When this is done the people who previously thought that they could hear subtle differences in long term sighted tests realise that when compared in instantaneous A/B switching tests there isn't really any difference in the sound between the mains cables. This has been proved many times and even you Cno admit to have never tried comparing cables in a blind test.

If anyone's interested why Cno was accused of scaremongering unnecessarily you can read it in the link below. Cno posted some incorrect information in post 15 on page 1 to which I replied.

http://www.whathifi.com/forum/hi-fi/just-put-in-some-mains-cable-upgrades

 

[CnoEvil]

But when it comes to things like replacement mains cables where in 99.9% of cases it will make no difference people should be aware of their own limitations when 'just listening'. In long term tests your auditory memory can easily play tricks on you especially in sighted comparisons where expectation bias can also come into play.

I agree that listening to equipment is of the upmost importance but with things like mains cables this is where methods such as instantaneous A/B switching come in useful during the selection process. When this is done the people who previously thought that they could hear subtle differences in long term sighted tests realise that when compared in instantaneous A/B switching tests there isn't really any difference in the sound between the mains cables. This has been proved many times and even you Cno admit to have never tried comparing cables in a blind test.

If anyone's interested why Cno was accused of scaremongering unnecessarily you can read it in the link below. Cno posted some incorrect information in post 15 on page 1 to which I replied.

http://www.whathifi.com/forum/hi-fi/just-put-in-some-mains-cable-upgrades

Steve, I respect you enough not to fall out over this.

What I will say is, "Everyone" on here who has tried TQ Black, has heard a difference....that is so statistically significant, that it should not be dismissed so easily.

There are highly experienced members on here who are not gullable, and can hear the difference.....but they choose not to post, so as to avoid this exact situation. i would also suggest that the majority of people who give it a go, hear the difference........and this is despite the "nay brigade" telling them it's impossible.

My objection is not what you believe, or even that you tell people that it makes no difference...........but that you tell them not to try. I actually give people a little more credit than that. You are in effect telling them what is, and what is not worth listening to, which is tantamount to censorship....and all from a position where you have either tried very little, or not at all.

A cable dem by the likes of Chord or TCI might actually make you see things differently; but as Jack Nicholson said in a Few good men - maybe "You can't handle the truth".

Now this is an ABX thread, so it's my last comment about cables.

 

[chebby]

...but as Jack Nicholson said in a Few good men - maybe "You can't handle the truth".

And what happened to him?

 

[TrevC]

What I find sad, is when all this obsession with testing stops people just going out and LISTENING.

Not so. It tells you what you need to listen to.

 

[the record spot]

By somebody else's definition it would seem.

 

[steve_1979]

Steve, I respect you enough not to fall out over this.

That's something which I like and respect about you Cno. We're able to have a difference of opinion and discuss things sensibly without being rude or getting upset with each other and we're still able to be friends and have a laugh and joke afterwards.

:cheers:

What I will say is, "Everyone" on here who has tried TQ Black, has heard a difference....that is so statistically significant, that it should not be dismissed so easily.

This is a perfect case example for this thread which is totally relevant here. I bet that 'everyone' who has tried TQ Black and heard a difference did so in a long term sighted comparison test. I'm not disregarding what these people say but under these comparison conditions your long term auditory memory can easily play tricks on you especially in sighted comparisons where expectation bias can also come into play too.

Now if they heard a difference in blind A/B comparisons or even passed an ABX test their claims would have far more credibility. We all know that long term sighted comparisons can be unreliable.

My objection is not what you believe, or even that you tell people that it makes no difference...........but that you tell them not to try. I actually give people a little more credit than that. You are in effect telling them what is, and what is not worth listening to, which is tantamount to censorship....and all from a position where you have either tried very little, or none at all.

I have no objection with people listening to things for themselves. In fact quite the opposite I urge them to go and listen to as much hifi equipment as possible.

While it's best to do blind A/B comparisons wherever possible I also understand that in real life situations this is sometimes too inconvient to do. This is why people should to be aware of their own ears/brains/hearing/memory limitations when comparing products in long term sighted comparisons which are notoriously unreliable.

but as Jack Nicholson said in a Few good men - maybe "You can't handle the truth".

Right back at yer Cno.

 

[Craig M.]

I'm probably an objectivist rather than a subjectivist when it comes to believing what is the best way to go about selecting hifi equipment, if the sound produced is the sole criteria. I'm on the fence a little regarding AB/X though because I'm willing to accept that audio memory isn't fantastic, my own preferred choice would be just instantaneous switching between A and B without me knowing if B was just A being used again or if a switch had actually taken place, with the question just being can I hear any difference between A and B and if so then obviously personal preference enters the equation.

While I'm happy to accept that audio memory can let a person down I do find it interesting that it doesn't seem so unreliable during sighted comparisons, when swapping between speaker cables (for instance) I've found it fairly easy to hear and remember differences and yet, using a rudimentary form of blind A/B switching, when I'm unaware of which cable is in use I've found it impossible to detect the same difference - or indeed any difference. The other interesting finding was that after the blind comparison, returning to a sighted comparison the cables still sounded identical - the earlier sighted differences were no longer apparent.

I'm not sure the argument that 'I didn't expect to hear a difference but I did so it must be real' stands up, surely if a person didn't think there could be a difference they wouldn't have bothered making the comparison in the first place? When I first got some interconnects to compare to the ones I already had I thought there probably wouldn't be a difference and felt a little silly asking the dealer if I could try some at home, and yet why was I going to the trouble of trying them? Because a part of me obviously thought there might be something in it, even if it seemed silly. I'm also not convinced that experience is any sort of guard against any sort of subconcious bias, there are a number of videos on youtube about something called the Mcgurk effect (were what you see directly influences what you hear), one of them by a chap who has been researching the Mcgurk effect for over 20 years - he is still fooled by it every time and there is nothing he can do to change that.

So that's my thought on the debate and my personal experience backs up the fact that, for me, my subjective opinion isn't especially reliable. I can well understand why mags/websites/reviewers tend not to use level-matched blind testing though, 'it all sounds pretty much the same' isn't going to sell many mags is it? In fact, according to one very well regarded magazine editor, many audiophiles actively want there to be differences and the mag wouldn't sell if they didn't print them.

 

[CnoEvil]

I'm probably an objectivist rather than a subjectivist when it comes to believing what is the best way to go about selecting hifi equipment, if the sound produced is the sole criteria. I'm on the fence a little regarding AB/X though because I'm willing to accept that audio memory isn't fantastic, my own preferred choice would be just instantaneous switching between A and B without me knowing if B was just A being used again or if a switch had actually taken place, with the question just being can I hear any difference between A and B and if so then obviously personal preference enters the equation.

While I'm happy to accept that audio memory can let a person down I do find it interesting that it doesn't seem so unreliable during sighted comparisons, when swapping between speaker cables (for instance) I've found it fairly easy to hear and remember differences and yet, using a rudimentary form of blind A/B switching, when I'm unaware of which cable is in use I've found it impossible to detect the same difference - or indeed any difference. The other interesting finding was that after the blind comparison, returning to a sighted comparison the cables still sounded identical - the earlier sighted differences were no longer apparent.

I'm not sure the argument that 'I didn't expect to hear a difference but I did so it must be real' stands up, surely if a person didn't think there could be a difference they wouldn't have bothered making the comparison in the first place? When I first got some interconnects to compare to the ones I already had I thought there probably wouldn't be a difference and felt a little silly asking the dealer if I could try some at home, and yet why was I going to the trouble of trying them? Because a part of me obviously thought there might be something in it, even if it seemed silly. I'm also not convinced that experience is any sort of guard against any sort of subconcious bias, there are a number of videos on youtube about something called the Mcgurk effect (were what you see directly influences what you hear), one of them by a chap who has been researching the Mcgurk effect for over 20 years - he is still fooled by it every time and there is nothing he can do to change that.

So that's my thought on the debate and my personal experience backs up the fact that, for me, my subjective opinion isn't especially reliable. I can well understand why mags/websites/reviewers tend not to use level-matched blind testing though, 'it all sounds pretty much the same' isn't going to sell many mags is it? In fact, according to one very well regarded magazine editor, many audiophiles actively want there to be differences and the mag wouldn't sell if they didn't print them.

I think that is a respectful, fair and reasonable approach....well most of it anyway.

We are all inclined to be drawn to, and influeneced by people who make sense of the world, in a way that ties in with how we naturally think. It is genuinly hard to remain open minded, as it is all too easy to reject anything that doesn't fit conveniently with what we see as reality.......and I very much include myself in this.

As I age, I take a more tolerant, less "black and white" view of the world, and have found things are never quite as simple as I first thought.

 

[CnoEvil]

...but as Jack Nicholson said in a Few good men - maybe "You can't handle the truth".

And what happened to him?

Martyrdom isn't for everybody. :shifty:

 

[TrevC]

OK, to put busb's remark somewhat sharper (may I, busb?): what if the participants all suffered neurosensory deafness ànd were really honest? Not a single positive result would be reported, so there is no difference, right? Utter :hs:, of course! I'm aware this is a daft example; it's just to make the point clear.

This is a very good point that you and busb make. ABX testing by definition has one very big limitation - it can only prove a positive result but it can't prove a negative one.

Basically if you really can hear a difference then you will get a positive result which categorically proves that you heard a difference. That's all fine and dandy and provides good quality useful data. However the problem with ABX testing is that you can't prove a negative result. If you can't hear any difference it doesn't prove that a difference can't be heard. It just shows that those people on that particular occasion didn't hear a difference. If they tried it again another time they might be able to hear a difference.

I fully agree with you that ABX testing isn't perfect. But it's still the most reliable comparison method available for checking whether or not you can hear any differences between two things. Don't forget that it's not just ABX which can't prove a negative result but all other listening comparison methods can't either.

I'd say ABX as a methodology is so flawed - it's almost pointless.

So everything sounds the same? Really?

 

[CnoEvil]

What I find sad, is when all this obsession with testing stops people just going out and LISTENING.

Not so. It tells you what you need to listen to.

Are you naturally like this, or do you have to work at it?

 

[TrevC]

What I find sad, is when all this obsession with testing stops people just going out and LISTENING.

Not so. It tells you what you need to listen to.

Are you naturally like this, or do you have to work at it?

Naturally intelligent?

 

[byakuya83]

Frog, I see no practical way of conducting a test without the participants being aware. Whether they consider ABX comparisons a 'test' or not doesn't change the outcome IMO as it influences either outcome the same. It's a tried and tested method and I'm not aware of any realistic alternatives.

 

[CnoEvil]

Naturally intelligent?

I suppose that's one way of putting it.

......and I see you have a sense of humour. :shifty:

 

[CnoEvil]

A right of reply.

My nemesis over on the Darkside has made some big assumptions about what, and how I think, and has somehow made "Listening, and making up one's own mind" sound like a ludicrous thing to do.....though I can see how encouraging this in certain circumstances, can have its advantages.

I'm intrigued that I'm seen as the one with the entrenched opinion, when it is others who lay down the law as to what is science, what is Foo and won't try for themselves, all while dogmatically telling others that disagree with them, that they are foolish and gullible......now I wonder where that approach comes from.

My history that lead to my position is as follows: I had read the arguments on both sides, and was in fact very sceptical. I decided to take a measured approach, and actually went about it in a fashion that was quite close to the method mentioned (despite the implication that I wouldn't). I took a bunch of cables at a variety of prices, waited till everyone was out and started messing about.

I had mains cables from Cardas, Audioquest, TCI, Clearer Audio and Atlas. I did a lot of swapping about, and what I found was that it made a worthwhile difference to my amps, and no difference to the Linn DS (some made it worse). I found this strange because I had always been told that it would have the biggest effect on the Source. I suspect that it was something to do with the fact that the Linn Dynamik power supply is doing a great job.

To confirm my findings, I went to my most sceptical friend, who was one of the few people I knew that was into hifi. He made all the same arguments that are played out ad infinitum, on most cable threads.

We played some music on the AVR600 with the standard lead, then changed to the Cardas one. The difference was so apparent, that he looked at me and accused me of making some underhand tweak. He wouldn't believe me until he personally changed the leads back and forth himself. He went away scratching his head as it flew in the face of everything he believed in (due to a science background). He now has made up his own cables, and can hear the benefit.

I know it's a cliché, but I also brought my daughter and Mrs. Cno into the fray, and they could tell when an after-market cable was put into the amp (without knowing), provided I didn't take too long to change it over.

I have also been to cable demos by TCI, Atlas, Vertere and TQ...and the result was the same.

I have never taken anybody's word for anything and always experiment. I have heard cables make no difference, and in some cases make things worse.....so this doesn't equate with the accusation that I'm simply hearing an improvement due to expectation bias.

Am I entrenched....possibly; though it's based on doing a lot of practical research...the results of which have been far from predictable. I feel it at least gives me the right to put my view across.

My position is this; imagine you are standing looking at an Aardvark, while typing so on a forum. Then people on the forum queue up to tell you that it's not possible, as they don't exist in your country. You reply that it is, as you are looking directly at it.

You ask if they have ever either been to your country, or even seen an Aardvark. To which they reply that they haven't (on both counts), but they don't need to, as they know it's not possible. The suggestion is that you need to get your eyes checked, and not to post any further comments, as it will mislead anyone who reads it (into believing such nonsense).

Now you don't need to tell me that the above example is a little OTT, but it makes the point from my perspective.

I know none of this amounts to proof, which is why I always state it as my opinion......and tell people to try for themselves.

The "Foo Fighters" make sure that they argue in such a dogmatic way, that they have an answer for every eventuality

- We are right, because Science says so

- We don't need to listen because there is no need, as science says it's impossible

- If you hear a difference, it is imaginary due to "expectation bias."

- Anyone recommending that you even try a cable is being irresponsible.

- Any proof you think you have isn't reliable

- Everything that a cable manufacturer is nonsense

- The people who say there is a benefit are brainwashing the reader, so anything they hear is unreliable.

Cno (absolute bastard) - over and out.

Oops, didn't mean to talk about cables again.

 

[the record spot]

Great post Cno. I tend to agree. A bunch of people who get their facts off what they've read from Google or similar spout like they knew the stuff all along.

I must say, I am deeply grateful as there's no way I could remotely have accessed Google myself, or that I couldn't have worked this out back in 2006.

 

[CnoEvil]

Great post Cno. I tend to agree. A bunch of people who get their facts off what they've read from Google or similar spout like they knew the stuff all along.

I must say, I am deeply grateful as there's no way I could remotely have accessed Google myself, or that I couldn't have worked this out back in 2006.

Cheers RS, but I suspect others won't think so......putting the kevlar vest on as we speak and running to the airport.

 

[steve_1979]

My nemesis over on the Darkside

Does this mean me?

 

[MakkaPakka]

I don't consider myself in a particular camp but I question everything where someone is trying to make me part with my hard earned money.

At the very least I want some sort of logical explanation of how and why something will work so that I can make an informed choice. Such explanations usually seem to be lacking when it comes to hi-fi.