ABX Testing

[busb]

It's an obvious request to ask those who state that they can hear audible differences or more importantly, inprovements between cables for some evidence for their claims, especially when there's no science to support them.

The propposed method is double blind ABX tests where the person(s) conducting the test are unaware of the exact sequence so can't subliminally influence the terst subjects. This methodology seems pretty obvious. What is less obvious is that any participants need to have faith that the method is "sound". If I was asked to take part in a DB ABX test of 2 camera lenses of the same basic specification (same focal length & minimum apperture), I'd have no quarms. If asked to take part in tests for cables, I would need convincing that the method works. The reason I'm sceptical is that these tests never seem to have a positive result. Many of you will sigh & say "Of course they won't!" My point is what if the test method is flawed? I say to those who rightly demand proof of differences are real that I want proof that the testing works.

So, can someone please provide links to scientifically verifiable data that DB ABX testing for audible differences works rather an merely assuming it does. Hopefully such proof will include raw data rather than just interpretation.

 

[The_Lhc]

Sorry but that's just another excuse, why would the same testing method be sound for lenses but flawed for cables?

 

[keeper of the quays]

I have a 1959 Dixieland jazz LP it's a special hifi recording and it sounds exceptionally good on my wide range equipment..but interestingly it says in sleeve notes that the recording process goes further than the human ear can hear! But the view of the manufacturer of said LP says even though it's beyond our hearing it adds a sweetness to the sound? And that's from 1959?

 

[lpv]

go to harbeth forum - it's been discussed to death with all the science you want.

 

[ID.]

So what's your alternative way of testing?

i kind of agree with drummerman, but to me that just says that if there are differences they aren't nearly as great as people make out.

Of course, the best would be to take measurements using equipment rather than just the human ear.

Now excuse me, I'm off to trademark ABX for my brand of hi fi equipment and cables.

 

[drummerman]

'AB ... ' tests, a favorite buzzword and yet most have never participated in one.

I am a sceptic as to their accuracy in certain circumstances.

Perhaps if done in familiar surroundings on a familiar system with familiar music they have relevance.

In any other circumstances expectation pressure, anxiety/stress and to many unknown factors will probably distort any result to unusable levels especially if it concerns subtle differences such as cables and perhaps even digital sources.

I did do one many a moon ago and the 'stress' of finding the place, the excitement of meeting certain people and totally unfamiliar surroundings/system/music etc will probably have made a mockery of the result.

This is why reviewers have reference systems, treated rooms etc as known constant to assess products. It's easy to take make fun of reviewers but they have one up on most of us when it comes to find differences, often 'hifiesque' language and personal preferences notwithstanding.

 

[nopiano]

There seems to be difference between longterm and short-term differences. We are always told what a short auditory memory we have, such that switches need to be almost immediate. (In the 70s, it was common to switch pairs of speakers in a retailer via a switch box, with ten or more pairs connected. Later, it was decided that speakers sounded better when they aren't vibrating 20 other boxes at the same time)

Some changes I've made over the years seem to creep up on me. I suddenly hear a detail I'd not oreviously heard, for example. When you recently upgraded something, you naturally attribute that to the new cartridge, cable or whatever. Having not changed much for a few years, I still find something new from time to time; now I associate that with my mood, or worsening hearing!

I do believe the true qualities of a system take days if not weeks of listening. But with experience, good decisions can still be made with a quick blast. The pitfalls have been well set out here, particularly in ensuring the volume stays the same, though that fails if comparing transducers with differing sensitivity or output.

 

[MajorFubar]

So, can someone please provide links to scientifically verifiable data that DB ABX testing for audible differences works rather an merely assuming it does. Hopefully such proof will include raw data rather than just interpretation.

Conversely I'd ask you to provide proof that it doesn't/cannot work. If you're going to test something, the only way to get a meaningful result is to remove any variables which might influence the outcome. It seems an obvious concept to me.

But equally I do agree with nopiano, sometimes it does take a while for some differences to be appreciated.

 

[busb]

Sorry but that's just another excuse, why would the same testing method be sound for lenses but flawed for cables?

Let's consider the lenses. We are presented with 2 photos of the same scene & same settings & asked to compare for differences between them - we examine both size by side. We consider a number of possible differences such as distortion, sharpness, noise, contrast, fringing artefacts etc. We would probably consider one atribute at a time & compare the same part of the photo in turn, alternating between the 2 such as in the case of a human portrait: their eyes for sharpness. The time to look at one photo & the other can be measured in fractions of a second. We could start the comparison by standing back far enough to ascertain if their were any obvious differences in colour redition as an example then homing in.

With audio testing, we have to compare the same piece of music sequentially in turn. The photo is a static object where we could cut out a small section of each photo & compare them side by side more or less simultaneously but for eye movement. I don't think anyone is suggesting doing the equivalent with audio such as listening to the left channel through one cable & the right through the other at the same time! We have to split the comparison into 2 events separated by the least time as is possible because we are relying on sequential memory. We cannot "freeze" a sound to be able to compare with another "frozen" sound. The photos are objects but the sounds are events in time. I'm suggesting that comparing "events" is more difficult than objects in the context of the comparison between sound & pictures. I not suggesting it's impossible but needs to be carefully thought through. I'm obviously aware that some will think of video but you can pause the picture but not the sound!

However, I can think of potential ways of testing audio ABX as a method that has to be able to verify positive or negative results equally effectively. We could do this by adding some measurable distortions such channel imbalance, increased noise, harmonic products, intermodulation etc. The degree of these quantifiable distortions could be increased to the point where they are noticeable. If for instance, test subjects can't hear a 3dB drop in one channel (as an example) we have to ask ourselves what chance have they got determining supposedly immeasurable aspects of SQ? If that proved to be the case, it would mean returning to the drawing board as far as proof is concerned, IMO.

 

[busb]

There seems to be difference between longterm and short-term differences. We are always told what a short auditory memory we have, such that switches need to be almost immediate. (In the 70s, it was common to switch pairs of speakers in a retailer via a switch box, with ten or more pairs connected. Later, it was decided that speakers sounded better when they aren't vibrating 20 other boxes at the same time)

Some changes I've made over the years seem to creep up on me. I suddenly hear a detail I'd not oreviously heard, for example. When you recently upgraded something, you naturally attribute that to the new cartridge, cable or whatever. Having not changed much for a few years, I still find something new from time to time; now I associate that with my mood, or worsening hearing!

I do believe the true qualities of a system take days if not weeks of listening. But with experience, good decisions can still be made with a quick blast. The pitfalls have been well set out here, particularly in ensuring the volume stays the same, though that fails if comparing transducers with differing sensitivity or output.

Very valid observations - I bought a new amplifier many years ago & was initially disappointed in not hearing a noticeable improvement. It was later when playing certain pieces of music that I heard more detail or less spikey treble maybe a month or 2 after purchase. So both long & short-term memory play a part.

 

[busb]

So, can someone please provide links to scientifically verifiable data that DB ABX testing for audible differences works rather an merely assuming it does. Hopefully such proof will include raw data rather than just interpretation.

Conversely I'd ask you to provide proof that it doesn't/cannot work. If you're going to test something, the only way to get a meaningful result is to remove any variables which might influence the outcome. It seems an obvious concept to me.

But equally I do agree with nopiano, sometimes it does take a while for some differences to be appreciated.

The last paragraph of my reply to The_Lhc may go some way to providing proof.

 

[pyrrhon]

You can't quantify everything. Science is great only if we know where to apply it and where to stop. But some small minds take huge pride in the false paradigm pretending that it's truth fighting voodooism. Reality takes a multifaceted mind. Music is beyond science.

 

[busb]

You can't quantify everything. Science is great only if we know where to apply it and where to stop. But some small minds take huge pride in the false paradigm pretending that it's truth fighting voodooism. Reality takes a multifaceted mind. Music is beyond science.

Music may well be beyond science but its reproduction shouldn't be.

One particular regular contributor springs to mind - it's their duty to evangelically correct those that do keep hearing things! On a more serious note, I am trying to further the topic a bit but with little success. I'm hoping someone can cite some creditable tests but I ain't going to hold my breathe.

 

[abacus]

You can't quantify everything. Science is great only if we know where to apply it and where to stop. But some small minds take huge pride in the false paradigm pretending that it's truth fighting voodooism. Reality takes a multifaceted mind. Music is beyond science.

Music may well be beyond science but its reproduction shouldn't be.

One particular regular contributor springs to mind - it's their duty to evangelically correct those that do keep hearing things! On a more serious note, I am trying to further the topic a bit but with little success. I'm hoping someone can cite some creditable tests but I ain't going to hold my breathe.

Type “hi fi abx tests” into google and you will find all the info you need.

Hope this helps

Bill

 

[davedotco]

I think it is important to understand exactly what an ABX test is and what it does, it is clear that some posters have no idea.

First of all it is designed to see if the listener can reliably pick one component (or cable) from another, there is no attempt to determine which is better.

Secondly it is specifically designed in it's methodology to remove stress and to allow multiple comparisons that are under the control of the listener.

It works like this. The listener is played a music selection on component/cable A, then on component/cable B and finally on component/cable X.

Component/cable X is either A or B. Simple as that, no fake switching or other traps, just A or B.

In order to make this a easy and stress free for the listener an 'ABX Box' is used to provide instant switching. The listener can switch back and forth as often as he likes, for as long as he likes, in some cases even operating the switch himself. At all times the choice is simple, is component/cable X actually A or B.

Simple as that.

 

[busb]

You can't quantify everything. Science is great only if we know where to apply it and where to stop. But some small minds take huge pride in the false paradigm pretending that it's truth fighting voodooism. Reality takes a multifaceted mind. Music is beyond science.

Music may well be beyond science but its reproduction shouldn't be.

One particular regular contributor springs to mind - it's their duty to evangelically correct those that do keep hearing things! On a more serious note, I am trying to further the topic a bit but with little success. I'm hoping someone can cite some creditable tests but I ain't going to hold my breathe.

Type “hi fi abx tests” into google and you will find all the info you need.

Hope this helps

Bill

Thanks Bill but most of it is exemplified by the Atkinson v Krueger debate that slings a lot of mud but little understanding. I'm after some hard facts regarding the efectiveness of DB ABX testing rather than perpetuating that method without question as many seem to do. I've got more chance of finding out whether or not we should leave the EU than reaching any clarity on this debate! I shall do some more searching.

 

[MajorFubar]

I'm after some hard facts regarding the efectiveness of DB ABX testing rather than perpetuating that method without question as many seem to do. I've got more chance of finding out whether or not we should leave the EU than reaching any clarity on this debate! I shall do some more searching.

Perhaps you're asking for the impossible. You may as well go in search of trying to find a study which proves that convicting or aquitting an accused man via a fair trial gets it right more often than a simple gut instinct on whether he looks shifty or not. There won't be such a study for that because the suggestion is beyond ludicrous.

You could come along and tell me you've made the best sounding CD player known to man, but for that statement to be proved with any credibility requires your CD player to be tested against its peers with all other technical/objective and subjective variables removed that might influence my perception of its performance, positively or negatively. That's what ABX testing ultimately does and there's nothing to question. The polar opposite is I listen to your competitors' products through the worlds best amps and speakers in an ideal listening environment, but I only listen to your CD player by overhearing it while it's plugged into the Matsui microsystem in the neighbour's garden shed, and from that test I conclude that frankly your CD player is rubbish.

 

[davedotco]

If you read my description of what and how ABX testing is actually designed to work, you can see that it is totally effective. It is simple scientific method reduced to it's most basic, it is how the world works!

It is worth pointing out that if no difference can be heard, the the concept of better or worse has no meaning.

However, tests of this type are time consuming and expensive to conduct, the hi-fi industry has no interest in setting them up, for obvious reasons. Companies like Harman are known to use them but do not, often, publish the results.

One of the most enlightening experiences is, in my view, taking part in a third party blind test. There is no need for it to be rigorous enough to satisfy scientific scrutiny, just a simple, level matched test where the specific components are not known to the listener.

I have been involved in a number of such tests, both as operator and listener and, to be honest, the results are startling, pretty much every time!

 

[lpv]

If you read my description of what and how ABX testing is actually designed to work, you can see that it is totally effective. It is simple scientific method reduced to it's most basic, it is how the world works!

It is worth pointing out that if no difference can be heard, the the concept of better or worse has no meaning.

However, tests of this type are time consuming and expensive to conduct, the hi-fi industry has no interest in setting them up, for obvious reasons. Companies like Harman are known to use them but do not, often, publish the results.

One of the most enlightening experiences is, in my view, taking part in a third party blind test. There is no need for it to be rigorous enough to satisfy scientific scrutiny, just a simple, level matched test where the specific components are not known to the listener.

I have been involved in a number of such tests, both as operator and listener and, to be honest, the results are startling, pretty much every time!

any chance you can describe the results of one or two most interesting/ shocking/ startling ones?

 

[pauln]

@busb

Have you tried the ABX comparator in foobar? Obviously you can't compare equipment but you can compare mp3 vs flac vs wav or remastered tracks vs originals and get a feel for how the testing works. You would also need some basic understanding of statistics to interpret the results - some might think getting it right half the time was 'not bad' rather than the result that would be expected from randomly guessing.

Some claim that abx testing is flawed because it puts people under 'stress' - the only stress I can see is that some people may be in fear of being found out. If differences are described as being like 'night and day' or 'a veil being lifted' then one would expect that those differences could easily be discerned in virtually any situation.

For the record I could distinguish between a high DR and a lower DR remaster of the same track but not between a flac and a 320 mp3 version of a track. (Ripped and compressed from CD on my laptop and listened to with Sennheiser HD650 headphones via an ODAC)

 

[davedotco]

If you read my description of what and how ABX testing is actually designed to work, you can see that it is totally effective. It is simple scientific method reduced to it's most basic, it is how the world works!

It is worth pointing out that if no difference can be heard, the the concept of better or worse has no meaning.

However, tests of this type are time consuming and expensive to conduct, the hi-fi industry has no interest in setting them up, for obvious reasons. Companies like Harman are known to use them but do not, often, publish the results.

One of the most enlightening experiences is, in my view, taking part in a third party blind test. There is no need for it to be rigorous enough to satisfy scientific scrutiny, just a simple, level matched test where the specific components are not known to the listener.

I have been involved in a number of such tests, both as operator and listener and, to be honest, the results are startling, pretty much every time!

any chance you can describe the results of one or two most interesting/ shocking/ startling ones?

The first is quite famous, it took place in 1978, I was not involved but several journalists and industry figues were. It involves a comparitive test of 3 very different amplifiers 'known' to have very different sounds and to have generated many articles and discussion about the merits of each. It is easily researched.

The first was a classic valve, push/pull design delivering around 8-12 watts depending on the distortion level considered acceptable, the second a regular class AB solid state design delivering up to 45 watts and the third was an innovative (for it's day) Current Dumping design with 100 watts.

A system was set up using revealing electrostatic loudspeakers and levels carefully matched, at normal (non overload) levels, no one could tell which amplifier was playing, despite pretty much everyone present being aware of the 'known' differences.

Secondly, around the same time, I was involved in a number of blind tests carried out for the early editions of Hi-fi Choice. These were not scientifically rigorous tests, we listened and discussed as a group and in most cases knew what items were being tested but not the order in which they were being played.

Once again the shocking results were just how difficult it was to hear differences in amplifiers, I mean really difficult, despite knowing that, in 'normal use', some of the amplifiers to be as different as 'chalk and cheese', to use the terminology of the day.

What was even more startling was when the testing was of loudspeakers. Levels were matched using an SPL meter with the result that several expensive, highly regarded models, sounded no better than some much cheaper models. This caused some consternation, everybody knew that speakers made a big difference but the tests told us a very different story. Sure there were a handful of designs that stood out, usually by being quite poor, but the difference between compedent designs of a similar type was a lot less than you might think.

Again these early Hi-fi Choice 'Group Tests' are researchable on line.

 

[TrevC]

You can't quantify everything. Science is great only if we know where to apply it and where to stop. But some small minds take huge pride in the false paradigm pretending that it's truth fighting voodooism. Reality takes a multifaceted mind. Music is beyond science.

Ah, but the reproduction of music is science! If you take foo cable x, say a Moondust Excaliber Orange, and measure its electrical characteristics you could easily duplicate it with bog standard cheap as chips chinese wire fairly easily. But the good news about interconnects and speaker wires is you don't need to bother, because providing the speaker wire is of sufficient gauge and the interconnect wire {used at line levels*) makes contact it will sound the same because ordinary wire doesn't vary enough to change the sound.

* Changing capacitance of wires on an MM cartridge can change the sound significantly.

 

[busb]

You can't quantify everything. Science is great only if we know where to apply it and where to stop. But some small minds take huge pride in the false paradigm pretending that it's truth fighting voodooism. Reality takes a multifaceted mind. Music is beyond science.

Ah, but the reproduction of music is science! If you take foo cable x, say a Moondust Excaliber Orange, and measure its electrical characteristics you could easily duplicate it with bog standard cheap as chips chinese wire fairly easily. But the good news about interconnects and speaker wires is you don't need to bother, because providing the speaker wire is of sufficient gauge and the interconnect wire {used at line levels*) makes contact it will sound the same because ordinary wire doesn't vary enough to change the sound.

* Changing capacitance of wires on an MM cartridge can change the sound significantly.

Well TrevC we do agree on something! In my reply to pyrrhon (post 13), I wrote:

"Music may well be beyond science but its reproduction shouldn't be."

I also agree that if it was proved beyond reasonable doubt (which I'm disputing, of course), it would indeed be good news for consumers who needn't pay very much for cables. As for interconnects, the only time I've heard a difference (or as you would prfer I say): thought I heard any difference - was beween a give-away & mid-priced Chord set of RCA leads. I also am willing to conjecture that many claims made are either completely bogus or the product of expectation bias.

I have taken part in ABX tests where I stated that I could not hear any real differences between NAS drives (not that I'd expect to!) i also firmly believe that cable manufacturers should desist from their rediculous & unsubstantiated pseudo-scientific mumbo-jumbo & simply say they don't know why their cables sound better & just loan the stuff or have a money-back guaraantee. If someone was to claim that painting the room above their livingroom a particular shade of pink made jazz sound better, I would dismiss that claim outright. Nothing could make jazz sound better!

 

[davedotco]

Perhaps you could give some idea of the methodology of the ABX test you took part in?

Did you listen alternately? With instant switching? How did you actually compare?

As someone who spent the morning pottering around the house, (and pausing from time to listen) to the fabulous music produced on the 1967 tour of europe by Miles Davis, Wayne Shorter, Herbie Hancock, Dan Carter and Tony Williams, I can assure you that your lack of interest in such music is entirely your loss.

 

[busb]

If you read my description of what and how ABX testing is actually designed to work, you can see that it is totally effective. It is simple scientific method reduced to it's most basic, it is how the world works!

It is worth pointing out that if no difference can be heard, the the concept of better or worse has no meaning.

However, tests of this type are time consuming and expensive to conduct, the hi-fi industry has no interest in setting them up, for obvious reasons. Companies like Harman are known to use them but do not, often, publish the results.

One of the most enlightening experiences is, in my view, taking part in a third party blind test. There is no need for it to be rigorous enough to satisfy scientific scrutiny, just a simple, level matched test where the specific components are not known to the listener.

I have been involved in a number of such tests, both as operator and listener and, to be honest, the results are startling, pretty much every time!

Dave, I have taken part in fairly informal ABX tests myself. Due to the nature of what was being compared, I had expectation bias from the start. However, I got a free lunch out of WHF & a fascinating day in Teddington. If I didn't believe that ABX testing wasn't pointless, I would suggest that the recording stage of subject's conclusions be conducted on paper otherwise, as soon as others state they hear a difference, no one wants to feel they were cloth-eared by saying they couldn't, due to peer preasure.

My beef with ABX tests is that just because it works very well for stuff like comparing camera lenses, it must work for audio. I dispute that is the case & want prove that it's effective. Am I being unreasonable?

Lets take the lens example where subjects are asked if they can see any differences between the photos taken on 2 different models where everything else is equal. Let's conjecture that the results were very inconclusive (statistically insignificant) & 2 possibilities existed. The 1st being that any differences were undectable but one manufacturer argued that the test method was flawed so preposed a test for the test. That additional test invloved degrading one of 2 otherwise identical photos then repeating the test to see if the subjects could spot the differences. If they couldn't, the test method itself was dubious (this ain't no scientific paper so we have to ignore the degree of degredation before a threshhold is reached). Conversely, if the distribution of results was wide but random, indentical photos could be slipped in to see if the distribution converged. These secondary sequences weed out erroneous answers & prove the method or not.

Properly conducted ABX testing can become extremely tedious for sure. If we are to use science & good engineering practice, lets not make assumptions that it must work for audio but test that assertion. ABX testing needs to be able to prove negative & positive results otherwise its like asking if God exists & drawing up a test where he/she is invited to reveal themself. If God shows up it proves the positive but if God doesn't show, does it prove non-existance? What if God always declines party invitations?

Please, somone point me to a paper where deliberately introduced distortions have been used & heard by test subjects & I'll shut the hell up about ABX or point out the flaws in my arguments.