Sighted vs blind vs ABX.

[idc]

Sighted listening - you know exactly what you are listening to as you can see it. You will evaluate sound quality based on all of your senses and can/will be influenced by cost, looks, image.

Blind listening - you do not know what you are listening to as you cannot see it. You are asked to listen for any differences and comment on them. You are evaluating the sound alone as other influences have been removed.

ABX - you as asked to listen to component A, then component B, you cannot see which one you are listening to and are you are then asked to listen to X, which is either A or B and you are asked to say if X is A or B.

I am all up for all three types of listening. However, many audiophiles flatly refuse to do ABX as such listening often challenges the results of sighted and blind listening. ABX listening finds that differences in sound cannot be identified anymore, or are very difficult to identify. That flies in the face of many a 'night and day', 'vast' or whatever improvement or difference in sound.

Indeed ABX tests have yet to find any audible difference in cables, where the cable works and lengths are not excessive (i.e in domestic applications). ABX has found that amps and CDPs difficult to tell apart. Amps have been shown to be identical if equalised. Speakers are the one hifi product where ABX has found readily identifiable differences.

I therefore think that ABX is the most important form of listening. Afterall, if we are going to make claims about sound quality, we should at least be able to differentiate between products by just listening to them.

 

[chebby]

Who are "we" and "you" referring to in your post?

Hopefully not me! I'm not into all that hard work.

 

[Anonymous]

Chebby, I think you're being unnecessarily dismissive of idc's post. i don't agree with the methods he proposes, but the underlying aim, which is to debunk some of the common myths in HiFi, is estimable.

The problem I have with the testing described is it remains subjective, and it is easily possible to manipulate and invalidate the results. Let's take, for example, two groups of 10 people who are told that they will be doing cable listening tests. The first group are told that in the 10 tests 10 different cables will be used. The second group are told that 5 different cables may be used, but any cable may be used any number of times. Assume the cable is never swapped throughout. The first group will be far more likely to hear differences, not because they occur, but because they have been told to expect differences, so they listen for anything that can be conceived to be different. You would expect the results to be something like 70% being able to identify differences.

The second group, being aware that they may be listening to one cable any number of times, are more likely to find differences in around 30% of the tests. In both cases, because the tests are subjective, and because the human mind is what it is, the results show little more than how open ABX or blind testing is fallible in conception. By the way, my figures are estimates because it's a long time since I did this sort of thing at University.

However, were the tests to switch to objective, as in using laboratory measuring techniques, it would be very easy to prove whether there really were differences or not. In the case of cables (and I have to be vague about the testing because WHFSAV has strict copyright rules on this site so don't want to describe the tests in detail) it is easy to test whether cables sound different, the degree to which they sound different, and in which areas of the audio spectrum you would expect to hear differences. No magic: WHFSAV's sister publication, Gramophone, used to publish just such tests.

I'm not an objectivist, nor do I dispute that items of HiFi equipment can theoretically sound different, although often less than a purely subjective assessment would indicate. It just seems to me that, even using blind and blind ABX testing, the conclusions can never be definitive, whereas applying stringent laboratory tests could provide definitive answers.

 

[idc]

By 'you' etc I mean people in general. As for ABX testing I do not count tests involving trickery and refer only to properly run tests, where any tester influence is reduced to the absolute minimum.

I am interested that audiophile claims of hearing a difference start to fail when they only get to listen for a difference.

Why is that? Why is there resistence to conduct such listening tests? What does that tell us about reviews and the audio industry in general?

 

[Alec]

applying stringent laboratory tests could provide definitive answers.

What would such tests look like?

 

[chebby]

Why is that? Why is there resistence to conduct such listening tests?

Time, manpower, money, and the fact that hardly anyone will want to pay to read about such tests no matter how 'worthy' an exercise it might be.

Some manufacturers conduct their own controlled and rigorous tests (HK and B&O are two I have read this about) but their findings will be lengthy and technical/statistical and specific to that manufacturer's product research needs and also commercially sensitive (let alone boring to the audience of a mass-market AV/hifi magazine).

 

[Craig M.]

i think in terms of reviews chebby is correct, i'm sure i once read something by alan sircom explaining that magazines that follow that route usually see their reader numbers fall through the floor. which is kind of understandable.

harmon kardon have done lots of blind testing, some of the results are available online. i liked the one that showed the majority of listeners preferred a neutral, accurate sound rather than something 'coloured'.

 

[idc]

Lab measurements are interesting in terms of listening tests when; how do we explain that certain people state they can hear a difference when measurements show either no difference or a difference which is known to be inaudible? Such instances very much tally with failed ABX testing. So ABX helps to show up spurious claims. The best example I can think of at the moment is jitter.

The benefits for me that ABX testing have produced are

- totally debunking cable claims, so I no longer worry about cables and how they supposedly affect my system and the cost benefits of not feeling the urge to upgrade any more.

- casting reasonable doubt on claims being made to see hifi such as jitter reduction. Is it really a problem that needs solving with well made DACs etc?

- making me concentrate on what matters most in a hifi, the speakers (in my case headphones).

- realising that build quality and features are more important than supposed sound improvements, particularly with some components in the hifi chain, such as amps.

- realising that with digital sources such as CDPs, music files, computers etc the actual transmission of the digital signal (so long as it works) is not important. The file size and codec do matter and so does the analogue part of a DAC.

- volume plays a major part in some components sounding supposedly better than others. That comes from ABX tests where the volume is equalised and at that point no difference can be identified.

 

[jaxwired]

In my book, ABX testing has been debunked. ABX testing has shown repeatedly that hifi amps are indistinquishable. I don't believe this to be true. Thousands of hifi enthusiasts don't believe this to be true. Hundreds of reviewers and full time audio industry insiders don't believe this to be true. Therefore, there is some problem with ABX testing that renders it unreliable.

If a test negates a known truth, then the test is flawed.

 

[Craig M.]

The benefits for me that ABX testing have produced are

- casting reasonable doubt on claims being made to see hifi such as jitter reduction. Is it really a problem that needs solving with well made DACs etc?

- realising that build quality and features are more important than supposed sound improvements, particularly with some components in the hifi chain, such as amps.

i'm not sure i agree totally with the above. the only dac i've really noticed a genuine difference between my mac and a cdp as transport, was the cyrus dacxp, which i think is a design from before computers were becoming popular as transports. the only reason i can think of as to why there was such a difference is jitter, all the more recent dacs i have tried haven't shown this difference. i think the dacxp has been superceded by the dacxp+, which i think has improved clocking - maybe to be more sympathetic to a higher jitter source.

as most amps have a flat frequency response they should sound very similar, i agree with that, but a lot of speakers, particularly higher end ones, can be a difficult load for the amp which is going to leave a weedy amp floundering. and a struggling amp is definitely audible.

there is definitely more than just your hearing going on when evaluating kit though.

 

[Anonymous]

The benefits for me that ABX testing have produced are

- casting reasonable doubt on claims being made to see hifi such as jitter reduction. Is it really a problem that needs solving with well made DACs etc?

- realising that build quality and features are more important than supposed sound improvements, particularly with some components in the hifi chain, such as amps.

as most amps have a flat frequency response they should sound very similar, i agree with that, but a lot of speakers, particularly higher end ones, can be a difficult load for the amp which is going to leave a weedy amp floundering. and a struggling amp is definitely audible.

not sure i agree with this...why not just use 2000w pa amps for the same price as a 80w hifi one? Any hifi stuff will be "weedy" next to pa.

 

[Anonymous]

idc, you're probably posting this on the wrong forum. Given the two highly disturbing responses suggesting, with little evidence, that commercial interests ensure any technical or subjective blind testing will be seen as a negative and diminish profits accordingly.

Unfortunately there is a fair amount of money that says the last thing the HiFi industry wants is to have one or several sectors threatened by serious testing. Reviewers are paid to review, and to have a lucrative area such as cables prove to be largely not worth revewing is to remove a significant part of their raison d'etre. This is not to say there is any conspiracy to delude a gullible audience. The nice thing about human intelligence is just how open it is to suggestion, which means, as pointed out earlier, if a certain group of people are told to find differences in blind or otherwise listening tests a significant proportion probably will even where no differences exist. There is no doubt that reviewers and general listeners genuinely do hear and see differences. That this may have more to do with the human psyche thal the scientific reality to most does not come int it.

The whole phenomenon is a reversal of other beliefs. We mostly accept, on scientific evidence, that the world is round, even though to us it appears flat. In other words we believe the objective over the subjective. Yet, even though the scientific evidence says cables will pretty much have the same performance, with cables a significant number accept the subjective over the objective.

 

[professorhat]

I know this won't make any difference, but it is worth just reminding some people here that hi-fi and home cinema are hobbies. In the same way as you won't find a wine enthusiast using a spectrophotometer and a centrifuge to analyse his wine collection, hi-fi / home cinema enthusiasts tend to listen to music or watch movies as opposed to looking at oscilloscopes and creating graphs. This will be why you find people involved in that hobby doing the former when they're testing as opposed to the latter, no matter how unscientific this may be.

It just tends to be a bit more fun. You know, like a hobby.

 

[Bodfish]

I know this won't make any difference, but it is worth just reminding some people here that hi-fi and home cinema are hobbies. In the same way as you won't find a wine enthusiast using a spectrophotometer and a centrifuge to analyse his wine collection, hi-fi / home cinema enthusiasts tend to listen to music or watch movies as opposed to looking at oscilloscopes and creating graphs. This will be why you find people involved in that hobby doing the former when they're testing as opposed to the latter, no matter how unscientific this may be.

It just tends to be a bit more fun. You know, like a hobby.

Bingo.

 

[chebby]

Professorhat

 

[Gerrardasnails]

I know this won't make any difference, but it is worth just reminding some people here that hi-fi and home cinema are hobbies. In the same way as you won't find a wine enthusiast using a spectrophotometer and a centrifuge to analyse his wine collection, hi-fi / home cinema enthusiasts tend to listen to music or watch movies as opposed to looking at oscilloscopes and creating graphs. This will be why you find people involved in that hobby doing the former when they're testing as opposed to the latter, no matter how unscientific this may be.

It just tends to be a bit more fun. You know, like a hobby.

Bingo.

Have you noticed how the sceptics use science, graphs, surveys but never their ears. IDC, how many of these ABX events have you attended? Anyone that believes all amps sound the same might as well give up the hobby!

 

[hammill]

I know this won't make any difference, but it is worth just reminding some people here that hi-fi and home cinema are hobbies. In the same way as you won't find a wine enthusiast using a spectrophotometer and a centrifuge to analyse his wine collection, hi-fi / home cinema enthusiasts tend to listen to music or watch movies as opposed to looking at oscilloscopes and creating graphs. This will be why you find people involved in that hobby doing the former when they're testing as opposed to the latter, no matter how unscientific this may be.

It just tends to be a bit more fun. You know, like a hobby.

What you say is true, but not really the point. I don't think anyone is suggesting that any hobbiest would perform these tests. That is why you buy a magazine. When Which test a vaccuum cleaner they measure the pick up of a specified weight of dirt from a variety of surfaces with dozens of vacuum cleaners. Sounds very boring work, but is a very good way of providing accurate information to the reader of the magazine.

I would also say that for me Hifi or Home Cinema are not hobbys, in the same way that skis and jackets are not my hobbys either but skiing is. My interest is listening/watching music and films and HiFi/AV is only a means to an end. The idea of changing my equipment every year (let alone my cables) has no appeal whatsoever. What I want is every few years when a big technological jump occurs (Blu-Ray being the last one of interest to me) or my equipment breaks is to be able to pick up a magazine which says that for this price range, in a listening room like mine, with my requirements, this/these are best value. I don't find HiFi magazines very interesting (no disrespect, I think it is inevitable with the subject matter) so I would he happy to see ABX testing if it gave more believable results, even if it made the copy a bit dry.

 

[professorhat]

Which do you think will be the more popular wine magazine?

1. One which lists the chemical components of the wine and from this, makes a scientific analysis of what they will taste like and which one will taste better.

2. One which uses subjective tasters who use all sorts of flowery language to describe each wine.

I'm not arguing against the merits of scientific tests on this occasion, but for a hobbyist, they're not particularly interesting to read. And I personally don't include vacuum cleaner testing as a hobby interest - very few people I'd imagine enjoy vacuuming their carpet and have subjective opinions on which vacuum cleaner does this best. So the methodolgy for testing something like this, and the methodology for testing something which is subjective (like music) isn't really related.

On the side of blind listening testing, it has been stated again and again by the WHF team that they do use blind listening and ABX testing - something which always seems to be conveniently ignored or forgotten.

 

[hammill]

Which do you think will be the more popular wine magazine?

1. One which lists the chemical components of the wine and from this, makes a scientific analysis of what they will taste like and which one will taste better.

2. One which uses subjective tasters who use all sorts of flowery language to describe each wine.

I'm not arguing against the merits of scientific tests, but for a hobbyist, they're not particularly interesting to read.

On the side of blind listening testing, it has been stated again and again by the WHF team that they do use blind listening and ABX testing - something which always seems to be conveniently ignored or forgotten.

To be honest, I can't think of anything more dull than a wine magazine -well perhaps What Concrete and I drink more wine than is good for me. Wine tasters are usually supertasters and what they can taste in a wine (blue cheese, strawberries, furniture polish, petrol) most people have no hope of tasting. The best you can hope for is finding someone who likes the sort of wines you do and then buy their recommendations - reading the flowery language is just irritating. The only wine writer I can read is Malcolm Gluck.

As previously noted, I buy Which and it is not an exciting read, but when I want a new vacuum cleaner, washing machine etc it gives me useful advice and that is why I buy it.

I have made no statements about what testing What Hifi do or don't do. I don't recall reading that they do ABX testing, but if they do, I am pleased that is the case.

 

[hammill]

Serendipity strikes. http://www.bbc.co.uk/news/uk-13072745. Maybe wine magazines should try ABX testing too.

 

[professorhat]

Serendipity strikes. http://www.bbc.co.uk/news/uk-13072745. Maybe wine magazines should try ABX testing too.

I drink a reasonable amount of wine, but couldn't tell by taste anything about it. There are others who could, from one taste, tell you the region and the vintage. Yet in a blind taste test, both of our opinions would be equally valid.

 

[professorhat]

To be honest, I can't think of anything more dull than a wine magazine -well perhaps What Concrete and I drink more wine than is good for me. Wine tasters are usually supertasters and what they can taste in a wine (blue cheese, strawberries, furniture polish, petrol) most people have no hope of tasting. The best you can hope for is finding someone who likes the sort of wines you do and then buy their recommendations - reading the flowery language is just irritating. The only wine writer I can read is Malcolm Gluck.

Lorks a mercy! Okay, it may not be your particular hobby, but I'd imagine you are able to empathise with the fact that some people do enjoy reading wine magazines and those people are more likely to enjoy the subjective musings of a wine taster, rather than a list of chemical components and a scientific theory about how these components will affect the human palate?

As previously noted, I buy Which and it is not an exciting read, but when I want a new vacuum cleaner, washing machine etc it gives me useful advice and that is why I buy it.

Great stuff, but as pointed out, that is objective testing (how clean are my carpets / clothes) which is suitable for the likes of vacuum cleaners and washing machines. This methodology is not suitable for something which is subjective, like wine and indeed listening to music. You can't analyse a wine or any bit of hi-fi and, through scientific measurement, conclusively say that this tastes the best / sounds the best to everyone.

I have made no statements about what testing What Hifi do or don't do. I don't recall reading that they do ABX testing, but if they do, I am pleased that is the case.

Indeed, but the response was to the thread in general, not just to your post, so please don't feel like I'm targetting your views personally

 

[Anonymous]

I find it strange that people happily accept science and its findings without question in their daily life, but when it comes to HiFi seem able to suspend that totally. Remember only a small and insignificant part of HiFi is really being discussed, accessories.

This 'it's only a hobby, we don't care about the science' line is a strange one. Without the science there would be no hobby, so like it or not it is an integral part. If science says and proves that, for example, cable burn-in does not exist, then what is so wrong with accepting that?

As to the 'well, i don't want to read about it anyway' line I'd think that an investigative article or articles that proved an area within HiFi was based on a fallacy would make fascinating reading.

Are people really no longer interested in scientific investigation, in finding out what makes things tick? Next thing you'll (that's a general, not a specific you) be telling me that Top Gear is entirely spontaneous...

 

[hammill]

Frank Zappa is claimed to have said "Writing about music is like dancing about architecture". I think that the same probably applies to wine writing but what ever rocks your boat.

 

[hammill]

Serendipity strikes. http://www.bbc.co.uk/news/uk-13072745. Maybe wine magazines should try ABX testing too.

I drink a reasonable amount of wine, but couldn't tell by taste anything about it. There are others who could, from one taste, tell you the region and the vintage. Yet in a blind taste test, both of our opinions would be equally valid.

Indeed they would. Which is why I tend not to find other peoples opinion on wine useful. At the heart of the matter is whether you consider HiFi to be more like wine or vacuum cleaners - I tend to the latter but again that is a matter of opinion.