An interesting experiment

[bigblue235]

Wouldn't it make more sense not to participate in threads that you don't approve of or are not interested in? My view is that steve_1979 is willing to ask the big questions that some people on this forum only answer is to is to be flippant. I share his desire to get to the bottom of whether or not expensive cables make the difference some claim, if 24/192 really sounds better than redbook & at what bitrate threshhold we cannot hear differences.

Where I differ is that I don't have much faith in ABX testing telling us anything useful about SQ - I ernestly wish it did. I prefer Wadworth's Bishops Tipple to Fullers London Pride. If I was to participate in ABX tasting, I'd bet I'd fail for exactly the same reasons (short-term memory inaccuracy) not because my preference is bogus but because the test methodology is flawed IMO. Proving it is flawed is just as difficult as proving otherwise. Shotgun ABX tests spread over short time periods don't prove anything but do give sceptics plenty of fuel to burn. So what if the concept of expectation bias isn't new - it's still a valid observation & probably points to many differences are so close to call as debating how many fairies can dance on the head of a pin.

I'm not uninterested, and I don't disapprove.

With the greatest of respect to Steve, I'm not sure he'll prove anything about cables or bitrates, as these things have been argued about a million times before and will be argued about a million times in future. If you've seen people giving flippant answers in these types of threads, it's probably because it's the same old, same old. Try an HDMI thread, whether Steve or anyone else starts it, it'll likely get locked.

I just wondered what Steve was trying to say with the repeated highlighting of expectation bias in that manner, and his suggestion that we should think about it. I genuinely didn't know what we had to think about as he's come to pretty much the same conclusions as are generally accepted: Your expectations can make a difference to what you think you hear. (Also referred to as: 'placebo', 'snake oil, 'audiophoolery', etc). And, rightly or wrongly, I put at least part of this down to Steve's fondness for the AVI way, as much of what he says, in this thread and elsewhere, is similar to what's preached.

And your Ale analogy suggests that the only way you'll get answers to your big questions is to do your own experimenting. Steve's ears may differ from yours

 

[fr0g]

this is probably worth a re-post...

320 v 128... can you tell? (10 times in a row)

Isn't that - to coin a phrase - boiling a fr0g?

doh, I forgot the link...

http://mp3ornot.com/

 

[fr0g]

Expectation bias isn't a new concept, it's been known about for ages. My experiment is just another demonstration that shows how easy it is to fool people into thinking that they can hear differences that aren't really there.

This is why it's important to use blind rather than sighted testing whenever you're demoing hifi equiptment. If you can see what you're listening to you may be biased without even realising it.

So why are you just bringing up something that's well known about and suggesting we all need to think about it now? Is it just the basis for banging the blind testing drum? I think those that agree, will. Those that don't, won't.

I know I was joking earlier about the AVI forum thing, but couldn't you maybe make a thread about something which isn't pushed over there, just for a change?

Wouldn't it make more sense not to participate in threads that you don't approve of or are not interested in? My view is that steve_1979 is willing to ask the big questions that some people on this forum only answer is to is to be flippant. I share his desire to get to the bottom of whether or not expensive cables make the difference some claim, if 24/192 really sounds better than redbook & at what bitrate threshhold we cannot hear differences.

Where I differ is that I don't have much faith in ABX testing telling us anything useful about SQ - I ernestly wish it did. I prefer Wadworth's Bishops Tipple to Fullers London Pride. If I was to participate in ABX tasting, I'd bet I'd fail for exactly the same reasons (short-term memory inaccuracy) not because my preference is bogus but because the test methodology is flawed IMO. Proving it is flawed is just as difficult as proving otherwise. Shotgun ABX tests spread over short time periods don't prove anything but do give sceptics plenty of fuel to burn. So what if the concept of expectation bias isn't new - it's still a valid observation & probably points to many differences are so close to call as debating how many fairies can dance on the head of a pin.

While I don't disagree that ABX is not perfect, it is pretty scientific and a sure fire method for proving a positive beyond doubt. And the fact that so many audiophiles refuse to participate is even more fuel for the sceptics imo.

When someone claims A is better than B, and cannot or as more often the case "will not" take a simple test to discount expectation bias, I take those findings with a large pinch of salt.

You say that your short term memory would fail you. Yet I wonder if you plug in a new, improved amplifier (for example) that you would instantly be able to say "better dynamics, wider soundstage, reduced sibilance." ? If so, then why not when swapping from one to another in an ABX test?

Isn't this all we do when we demo gear? So if we can't trust our short-term memory as to what is best, then why demo different kit at all?

 

[Exshopguy]

http://www.youtube.com/watch?v=G-lN8vWm3m0

Explains how our perceptions can be skewed very easily.

 

[shooter]

Isn't this all we do when we demo gear? So if we can't trust our short-term memory as to what is best, then why demo different kit at all?

Its not all down to sound quality, for me aesthetics plays a large part in what i will buy. I may compromise on a sound because it looks better in the room and suits me better. I do most of my buying without hearing it, the last time i demoed then bought would of been over 10 years ago. Why you may ask, simple really, because none that i have bought have been bad at all, just different.

Back to the OP, i'll like to try the Lossless vs Lossy test, maybe a group of us could get together, not that it would make one jot to how i listen, but may proove things, i'll bring my ear syringes

 

[fr0g]

Isn't this all we do when we demo gear? So if we can't trust our short-term memory as to what is best, then why demo different kit at all?

Its not all down to sound quality, for me aesthetics plays a large part in what i will buy. I may compromise on a sound because it looks better in the room and suits me better. I do most of my buying without hearing it, the last time i demoed then bought would of been over 10 years ago. Why you may ask, simple really, because none that i have bought have been bad at all, just different,

Back to the OP, i'll like to try the Lossless vs Lossy test, maybe a group of us could get together, not that it would make one jot to how i listen, but may proove things. i'll bring my ear syringes

Actually I'm rather particular about my sponds

But yes, the reply was more to people in general rather just yourself. Many people who demo also use the same reasons.

 

[shooter]

Actually I'm rather particular about my sponds

But yes, the reply was more to people in general rather just yourself. Many people who demo also use the same reasons.

Agreed, i'm a odd one in that respect; my sponds have now been edited

 

[John Duncan]

There is another side to this experiment. It proves that if you think something is lossless it sounds better, right? So whilst this is all psychoacoustics, unless I get Mrs JD to rip my CDs at random bitrates without telling me at what quality they've been done, my brain is going to fool me into thinking that lossy rips are inferior. So, disk space allowing, I might as well rip lossless...

 

[fr0g]

There is another side to this experiment. It proves that if you think something is lossless it sounds better, right? So whilst this is all psychoacoustics, unless I get Mrs JD to rip my CDs at random bitrates without telling me at what quality they've been done, my brain is going to fool me into thinking that lossy rips are inferior. So, disk space allowing, I might as well rip lossless...

I would always rip lossless. I don't think that's the point though. I think the point is to dispell the continued myth that somehow high bitrate lossy MP3 (as oppossed to low bitrate 128 and below) is somehow inferior.

So often we see MP3 dismissed as being for the plebs who don't care about audio quality, when in reality they are pretty much indistinguishable from the CD, and in turn (for another discussion) indistinguishable from HD bitrates (when taken from the same recording).

And while I always say people should make their own choice, it would be nice to have the truth stated on this one for once.

I have a couple of Studio master downloads (from a previous life before I had my eyes opened) and since then I have downsampled them and ABX'd. I cannot distinguish them at anything equal to or greater than 190 VBR MP3.

When I had a play with a B&W 800D /Lyngdorf /Classe system I used 190 VBR MP3s burned to a CD...Sounded stunning.

 

[John Duncan]

I would always rip lossless. I don't think that's the point though. I think the point is to dispell the continued myth that somehow high bitrate lossy MP3 (as oppossed to low bitrate 128 and below) is somehow inferior.

So often we see MP3 dismissed as being for the plebs who don't care about audio quality, when in reality they are pretty much indistinguishable from the CD, and in turn (for another discussion) indistinguishable from HD bitrates (when taken from the same recording).

And while I always say people should make their own choice, it would be nice to have the truth stated on this one for once.

I have a couple of Studio master downloads (from a previous life before I had my eyes opened) and since then I have downsampled them and ABX'd. I cannot distinguish them at anything equal to or greater than 190 VBR MP3.

When I had a play with a B&W 800D /Lyngdorf /Classe system I used 190 VBR MP3s burned to a CD...Sounded stunning.

Define 'truth'.

Well I've always said I can't tell the difference, and it's not like there's any 24/96 music out there that anybody wants to listen to. My point is though, as idc often states, it's not that expectation bias fools people into thinking stuff sounds better, it does make it sound better in 'sighted' tests (which, if you know - or think - a file has been ripped to particular bitrate, is effectively what you;re doing). I'm not denying that one can't ABX a difference, but you're never ABXing.

I also contend, unlike some people, that I can tell the difference between 128k and lossless most of the time (though it's dependent on the complexity of the music, as you might expect).

 

[fr0g]

I would always rip lossless. I don't think that's the point though. I think the point is to dispell the continued myth that somehow high bitrate lossy MP3 (as oppossed to low bitrate 128 and below) is somehow inferior.

So often we see MP3 dismissed as being for the plebs who don't care about audio quality, when in reality they are pretty much indistinguishable from the CD, and in turn (for another discussion) indistinguishable from HD bitrates (when taken from the same recording).

And while I always say people should make their own choice, it would be nice to have the truth stated on this one for once.

I have a couple of Studio master downloads (from a previous life before I had my eyes opened) and since then I have downsampled them and ABX'd. I cannot distinguish them at anything equal to or greater than 190 VBR MP3.

When I had a play with a B&W 800D /Lyngdorf /Classe system I used 190 VBR MP3s burned to a CD...Sounded stunning.

Define 'truth'.

Well I've always said I can't tell the difference, and it's not like there's any 24/96 music out there that anybody wants to listen to. My point is though, as idc often states, it's not that expectation bias fools people into thinking stuff sounds better, it does make it sound better in 'sighted' tests (which, if you know - or think - a file has been ripped to particular bitrate, is effectively what you;re doing). I'm not denying that one can't ABX a difference, but you're never ABXing.

I also contend, unlike some people, that I can tell the difference between 128k and lossless most of the time (though it's dependent on the complexity of the music, as you might expect).

Truth. In this case that there is a difference that they could perceive blindly. I think we broadly agree too. I can ABX 128K most of the time, although I still think it sounds good (I listen to a lot of internet radio at 128 and it's great quality).

You are right. If I *think* it sounds better, it does. Simple. But the reason I think it sounds better can be dispelled. (As it has been for me).

There are cases where it doesn't matter...as I said, with my own CDs I generally listen to the FLAC version. HDD space is cheap, so why not indeed? Where it does matter is where a) People are being conned into spending more than they need to, or maybe more than they can afford and b) Where people make dismissive comments about someones choice because of a false presumtion of reduced sound quality.

On a broader scale I'd like to see tests between £5000 DACs and £100, £200 DACs. Proper, scientific tests, not sighted where the listener gets to see the huge chunk of aircraft grade steel glistening in the light against a less fancy looking box. According to some quarters, the Cambridge DacMagic measures as well as anything. I would imagine it actually sounds as good as anything too, regardless of cost.

These days many people are waking up to the fact that spending that £5000 is doing nothing more than feeding an ego or a box-lust...certainly not increasing the sound quality of a system.

 

[busb]

Wouldn't it make more sense not to participate in threads that you don't approve of or are not interested in? My view is that steve_1979 is willing to ask the big questions that some people on this forum only answer is to is to be flippant. I share his desire to get to the bottom of whether or not expensive cables make the difference some claim, if 24/192 really sounds better than redbook & at what bitrate threshhold we cannot hear differences.

Where I differ is that I don't have much faith in ABX testing telling us anything useful about SQ - I ernestly wish it did. I prefer Wadworth's Bishops Tipple to Fullers London Pride. If I was to participate in ABX tasting, I'd bet I'd fail for exactly the same reasons (short-term memory inaccuracy) not because my preference is bogus but because the test methodology is flawed IMO. Proving it is flawed is just as difficult as proving otherwise. Shotgun ABX tests spread over short time periods don't prove anything but do give sceptics plenty of fuel to burn. So what if the concept of expectation bias isn't new - it's still a valid observation & probably points to many differences are so close to call as debating how many fairies can dance on the head of a pin.

I'm not uninterested, and I don't disapprove.

With the greatest of respect to Steve, I'm not sure he'll prove anything about cables or bitrates, as these things have been argued about a million times before and will be argued about a million times in future. If you've seen people giving flippant answers in these types of threads, it's probably because it's the same old, same old. Try an HDMI thread, whether Steve or anyone else starts it, it'll likely get locked.

I just wondered what Steve was trying to say with the repeated highlighting of expectation bias in that manner, and his suggestion that we should think about it. I genuinely didn't know what we had to think about as he's come to pretty much the same conclusions as are generally accepted: Your expectations can make a difference to what you think you hear. (Also referred to as: 'placebo', 'snake oil, 'audiophoolery', etc). And, rightly or wrongly, I put at least part of this down to Steve's fondness for the AVI way, as much of what he says, in this thread and elsewhere, is similar to what's preached.

And your Ale analogy suggests that the only way you'll get answers to your big questions is to do your own experimenting. Steve's ears may differ from yours

I used tp participate on one audio newsgroup some years ago & these very questions were being asked. I can remember one particular thread about sighted testing v unsighted where many people strongly objected to the whole idea of unsighted tests being unfair! I, & many others couldn't understand the objections. No one used the term Expectation Bias back then so we have inched forward a little. I understand the point about lack of resolution of some of these questions that some find lets say tedious as we appear to be no closer to an answer.

The problem with audio ABX testing is that most most of the time, it's only going to prove a positive (or not!) so is nearly always going to get skewed. Lets imagine a test involving 2 amplifiers & ask the participants if one of them sounds better or different from the other. Any of those participants who believe before the tests start that most amps sound the same can just say no to each test sequence. They may or may not privately acknowledge hearing differences but expectation bias of those with preconceived notions will skew the results. One way around this is to include in the sequence abberations such as distortion or level shifts - anyone marking each test with a negative gets excluded from the results. It's vital that any test proves both postives & negatives. One test that Steve mentions naturally falls into this bracket by asking at what bit rate threshhold does the SQ become indistinguishable from lossless where low bitrates should sound "poorer". This sort of test stands a far better chance of telling us something useful. However, we still have the problem that participants will both tire & get bored by the whole process & just get very confused.

I firmly believe we are inching towards answers by devising better tests & the fact that the term Expactation Bias gets mentioned & understood is in itself a step forward. When vendors charge & sucessfully sell equipment & cables at vastly inflated prices, its neccessary to carry on asking these same questions. Some will rightly point out that if some people want to buy really expensive cables is their choice but I feel less comfortable that they may being getting ripped off or buying into pointless insurance.

 

[shooter]

According to some quarters, the Cambridge DacMagic measures as well as anything. I would imagine it actually sounds as good as anything too, regardless of cost.

So does the MF DAC 1, state of the art Sterophile called it. I would like to hear that myself but as things go i'm not sure that well measured equals good sound, would you design an audio unit using plots and graphs alone? I very much doubt that happens, after all our ears evaluate sound.

 

[shooter]

I downloaded an album from iTunes, namely These New Puritans as the CD wasnt available at HMV for some reason. I'm not sure of the download quality, all i know its lossy but its quite clear its inferior, it lacks the intensity and clarity of the CD version that i now have, just sounds a bit flat and uninvolving in comparison. Unsure if that is a trend but it put me off downloading iTunes material as i'd like to hear the best available for home listening, why compromise you still need to buy the album to start?

Had the brother in-law around at the weekend with his wife. He's a Apple download man onto his iPhone/Cloud and they were enquiring about a system for there new flat. He was thinking B&W Zep, i was thinking a pre owned AVI's and a Apple TV, he agreed on my thoughts. All im saying really that everything has its place and if your happy then thats good enough.

 

[fr0g]

According to some quarters, the Cambridge DacMagic measures as well as anything. I would imagine it actually sounds as good as anything too, regardless of cost.

So does the MF DAC 1, state of the art Sterophile called it. I would like to hear that myself but as things go i'm not sure that well measured equals good sound, would you design an audio unit using plots and graphs alone? I very much doubt that happens, after all our ears evaluate sound.

Maybe not when it comes to speakers, but when designing something that has to take a digital audio signal and recreate as faithfully as possible, the analogue signal it represents, then yes, why not?

 

[shooter]

According to some quarters, the Cambridge DacMagic measures as well as anything. I would imagine it actually sounds as good as anything too, regardless of cost.

So does the MF DAC 1, state of the art Sterophile called it. I would like to hear that myself but as things go i'm not sure that well measured equals good sound, would you design an audio unit using plots and graphs alone? I very much doubt that happens, after all our ears evaluate sound.

Maybe not when it comes to speakers, but when designing something that has to take a digital audio signal and recreate as faithfully as possible, the analogue signal it represents, then yes, why not?

I'd say because you wont know the final outcome when it comes to sound quality. You may have the best measured components in the world today and put together measure perfectly [i'm thinking dac but it could be anything] but what if the resulting sound is awful and unlistenable?

Plots and graphs are formed by tones sent and recieved, music is more complex than a 1 khz sinewave.

 

[idc]

Have we got a causal link between measurements and sound quality that means the way something is made is guaranteed to get great reviews and lots of happy customers?

 

[professorhat]

I'm surprised no one has yet pointed out the obvious i.e. try this experiment out on three other people and it's very possible you'll get different results. What does this tell me? It tells me that each and every person must demo things themselves to see what they think and the results of other people's experiences are not generally that relevant.

I'm also with JD on this - unless I intend to listen / watch everything on my system in an unsighted manner, then the sighted test is the more significant one, since it reflects the results I'll be getting in real life when I'll know exactly what each and every component is etc.

And again, to stress, this doesn't mean I'm right and other people are wrong. What this means is, sighted tests are best for me for the reasons stated above. Unsighted tests are right for others for the reasons they've stated. There's no need to convince each other.

 

[shooter]

Have we got a causal link between measurements and sound quality that means the way something is made is guaranteed to get great reviews and lots of happy customers?

Quite possibly but on measurments alone i wouldn't of bought my last amp, the Chord, and i would of missed out on many years of amasing audio sound. It had flaws but some things it used to do was just magic. So for me though measurments are good to a degree, and i'm not an anti measurement guy, there is more to it than that and im not sure i could trust or buy into a company building products using plots and graphs alone because for the most part our ears decide.

 

[busb]

I'm surprised no one has yet pointed out the obvious i.e. try this experiment out on three other people and it's very possible you'll get different results. What does this tell me? It tells me that each and every person must demo things themselves to see what they think and the results of other people's experiences are not generally that relevant.

I'm also with JD on this - unless I intend to listen / watch everything on my system in an unsighted manner, then the sighted test is the more significant one, since it reflects the results I'll be getting in real life when I'll know exactly what each and every component is etc.

And again, to stress, this doesn't mean I'm right and other people are wrong. What this means is, sighted tests are best for me for the reasons stated above. Unsighted tests are right for others for the reasons they've stated. There's no need to convince each other.

I don't entirely undestand your point regarding sighted v unsighted. When I'm at home enjoying listening to music, I generally have my eyes open out of habit (whilst some prefer to close their eyes). I know the system 'cos it's mine & have no point to prove to myself or others - I'm just listening to music.

If I participate in the compeletely different task of trying to determine if something sounds better than something else, either by taking part in an experiment or wanting to purchase something new in a shop, I want to factor out how appealing a certain item looks from how it sounds then make a decision based on SQ, looks, reputation of the manufacturer, easy of matching etc. I want to be able to differentiate why I prefer something over something else. I may decide to purchase on appearance if something looked so damn cool but didn't sound quite as good as an alternative - I would be making a personal but imformed choice. To be choosing & just enjoying are entirely different processes.

As for statisical validity, generally the larger the sample size, the more accurate the results. Lets consider conducting 120 tests then dividing that initial sample into 3 then do the same maths on those 3 sets. If the results deviate substantially, the sample size was too small.

 

[professorhat]

I don't entirely undestand your point regarding sighted v unsighted. When I'm at home enjoying listening to music, I generally have my eyes open out of habit (whilst some prefer to close their eyes). I know the system 'cos it's mine & have no point to prove to myself or others - I'm just listening to music.

If I participate in the compeletely different task of trying to determine if something sounds better than something else, either by taking part in an experiment or wanting to purchase something new in a shop, I want to factor out how appealing a certain item looks from how it sounds then make a decision based on SQ, looks, reputation of the manufacturer, easy of matching etc. I want to be able to differentiate why I prefer something over something else. I may decide to purchase on appearance if something looked so damn cool but didn't sound quite as good as an alternative - I would be making a personal but imformed choice. To be choosing & just enjoying are entirely different processes.

Very simply, when I'm at home listening to my system, I am completely aware of what components are installed, and if I'm streaming, I'm completely aware what bitrate the songs are ripped to. So, if when I test "sighted" (i.e. aware rather than anything to do with whether my eyes are open or closed or not), I consistently think one component is better than another, this is more important information for my decision than any result "unsighted" (i.e. unaware). The reason is, in general use at home, I will be completely aware of which component I'm using / what bitrate the song is playing at. So the "sighted" result is more relevant than the "unsighted" result for that reason.

As I say, you don't have to agree with my thinking here - you might think the "unsighted" result is more important information. I get that and I can understand why you would think that way. It's just not the way I think.

As for statisical validity, generally the larger the sample size, the more accurate the results. Lets consider conducting 120 tests then dividing that initial sample into 3 then do the same maths on those 3 sets. If the results deviate substantially, the sample size was too small.

Don't get me wrong, statistics do have their use, but my point is, they're not particularly important in this scenario - and here's the reason why. Let's say you tested 10,000 people and 9,999 people consistently reported A was better than B, but 1 person consistently reported B was better than A. Statistically therefore, you'd say A was better than B. But what if you were that one person? You'd think B was better than A. What the other 9,999 people reported is completely irrelevant to you.

So whilst statistics are great for showing general trends, to actually find out what is relevant for you (and you alone), there is only one choice and that's to do the test yourself.

 

[fr0g]

Very simply, when I'm at home listening to my system, I am completely aware of what components are installed, and if I'm streaming, I'm completely aware what bitrate the songs are ripped to. So, if when I test "sighted" (i.e. aware rather than anything to do with whether my eyes are open or closed or not), I consistently think one component is better than another, this is more important information for my decision than any result "unsighted" (i.e. unaware). The reason is, in general use at home, I will be completely aware of which component I'm using / what bitrate the song is playing at. So the "sighted" result is more relevant than the "unsighted" result for that reason.

As I say, you don't have to agree with my thinking here - you might think the "unsighted" result is more important information. I get that and I can understand why you would think that way. It's just not the way I think.

The thing you are ignoring is that if you "know" that the lower bitrate sounds the same you will stop worrying...That's what happened to me.

It's like a placebo medicine. It works until you know it's placebo. Once you have registered the truth about it, it no longer has the same effect.

 

[professorhat]

Unless of course you know that the placebo effect does have a real effect. Then you know that it will have that effect so it works again. Until you realise that you know that you know you know...

:help:

 

[steve_1979]

320 v 128... can you tell? (10 times in a row)

I think that this might depend on the music. With some songs I could probably guess right 10/10 but with others I'm not so sure. I haven't actually tried it yet but I'll give it a go when I have some spare time later in the week.

 

[steve_1979]

I'm surprised no one has yet pointed out the obvious i.e. try this experiment out on three other people and it's very possible you'll get different results.

This is an exellent point. Three people is too small a sample group to give any conclusive results. My experiment is very suggestive but more people would need to be tested before any definite conclusion could be reached.

My experiment would be easy for anyone to duplicate and I'd be interested to hear what results other people get. Remember that it's important that the test subjects don't know what you're testing for untill after the experiment has been finished.