Confused by all the bits

[Benedict_Arnold]

I'd agree, though with one reservation.

Listening tests of this kind should properly involve two kinds of expertise: audio engineering expertise and expertise in psychology/psychoacoustics. However, there's no evidence of any understanding of psychology/psychoacoustics in these tests. In particular, the set-up of the tests seems very amateurish. The selection of subjects was done informally. No proper clinical tests of the subjects' hearing ability were done. The tests were carried out in a number of different venues, with very different acoustic properties. We're not told whether the results were consistent across all these venues or whether they varied from one venue to another. If this research had been submitted to a psychology journal, it would have been filed under B for bin.

Having said that, and in the absence of any more robust test results, these tests may reflect reality. But the fact is, we just don't know. My position on all this is firmly agnostic. As you say ...

I believe that's one example of what's called a "double blind" test, as used extensively, for example, in medical trials. The lab rats (read listeners) are "blind" to the medicince (read audio) they are receiving and the medicine (read audio) is "blind" to the lab rats (listeners). It's considered a very very important factor in medical trials. In a true double blind test the people running the test wouldn't know whether source A was an Amstrad or an Arcam either, removing any bias (and possible conveyance of such) to the lab rats as well, i.e they couldn't tell the rats "and now this is the more expensive [hint better] source".

Pre-selecting the listeners for their abilities to clinically discern one type of digital signal from another (or not) would have invalidated the tests IMHO. Just imagine doing the trials with a room full of NEDs raised on boomboxes, ear-bud MP3 players or whatever, versus a room full of symphony orchestra conductors, virtuoso violinists, etc. The former probably couldn't tell a good hifi from the sound of a pneumatic drill, the latter would be listening for every bum note, whether caused by the musician / singer or the equipment. A middle of the road or broad spectrum group of lab rats, on the other hand, would probably give a more balanced opinion on the overall sound.

 

[DocG]

The selection of subjects was done informally. No proper clinical tests of the subjects' hearing ability were done. The tests were carried out in a number of different venues, with very different acoustic properties. We're not told whether the results were consistent across all these venues or whether they varied from one venue to another. If this research had been submitted to a psychology journal, it would have been filed under B for bin.

Pre-selecting the listeners for their abilities to clinically discern one type of digital signal from another (or not) would have invalidated the tests IMHO.

I agree. And I'd like to add: the fact the tests were performed in a number of different venues with different acoustic properties isn't a drawback, it's a strength. Comparing to medicine again: multi-center studies are more convincing than single-center trials, just because the conclusion can be drawn for a broader, more diverse population. The most robust trials include a huge number of subjects, from both sexes, from all over the world, with different ethnicities, etc. It is key, of course, to then randomise them properly to the distinct subgroups, but that's another matter.

 

[andyjm]

As a newly minted engineer, I participated in a test in the research department of a well known broadcaster. The apparatus consisted of A2D converter followed by a D2A converter. The resolution of the the digital link between A2D and D2A was selectable from (I recall) 8 to 16 bits. The test consisted of playing programme material either direct, or through the test box. Different resolutions were selected and test participants asked whether they could tell the difference between two signals played to them (blind). The signals could be two of the same, two straight analogue, two digital or one of each. Everying was level matched, and played in an acoustically treated listening room. The results were then collated.

Of mild interest, the most telling signal was the decay of a single piano note. In the last dying second before the note dissapeared, lower bit depths could be detected by a very slight 'tizz' just as the note died.

The results were that 12 bits and below was detectable, some could just detect 13 bits with very telling material (the piano note for example), 14 bits and above was not detected in a statistically meaningful manner.

This equipment had been used to validate the choice of 13 bits for the distribution of stereo FM to transmitter sites.

Anyway, the moral of the tale is that with much younger and much better ears, in a properly conducted test with level matched equipment, I was unable to tell the difference between 14 bits and a straight analogue signal. My guess is those who maintain they hear a difference between 16 and 24 bits may well be hearing a difference, but not because of the bit depth.

I have been able to detect differences in downloaded 16 and 24 bit files, but not differences between 24 bit files that I have compared with the same file downsampled to 16 bits. My guess is (as the posters above have suggested) that the difference in the downloads is in the master, and 24 bit files are more marketing than anything else.

 

[matt49]

Pre-selecting the listeners for their abilities to clinically discern one type of digital signal from another (or not) would have invalidated the tests IMHO.

But the experiment's authors did preselect the listeners, and they say so explicitly in the article. My point is that they did so informally and with no clear rationale.

 

[matt49]

And I'd like to add: the fact the tests were performed in a number of different venues with different acoustic properties isn't a drawback, it's a strength. Comparing to medicine again: multi-center studies are more convincing than single-center trials, just because the conclusion can be drawn for a broader, more diverse population. The most robust trials include a huge number of subjects, from both sexes, from all over the world, with different ethnicities, etc. It is key, of course, to then randomise them properly to the distinct subgroups, but that's another matter.

You're right about multi-centre testing -- at least as a means to ensure a proper social and ethnic range of test subjects. My point was a different one and had to do with the test environment. In this case the test environment (room acoustics, ambient noise) is part of what's being tested. One could reasonably presume that if there are any discernible differences between the digital sources, these might show up to different degrees in different acoustic environments, and therefore it would make sense to report on each of the results separately. It's really just a matter of making the variables in the experiment visible and making the results of the tests more transparent to readers.

BTW I'm not writing this as an advocate of 24-bit recordings. As I said in a previous post, I just think it's too early to say with any certainty whether 24-bit recordings offer any perceptible improvement over 16-bit, and the same applies to debates in several other areas of hi-fi. And scepticism about manufacturers' claims is always a very good idea.

 

[DocG]

And I'd like to add: the fact the tests were performed in a number of different venues with different acoustic properties isn't a drawback, it's a strength. Comparing to medicine again: multi-center studies are more convincing than single-center trials, just because the conclusion can be drawn for a broader, more diverse population. The most robust trials include a huge number of subjects, from both sexes, from all over the world, with different ethnicities, etc. It is key, of course, to then randomise them properly to the distinct subgroups, but that's another matter.

You're right about multi-centre testing -- at least as a means to ensure a proper social and ethnic range of test subjects. My point was a different one and had to do with the test environment. In this case the test environment (room acoustics, ambient noise) is part of what's being tested. One could reasonably presume that if there are any discernible differences between the digital sources, these might show up to different degrees in different acoustic environments, and therefore it would make sense to report on each of the results separately. It's really just a matter of making the variables in the experiment visible and making the results of the tests more transparent to readers.

OK, I see. Point taken!

 

[davedotco]

And I'd like to add: the fact the tests were performed in a number of different venues with different acoustic properties isn't a drawback, it's a strength. Comparing to medicine again: multi-center studies are more convincing than single-center trials, just because the conclusion can be drawn for a broader, more diverse population. The most robust trials include a huge number of subjects, from both sexes, from all over the world, with different ethnicities, etc. It is key, of course, to then randomise them properly to the distinct subgroups, but that's another matter.

You're right about multi-centre testing -- at least as a means to ensure a proper social and ethnic range of test subjects. My point was a different one and had to do with the test environment. In this case the test environment (room acoustics, ambient noise) is part of what's being tested. One could reasonably presume that if there are any discernible differences between the digital sources, these might show up to different degrees in different acoustic environments, and therefore it would make sense to report on each of the results separately. It's really just a matter of making the variables in the experiment visible and making the results of the tests more transparent to readers.

OK, I see. Point taken!

For me the crux of the matter resides in making absolutely sure that the (real) results and differences that you hear are actually caused by the changes that are being made to the item under test. Far too often conclusions are drawn from results that reflect other factors in the test that are being changed unintentially

Simple examples include the different mastering techniques used on different releases as discussed elsewhere and the cleaning effect of plugging and unplugging cables. Such factors can often confuse the issue quite thoroughly.

And these are simple examples only, what happen to electronic components in a dynamic environment such as a music playing system is far more complex, one recent example, a highly regarded mid price dac sounding less good than a very cheap Fiio, in this case because of a 'noisey' transport upsetting the better dac and not the cheap Fiio.

Took a lot of working out that one!

 

[igloo audio]

I agree, it's not always straightforward.

We've been through 30+ DAC's over the years, and the most reliable installs are with iTunes running on a MAC. PC using JRiver, Foobar etc., takes a little more effort, and in both instances running long USB cables (always advised against) can be problematic.

It's worth noting there's little to no point buying a DAC that doesn't support Asynchronous USB IMO, though there are a few exceptions like the Weiss converters, which use Firewire. Equally, you can use an interface converter supporting S/PDIF or AES to your DAC, something like the Stello U3 (which uses the XMOS chip) works great with the Naim DAC. The Eximus DP1 for a complete multitasking solution.

Many good, a few great solutions, but a lot of average ones given the marketing hysteria.

Peter

 

[kitkat]

Thank you for helping me understand that, some really good answers, I thought that getting a 24 or 32 bit player would give me much more detail but now I see thats not always true.

 

[MakkaPakka]

I've been caught out by sighted tests before. I used to swear that I could hear a difference between 320kbps MP3 and lossless FLAC files. As soon as I tried doing a proper ABX comparison using Foobar the differences disappeared.

I used the same foobar testing to compare a 32 bit WAV download to a FLAC CD rip - using a 24 bit DAC

After several failures I realised I had to listen to a good three or four minutes each time going back and forth again and really labouring over it.

I got it right three times in a row before it was just too fatiguing to keep going. My concentration/endurance would not have lasted long enough to come up with something statistically meaningful. The experience was such hard work I've had no desire to try it again.

My conclusion was that I could hear a difference but it was irrelevant - it could only be picked up by listening to the same bit again and again and again with a level of concentration that was a world away from actually listening to music.

In any 'normal' blind test with thirty seconds or so then I would have had no chance.

This is a good test for people who are convinced in differences:

http://mp3ornot.com/

 

[davedotco]

Thank you for helping me understand that, some really good answers, I thought that getting a 24 or 32 bit player would give me much more detail but now I see thats not always true.

This is worth repeating, no player or processor 'plays' 24 or 32 bits. This is simply the processing capability of the device in question.

CD standard 16bit/44.1khz is entirely capable of reconstructing the original music with audibly perfect results. The use of 'higher resolution' formats is pointless, the recording industry would improve the quality of it's output far more by simply mastering it's regular CD standard releases correctly.

This would of course require the music industry to actually care about the quality of it's product, which (mostly) it does not, unless it is able to con you into paying yet again for a 'higher quality' version of recordings you already own.

 

[matt49]

I used the same foobar testing to compare a 32 bit WAV download to a FLAC CD rip - using a 24 bit DAC

After several failures I realised I had to listen to a good three or four minutes each time going back and forth again and really labouring over it.

I got it right three times in a row before it was just too fatiguing to keep going. My concentration/endurance would not have lasted long enough to come up with something statistically meaningful. The experience was such hard work I've had no desire to try it again.

My conclusion was that I could hear a difference but it was irrelevant - it could only be picked up by listening to the same bit again and again and again with a level of concentration that was a world away from actually listening to music.

In any 'normal' blind test with thirty seconds or so then I would have had no chance.

MakkaPakka,

I don't doubt your experience for a minute, in fact it rings very true with me.

But I wonder if there isn't something else going on that we're not admitting to ourselves. A lot of our hearing experience is subliminal; that's to say, we experience sounds that we're not aware of, and they affect us on an emotional level. We may not be able to identify the experiences consciously, but they still have an effect on us, and we come away from them with feelings of pleasure or displeasure, of heightened or depressed mood. So when people say 'if you can't identify it in a blind test, then it doesn't matter', I'm a bit sceptical.

A case in point is the psychoacoustic phenomenon of 'roughness'. We experience roughness as a result of infinitesimally rapid changes in amplitude. For instance, if you drag a stiff-bristled brush across a surface such as brick you create an unpleasant sensation of roughness due to the very many rapid shifts in volume. It's been shown in psychoacoustic experiments that subliminal levels of roughness can make people feel uneasy, even though they're not consciously aware of the sensation.

If this is true (and it's a big if for me -- not least because I haven't read nearly as much of the literature on it as I'd like to), it might be relevant to the debate about bit rates (though I remain agnostic and am in no hurry to buy into 24-bit myself).

Matt

 

[davedotco]

I've been caught out by sighted tests before. I used to swear that I could hear a difference between 320kbps MP3 and lossless FLAC files. As soon as I tried doing a proper ABX comparison using Foobar the differences disappeared.

I used the same foobar testing to compare a 32 bit WAV download to a FLAC CD rip - using a 24 bit DAC

After several failures I realised I had to listen to a good three or four minutes each time going back and forth again and really labouring over it.

I got it right three times in a row before it was just too fatiguing to keep going. My concentration/endurance would not have lasted long enough to come up with something statistically meaningful. The experience was such hard work I've had no desire to try it again.

My conclusion was that I could hear a difference but it was irrelevant - it could only be picked up by listening to the same bit again and again and again with a level of concentration that was a world away from actually listening to music.

In any 'normal' blind test with thirty seconds or so then I would have had no chance.

This is a good test for people who are convinced in differences:

http://mp3ornot.com/

Can I just confirm something in your post?

You compared a hi-resolution download with a a flac file ripped from a CD, is that correct?

And you could, just, tell the difference, is that correct?

 

[MakkaPakka]

The foobar add-on mentioned is very good. It's designed specifically to be flexible and not give you any idea which file is playing. I've found any other type of testing worthless really because of the difficulties of swtiching quickly enough, level matching, etc. None of the tests I did switching between CD players for example led to me believing there was any difference. Every time I convinced myself there was, I wasn't so sure the next time.

The download was a 32-bit WAV at 48Khz . The FLAC rip was the CD. There is no difference between the versions. Obviously I could have converted the WAV down but with instant switching they sounded absolutely identical so I didn't see the need. It's not the most technically impressive specs on the WAV but it's a nicely mastered album with decent dynamic range.

To say I 'just' head a difference might be overstating it. We're talking about the most minimal, minimal difference in clarity - if I'd have been listening any harder I'd have had a nose bleed I suspect.

I do have some 32-bit 192Khz files and, when I can find the motivation (and have finished off the acoustic treatment of my room) will do some tests with them as I still need more convincing. I certaintly wouldn't pay a premium for a hi-res download if it was the same master as the CD based on my current thinking.

 

[BigH]

CD standard 16bit/44.1khz is entirely capable of reconstructing the original music with audibly perfect results. The use of 'higher resolution' formats is pointless, the recording industry would improve the quality of it's output far more by simply mastering it's regular CD standard releases correctly.

This would of course require the music industry to actually care about the quality of it's product, which (mostly) it does not, unless it is able to con you into paying yet again for a 'higher quality' version of recordings you already own.

I quite agree all these bits are a red herring or a con to get people to buy the same music over and over again. The problem is the mastering esp. the compression (DR), I know all the record industry cares about is sales. I have been advocating that they produce 2 versions of each album, 1 for the mp3/car player and one for the hifi market, I would not mind paying a bit extra for higher quality, whats the point of buying cds that you don't play because of the poor quality.

 

[davedotco]

The foobar add-on mentioned is very good. It's designed specifically to be flexible and not give you any idea which file is playing. I've found any other type of testing worthless really because of the difficulties of swtiching quickly enough, level matching, etc. None of the tests I did switching between CD players for example led to me believing there was any difference. Every time I convinced myself there was, I wasn't so sure the next time.

The download was a 32-bit WAV at 48Khz . The FLAC rip was the CD. There is no difference between the versions. Obviously I could have converted the WAV down but with instant switching they sounded absolutely identical so I didn't see the need. It's not the most technically impressive specs on the WAV but it's a nicely mastered album with decent dynamic range.

To say I 'just' head a difference might be overstating it. We're talking about the most minimal, minimal difference in clarity - if I'd have been listening any harder I'd have had a nose bleed I suspect.

I do have some 32-bit 192Khz files and, when I can find the motivation (and have finished off the acoustic treatment of my room) will do some tests with them as I still need more convincing. I certaintly wouldn't pay a premium for a hi-res download if it was the same master as the CD based on my current thinking.

Thanks for the clarification.

Your methodology may be deeply flawed, you have no idea as to whether the mastering for the WAV file is the same as he mastering from the CD, in fact they could be from very different masters or did you check that? You seem to be aware of the issue so I wondered why you chose this methodology.

Th best test is to take one of your hi-res files and use a competent processor (software or hardware) to down sample the file to 16bit/48khz and perform the tests again, that would be a viable experiment.

Do not try to down sample to 16bit/44.1khz, this is a more complex process (not whole numbers) so you end up listening to the abilities of the processor rather than any difference in the sound quality of th files.

 

[Benedict_Arnold]

On a slight tangent here.

How would one know if a downloaded FLAC, for example, was actually a pukka 192 kHz / 24 bit FLAC downsampled from the studio master or just an upsampled rip of a CD? Other than trusting the source how would a punter like me know?

A glass of "Peckham Spring" anyone?

 

[davedotco]

On a slight tangent here.

How would one know if a downloaded FLAC, for example, was actually a pukka 192 kHz / 24 bit FLAC downsampled from the studio master or just an upsampled rip of a CD? Other than trusting the source how would a punter like me know?

A glass of "Peckham Spring" anyone?

You probably would not know.

And it wouldn't matter anyway.

As someone pointed out, recording quality and mastering are what matter, but the record industry does not care as the market share for quality recordings is tiny. CDs of modern 'pop' music are no more than extended ringtones.

 

[MakkaPakka]

I used the files that I had for convenience having confirmed their respective dynamic ranges (which I did over the whole album) - it wasn't intended to be a specific thought-out test at the time. As I said, they sounded identical in normal listening. I may loads the files into audacity at some point and see if there's anything different there.

The album was mixed by the artist and doesn't have a mastering credit (though it's allegedly Steve Hoffman who apparently said he barely touched it)

I'm as sure as I can be that there's only one master because the artist oversaw the whole thing and did liner notes commenting on the dynamic range of the album.

 

[Benedict_Arnold]

I used the files that I had for convenience having confirmed their respective dynamic ranges (which I did over the whole album) - it wasn't intended to be a specific thought-out test at the time. As I said, they sounded identical in normal listening. I may loads the files into audacity at some point and see if there's anything different there.

The album was mixed by the artist and doesn't have a mastering credit (though it's allegedly Steve Hoffman who apparently said he barely touched it)

I'm as sure as I can be that there's only one master because the artist oversaw the whole thing and did liner notes commenting on the dynamic range of the album.

That's great for that album, but say I order up, I dunno, a Kate Bush (big fan here) FLAC puporting to be downsampled from the studio master. If it's not signed in blood by Kate hereself how are we to know it's not just a rip off (and rip-off) the CD?

 

[MakkaPakka]

How do you know your steak was hung for 28 days as claimed and not 26 or 27?

Like with a lot of things, you're relying on good faith of the seller and the trade descriptions act.

 

[davedotco]

I used the files that I had for convenience having confirmed their respective dynamic ranges (which I did over the whole album) - it wasn't intended to be a specific thought-out test at the time. As I said, they sounded identical in normal listening. I may loads the files into audacity at some point and see if there's anything different there.

The album was mixed by the artist and doesn't have a mastering credit (though it's allegedly Steve Hoffman who apparently said he barely touched it)

I'm as sure as I can be that there's only one master because the artist oversaw the whole thing and did liner notes commenting on the dynamic range of the album.

Thats fine as far as it goes, maybe I'm just a bit more skeptical. Mind you I have worked in the industry so I have good reason.

 

[Benedict_Arnold]

How do you know your steak was hung for 28 days as claimed and not 26 or 27?

Like with a lot of things, you're relying on good faith of the seller and the trade descriptions act.

There's a big difference between dealing with the local butcher / abbatoir where you can go around and make a scene if your T-bone isn't up to snuff vs some on-line ne'er do well / fly by night. Moreover a trading standards officer / magistrate can far more easily deal with something tangible like a steak (or a fried amplifier, for example) vs. the bandwidth of a FLAC. Might as well as the TSO or magistrate how to rebuild a space shuttle.

 

[MakkaPakka]

I am a complete skeptic, that's why my posts are littered wth caveats! I'm also very reluctant to spend money so I have to be entirely sure something is worthwhile - I'm still on the fence on the subject as a whole but am satisfied there was a miniscule difference here.

 

[MakkaPakka]

There's a big difference between dealing with the local butcher / abbatoir where you can go around and make a scene if your T-bone isn't up to snuff vs some on-line ne'er do well / fly by night. Moreover a trading standards officer / magistrate can far more easily deal with something tangible like a steak (or a fried amplifier, for example) vs. the bandwidth of a FLAC. Might as well as the TSO or magistrate how to rebuild a space shuttle.

I think Trading standard did the job with Russ Andews and that Black Ravioli stuff - something has to be proven if challenged.

As the posts in this thread suggest, it's really not worht worrying about the exact technical specs. I wouldn't pay a premium for a hi-res download unless it was my absolute favourite album.