I believe that's one example of what's called a "double blind" test, as used extensively, for example, in medical trials. The lab rats (read listeners) are "blind" to the medicince (read audio) they are receiving and the medicine (read audio) is "blind" to the lab rats (listeners). It's considered a very very important factor in medical trials. In a true double blind test the people running the test wouldn't know whether source A was an Amstrad or an Arcam either, removing any bias (and possible conveyance of such) to the lab rats as well, i.e they couldn't tell the rats "and now this is the more expensive [hint better] source".matt49 said:I'd agree, though with one reservation.
Listening tests of this kind should properly involve two kinds of expertise: audio engineering expertise and expertise in psychology/psychoacoustics. However, there's no evidence of any understanding of psychology/psychoacoustics in these tests. In particular, the set-up of the tests seems very amateurish. The selection of subjects was done informally. No proper clinical tests of the subjects' hearing ability were done. The tests were carried out in a number of different venues, with very different acoustic properties. We're not told whether the results were consistent across all these venues or whether they varied from one venue to another. If this research had been submitted to a psychology journal, it would have been filed under B for bin.
Having said that, and in the absence of any more robust test results, these tests may reflect reality. But the fact is, we just don't know. My position on all this is firmly agnostic. As you say ...
Pre-selecting the listeners for their abilities to clinically discern one type of digital signal from another (or not) would have invalidated the tests IMHO. Just imagine doing the trials with a room full of NEDs raised on boomboxes, ear-bud MP3 players or whatever, versus a room full of symphony orchestra conductors, virtuoso violinists, etc. The former probably couldn't tell a good hifi from the sound of a pneumatic drill, the latter would be listening for every bum note, whether caused by the musician / singer or the equipment. A middle of the road or broad spectrum group of lab rats, on the other hand, would probably give a more balanced opinion on the overall sound.