I want some discussion on testing, whether it be purely subjective, purely objective or a mixture of the two.
I did try to have this discussion on another forum, but my posts were “moderated away” and the topic locked. Seemed there was to be no doubt at all cast on AB AB/X testing. Shame, it had stayed sensible and unheated, just some difference in opinion and misunderstanding of what was being said (IMO).
I'll start with where I stand. Many will have a clue already, but I fully approve of, and endorse the idea of blind testing, especially AB/X testing. AB/X is the only truly scientific method of comparing 2 similar sounding components (whether it be hardware, or software (in the case of codecs)).
My feeling is that if enough people do the same test, and can find no statistically meaningful difference between 2 components, then they are to all intents and purposes, the same (in terms of audibility).
With these tests I have proven (to myself) that it is worthless changing cables (of any description) beyond ensuring they are fit for the purpose...i.e. they aren't far too thin and high in resistance and they are well built and not prone to failure. I have also proven to myself that 256 Kbps and above, done properly in lossy MP3, AAC, OGG is more than enough to be transparent to the original and that anything over 16 /44.1 is pointless. Any changes in quality here are down to the original mastering.
So, while I think it is a small percentage of people, I believe AB/X testing results can be affected by the very fact that we “know” we are trying to test something, rather than simply sitting and enjoying the music.
However, I also think that single tests, or tests where there are not many participants can be flawed, flawed for the very same reasons as I think that non-blind AB testing is flawed, i.e. our brain plays tricks on us, and our ears hear things differently in different circumstances. Tests done with wine have shown that 2 identical wines, labelled differently, one cheap, one expensive, will receive different opinions by the same person. People will claim that the expensive one “tastes better”. But they are the same...However, the “expensive labelled version” does indeed taste better, as brain scans have shown more activity in the brain's pleasure centres whilst drinking it. So not only is the brain being fooled, it is reacting physically differently to 2 identical yet differently labelled bottles.
I think it's quite fair to extrapolate this to audio and say that 2 cables, one a super expensive one, one a cheap one can physically sound different to a person listening, even if they produce an identical result.
And example I gave (or tried to) was that my mother herself did not pass any exams at school, yet she was and is a clever person, however she was terrified of exams, and froze. Worry about "the future" I feel is no issue here as in those days, it wasn't so important for women to actually succeed in school either as most were expected to get married, have kids and stay at home.
Yes, this is an anecdotal case, and I have no idea if it is applicable, but it feels like it may be, at least to me.
So, on the one hand, I embrace blind testing, especially AB/X, but on the other say that they are rarely conclusive, and only begin to be with a large pool of participants.
Not only that, I can understand why some people don't bother. I reported a pass for 320 Kbps MP3, (which I think is remarkable myself), but the first response I got was “I would suspect the testing method” . I think it's responses such as that that make people think “why bother?”.
So, what do you people reckon?
Are they worth it to you?
Do you really trust your ears in a sighted test?
Do you think that the act of testing and knowing you are testing can alter the results?
Anything else?
I did try to have this discussion on another forum, but my posts were “moderated away” and the topic locked. Seemed there was to be no doubt at all cast on AB AB/X testing. Shame, it had stayed sensible and unheated, just some difference in opinion and misunderstanding of what was being said (IMO).
I'll start with where I stand. Many will have a clue already, but I fully approve of, and endorse the idea of blind testing, especially AB/X testing. AB/X is the only truly scientific method of comparing 2 similar sounding components (whether it be hardware, or software (in the case of codecs)).
My feeling is that if enough people do the same test, and can find no statistically meaningful difference between 2 components, then they are to all intents and purposes, the same (in terms of audibility).
With these tests I have proven (to myself) that it is worthless changing cables (of any description) beyond ensuring they are fit for the purpose...i.e. they aren't far too thin and high in resistance and they are well built and not prone to failure. I have also proven to myself that 256 Kbps and above, done properly in lossy MP3, AAC, OGG is more than enough to be transparent to the original and that anything over 16 /44.1 is pointless. Any changes in quality here are down to the original mastering.
So, while I think it is a small percentage of people, I believe AB/X testing results can be affected by the very fact that we “know” we are trying to test something, rather than simply sitting and enjoying the music.
However, I also think that single tests, or tests where there are not many participants can be flawed, flawed for the very same reasons as I think that non-blind AB testing is flawed, i.e. our brain plays tricks on us, and our ears hear things differently in different circumstances. Tests done with wine have shown that 2 identical wines, labelled differently, one cheap, one expensive, will receive different opinions by the same person. People will claim that the expensive one “tastes better”. But they are the same...However, the “expensive labelled version” does indeed taste better, as brain scans have shown more activity in the brain's pleasure centres whilst drinking it. So not only is the brain being fooled, it is reacting physically differently to 2 identical yet differently labelled bottles.
I think it's quite fair to extrapolate this to audio and say that 2 cables, one a super expensive one, one a cheap one can physically sound different to a person listening, even if they produce an identical result.
And example I gave (or tried to) was that my mother herself did not pass any exams at school, yet she was and is a clever person, however she was terrified of exams, and froze. Worry about "the future" I feel is no issue here as in those days, it wasn't so important for women to actually succeed in school either as most were expected to get married, have kids and stay at home.
Yes, this is an anecdotal case, and I have no idea if it is applicable, but it feels like it may be, at least to me.
So, on the one hand, I embrace blind testing, especially AB/X, but on the other say that they are rarely conclusive, and only begin to be with a large pool of participants.
Not only that, I can understand why some people don't bother. I reported a pass for 320 Kbps MP3, (which I think is remarkable myself), but the first response I got was “I would suspect the testing method” . I think it's responses such as that that make people think “why bother?”.
So, what do you people reckon?
Are they worth it to you?
Do you really trust your ears in a sighted test?
Do you think that the act of testing and knowing you are testing can alter the results?
Anything else?
Facebook
Twitter
Instagram