Streaming over Ethernet testing

[Crossie]

The problem with all "big question" tests is that the results are based on the subjective opinions of only three listeners. To get any sort of reliable result you need a much larger sample of listeners, but of course this gets expensive and time consuming. For my two pennies worth the differences are obviously small and subtle and most of us would be better off concentrating on the things that really make a big difference - electronics, speakers and quality recordings.

 

[BenLaw]

Why is this peculiar though? We're talking about people's preferences here in a blind listening situation - this isn't something that can be defined by logic, even if some people would like it to be.

Different preferences - clearly not peculiar, if the sound really is different. You make a correct point in that regard. Others are questioning tho if there is a difference at all (or whether this is the placebo effect) and if there is a difference what is the cause of that difference.

Crossie makes a very good point, but people clearly do get excited by cables and the like and they get frustrated when no-one can or does explain the reasons behind any differences.

I also agree with Crossie that small-sample non-scientific tests aren't really going to tel us anything. Even on IDC's headfi thread the interesting articles deal with 2 or 3 people. Only way to resolve it is for dedicated WHF forumites to set up a fund to rent space at the next WHF Show and conduct proper tests with a large sample of visitors.....

Maybe Max can chip in with his E6000 gambling winnings or the E7000 he took from the credit card company

 

[hammill]

Why is it peculiar? I thought that was obvious, but clearly not. We have three means of data transmission. All of these should be able to transmit all of the data with 100% accuracy and in a timely manner - the amount of data in question should be trivial for all of them. If this is the case, then the listeners should hear not differance at all. Ah you say, but a wireless network can perform very badly if there are other wireless networks using the same address and this might cause a problem. Well I would have expected that to be ruled out by the testers configuring the system carefully but I accept it is possible in which case wireless might just sound worse. You then might point out that some people claim that homeplugs can cause RFI which might pe prejudicial to the sound in some undefined way so again there is a faint possibility it might sound worse. If the tests had shown the wired connection as superior I would have been surprised that anyone could hear the difference but it is just about plausible. But the results show that only one in three listeners preferred what should have been the technically superior solution. This suggests to me that either the tests are flawed in some way or that there is a problem with the equipment. To accept results like this without comment, or further investigation makes me wonder why I bother to pay for a magazine - results like this require explanation otherwise every other thing the magazine has to say loses any credibility to me - actually in my case it already has.

 

[Anonymous]

What was being tested by my colleagues and the readers was [..] whether the readers involved thought there were difference in the sound.

I thought the participants were not to be harmedtested?

With a result this peculiar (only 1 listener preferred the directly wired connection for example) I would expect WHF to be all over the equipment trying to find out what was happening. This general attitude is why I am cancelling my subscription.

Huh? Why do you think this result is peculiar?

Let's see... We have three minimally dissimilar test setups, and three listeners. Assuming the differences are too small to be noticed by humans, you would expect a completely random distribution. Let's call this the null hypothesis, the one that we actually want to disprove. The null hypothesis states that there is a 22% chance that all participants choose a different setup as their favourite (2/9), an 11% chance that all participants choose the same setup (1/9) and hence there is a whopping 67% chance of a result like the one you refer to (a 1-2-0 score).

edit: in summary, all of this proves that nothing was proven. The results are exactly as predicted by common sense.

 

[hammill]

What was being tested by my colleagues and the readers was [..] whether the readers involved thought there were difference in the sound.

I thought the participants were not to be harmedtested?

With a result this peculiar (only 1 listener preferred the directly wired connection for example) I would expect WHF to be all over the equipment trying to find out what was happening. This general attitude is why I am cancelling my subscription.

Huh? Why do you think this result is peculiar?

Let's see... We have three minimally dissimilar test setups, and three listeners. Assuming the differences are too small to be noticed by humans, you would expect a completely random distribution. Let's call this the null hypothesis, the one that we actually want to disprove. The null hypothesis states that there is a 22% chance that all participants choose a different setup as their favourite (2/9), an 11% chance that all participants choose the same setup (1/9) and hence there is a whopping 67% chance of a result like the one you refer to (a 1-2-0 score).

I'll ask again: why do you think the outcome is in any way peculiar?

If you are suggesting that the large differences between the three sources found by the listeners are completely false and that the result is therefore just random, then I guess it is not peculiar. I was coming from the angle that the listeners were hearing real differences and that these needed to be explained, but of course they could all be imagining it. I have to say that I doubt the listening panel will like your theory though and it would make the big question and probably most listening tests somewhat pointless.

 

[Anonymous]

If you are suggesting that the large differences between the three sources found by the listeners are completely false and that the result is therefore just random, then I guess it is not peculiar. I was coming from the angle that the listeners were hearing real differences and that these needed to be explained, but of course they could all be imagining it. I have to say that I doubt the listening panel will like your theory though and it would make the big question and probably most listening tests somewhat pointless.

No, you misunderstand me (although I'll blame my writing). I'm not suggesting anything about the differences found by the listeners, because I wasn't there and I wasn't using their ears.

All I'm saying is that the tests do not prove anything. To be entirely accurate: the test has failed to prove that the choice of transmission has any effect at all on the sounds perceived by the listeners, because the results are indistinguishable from a pure random distribution. My beef is with people that use this test to say "See! There are differences!" -- not with the testers.

I've said it before in this thread (or some other), these tests are purely for entertainment. The testers have spent a lovely afternoon gawking at marvellous new equipment, and the readers get to share in that by virtue of the magazine's editors. That's it. But since you asked, yes, from a scientific viewpoint, the tests are pointless.

 

[professorhat]

The problem with all "big question" tests is that the results are based on the subjective opinions of only three listeners. To get any sort of reliable result you need a much larger sample of listeners, but of course this gets expensive and time consuming.

My understanding of the Big Question is it's not really designed to present any sort of proof to anyone - what it's there to do is get people talking about things and question what they might think they "know" e.g. we all know a wired connection is best so use that - oh look, when we actually blind tested three people who previously "knew" this, two of them found they actually preferred a different type of connection - maybe this is something you'd like to investigate for yourself and see what you think?

BQ certainly does get people talking (which is good) and whether people actually do go out and try these things for themselves or whether they brand it as non-scientific rubbish is of course completely up to them. As I've said before, this is a hobby for most of us, not a scientific experiment - take from these things what you will and leave the rest. Most importantly, just enjoy your system and don't worry too much about what others are doing if you don't agree with them!

 

[professorhat]

Ah you say, but a wireless network can perform very badly if there are other wireless networks using the same address and this might cause a problem. Well I would have expected that to be ruled out by the testers configuring the system carefully but I accept it is possible in which case wireless might just sound worse. You then might point out that some people claim that homeplugs can cause RFI which might pe prejudicial to the sound in some undefined way so again there is a faint possibility it might sound worse.

This is my point though - worse in whose opinion? Let's for a minute assume there were errors in the wireless transmission and there was interference on the homeplugs and thus differences could be detected between all three (leaving aside all arguments as to why this can't be the case etc. etc. for just a moment). Technically you'd think they would sound worse to everyone - but why is this? We can definitely say they would all sound different, but why would everyone think the wireless and homeplugs solutions sounded worse? Much like the Pepsi vs Coca Cola test, logic doesn't follow when you're talking about subjective opinions. Logic suggests everyone should find the most expensive wine the best, but that rarely holds water in real life under blind tasting.

All I'm saying is, were differences to be present in all three solutions (and I'm by no means saying there were - as has been pointed out, I wasn't there so can't comment on this), but were that to be the case, it isn't by any means peculiar that people would have differing opinions on what sounded better and what sounded worst. If it were true that we had the same opinion, we'd all be buying exactly the same kit in whichever price bracket we happened to be spending in as we'd all agree it sounded best.

Anyway, I'm sure similar things will come up from next month's Big Question and I'll then be in a much better position to comment!

 

[Andrew Everard]

I often wonder that is the test was repeated, but with making no changes to the transmission method at all, what the results would be!

Please see earlier in thread.

 

[Anonymous]

The 3 found significant differences though.

More dynamic, too much separation, cymbals sharp, emotional, violins screechy, wider soundstage, less bright, prominent midrange, muddier.

 

[John Duncan]

The 3 found significant differences though.

More dynamic, too much separation, cymbals sharp, emotional, violins screechy, wider soundstage, less bright, prominent midrange, muddier.

Are you surprised that there are differences between a wired and HomePlug connection, for example? I'm not, having heard what it can do to the mains. That said, since it's a fairly brief, subjective test (and because some people like Arcam and some people like Cyrus), I'd be astonished if proletariat didn't prefer different things.

EDIT - I'm leaving that autocorrect in because it's funnier...

 

[Anonymous]

For information:

Netgear AV+ 500 have a speed of up to 500Mbs, a wav file is 1.411Mbs.

 

[Anonymous]

The main issue is can you HEAR the difference.?

Do not listen to specs. ,listen to the music

Linn DSI ,Dali Mentor 5 speakers, bi wired with LK 400 cable on separate electrical loop

 

[professorhat]

EDIT - I'm leaving that autocorrect in because it's funnier...

 

[aliEnRIK]

A question for the WHF staff who ran the Big Question please:

Was any form of mains filtering used or where the homeplugs allowed to swamp the mains with RFI?

 

[idc]

So what's the technical explanation for someone saying streaming via Ethernet via homeplugs (and obviously some Ethernet cables) sounds better?

Sound waves are created from the speakers connected to an amplifier, which is connected into three systems - one where music is streamed via ethernet cables, another where music is streamed via homeplugs and a final one where music is streamed wirelessly. Once any of these sound waves hit the eardrum, it vibrates and causes a chain reaction resulting in these vibrations being passed along the cochlea. Here, there are thousands of cilia (which are hair like nerve endings) and as the vibrations occur, these move. The brain interprets these movements as sound. Things get a bit fuzzy from here on in I'm afraid, but the best way I can put it is that when some brains "hear" the sound produced by the music streamed via homeplugs or music streamed wirelessly, this results in greater satisfaction than that produced by the music streamed via ethernet. The opposite is also true for some other brains.

Hope this helps.

Spot on professor. When the brain knows what it is listening to it will then use other factors to differentiate between the different types of connection. When the brain does not know what it is listening to, it can't.

That is why we find a consistent difference between sighted, blind and ABX test results. Sighted and differences are reported (including night and day), blind differences are reported but they are much smaller, ABX and no difference.

That shows how influencial sight and knowledge is on how we hear. It also shows how much hifi hype and marketing is important to the industry as without it, all hifi sounds pretty much the same.

 

[professorhat]

Spot on professor. When the brain knows what it is listening to it will then use other factors to differentiate between the different types of connection. When the brain does not know what it is listening to, it can't.

That is why we find a consistent difference between sighted, blind and ABX test results. Sighted and differences are reported (including night and day), blind differences are reported but they are much smaller, ABX and no difference.

That shows how influencial sight and knowledge is on how we hear. It also shows how much hifi hype and marketing is important to the industry as without it, all hifi sounds pretty much the same.

Erm, have you read the article in question? The tests were done blind - the guys had no idea what the difference was between them, yet they noticed "profound" and "massive" differences between the systems.

Besides, my post was just me being facetious.

 

[idc]

The results would have been different if the same test was conducted sighted and ABX as well as blind. I am sure of that because consistently that has been the case with all testing, of all parts of the hifi chain.

You are still spot on by identifying the differences are in the listeners head.

No one has been able to show how a digital signal can vary and that variance affect sound quality. It is possible to show variances in a digital signal with the likes of jitter, but testing of the audibility of jitter has been inconclusive.

What is known for certain is that digital transmission either works or it does not and when it does not there are breaks in the signal, some of which may not have any apperent effect and others will appear as clicks, pops whatever.

So all three transmision types worked in the test. So the difference is in the listeners, further proved by their inconsistent reactions to the test.

 

[John Duncan]

That shows how influencial sight and knowledge is on how we hear. It also shows how much hifi hype and marketing is important to the industry as without it, all hifi sounds pretty much the same.

Remind me how many pairs of headphones you have again, idc?

 

[professorhat]

You are still spot on by identifying the differences are in the listeners head.

Given that this was a blind listening test, I'm struggling to think where else the differences could have been detected...

No one has been able to show how a digital signal can vary and that variance affect sound quality. It is possible to show variances in a digital signal with the likes of jitter, but testing of the audibility of jitter has been inconclusive.

No one has been able to detect the Higgs Boson particle, but many are convinced it exists. Convinced enough to spend several billions of pounds to build a device to try and detect it.

But I digress.

 

[idc]

That shows how influencial sight and knowledge is on how we hear. It also shows how much hifi hype and marketing is important to the industry as without it, all hifi sounds pretty much the same.

Remind me how many pairs of headphones you have again, idc?

Sixteen. ABX testing finds the greatest differences with speakers

. Anyway, I collect them for sighted purposes as well as they are things of beauty.

 

[idc]

You are still spot on by identifying the differences are in the listeners head.

Given that this was a blind listening test, I'm struggling to think where else the differences could have been detected...

No one has been able to show how a digital signal can vary and that variance affect sound quality. It is possible to show variances in a digital signal with the likes of jitter, but testing of the audibility of jitter has been inconclusive.

No one has been able to detect the Higgs Boson particle, but many are convinced it exists. Convinced enough to spend several billions of pounds to build a device to try and detect it.

But I digress.

I mean to say that the test helps to show the difference is not with the means of transmission, just as there is no difference inherantly with any cable. The difference is all accounted for with the listener.

The Big Question has answered that it is down to personal preference as to which type of connection suits you best. Inherantly none is better than the other in terms of sound quality.

That is the consistent answer of the Big Question as it shows how blind testing influences the listener.

 

[professorhat]

I mean to say that the test helps to show the difference is not with the means of transmission, just as there is no difference inherantly with any cable. The difference is all accounted for with the listener.

The Big Question has answered that it is down to personal preference as to which type of connection suits you best. Inherantly none is better than the other in terms of sound quality.

That is the consistent answer of the Big Question as it shows how blind testing influences the listener.

That may well be your interpretation of what the tests show. I suspect others will disagree (since I do!).

 

[Andrew Everard]

Although all the above does overlook the fact that the listeners did hear differences, and each preferred one presentation over another, where many would suggest there should be none.

 

[Excitable Boy]

Hi - I have been fascinated to read this heated exchange, as I took part in the test in question. As I said in the piece, I was genuinely amazed with the results, and surprised at how different each set-up sounded. It was done "blind" - we had no idea what was being changed each time, and all thought a piece of equipment was being swapped. The differences were clear and profound. By the end we could all tell whether set-up 1, 2 or 3 was being used.

The fact that we each preferred a different system was strange, but was exactly as it happened. Generally, I am not a fan of wireless, but it was by far my favourite sound on the day. The wired system was lifeless and "cold" to my ears, and the mains based set-up was lacking in clarity.

Explaining why they sounded so different is very hard, and I think it's a bit harsh to slam WHF for not doing so as some of you have done. Can any of you really explain why speaker A sounds better to you than speaker B from a technical perspective ? Does it matter anyway - just go with what you prefer and enjoy. The point of the day and the article was to challenge your pre-conceptions which it certainly did so.

Finally, I would urge all of you to take part in a "Big Question" session if you possibly can. It's fun, it's intertesting, and the WHF team are great hosts. It might also help to change some of your views on the wonderful world of hifi.