Put simply to the OP yes it does make a difference, but more to the point, if you can't tell any difference then why worry? Just carry on with 320kbps files and be happy, that is unless you are doubting yourself.
I personally think that if anyone is going through all the trouble of getting your...