Hi Gazzip, I just wanted to provide a few more thoughts about your setup.
By all means, you have an impressive kit. But I think that somehow everything about it seems to be laid out to get exactly the problems that you are having.
At the current gain setting in your power amp, its input sensitivity is 4.6V according to the manual (I calculated 4.86V, but whatever). But then, your preamp is capable of producing 15V out of the single-ended output. This means that the usable rotation of the volume pot in the preamp is actually quite little in your situation. If its gain is linear in relation to the position of the pot, at around 30% rotation you will already be feeding the maximum voltage that the power amp can handle, and it will be working at full power—this also depends on other variables, but more or less it's that.
Add to this that your speakers are not exactly on the insensitive side of things at 91dB, and what you get is very loud sound by just touching the pot a little. Less gain would improve matters.
It's a pity that the attenuators didn't work well for you, but I think that the reason may be related to the impedance mismatch issue that Vlad raised. The load impedance of the power amp should be sufficiently high in relation to the output impedance of the preamp to minimise the voltage loss. I remember hearing a couple of times that a difference of a 1,000 order was optimum—so for example, a preamp with 50Ω output impedance would be matched with a power amp that would present 50kΩ load impedance. But it's more common to find pre/power amps combinations in which the load impedance is 300 or 400 times larger than the output impedance.
Now, your preamp has an output impedance of 600Ω, which is actually high for a preamp. Then your power amp presents a load impedance of 10kΩ on the single-ended input (according to the manual; the brochure says 15kΩ), which is on the low side of things (either). Separately these numbers don't mean much, but when you put them together you are looking at a load impedance that is about 17 times larger than the output impedance of your preamp.
Actually, the product page of your preamp specifies a requirement of 20kΩ minimum load impedance—double what your power amp provides. I think it's also interesting to see that the load impedances of Audio Research power amps, with which your preamp was likely designed to be matched, are usually 150kΩ or larger—fifteen times more than your power amp. And I'd bet that Bryston preamps correspondingly have very low output impedances.
As for your question about whether the sound quality of your setup could be affected, I think that that could actually be the case. You already saw with the attenuators the importance of maintaining the relationship between impedances in the preamp and power amp. It could well be that the sound of your system is currently affected by the impedance mismatch, although to a lesser extent than what you heard with the attenuators.
The gain mismatch could also be a problem. Your preamp is working so little that the audio signal is likely close to the background noise of the preamp, and all that together is what the amp is amplifying. Should this be another preamp, the lack of detail might be apparent—but then, maybe background noise is simply not an issue with a preamp of this class, I don't know.