You do not have to play a high end system loud to get the best from it. If anything, it's more important that it sounds good when played quietly as you can't possibly listen at eardrum splitting levels at all times.
The absolute biggest factor in sound quality irrespective of the level of your system is the recording you feed it. I have a high end system, and believe me, it is very unforgiving when it is fed a poor recording: it sounds awful as it is only representing what is there, it cannot "improve" what is poor to start with.
Quite a few decent amps handle two ohms, as the lab tests in Hifi News magazine reveal. But you’re right to highlight that it’s the variable impedance curve of loudspeakers that causes the stress in some cases.