That was why I switched to headphones, as I could never get speakers to fit a room around its other uses and family.
I was then given some advice by the owner of Musical Fidelity that the best sound comes when the amp can easily drive the speakers, which I have found to be true in practice. Some high impedance headphones that sounded flat with one headphone amp, came alive with a more powerful one. Headphones that are terrible when plugged into a computer, sound great run off an amp.
It is easier to switch headphones around like that, than it is speakers, so the effect amp power and speaker impedance has, is more noticeable. But the principle applies to speakers, so as a rule of thumb, buy a powerful amp and low impedance speakers to stand the best chance of improved SQ.