I am not a big fan of the Rotel sound, if that's what it is. But it is odd to have a digital input clipping an amp like that.
The big screaming detail in your original post is the 'bass and treble up to 4.' That is by far the most obvious explanation for what is happening. I had a pair of Q300s and I do not have a single negative thing to say about them - brilliant speakers for the money, as hyped. But my Q300s spent most of their time hooked up to a Naim Nait 5 (45W x 2, similar to the Rotel), and I can say that they need a lot of juice. The Nait drove them really well, but the volume knob was well into the early afternoon to get big sound out of them. That Rotel does not have the grunt of a Nait 5, and if you are pushing the amp at all, with a low-efficiency speaker like the KEFs, the Rotel's 40 watts are pumping. Then you roll the bass knob up to four. I couldn't find an impedence graph for the Q300 online but other speakers based on the Uni-Q driver have thier lowest impedence dips right around 100-200 Hz and at around 10K - basically exactly where the extra boost from the bass/treble knobs is. You are asking too much of a 40W amp.
Have you tried the old setup without the bass and treble boost? If you are saying that the bass is so bad that you have no choice, then something is wrong - especially if there are problems at low volumes. But I do present the following audiophile axiom: If you can only deal with your amp because it has bass and treble controls, it is time for a new amp. Like, sit down with your wife and declare an emergency.