I don't know the HK at all and have limited experience of AV kit, so won't join that debate (not that it usually stops people here!)
But I may be able to add a bit to your decision process. In my experience it's better to spend more on the speakers than the amp than vice versa. Contrary to the source first 'rule' that used to be in vogue.
The experience is based on when I demo'd B&W 802 and 803 Nautilus speakers. To keep the budget similar I listened to 803's with dearer amps (Chord) and the 802's with lower cost amps (Meridian). It was night and day in favour of the 802 / Meridian combination, which I duly bought.
I kept this pairing until I eventually downsized, several years later. Getting to know the 802's well I was never sure they were being driven properly by the Meridian though, and would have liked to upgrade. My original decision was still valid, it's just that the urge to upgrade was quite strong fairly early on. So if the HK will drive the 804's then go for it, just start working on the business-case-to-self-for-the-upgrade soon!
Ps new 804's, or second hand 803's? In fact go the second hand route, save at least a grand, bingo, there's the new amp. Or are you thing used 804's anyway? Just a thought.