asking what is the most important isn't of great relevance, though I might agree with 'the room' as a general win😉
A given hifi's shortcomings are always about the weak link, the part of the system dragging the rest down in quality. I'd say an average hifi setup might sound just as good as one twice the price if you have corroded terminals and a piece of chickenwire as your line level interconnect. But a rustbucket amp would achieve the same, so would a cardboard box pair of speakers.
But if you mean which would you spend more on when buying a whole new system.. then I'd say it comes down to your own preference, what do you want to be the strong points of your sound. Maybe you want it to be an all rounder, or maybe extra tight sound from that more expensive amp, or that extra physical push of sound you get from bigger speakers. Or whatever you see...
Me, I prefer bigger speakers, when It's all together I'll have an audiolab 8000c+p with some monitor audio GR20's. Sound source is still pending a decision, but right now it's my PC (SB x-fi elite-pro) and an old pioneer PD-103 that I got for a tenner. Clearly my 'paying weight' is on the speakers, but I bet you others here are different.