Talk about opening a can of worms, haha. Basically this is asking the question "Is hifi worth it"? Which is an easy question to ask but not the easiest question to answer, because there are a lot of reasons. Well, I mean you can answer it very briefly indeed: "Yes, it is worth it". But that isn't the most illuminating response. I think you can tell the impact of the higher-end gear, but to sweeten the pill, perhaps the most useful information is that it is possible to secure a higher-end performance for a mid-price expenditure, IF you are careful and canny about it.
One reason higher performance has a tendency to cost more, is because the process of getting sound from the source to our ears is a complex one, with several steps, each wih several factors affecting the outcome. Whenever you have a situation where many factors affect the outcome, it's pretty inevitable that some can be dealt with cheaply, and some will require expense. Cumulatively the effect of the factors that require expense will up the price considerably, and as the price rises, the proportion of people willing to pay it falls, so economies of scale can't be leveraged and the price winds up still higher as a result. To add to the misery, some improvements not only necessitate greater expense in themselves, but force knock-on expenses upon you. Want the benefits of electrostatic speakers? You're going to need to fork out for the more poweful amp needed to drive them.
The laws of physics necessitate greater expense in some areas to get a better sound. Want more responsive speaker drivers? Fundamentally, you will therefore want drivers of less mass. Making something lighter while retaining sufficient strength is an engineering challenge requiring new, more exotic materials liable to result in more expense. Want the bass to be reproduced better? You are liable to need a bigger and more expensive cabinet. External vibrations can affect the sound? Splash out on more isolation.
This expense is ramped up still more by the problem that we are condemned to use sources that are some way from perfection. You wouldn't hold up the CD as an example of a source optimised to make the process of achieving fidelity a cheap and easy one. The source we are given is vulnerable to jitter? Now we need to spend more to ameliorate the effects. 16-bit resolution a bit course, especially at low volumes, and lacking in dynamic range? Maybe we'll add some dithering or interpolate to 24-bit and pay more for the pleasure.
Issues like jitter and dithering were first taken most seriously in the world of music production, which often natually pioneers many of the improvements we find in modern hifi, though the process has worked the other way too. Recording engineers and producers know the value of making those cumulative improvements in the sound, the little things which add up to a lot, and will go to considerable effort and expense to secure them. And you can hear the effect for yourself, by simply comparing early recordings with later ones and noting the breathtaking advances made along the way as a result. Anyone can hear them.
Actually, people are affected more by sound quality than they realise, but much of the time the impact is attributed to the music itself rather than the reproducing equipment. If something sounds better, many are liable to think the MUSIC is better, rather than the gear. Only extended listening to different material will show them the truth. And psychoacoustic effects can complicate things: something a bit louder can sound subjectively "better" if you don't do a proper comparison. Recording professionals know this and exploit the effects in their productions, and so indeed can some hifi demonstrators. Also, many musical sources don't exploit the full capabilities of high-end gear. Got a system with greater dynamic range? You're not going to hear the benefits so much if you only listen to heavily compressed pop.
And to cap it all, although the term "hifi" implies a request for fidelity, a faithful reproduction of the original, often that isn't really what people want. If they did, they'd all buy studio monitoring equipment: flat, neutral, and ruthlessly revealing. What they REALLY want, a lot of the time, is something which ENHANCES the sound in a way they find engaging, whether it's enhanced warmth, or exaggerated detail. If the game is now to enhance the original, where does that process - and the associated expense - end?
Many people don't invest in high-end gear purely to "improve" the sound anyway, but because they also find the exploration of different design philosophies interesting (especially if they are involved in making gear themselves), and to get different systems suited to different purposes. But if you want a way out of the madness, set yourself the goal of trying to achieve higher-end performance on a mid-priced budget. It can be done...