I'd agree there is an amount of HiFi snobbism, but there are other factors, too.
Apart from anything else, there's the fact that, if you don't like the effects filled, loud, shoot-em up type movies, then the benefits A/V diminish somewhat. The slow, thoughtful films tend to use the front three speakers far more than the effects ones, which at best are used to represent the buzz of conversation in a caf‚ if they are used at all. This greatly cuts back the advantages of A/V, since HiFi can represent more than adequately the front three speakers.
Then there's the sheer mass. Unless you go for on of those compact designer systems and unless you have a separate room, your living area becomes cluttered with up to seven speakers and, of course, their cables. Small apartments aren't designed for A/V. Also, as most would agree, the compact systems can't cut it when it comes to music.
There's also the matter of your tastes. I listen to music 75% of the time, so any A/V system for me should be strongly biased toward music. Most aren't. The A/V amplifier I bought was reputed to be highly musical, but, after 6 months of great dissatisfaction and total lack of ability to match the level of even the cheapest stereo amp, I bought a NAD, sold the A/V amp and haven't looked back since.
One day, in a bigger apartment or house, I'd go back to home theatre, but in a separate room and away from the stereo, perhaps. Yes, a compromise is to put a stereo amp driving the front speakers into the mix, and I've suggested such, but for me it's an extra box, and still means 7 speakers dominating the living space, since the stereo would always be in the living room.