For argument sake we can caricature this, audiophiles listen to equipment and music lovers listen to music. Music lovers explore new music and audiophiles explore new gear with the same music.
An audiophile is addicted to upgrades and tweaks because it delivers happy endorphins from his hunter-gatherer activity, and that mixes up with the music listening and saturates it as better overal. When the new gear novelty wears off, he looks for another fix, another upgrade. Buy new speakers, new CDP, new TT belt, new wires, new racks, then listen to Dire Straits again and again and again.
Of course not all audiophiles are like this, that is the extreeme oniomaniac version. How can you tell how much music lover is left inside you, look at your spending habbits and new music explorations. Introspect and decide.
If gear made listening to music more enjoyable because of better sound quality, men with expensive hi-fi would enjoy music more than men with cheap earbuds, whic isn't the case. Do not misinterpret enjoying consumerism and shopping with the actual pleasure derived from music alone. No reason why both shouldn't coexist, but I see the OP is worried if one can suffer at the expense of the other. IMO, it does.
Disclaimer: No reverse snobery posturing intended. Just personal thoughts by someones who is into hi-fi since 12 y/o.