I think the time to stop upgrading is when you feel that the increases in sound quality that you're getting are disproportional to your income, if you see what I mean. But only you know where to draw the line.
Sitting at work thinking "hmm... I'll earn one twentieth of that new CD player today" is a truly demoralising experience and starts to make your work less rewarding than just getting on with your job and telling yourself "I have to do this job because I just do, OK?". Don't be a slave to money, it's too easy to fall into that trap.
Of course, if you have a family then that's a different matter, but I am fortunate enough not to have people getting in the way of my hobby. Or is it a way of life? I'm inclined towards the latter, you either "get" the allure of hi-fi or you don't.