drummerman:
You can chop and change but from what I've read this is not a problem of one component not matching. IMO you've reached a ceiling with everything you have ie. whatever you change, the others will hold the system back to an extend that even if you buy a vastly better part, it wont show what it can do, whether this is a player, amp or speaker.
Reading this makes me think I've reached this point with my system currently. I think I probably need to make the next step up (I've posted similar queries about CD players, all manner of DACs, etc, recently). Unfortunately, 10 month old twins, a limited budget but ambitions to get a better sound aren't a good foundation for making wholesale hi-fi upgrade changes!
I can relate to where the OP is at though. Good luck whatever you decide to do!