Hi,
Your problem is easy to describe and solve.
Your computer is a sea of electrical noise and some of that noise is spilling into your headphone out circuit on your computer. It happens to a varying extent on all computers.
It's most noticeable when you have the volume cranked up and you pause or stop a track, then use your mouse or track pad.
Buying a headphone or regular amp will not solve your problem - but you will need one.
What you need to do is buy a good DAC. Plug it into the usb socket on your computer and then connect the output sockets on the DAC to your amp.
So what's going on here? Simple. Inside your computer is a cheap component called a DAC. It's converting the stream of 1's and 0's that make up all sounds on your computer and changing them into an analogue audio signal. That signal is then amplified by another cheap and nasty little component - the headphone out amp inside your computer. I've got a mac, but even with them the headphone out amp is still at the very cheap end of the scale. Hence the size of their profits!
Given the amount of electrical activity going on inside your computer, electrical noise is very loud and will as mentioned earlier spill into audio circuits - it's very difficult to avoid and keep costs down.
By having your audio signal sent from your computer in it's simplest form i.e. binary and then having the conversion to an analogue signal being done by an external DAC you can reduce or even completely remove all noise.
The difference will astound you.
Some recommendations then. I'll assume your on a budget. The best cheap DAC is the HRT Music streamer II - £145. Used with a Slee Novo headphone amp - £250 and you'll never want to go back to what you had before. Money no option - you wont get much better than the Lehmann Audio Black Cube USB -£950.
Enjoy your sounds!