Your description suggests it's a mechanical problem in the speaker that's buzzing: there's nothing active in there that could cause a electronic buzz, and the swapping of the connections rules out the amp or source components. Does the buzz occur at certain frequencies or at very high levels?
The most probable cause is a rubbing voice-coil, a damaged cone surround or maybe even a loose driver fixing: if you have a speaker with visible screws on the front, you could try - very carefully - tightening them slightly, as they tend to work loose with extended enthusaistic use.
Make sure you have the right size of screwdriver or Torx/Hex bit, and be careful not to let it slip: it's all too easy to damage a drive unit with an accidental slip. I'd advise holding the shaft of the screwdriver with one hand while turning it with the other, just to be on the safe side.
If the fixings are tight, and the rubber surround on the bass unit is intact, the most likely cause is a rubbing voice-coil in one of the drivers, usually the result of damage caused by too much level and distortion causing some melting of the wires in the coil. Have you been caning the speakers?
Listen closely to the speaker, and you should soon be able to identify which driver is buzzing
If this is the case, a new driver will put things right - it's best if you change the drivers as a pair, so if it's the bass unit you should change the driver in both speakers. The speaker manufacturer or your local retailer should be able to advise you of the cost.