For new vinyl I buy, where possible, direct from the label, usually in the US - many of these now come with a coupon so you can easily and legally download mp3 versions for Ipod, phone, etc. Great idea which all companies should follow. Mainly seems to be the smaller labels that do this where vinyl sales are going from strength to strength.
Now because of the exchange rate it is cheaper to buy import vinyl from independent shops here in the UK such as Rough Trade, Piccadilly and Action Records.
CD's are bought from Amazon (marketplace as well), Ebay, Play, HMV online or any of the independent shops if they have exclusives. Rough Trade are particullarly good for this as they regularly have exclusive bonus discs to give away. The Fleet Foxes debut came with a bonus disc (well before the recent 2 disc reissue) as did the current Andrew Bird album.
Because of Spotify I cancelled my Emusic subscription and now use that as a listening / review tool for new purchases. If the albums I downloaded from Emusic were particularly good I would buy the cd as well. Emusic allowed me to make my own mind up rather than rely on magazine reviews.
I have only ever paid for two download only albums from Itunes when the bitrate was quite low. It was the only option I had to own them. Still a bit of a completist when it comes to my favourite bands / labels.
Downloads are still too expensive. You can generally get new releases, on cd, online for about £9 and the download version, at low bitrates, is normally only a £1 or so cheaper. I can rip my own cds at my preferred bitrate (Apple lossless) and have a physical copy if my hard drive should ever crash.
I spend between £100 and £200 monthly on physical releases and because of Spotify, £nil on downloads.