I listen to headphones a lot at work and immensely enjoy it, at least until somebody interrupts me. - At home I sometimes use my Grado's late night or if I fancy loud and enjoy it too.
I have absolutely no problems switching between speakers and H/phones.
My system is far from high end but my...