Firstly XLR connectors are significantly more expensive than their rca/phono counterparts and they take up a lot of space, so you really have to have a good reason to use them. So why?
Firstly, XLRs are superior connectors to rca/phonos, not sonically, that is not proven, but in terms of robustness and the fact that when the connections are made or broken, the earth connection is the first to be made and the last to be broken, so no nasty noises or worse.
Secondly they have three pins, so a superior construction can be used where the signal 'send', the signal 'return' and the screen can all be kept separate, a technique I approve of, though again, improvements in sound quality are not proven.
Thirdly, they can be used as a balanced cable, the use of +ve and -ve phases run side by side is a good form of noise cancellation, particularly on low level signals over long runs.
However, this is complicated by the fact that most hi-fi components are not balanced, ie a single signal passes through the component from input to ouput. In order for this component to drive a balanced cable the signal needs to be split in two and the phase reversed on one half of the signal to provide a balanced output. This is then transmitted over the cable to the next component, which reverses the process to produce a single ended signal once again.
However, some hi-fi components are actually, themselves fully balanced, so (say) a fully balanced amplifier will have two completely separate signal paths per channel, one of which is out of phase with the other so that when they are combined, any noise/distortion will cancel out. Fully balanced components are fairly rare and tend to be quite expensive, except in many dacs, where the dac chip produces a differential (balanced) signal by default.