No that's hifi hocus pocus. It could be argued that 3.5mm jacks aren't built as robustly as phono sockets but unless it becomes worn with repated use or faulty with a bad connection because you've got a 3kg downfoce hanging off the end of the plug, there's no reason why it should add any more distortion or be less reliable. In fact in studios the 1/4" versions are standard connectors along with XLRs, not RCA plugs.