Hi Fi: In the old days, discrete components could not be made of good quality cheaply, so bypassing them where possible could give improved performance, however the quality these days (Not forgetting a lot of it is done in the digital domain) are such that differences are negligible, even if any at all.
AV: One of the biggest problems, is that unless you have a dedicated cinema room. (Or there are no complaints in rearranging the lounge with acoustic treatments) then the speaker/room combination can colour the sound, fortunately most AV Receivers can help reduce these problems by using DSP room compensation, thus you get a sound that is closer to the original recording. If you select pure direct in this case, then all room compensation is removed, and in most cases, you will get a much inferior sound.
There are no hard or fast rules to this, so best to use what sounds best to you, as without comparison with the original recording, it will be down to your personal preference.
Hope this helps
Bill