Hi pej1122. I tried my portable headphone amp to see if it worked with my TV and it did not. There was a scary humming noise. Grado SR80s straight out of the TV sounded OK. But the quality of sound is poor. MTV pop was awful and Kerrang! rock not much better. The adverts sounded the best! Compared to a decent source of CD or streamed music and a dedicated amp there is too much mush due to the compression and no soundstage. All the sound is in your head.
So, based on that experience I would look to the AV amp for sound. Grados are a great headphone and their dynamic make them exciting to listen through. Do you know they are open backed so they will leak sound that can be heard by others clearly? Grados are also pretty sensitive, so will work fine out of the amps headphone socket, presuming it has one.
If you get an amp, then connect it with phono cables to the amp using an aux connection.
However, if you source is sound from the TV, well you cannot polish a t***. I have no experience, but CDs and DVDs should be much better.