Are there any downsides to using the TV as a streamer, other than the lack of gapless playback?
I'd say go for the TV unless there is some feature in the dedicated streamer that you really want.
I think TVs make better playback control devices than any streamer separate because of the large screen that is easily seen from your listening position (assuming the TV is sited opposite that). I use Apple Music on an Apple TV instead of the Android TV software built into my Sony Bravia TV. That gives you gapless playback, but isn't the main reason I choose to use it.
I use a separate DAC before my analogue amplifier too, but you may not notice a significant difference by introducing another DAC before the AXR100.
It might be the settings on my Sony, but the only drawback I see is that everything on the optical out gets transcoded to 48kHz. Probably not an issue for me because I don't think my ears and system are good enough to notice the difference, but there is something unsatisfactory about Apple Music saying it is playing 16-bit/44.1kHz, 24-bit/96kHz or 24-bit/192kHz when my DAC says it is getting 48kHz from the TV regardless.
I'd consider whether or not the saved money would get you a better receiver/amplifier. I'm not suggesting there is anything wrong with the AXR100, but the money might bring options into range that are more to your liking.