homepod volume from 2 sources

Hi..

I have a stereo pair HomePod set up with a ATV 4K 3rd gen to my tv and all sound whichever input I’m on goes through the HomePods which is exactly how I want it.

One thing I like to do is play Xbox whilst listening to an album so have just gone to do that for the first time.. opened Music on my phone chose album sent audio to HomePods.. all good, then I went to play game and the sound comes out, great I thought HomePods smarter than I thought..

only problem is the tv sound is too loud and volume controls both sources.

is there a way to control volumes independently?

thanks