Phone randomly connects to someone else’s homepod

So my roommate and I both have HomePods in our rooms (we have separate rooms). They are each set up with different homes so they aren’t connected to each other in any way.

when my roommate plays music on his HomePod, however, it comes up on my phone and acts as though my phone is the source it’s playing from, allowing me to control his music and I have to turn on airplane mode to disconnect from it. Once I turn airplane mode back off though, it just does the same thing and I’m unable to play my own music. If I try to connect to my HomePod, it plays the music my roommate was playing and turns his music off.

Is there any way to stop this from happening?