Home app not responding on LTE

So I have been having an issue with my Homekit accessories not responding while my iPhone is on LTE.

I have numerous Philips Hue bulbs around my home and when I'm on my wifi network, they respond perfectly. I can control my lights from the Home app and the Hue app with zero issues.

However, whenever I leave my house and my phone switches to LTE, my lights and accessories all say "not responding" in the Home app.

I have tried everything I can think of to fix this but I'm out of ideas. My Apple TV is the main hub and I've reset that. I've reset the Philips Hue bridge. I've verified all of my router settings are correct but no matter what I do, my lights will not respond while on LTE.

The caveat to all of this is that I can still control my hue lights directly from the hue app itself while on LTE so that tells me it's not a problem with the Hue setup or the lights. It's the home app.

I've also downloaded Home+ from the app store to see if that would respond any differently and it responds the same was as the home app. Get on LTE and everything goes to No Response.

I am running the latest and greatest ios 14 on my iPhone 11 and I have reset the network settings on my iPhone as well. No dice.

Has anyone else had this issue and have you found a fix?