There have been a number of threads about similar issues. I also had trouble with a stereo pair with my AppleTV.
A recent thread is here:
HomeKit Connection Issues - Apple Community
There seems to be a number of possibilities for all these connectivity issues.
First, there's a lot of discussion about the Private Wifi ID (the devices hardware MAC address) feature. Turning that off might help. However, it should only create the ID once, and as long as you don't reset anything, or use the devices frequently enough, then the address shouldn't change often enough (once created it should never change) to be the cause of these issues. Apparently if you have this activated on the iPhone when you set up the speakers, it gets applied to the speakers too. Try turning it off in the iPhone before setting up the speakers (carry out a full reset if needed) to disable the feature in the HomePods.
Using a too low DHCP IPv4 lease time in your router might be a cause, depending on whether your router tries to keep the same address each time, or whether it allocates a new one too frequently. Again, normally this IP address doesn't change that often. In my router I have selected individual devices and set the router to keep the same IP for each (I have tried using a manual configuration for my DHCP, but wasn't fixing the issue, it's not possible to set that in the HomePods). Again I'm not convinced this was the cause, as the problem occurred more frequently than the likely changing of any IP addresses.
In my Router there is a ULA (Unique Local Address) feature, this applies to the IPv6 addressing protocol (its function is similar to the Private Wifi Address feature). The HomePods do have an IPv6 address in them, so I disabled this in the router. So far that one seems to be the key element that might have cause the issue. That's the newest setting I've tried changing, and I've had perfect connectivity since. This was a bit buried int eh Advanced settings, so might be hard to find (if your router has this feature).
I also switched my AppleTV from Ethernet to Wifi (just so everything was completely on the same part of the network)..
Also try changing the Room setting, I changed mine from the default 'Living Room' to 'Lounge', and make sure all the devices are in the same Room.
Some of those were collated from other users possible fixes, some if from my own attempts to fix it.
So in all, a very 'belt & braces' approach, but trying each of those individually had not fixed the issue. Seems that you need to eliminate any possible way that any address system can change the ID of a device (so disable Automatic DHCP addresses in IPv4, Disable ULA, and Disable Private Wifi ID).
