I think it is a physical hardware problem in my case— the rear ports do not provide a secure fit. Slight movement of a cable is enough to disconnect an external drive. In my case, I have a powered OWC 11-Port Thunderbolt Dock (with 2 drives attached) connected to a rear port. I've now tried 2 thunderbolt cables to connect the dock to the rear ports on the M4 Mac Mini and I've tried 2 of the 3 ports.* Both cables and both ports have disconnected with a slight movement of the cables (at the Mac Mini end of the cable). Both cables are new that came with products (i.e. external drive enclosures).
The last thing I'm going to check is to see if the insecure fit problem also exists with an Apple-branded Thunderbolt cable (iow, if the Mac Mini M4 doesn't play nice with non-Apple cables). I have a nearby Apple store, so probably will just take it in to have them see for themselves.
I'd suggest giving your cable the jiggle test to see if the physical port is the problem.
(*I haven't messed with the 3rd rear port because my Home directory is running on an external drive there; it also disconnected once, causing problems, and I'm leaving it untouched with my fingers crossed).