Adapter problems

Try booting into Safe Mode to see if you encounter the same issue. Keep in mind Safe Mode prevents the automatic launching of third party software (and even some macOS drivers) during boot & login, so some macOS functionality and hardware may be reduced or non-functional in Safe Mode. This test is to see whether you may have some third party software installed which may be interfering with the normal operation of macOS and causing problems with your USB headphones. Typical culprits are anti-virus apps, cleaning apps, and third party security software.

Also, try disconnecting all other external devices in case one of them is interfering.

Make sure the "Security & Privacy" System Preferences are still configured to allow access to the USB headphones.

Control access to the microphone on Mac - Apple Support

Just had another thought while writing the previous sentence, are you reconnecting the headphones to the same USB port or a different USB port? Maybe the security settings are seeing the headphones as different when connected to a different port. If you are using a USB-C port, then try rotating the USB-C connector 180 degrees upside down in case the USB-C port is half bad.

What happens if you leave the headphones connected until the next time you use them. Do you have the same issue?

Maybe even try a PRAM Reset (hold for at least two chimes, three if possible, but USB-C Macs only allow at most two chimes).

Are you leaving the USB adapter connected to the Mac? If so, unplug the USB adapter when not using the headphones, then plug them back in when you want to use the headphones to see if this helps.

Just stumbled across this thread which is also reporting some odd headphone issues, but I'm not sure if they are the same or related:

Using plugged headphones is causing issues - Apple Community