Lightning to 3.5mm Headphone Jack adapter stops working after one use, but still works on other devices

I recently bought a Lightning to 3.5mm Headphone Jack adapter for my iPhone SE 2nd Gen.

I used it once to test, then took it out to charge my phone. Upon reconnecting the adapter, it did not work anymore, and the sound played out of the speakers.

I've tried restarting the phone, tried other earphones, et cetera. The earphones are all fully functioning, and restarting didn't do anything.

Unofficial adapters are fully functioning, but those tend to play horrible static over sound.

I have cleaned the Lightning port, looked at all possible settings, and restarted again, to no avail.

The strange thing is, I tested the adapter on an iPhone 11 Plus, and it worked with no problems. I also tested it on another iPhone SE 2nd Gen, and it did not work.

The other strange thing is that it occasionally does work, it just takes about a million restarts of my phone and sheer luck.

Any idea why this is, and how to fix it?

Thank you.