Watch SE3 timers clearing/resetting themselves

I recently upgraded to an Apple Watch SE3.

Now, sometimes when I have a timer set, the timer goes off and as I raise my arm to look at it, it gets reset/cleared and is gone. I'm not sure why, I definitely am not touching it (could it just be my sleeve?).

I need this to stop because I use timers a LOT.