Cannot swipe control center on apple watch

Cannot swipe control center on apple watch. There was this problem so many times. I pressed the side buttons to restart then it’s work but it came back to the same problem for a few days. It has been a problem like this since I used. I tried to see the recommendation to go digital crown and open app then try to open control center but it’s not work.

It would appreciate if you recommend the correct way or what I did something wrong. I don’t want to restart every time the swipe error