Apple TV remote doesn't swipe at all

If your Apple TV remote isn’t working - Apple Support

Or you can also just try restarting your Apple TV

That would be something wrong with the mechanics of the remote so you should get it fixed, but until then you can use an iPhone, Apple Watch, iPad etc. to control the Apple TV through the remote app.

CooperN07.