How can I unpair Apple Watch from a different Apple ID?

Activation Lock is designed to prevent anyone else from using an Apple Watch if it's ever lost or stolen.

About Activation Lock on your Apple Watch - Apple Support

Activation Lock is an anti-theft feature that cannot be bypassed. Only the previous owner can remove the lock by using their Apple ID & password. Apple will not help you find the previous owner. If the previous owner cannot or will not remove the lock, then the watch is useless to you or anyone else. Try to get a refund (through eBay) or dispose of the watch properly. In the future, see the 'Before you buy a watch from a third-party' section in:

What to do before you sell, give away, or trade in your Apple Watch, or buy one from someone else - Apple Support

If you are the ORIGINAL owner, then see the 'Start a support request' section in:

How to remove Activation Lock - Apple Support

Also see lobsterghost1's tip:

The All Too Common SAD Reality of Buying … - Apple Community

Here are two ways the original owner can unpair the watch if you no longer have the paired iPhone:

• See the 'How to erase your Apple Watch if you don't have your iPhone' section of:

Unpair and erase your Apple Watch - Apple Support

• With a web browser, go to: https://www.iCloud.com and sign in with your Apple ID. Open Find My iPhone and select All Devices (at the top). Then click your Apple Watch, select 'Erase Apple Watch', and select Next until the device is erased. Finally click "X" next to remove your Apple Watch.

> I was wondering if there is anything I can do in this case. He will not take the watch back, but will not unpair it either.

Short answer, no. You will not be able to use that watch unless he cooperates and unpairs it from his account.