How do I access devices that are linked to my Apple ID but are not signed in?

I may not be understanding your question well enough, but my answer, in the context of two-factor authentication, is that you can't do that. But you can check to see if a device is signed in that should not be.

Assuming you are using two-factor authentication, Check your Apple ID device list (https://support.apple.com/en-us/HT205064) and remove any device that shouldn't be signed in using the steps in the link.

If there are none that shouldn’t be signed in, then you should not have a problem at this point. But, if an unauthorized device signed in, Change your Apple ID password https://support.apple.com/en-us/HT201355 

Here is What to do after you change your Apple ID or password https://support.apple.com/en-us/HT204071

And see these security-related Apple Documents:

 If you think your Apple ID has been compromised https://support.apple.com/en-us/HT201303

Security and your Apple ID https://support.apple.com/en-us/HT201303

iCloud security overview  https://support.apple.com/en-us/HT202303

Ways to keep your information safe on Mac https://support.apple.com/en-gb/guide/mac-help/mh11402/mac

Using app-specific passwords https://support.apple.com/en-us/HT204397

You need to directly access the device. You can try using Find My Phone for limited access.

Find My Phone - If your device is lost or stolen.      

Find My Phone - Lock and track your device using Lost Mode.