I didn’t bring my Apple TV remote with me

You would need a physical remote in order to setup the Apple TV on a network. You can't use the remote app if it's not on the same network as the Apple TV. The alternative would be to restore the Apple TV and pair with a BT keyboard.

What model of Apple TV is it exactly?

If its an HD (4th gen) or 4K Apple Tv, you can set up the Remote widget in control center directly, without the original Apple TV remote and without having the Apple Tv connected to Wifi.

Just open the Remote Widget in control center, make sure your Apple TV is turned on, with the iPhone near the Apple Tv and both bluetooth and wifi turned on on the iPhone, wait for the Apple TV to appear in the list of devices. Select it from the list and a code should appear on the TV screen. enter the code on your iPhone to complete the pairing.

If its a 3rd gen or earlier you will need to borrow a remote just for setup in the Remote App.