What is a modern replacment for Mac OS X Server's VPN service?

I use a VPN server embedded in the network firewall.

That requires a mid-grade firewall.

That implementation also has the advantage of allowing access even if the Mac is down (or the host-based VPN server has lost the plan), so you could (for instance) access and toggle a remote power switch and force-restart the Mac.

It also offloads the overhead and the shenanigans from the Mac out to the firewall, if the gremlins are poking at the VPN server hard.

And the advantage of keeping the VPN server configuration relatively disconnected from and separate from the Mac host configuration. Being based in the firewall, the administrator has to take specific steps to access and alter the configuration, where a configuration mistake somewhere on macOS itself can cause an embedded VPN server to fail.

If all you need/want is the file server and VPN, you could also consider replacing the Mac Pro entirely with a NAS device. There are configurations from simple SOHO to enterprise class, tower, rackmount, etc. Plus multi-drive configurations allow for various RAID set ups.

QNAP and Synology and maybe ASUSTor would be the most popular/highest rated vendors for NAS.

I’ve had success with Ubiquity firewalls in recent times, and with Zyxel USG-series firewalls in years past.

Ubiquity sells many parts that can be used to build an integrated network system, well beyond the firewall. In this case, possibly Cloud Gateway Ultra or Cloud Gateway Max. That based on your current use of an AirPort Extreme box. The latter can be configured as a video recorder for available security cameras, if that is of interest.

As for local storage and many other features, I've run Synology NAS boxes. The Synology boxes have largely replaced macOS server (packages), with a few features served from the Ubiquity gear. Macs can all serve files locally, so you can continue to run that configuration until you’re ready to migrate away from macOS Server. And Synology has packages available for offsite backups.

Actually, I use two QNAP NAS - one as a file and media (Plex) server and one as a Time Machine backup server. QNAP's Hybrid Backup service has built in Time Machine support. I suspect Synology has a similar service.

I also backup certain computers and parts of the file server to Backblaze and Backblaze B2 (using QNAP's Hybrid Backup Sync service) respectively. It's quite viable to make your setup work with NAS if you wanted to consider that option. No non-starters, at least.