I found a way to fix this without doing a factory reset. You’ll need a MacBook with Apple Configurator installed (available in the App Store). Here’s what to do (steps 2 and 3 are optional only if you haven't previously granted Trust to the MacBook you are using):
1. Connect the iPhone to the Mac.
2. Force reboot the iPhone (volume down + power button).
3. After the iPhone reboots, there should be a “Trust this computer” prompt—choose “Trust”.
4. Open Apple Configurator and find the connected iPhone.
5. You should now see the installed apps on the iPhone (you might only see the one in Guided Access).
6. Delete the app (by selecting the app and pressing backspace).
7. Force reboot the iPhone again (it will now be on the Home screen but unable to open any app, and the Guided Access passcode prompt will pop up continuously).
Done!