Can't type password to get into recovery mode on iMac

I've tried with a standard wired keyboard, a magic keyboard plugged in with the usb cable, and also a wireless keyboard with a usb dongle. None have worked, but they worked a few hours ago.

I also tried to boot to safe mode, but for some reason it won't do it. I've followed these instructions, again trying with all the keyboards, and it always boots normally.

I've managed to fix the issue by deleting the firmware password using the terminal

sudo firmwarepasswd -delete

This allowed me to both boot into safe mode and also back into recovery mode.

I'm holding shift from the moment I click restart, up until it asks for my password to log in. I've tried this many times, and it never worked.

I read that it cannot boot to safe mode if there's a firmware password, which was the issue I had at the beginning. That's the password it physically won't let me type in anymore.