Hi TechyTim,
welcome to the Apple (user-to-user) Community!
Re: "Voice control activating despite being turned off"
Wondering just where you have turned off Voice Control:
Check in Settings > Accessibility : Voice Control
or
Many accessibility features can be turned on and off with Siri. Use Siri on iPhone - Apple Supporti.
or
If you have set up Accessibility Shortcut: Check if Voice Control is on or off in Control Centre
(If needed: About the Accessibility Shortcut for iPhone, iPad, and iPod touch - Apple Support)
_______________________________________________________
Re: How do I turn it off permanently so it doesn’t come back on.
Seems you would have downloaded Voice Control to your iPhone in order to use it.
To check whether Voice Control is still on: see if its microphone symbol is displayed, or not.
Also: See if following the steps in Apple support's online article (link below) can help:
(ie: Choose what happens when you use the side button:)
Turn off Classic Voice Control on your iPhone, iPad, or iPod touch - Apple Support
All the best :-)