Siri on HomePod Mini answers when Im trying to ask Siri on my iPhone to do something specific to my iPhone

If I want Siri to set an alarm on my iPhone and I am close to my HomePod Mini, Siri on the HomePod will answer and it will set the alarm on the HomePod. Sometime I set the alarm as a reminder of something in a few hours and I may not even be in the house when I need the alarm to to off. I know I can disable "Hey Siri" on the HomePod but that defeats the whole purpose if I keep haing to tourn it on and off all the time.

Is there someway I can force a way to distinguish between them? or some other workaround?

Can I turn down the sesitiviy for Siri on the HomePod?