Message app keeps selecting secondary number for contact

Using Message app on iOS 18 I am often writing to a friend with two saved phone numbers in his contact card, only one of them has iMessage activated on it.

If I go from its contact card and click message > select the iMessage number I can write I message, the conversation is then for awhile using iMessage.

If I leave the Message app then reopen and go on the conversation thread it goes back to text and to the secondary number.

Clicking on the contact info shows the secondary number with a "recent" tag next to it.

In the contact card I've clicked on message then long press on the correct iMessage number but still this issue occurs.

I can't find anything on this issue.

Anyone with a clue?