You're not saving anything by doing what you're doing. It is nearly certain that the Mini is consuming more power than it would if you simply let it sleep according to Energy Saver.
When you restart a Mac, it has to load all those previously loaded apps, rebuild cache files, reestablish communication with any number of Internet services you use... etc. That takes time, during which there are greater power requirements than running in an idle state. Then there's the problem of replacing its NVRAM battery, used to retain some memory contents whenever mains power is removed. That will accelerate the need for its eventual replacement, which you'll have to pay for. Otherwise they will last for ten years at a minimum.
A sleeping Mini consumes less power than a night light. Half a watt. Negligible. Energy cost < 25¢ per year.
Apple thinks about these things. Actually, they obsess about them, so we don't have to.
In short the answer to your question yes the hardware is affected, and yes it's a problem. But it's your Mac, your energy bill, and your repair costs to bear, so do as you please.