The Siri Remote allows for various configuration options in Settings.
See if any of the other options (i.e. not Auto) work better for you.
Apple TV: Settings﹥Remotes and Devices﹥Volume Control﹥
﹥Auto (let tvOS automatically choose an option, what it thinks might be best)
﹥HDMI (for select AV receivers and select soundbars; not for use with built-in TV speakers, as TVs ignore that.)
﹥Receiver via IR (Siri Remote communicates with receiver directly)
﹥TV via IR (Siri Remote communicates with TV directly)
﹥Off (no volume control through the Siri Remote or Apple TV box)
﹥Edit Devices … (e.g. custom name the currently selected option as a new option)
﹥Learn New Device … (copy control signals from another device remote (IR), in case a device doesn’t use “standard” control signals.)