Delayed input monitoring response in Logic Pro on MacBook Pro

When I am playing back a recorded audio track and turn input monitoring off and on while the track is playing, the response will take a couple of seconds at first and then build up to tens of seconds to respond to on/off cycle of input monitoring.

I've read all the suggestions on using the red "R" record enable button instead of the "I" input monitoring button, but that shows the same cumulative delay error. It doesn't matter if the track is playing, the long response delays persist.

Restarting Logic may fix but may require restarting the M1 MacBook Pro and restarting the coreaudiod in terminal.

My project is very simple. Just a few short audio tracks. Nothing elaborate.

This is fairly repeatable.

[Re-Titled by Moderator]

Original Title: Input Monitoring repeatable bud