Do you have software monitoring enabled in Logic's Preferences/Audio Under the "General" tab.
Make sure Software Monitoring and Input Monitoring only for Focused tracks. This runs your signal through Logic's audio engine. Under the Devices tab your I/O buffer should be 128. Also, do not use Low Latency mode (if you have it on).
On a side note, the Scarlett should have a hardware Monitoring function. It means you monitor directly through the Scarlest before it goes through Logic's audio engine. Read the Scarlett manual see if they mention Low Latency Monitoring or Direct Monitoring. If you use direct monitoring Logic's software monitoring should be off.