Random Latency Issues In Logic

This has happened to me twice now in Logic. I'll be working on a project and everything is groovy. The next day when I go back to it all of a sudden I have latency issues when recording that I can't explain. I understand the issues with plug ins, buffer sizes and so on. I've tried muting and disabling all extraneous tracks that I don't absolutely need for recording. If I start a new project, create an audio track there's no latency. The only workaround I've found so far is to bounce the project to an MP3, load that into a new project, record what's needed and then load the new recording back into the original project.

Today I did some googling and found that logic has a low latency mode that disables certain plug ins which works but the only thing it disabled was the reverb I have on a vocal bus the I need cuz I'm recording vocals so that wont work for me.

Rebooting doesn't help either.

I use the Focusrite 2i2 btw. I don't use direct monitoring cuz I route the output of the focusrite to a headphone amp so I can get a bit more volume but if I plug my headphones into the focusrite with direct monitoring on it doesn't help either.

I did not try deleteing the preferences file yet as is suggested in Logic support.

here are screen shots of my preferences.