Things probably aren't as bad as you think.
The big question is what triggered your 'out of memory' warning, and whether that relates to storage space or RAM.
That's not to say there's no problem at all, just that what you're seeing may be perfectly normal.
I'm going to make some assumptions here, so bear with me, but I'll try to explain it in basic terms. First, I'll assume the 'out of memory' warning related to disk space and not RAM.
The second image you posted shows what I think is the 'normal' or 'true' state of your system - you have a ≈1TB drive and are using about 2/3rds of it (≈664GB). That's perfectly reasonable.
During normal usage, and when the OS does certain housekeeping activities, the OS automatically caches data - for example, if you recently searched for files, then the OS may cache, or 'remember' that search query in case you run it again... the subsequent query will use the cache and be much faster since the OS doesn't have to scan the entire disk again.
Similarly, individual applications may cache data for future re-use. The most common example people know is a web browser that saves recently downloaded content (especially images) in a cache, so the next time you visit the page it can display the image from cache/disk rather than have to download it again.
In short, caches are good, and a key part of good OS performance.
Now, the downside of these caches is that they take space on disk. However, the OS knows which files are caches, vs. your actual documents, and it knows that caches can be deleted with impunity - they'll just be rebuilt if they need to be. That's why, in your screenshot, it shows 'Available: 330.55 GB (287.64 GB purgeable)' showing that some 287GB of space is being used temporarily, but can be reclaimed and reused if needed.
The difference between the two screenshots indicates that the purgeable space couldn't be deleted at that point in time, likely because it was actually in use. Now, 287GB is a lot of space, and it's really hard to tell what it's actually being used for, but it likely relates to some application(s) you were running at the time.
For example, if you're running a video editing program such as Final Cut Pro, it will use a large amount of cache and temporary file storage for the footage, transitions, etc.. Similarly with image editing programs and the like - PhotoShop, for example, will retain an undo buffer, so you can undo multiple levels of image manipulation - each of the edits needs a certain amount of space.
The temporary files are typically deleted when the file is saved, or when the application is closed.
Many other programs also use caches and temporary files. While few may use that much themselves, if you have a dozen applications open, each with multiple documents that you haven't saved in a while (so the app is storing temp versions of the file with edits since the last save) it can add up.
So without knowing which applications were running at the time, it's impossible to be more specific, other than I think this is likely part of the normal workflow for your applications. Clearly it's transient, since the amount of purgeable space varies, and even drops to zero. The key is just working out which applications are contributing to it. Saving documents and closing apps that you're not using can help clue the OS into which files can be cleared out.
