Washed out colors on external displays on M4 Mac mini

Just got a new M4 Mac mini and have connected it up to two Dell S2417DG displays. I previously used these displays with a 2013 MacBook Pro. Most, but not all applications have washed out colors. It's especially noticeable on application icons and in Photos. In Photos, photos appear correctly for a second and then suddenly turn washed out.

My MacBook Pro had vibrant colors and deep blacks when connected to these displays. I've tried using different cables (multiple HDMI and USB C to DisplayPort) and setting different color profiles (I used Rec. ITU-R Bt.709-5 on my MacBook as that gave the best color) and it makes no difference, the Mac mini's color continues to be washed out.

Some applications, such as Firefox and Terminal don't appear to be affected and have proper colors. The wallpaper also has proper colors and deep blacks so this feels like it has to be a software bug and nothing to do with the hardware.

Anybody else experiencing washed out colors? Any potential solutions?