High WindowServer memory usage

I have new Macbook Pro 14 with M1 Max. I am connected with external monitor - Dell 38". I have a problem with still growing high memory usage of WindowServer process. When I restart system it uses ~1,2GB but after a few hours it grows up to 5 - 6GB. After two days of running system it grows up to 30 - 35GB. I have 64GB RAM so it is crazy that it used most of mine memory. It is not normal, do not want to restart my system each 3 days and setup applications from scratch. When it grows to 30-35GB I can not even smoothly switch between screens and applications.

I know that this process is responsible for open windows in the system so it might be related with connected external monitor. I am connected by USB-C.

Any ideas how can I reduce consumption of this process and fix this issue?