Zsh kills Python process with plenty of available VM

Interesting read. I was always told that Macios is not the same as IOS.

TT123B wrote:

Interesting read. I was always told that Macios is not the same as IOS.

They are not the same. Mac's run macOS and iPhone's/iPad's run iOS.

Probably if they are using the original iOS ?

So to summarize my understanding of this unspecified-error question (and here without intending to be sarcastic or derogatory), this proprietary third-party app runs out of virtual memory on Linux and that’s okay, but it runs out of virtual memory on Mac and that’s bad, because of the different ways the two systems end what was seemingly described as a virtual-memory-leaking run-away app?

Last I checked, recent macOS offers 18 exabytes of virtual memory, limited by how much backing storage is available (in the boot partition) for that usage. (Details)

As another source of assistance with this app, have you checked with the gurobipy forum?

• https://support.gurobi.com/hc/en-us/community/topics

There is no virtual memory system in existence that can exceed the aggregate capacity of available memory and available backing storage. How that virtual memory addressing limit is reached, and what resource quotas might be enforced, and where that backing storage might be allocated, and whether the virtual memory system supports dynamically increasing backing storage, and details such as memory compression, varies.

Put more succinctly, you get as much virtual memory as there is (non-wired) main memory and backing storage (swap) available.

Mapping how Linux does virtual memory management to another platform won’t work well. XNU descended from Mach, and mostly still follows the Mach memory management scheme and norms. Not those of Linux. If you want more control over virtual memory allocations and backing storage than macOS and quite possibly Linux for instance, maybe OpenVMS x86-64 is a better operating system choice.

Memory-related shell commands:

ps -o pid,rss,vsz,pmem,comm {pid}    # process information for {pid}
vmmap -w {pid} | head -10            # process-level virtual memory
vm_stat 1                            # system-wide system-wide statistics

# maybe also useful
sysctl vm.swapusage                  # swap usuage; Activity Monitor app easily shows this, too

More reading: Memory and Virtual Memory

Fundamentally, trying to fix an app with no source code and with what might seems a virtual memory leak — or an unusual app design — is likely best left to the app developers.

Most of what I know of virtual memory comes from operating system development work on an early 32-bit virtual memory system, and then later with 64-bit operating system development work. (Far fewer front panel lights on these.) No app that tries to consume all resources is going to be particularly predictable though, as there are always competing activities on (most) modern systems. And a Python app running under the control of the Python interpreter only adds to the complexity of trying to predict the inevitable failure.

gggg87 wrote:

Thank you for sharing the commands, what would be really helpful in this case then would be a command that, instead of vm usage, reports the residual available vm to the processes. It seems that, given what you say, either the getinfo run on the VM volume is just spitting out a theoretical value that is nowhere near the actual value, or there are other resources at play that I cannot look into (for that I'll wait a reply from Gurobi on how what this routine does is implemented on macOS). So, is there a way to get the actual value of residual remaining (i.e. available) space in the VM volume from terminal? If I run a standard du -sh I get the same values as those I get from getinfo, is the output of this command reliable in terms of describing the actual space remaining?

There isn’t a way to do that because there isn’t a way to do that. Yes, that reads like a tautology. But think about it. By then time the data needed to answer that question can be gathered, it will have changed.

If you are on a near -completely quiescent single-user non-multitasking system, with no other active apps either using storage directly or using storage indirectly for virtual memory paging maybe, and minimal or no system activity, probably. But you are not.

So you just spent system resources to get an answer that is somewhere between unreliable and wrong. Which means you’ll get what resources are currently available.

Pragmatically, the app can allocate virtual storage and catch the allocation failure that way, or can allocate main storage and catch the error. And whatever the Python memory management implementation is itself doing here just adds to the “fun”.

iOS virtual memory doesn’t page user-written memory: https://alwaysprocessing.blog/2022/02/20/size-matters

(continued due to 5000 character limit)

Here is the meaningful portion of my "df -H" output:

Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s3s1    995G     12G    197G     6%    451k  1.9G    0%   /
devfs             374k    374k      0B   100%    1.3k     0  100%   /dev
/dev/disk3s6      995G   3221M    197G     2%       3  1.9G    0%   /System/Volumes/VM
/dev/disk3s4      995G   8114M    197G     4%    1.3k  1.9G    0%   /System/Volumes/Preboot
/dev/disk3s2      995G     56M    197G     1%     110  1.9G    0%   /System/Volumes/Update
/dev/disk1s2      524M   6312k    504M     2%       1  4.9M    0%   /System/Volumes/xarts
/dev/disk1s1      524M   6042k    504M     2%      30  4.9M    0%   /System/Volumes/iSCPreboot
/dev/disk1s3      524M   2707k    504M     1%      64  4.9M    0%   /System/Volumes/Hardware
/dev/disk3s1      995G    773G    197G    80%    3.3M  1.9G    0%   /System/Volumes/Data
/dev/disk3s7      995G    143M    197G     1%     243  1.9G    0%   /Volumes/shared
map auto_home       0B      0B      0B   100%       0     0     -   /System/Volumes/Data/home

/dev doesn't count.

/System/Volumes/xarts, /System/Volumes/iSCPreboot, and /System/Volumes/Hardware don't count because they're part of a separate container. They do count, but more at a hardware usage level.

All the other volumes are shared volumes and the "Avail" column will be the same. That's my free storage. I'm well above 100 GB, so I'm doing alright. However, this is not true:

I do not have 212 GB free.

And this can be tricky. The case of the "-H" parameter is important. If I do "df -h", then it uses traditional math where 1K is 1024 instead of "new math" where 1K is 1000. But these are computers. It's really 1024, but that's not pretty, is it?

I will try to install a separate python version bypassing the conda environment and see if in a regular python environment this happens also, or if it is just conda that is messing with VM management.

I doubt conda is at fault in this respect. I was just speculating that conda was probably responsible for keeping zsh in there when you thought you had changed.

I was able to somewhat reproduce this problem using Xcode's Python. In my case, it only used compressed memory but still killed it eventually. I think that was probably because my Python script was fake. It was just allocating the RAM, not really using it. Virtual memory is a constant battle between the operating system and apps both lying to each other about the RAM that they need, want, and are actually given.

If you are positive that macOS does not have additional limits on VM, this should be worth a try and would at least rule something out, thanks again.

I'm not positive about anything. It's all guesswork. After all, you mentioned "Sequoia 15.7.1". I don't have that operating system available right now. I'm not even sure my Sequoia system is running that version. But I can tell you with absolute certainty that these fundamental behaviours can, and will, change even between these minor updates. I did my test with Python on Tahoe, so that alone might explain why I only got compressed RAM.

The only thing I am sure about is that a well-written program shouldn't consume memory without bounds. If you're attempting to run a poorly-written program, then you can't fault the operating system for detecting and killing it before it destabilizes the system.