Repository navigation
CLI process uses up all available machine memory #19020
Description
Description
Our application runs a long running CLI process. This process runs up to 24 hours, then dies and is restarted by a scheduler.
Recently this process started to consume massive amounts of memory - we noticed a usage of up to about 21GB (the machine has 24 GB RAM, and all other processes consume about 3GB). After starting the process, it first consumes a normal amount of memory (about 52MB) for some time. After some time it starts to rapidly increase its usage, until the system OOM-killer shuts it down. The time until this behaviour starts seems to depend on user activity (the process handles part of it), the time until the process is killed after the runaway memory usage starts is a few minutes.
Until the memory usage increases, the process seems to function as intended.
The PHP process runs in a docker container derived from "php:8.4-apache-bookworm" running PHP 8.4.8, but we have also reproduced it with PHP 8.4.7.
This behaviour started out of the blue, about one and a half week after the last update.
What we have done so far:
- Analysed the processes memory usage using valgrind to find out whether this memory usage comes from some library or PHP itself. It looks like it is PHP itself, in the VM stack. I'll add the massif output file, it was obtained by running
ZEND_DONT_UNLOAD_MODULES=1 USE_ZEND_ALLOC=0 valgrind --tool=massif --massif-out-file=./massif-malloc3.out <command>in a container with a PHP debug build (configure --enable-debug). - Analysed on the application side what might trigger this.
Added files:
- massif output
- censored output of
php -i(credentials censored, some application environment variables removed)
php.censored.ini.txt
massif-malloc3.out.txt
PHP Version
PHP 8.4.8 (cli) (built: Jul 1 2025 02:17:01) (NTS)
Copyright (c) The PHP Group
Built by https://github.com/docker-library/php
Zend Engine v4.4.8, Copyright (c) Zend Technologies
with Zend OPcache v8.4.8, Copyright (c), by Zend Technologies
Operating System
Linux 6056e9447d40 5.10.0-35-amd64 #1 SMP Debian 5.10.237-1 (2025-05-19) x86_64
Activity
[-]CLI Prozess uses up all available machine memory[/-][+]CLI process uses up all available machine memory[/+]jbaron-gingco commentedon Jul 4, 2025
If anything is missing, I'll gladly try to provide it.
If someone has any idea on how to debug this further: Please tell me :-)
SakiTakamachi commentedon Jul 4, 2025
@jbaron-gingco
PHP 8.4.10 was released yesterday. Could you test whether the issue still occurs in 8.4.10?
Also, since there’s a possibility that the issue is caused by a third-party extension, would it be possible to test in an environment where only the bundled extensions are installed? (Simply disabling third-party extensions is not sufficient, as they may still perform initialization just by being installed.)
Lastly, would you be able to provide a minimal reproducible code sample? It’s difficult for us to investigate unless we can reproduce the issue on our end.
jbaron-gingco commentedon Jul 9, 2025
Thanks for your help, we have now identified the problem - it was entirely on the application side:
Because we were only able to reproduce this in production, it took some time to find it, in order to not disrupt production systems further.
@SakiTakamachi: Thanks for your input!