Repository navigation
Inc GC in 3.14t can defer too much, resulting in high memory use #148937
Description
Bug report
Bug description:
In the process of testing the default build incremental GC, I noticed a problem with the 3.14t GC. Shortly before the 3.14.0 release, I added some additional logic to defer GC triggering if the process memory use has not increased by 10% since the last collection. Unfortunately that does not work well with how mimalloc handles memory. It does not promptly return memory to the OS or mark pages as unused (e.g. with madvise). So, the process size from the point of view of the OS does not decrease after the GC frees cyclic trash.
For programs that create even modest amounts of cyclic garbage, this is a major problem. It acts as if the GC threshold has been set to 40x the "threshold0" value. By default, that means full GC collection every 80,000 net new objects. This is not a continuous memory leak but it means the process uses uses way more memory than it should.
The simple fix would be to just remove that process size based defer. However, that would result in regressing on the GH-132917 issue, so I wouldn't recommend that. I have a fairly simple fix that "asks mimalloc" how much memory is being used, rather than asking the OS. Based on my testing, that works well.
CPython versions tested on:
3.14
Operating systems tested on:
Linux
Activity
pythonGH-148937: fix for free-threaded GC (RSS based defer)
pythonGH-148937: fix for free-threaded GC (RSS based defer)
nascheme commentedon May 6, 2026
After some more experimenting, I think just removing could be okay. The original performance issue reproducer was the script in gh-132917 (appending many tuples to a list). That performance issue was resolved with either gh-133464 or gh-142562. The reason gh-142562 fixes it is that by counting untracked tuples in the "long_lived_total", we allow the
count < gcstate->long_lived_total / 4condition to work correctly (avoiding quadratic runtime scaling).colesbury commentedon May 6, 2026
If you are okay with it, then let's go with removing the RSS based defer for now. We can add mimalloc-based heap size estimation later when we are confident in that approach.
nascheme commentedon May 6, 2026
Benchmark results from list_append_perf_test.py:
[-]GC in 3.14t can defer too much, resulting in high memory use[/-][+]Inc GC in 3.14t can defer too much, resulting in high memory use[/+]pythongh-148937: revert process RSS based GC deferral
gh-148937: revert process RSS based GC deferral (#149475)