Repository navigation
Make the garbage collector thread-safe in --disable-gil builds #112529
Description
Feature or enhancement
Python's cyclic garbage collector relies on the global interpreter lock for thread-safety. There are a number of changes needed to make the GC thread-safe in --disable-gil builds.
My intention is to implement these as a series of smaller changes.
- Garbage collection is guarded by
gcstate->collecting. This need to be made thread-safe. - The
--disable-gilbuilds should stop-the-world when finding garbage, but not when calling finalizers or destructors. (depends on Implement stop-the-world functionality (for--disable-gilbuilds) #111964) - The
--disable-gilbuilds should find GC-enabled objects by traversing the mimalloc heaps, because the_gc_next/_gc_prevlists are not thread-safe. (Note it's safe to use the lists during stop-the-world pauses; we just can't maintain them safely during normal Python execution). - The
--disable-gilbuilds should probably use only a single generation to avoid frequent stop-the-world pauses - Eventually, we can get rid of the GC pre-header in
--disable-gilbuilds to save memory. (This will require updating the trashcan mechanism.)
Remaining work:
- Fix scheduling of free-threaded GC (currently scheduled as if there is a young generation)
- Update Python docs
- Refactor out common code from
gc.candgc_free_threaded.c
See also #111964
Linked PRs
- gh-112529: Use atomic operations for
gcstate->collecting#112533 - gh-112529: Track if debug allocator is used as underlying allocator #113747
- gh-112529: Use GC heaps for GC allocations in free-threaded builds #114157
- gh-112529: Implement GC for free-threaded builds #114262
- gh-112529: Remove PyGC_Head from object pre-header in free-threaded build #114564
- gh-112529: Simplify PyObject_GC_IsTracked and PyObject_GC_IsFinalized #114732
- gh-112529: Stop the world around gc.get_referents #114823
- gh-112529: Make the GC scheduling thread-safe #114880
- gh-112529: Use _PyThread_Id() in mimalloc (free-threaded build) #115488
- gh-112529: Revert "Use _PyThread_Id() in mimalloc in free-threaded build" #115524
- gh-112529: Don't untrack tuples or dicts with zero refcount #117370
Activity
pythongh-112529: Use atomic operations for `gcstate->collecting`
gh-112529: Use atomic operations for `gcstate->collecting` (#112533)
Merge branch 'pythongh-113688' into pythongh-112529-gc
pythongh-112529: Track if debug allocator is used as underlying alloc…
pythongh-112529: Track if debug allocator is used as underlying alloc…
pythongh-112529: Use GC mimalloc heaps for GC objects
pythongh-112529: Track if debug allocator is used as underlying alloc…
pythongh-112529: Use GC mimalloc heaps for GC objects
pythongh-112529: Track if debug allocator is used as underlying alloc…
pythongh-112529: Implement GC for free-threaded builds
gh-112529: Track if debug allocator is used as underlying allocator (#…
36 remaining items
pythongh-112529: Simplify PyObject_GC_IsTracked and PyObject_GC_IsFin…
pythongh-112529: Don't untrack tuples or dicts with zero refcount
gh-112529: Don't untrack tuples or dicts with zero refcount (#117370)
Squashed commit of the following:
Squashed commit of the following:
pythongh-112529: Make the GC scheduling thread-safe (python#114880)
pythongh-112529: Simplify PyObject_GC_IsTracked and PyObject_GC_IsFin…
pythongh-112529: Don't untrack tuples or dicts with zero refcount (py…
colesbury commentedon May 7, 2024
I'm going to close this as the free-threaded GC is implemented now. We can open a new issue for refactoring out the common code.
pythongh-112529: Use atomic operations for `gcstate->collecting` (pyt…
pythongh-112529: Track if debug allocator is used as underlying alloc…
pythongh-112529: Use GC heaps for GC allocations in free-threaded bui…
pythongh-112529: Implement GC for free-threaded builds (python#114262)
pythongh-112529: Make the GC scheduling thread-safe (python#114880)