Repository navigation
Heap types implemented in C must have Py_TPFLAGS_HAVE_GC and implement tp_traverse #116946
Description
Activity
I use the following convention:
- When a list item has been addressed, I add a checkbox.
- When the behavior has been changed in main, I add ✅.
- If nothing needs to be done, I cross the name.
The following items may need to support the GC protocol or be made immutable. Once this is done, this first list must be empty and each of its member will be moved to one of the lists below (may be present in more than one list).
- _interpreters.CrossInterpreterBufferView (?)
The following types are true container types that should support the GC protocol and have reference cycles with public documented API only. In particular, the corresponding PRs need to be backported.
- ✅ _curses_panel.panel
- ✅ _tkinter.tkapp 1
- ✅ _tkinter.tktimertoken 1
- ✅ zlib.Compress
- ✅ zlib.Decompress
- ✅ zlib._ZlibDecompressor
The following types are true container types that should support the GC protocol. This list tracks, in addition, their immutability in main (if this is not already the case):
- ✅ _curses_panel.panel
- ✅ _tkinter.tkapp
- ✅ _tkinter.tktimertoken
- ✅ zlib.Compress
- ✅ zlib.Decompress
The following types are empty (or their attributes cannot be used to create cycles) and can be made immutable in main. Non-immutable trivial types are left as is in bugfix versions to alleviate the burden on the GC even if it's possible to create reference cycles manually (and require the user to dig into implementation details):
- ✅ posix.DirEntry
- ✅ posix.ScandirIterator
- ✅ _random.Random
- ✅ _remote_debugging.RemoteUnwinder
- ✅ select.epoll
- ✅ select.poll
- ✅ _tkinter.Tcl_Obj
The following types are immutable and empty (or cannot create cycles), hence do not need the GC. If they do implement the GC protocol, they should be simplified.
- _bz2.BZ2Compressor
- _bz2.BZ2Decompressor
- _hashlib.HASH
- _hashlib.HASHXOF
- _hashlib.HMAC
- _interpchannels.ChannelID
- _lzma.LZMACompressor
- _lzma.LZMADecompressor
- _ssl.Certificate
- _thread._localdummy
- _tokenize.TokenizerIter
The following types are immutable but not empty, but really internal.
Visiting them is delegated to someone else:- ✅
functools._lru_list_elem
Finally, the following types are explicitly required to be missing the GC (testing purposes):
- ✅
_testcapi.HeapCTypeSetattr - ✅
_testcapi.HeapCType - ✅
_testcapi.HeapCTypeSubclass - ✅
_testcapi.HeapCTypeSubclassWithFinalizer(inherits_testcapi.HeapCTypeSubclass)
The following should require the GC protocol. Currently the tests do not leak, but they could if we were to store the instance as one of its attribute. Nevertheless, since we're only testing them once, I suggest we leave them alone.
- _testcapi.HeapCTypeWithDict2
- _testcapi.HeapCTypeWithDict
- _testcapi.HeapCTypeWithNegativeDict
- _testcapi.HeapCTypeWithRelativeDict
The following testing types are also empty or trivial:
- ✅
_testcapi.HeapCTypeVectorcall - ✅
_testcapi.HeapCTypeWithBuffer - ✅
_testcapi.HeapDocCType - ✅
_testcapi.NullTpDocType - ✅
_testcapi.PyUnicodeWriter - ✅
_testcapi._test_structmembersType_NewAPI - ✅
_testimportexec.Str - ✅
_testlimitedcapi.LimitedRelativeVectorCallClass - ✅
_testlimitedcapi.LimitedVectorCallClass - ✅
xxlimited_35.Null - ✅
xxlimited_35.Str - ✅
xxlimited.Str
The following testing types are non-empty but do not own the references:
- ✅
_testcapi.HeapCTypeWithRelativeWeakref - ✅
_testcapi.HeapCTypeWithWeakref2 - ✅
_testcapi.HeapCTypeWithWeakref
Footnotes
-
Backports delayed until https://github.com/python/cpython/issues/138791 is fixed. ↩ ↩2
64 remaining items
I'm wondering why _thread._ThreadHandle (immutable heaptype) caused ref leaks without GC support in #116915.
That wasn't a real refleak and it occurred because register_at_fork was called, which the refleak checker mistakenly identifies as a leak. This is a known, longstanding issue in CPython that fork handlers cannot be cleared.
See this comment
cpython/Lib/test/test_posix.py
Lines 243 to 245 in ce23eea
- added 3 commits that reference this issue
on Nov 12, 2025 I'm in the process of trying to convert a single phase init module (https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/tree/bindings/python/tests/gpiosim/ext.c) over to multi-phase init in light of PEP 793 looking to soft deprecate PyInit_*.
I'm trying to keep up with the back and forth on
Py_TPFLAGS_IMMUTABLETYPEand types created viaPyType_FromModuleAndSpecwhich arePy_TPFLAGS_HEAPTYPE.In our case, the type does need access to module state, so needs to be a heap type after the transition.
What's not clear is whether I also need to define this as
Py_TPFLAGS_HAVE_GCand definetp_traverse. It looks like the standard modules likebz2(which is probably our closest analogue) has been marked GC, unmarked GC, and maybe still under consideration to be added back #139073 (comment)If I'm reading #118021 (comment) correctly, if a heap type is created via
PyType_FromSpecand thus has no module ref, it may be possible to avoid the GC flag, otherwise if using methods likePyType_FromModuleAndSpecorPyType_FromMetaclasswith a module argument, then there may be a need to add the GC flag and atp_traversemethod?Pep 573 https://peps.python.org/pep-0573/#adding-module-references-to-heap-types:
Usually, creating a class with ht_module set will create a reference cycle involving the class and the module. This is not a problem, as tearing down modules is not a performance-sensitive operation, and module-level functions typically also create reference cycles. The existing “set all module globals to None” code that breaks function cycles through f_globals will also break the new cycles through ht_module.
Have there been any new discoveries or is there any clarification on what's recommended for C extension maintainers? Just looking for some guidance, otherwise I may just punt on converting this until this issue has been resolved.
PR #152310 reimplements the GC protocol for
tkappandtktimertoken, addressing the two problems that led to the gh-138331 revert (gh-138807):- The shutdown crash (gh-138791: fix crash when visiting a non-consumed timer handler #138789): the GC slots use a plain cast instead of the type-checking macro, since the collector can visit a surviving object after
module_clear()has reset the global type pointers. - The cancelled-timer regression: a pending timer is kept alive by the Tcl event loop, so it is treated as a GC root (only its callback is traversed) and collecting it no longer cancels a live timer.
Both are covered by regression tests.
- The shutdown crash (gh-138791: fix crash when visiting a non-consumed timer handler #138789): the GC slots use a plain cast instead of the type-checking macro, since the collector can visit a surviving object after
- added a commit that references this issue
on Jul 10, 2026 Have there been any new discoveries or is there any clarification on what's recommended for C extension maintainers?
I think we should create a separate documentation issue. @serhiy-storchaka ?
As currently we have following advice in docs:
Instances of heap types hold a reference to their type. This ensures that the type isn’t destroyed before all its instances are, but may result in reference cycles that need to be broken by the garbage collector.
To avoid memory leaks, instances of heap types must implement the garbage collection protocol. That is, heap types should:
Have the Py_TPFLAGS_HAVE_GC flag.
Define a traverse function using Py_tp_traverse, which visits the type (e.g. using Py_VISIT(Py_TYPE(self))).
If that's actually optional, I believe it worth to be documented. Using GC-enabled immutable heap types come with a noticeable performance loss. Here is an example for the
_decimalmodule: skirpichev#24
If a heap type does not have Py_TPFLAGS_HAVE_GC or doesn't not implement tp_traverse, the garbage collector may fail to collect the type and most of its instances at Python exit. The problem is worse when sub-interpreters are involved: if a sub-interpreter exits and does not clear all of its memory, the other interpreters will continue running with these leaks.
See issue gh-87138 for the rationale.
I wrote an article on the lock type which had this bug and caused a memory leak: https://vstinner.github.io/subinterpreter-leaks.html
Linked PRs
zlibobjects #138290zlibobjects (GH-138290) #138327zlibobjects (GH-138290) #138328_tkinter.Tk{app,tt}Object#138331_curses_panel.panel#138333_curses_panel.panel(GH-138333) #138427_curses_panel.panel(GH-138333) #138428_functools._lru_list_elem#138334Py_TPFLAGS_IMMUTABLETYPEtoselect.[e]poll#138340Py_TPFLAGS_IMMUTABLETYPEto_random.Random#138341Py_TPFLAGS_IMMUTABLETYPEto several internal types #138582_tkinter.Tk{app,tt}Object(#138331)" #138807_thread.lockand_thread.RLock#141268_thread.lockand_thread.RLock#141448The following PRs were directly closed because the types are immutable and empty:
_thread._localdummy#138329_tokenize.TokenizerIter#138330_ssl.Certificate#138332The following PRs were wrongly merged so we reverted them:
bz2objects #138266bz2objects (GH-138266) #138320bz2objects (GH-138266) #138322lzmaobjects #138288lzmaobjects (GH-138288) #138321lzmaobjects (GH-138288) #138323_hashlibobjects #138289_hashlibobjects (GH-138289) #138324_hashlibobjects (GH-138289) #138326PRs for reverting the above commits: