Repository navigation
Reference count variations after getting __dict__, getattr, and gc.collect #115822
Description
The following program accesses __dict__ (!) and some attributes of two different classes, in a loop.
After some of the loops, the total reference count is exactly 2 less than usual. In this case, it's every 5th run of the outer loop:
import gc
import sys
class A:
class_attr = None
class B:
class_attr = None
a = A()
b = B()
for i in range(100):
for obj in a, b:
obj.__dict__
for i in range(20):
obj.class_attr
gc.collect()
print(sys.gettotalrefcount())24189
24189
24187
24189
24189
24189
24189
24187
24189
24189
24189
24189
24187
...
The code is a highly simplified version running test_typing in refleak mode (-R:), where a and b correspond to typing.Tuple and typing.List. In the actual test, this happens with more classes, and more overlapping “periods”. Sometimes, the test doesn't meet our (currently rather generous) heuristic, and fails.
The behaviour is ultimately benign, but it interferes with reference leak testing. We should find out what's happening and make regrtest clear any caches and revert to a known state.
Activity
[-]Reference count variations after gc.collect[/-][+]Reference count variations after getting __dict__, getattr, and gc.collect[/+]encukou commentedon Feb 22, 2024
I'll bisect this.
Sharing in case it rings a bell for someone :)
AlexWaygood commentedon Feb 22, 2024
Could look at a8b9350
encukou commentedon Feb 22, 2024
Bisected to #106539 -- “Dematerialize instance dictionaries when possible”. That makes sense :)
Don't know if we can meaningfully control dict (de)materialization.
But it seems that it would be best to use attributes, and avoid accessing
__dict__(which needs to create a dict object).encukou commentedon Feb 22, 2024
I'll prepare a PR.
encukou commentedon Feb 22, 2024
Ah, not that easy: getattr searches more than just the dict.
Maybe we need a function in
operatorthat's likegetattrbut doesn't search the object's class.@brandtbucher, any ideas would be welcome :)
encukou commentedon Feb 27, 2024
I've spent some time reading up on the dict dematerialization implementation, and the public discussion. Now, it seems there's not much to be done about the refcount oscillation. The original proposal about avoiding
__dict__--__dict_copy__and__dict_update__-- would probably need to be extended with__dict_get__and__dict_contains__to be useful inpickle,enumandtyping(and third-party code that does similar things and wants to be fast). Sort of like an alternate Mapping API for non-materialized instance dicts, which sounds rather ridiculous.Perhaps it'll be better to teach the tests about what kind of “leaks” are expected.
I've merged #115726 to make the behaviour easier to see in the test output.
I plan to focus on other things for a while, and return to this issue when the buildbot fails again, or in a few weeks, with a fresh brain :)
brandtbucher commentedon Feb 27, 2024
There's a chance we may ditch dict dematerialization for a simpler scheme in the future. So this problem may resolve itself for 3.13 or 3.14.
[-]Reference count variations after getting __dict__, getattr, and gc.collect[/-][+]Reference count variations after getting `__dict__`, `getattr`, and `gc.collect`[/+]