Repository navigation
tp_bases of object is NULL: undocumented or unintentional behavior change in 3.12? #105020
Description
Activity
NULL means it is stored in the interpreter. To get the correct value, you will need an internal function
Objects/typeobject.c: lookup_tp_basesor
Objects/typeobject.c: _PyType_GetBasesThese changes and APIs usages are private and undocumented.
The same appears to be true for
tp_dictof static builtin classes, and that change is also not documented.For both changes there's no clean alternative other than using the private API (
_PyType_GetDictfortp_dictand_PyType_GetBasesfortp_bases). One could usePyAttr_GetAtring(type, "__dict__")to accesstp_dictinstead, but that's not necessarily equivalent when subtypes are involved.I'm running into this issue while porting PyObjC to 3.12.
Reacted by Gregory P. SmithThis is related to #94673
@ericsnowcurrently, do you recall why there are no public accessors to replace directly accessing
tp_basesandtp_dict?Reacted by Gregory P. Smith- added a commit that references this issue
on May 28, 2023 PyObjC 9.2 will use
_PyType_GetDictwith Python 3.12 to get it to work without invasive changes. It would be nice if that API could be moved to the unstable or even stable tier as there's currently no way to get the same functionality with a public API. The same is true for_PyType_GetBases(use case for the OP).In particular, the only public way I've found to get this information is to use
PyObject_GetAttrfor the corresponding dunder name. That doesn't fit my use-case for two reasons:-
Semantically this is not the same because types can override
__getattribute__and could return different information.In PyObjC the
PyObject_GetAttrpath could lead to unintentional recursion because I have a custom__dict__descriptor on some types to lazily initialise those types. -
The
PyObject_GetAttralternative results in differences in refcount (its result is a new reference, while accessing the correspondingtp_slot is a borrowed reference). That leeds to fairly large changes to correctly handle refcounts in all paths through the code (some of which is from the Python 2.1 era).
@rwgk, sorry about hijacking your issue, but the problem we're running into is basically the same with to different slots on PyTypeObject and should have a similar solution.
-
sorry about hijacking your issue
No worries about that, on the contrary, I feel like I should edit the title. Something like:
Urgently needed: Guidance for binding tool maintainers: How to deal with the consequences of #103912
Based on the comments I saw thus far, I'm sorry to say, it seems like this wasn't actually planned out.
I'm working on a fix.
Reacted by Gregory P. Smithdo you recall why there are no public accessors to replace directly accessing
tp_basesandtp_dict?I don't, though it seem very sensible for us to add getter functions to the public API.
gh-105115 restores tp_bases (and tp_mro) back to where they were pre- gh-103912, by making them immortal for static builtin types. That should resolve the reported problems.
FYI, I totally missed that the two fields were part of the documented public API for
PyTypeObject. Breaking compatibility is on me. Sorry about that.Reacted by Gregory P. Smith and sunmy2019Awesome, thank you @ericsnowcurrently!
Reacted by Eric Snow- added a commit that references this issue
on May 31, 2023 Thanks! @ericsnowcurrently
@ericsnowcurrently, what's the right way forward for accessing
tp_dict?I'm currently using
_PyType_GetDict, but that's a private API and I'd prefer to avoid those.Three options:
-
add
PyType_GetDictas public API and keep_PyType_GetDictas an alias for code that's already adjust to the beta -
rename the private API to
PyType_GetDict -
rename the private API to an unstable one like
PyUnstable_Type_GetDict
I'd prefer the second option unless there's a good reason to assume that returning a borrowed reference from
PyType_GetDictwill be problematic in the future.Also: Is this a change that can still get into 3.12, even during the beta cycle?
-
what's the right way forward for accessing tp_dict?
If we keep it in the internal headers then we can re-name it however we like, whenever we like. 😄 Moving it to the public API would be okay if it returned the dict proxy object. However, it returns the actual dict, which complicates things (but doesn't necessarily rule it out). Doing so would require @Yhg1s to approve.
As to the complexity--when it comes to the static builtin types, giving the actual dict to C-API users would potentially expose us to future pain. For example, we may eventually statically initialize that dict, which means it must be treated as immutable. It's unclear what the actual impact would really be though, so it doesn't seem like a simple answer is reachable quickly. 😞
The safest bet would be to leave the getter function in the internal C-API for 3.12, possibly dropping the leading underscore.
Leaving the function in the internal API leaves me in the position that I have to use a private API to do something that used to be part of the (implicit) public API, just one small step removed from having to resort to poking in internals when someone notices that the API is not used by stdlib extension modules and removes the export for the symbol (e.g. visibility=hidden).
BTW. Returning a dict proxy from a public API for this wouldn't work for me, both because of the refcounting implications mention in an earlier comment and because AFAIK the PyDict_ API doesn't work on a mapping proxy. Both could be worked around, but would require significant changes to code that has been stable for a long time.
I'll try to raise this issue with our RM over the weekend, trying to communicate in a couple of minutes each morning and/or evening doesn't really work.
Just to make sure we're on the same page, in what cases do you need access to the tp_dict of static builtin types? They are the only ones where tp_dict is NULL.
- added a commit that references this issue
on Jun 1, 2023 @ronaldoussoren, it would probably be best to move this discussion to a separate issue. Feel free to create one, or I'll do it tomorrow if you haven't already. Thanks!
This is to confirm that current pybind11 master builds with the 3.12 branch, and all pybind11 unit tests, excl. the tests that depend on numpy, also pass.
Thanks again @ericsnowcurrently!
cpython commit e6d5e63 (HEAD -> 3.12, upstream/3.12)
pybind11 commit d0232b119f5f0cf2bf6cecacaf099c301c811ca8 (HEAD -> master, upstream/master)
gcc (Debian 12.2.0-14) 12.2.0Reacted by sunmy2019 and Eric SnowThanks for letting us know about this!
Documentation
See #103912 (comment) for full context.
Specific problem:
It appears
tp_basesofobjectwas an empty tuple in the past, but is nowNULL, could that be? Is that intentional?If intentional, could it please be documented? I think the behavior change is likely to trip up fair a number of people.
I looked in https://github.com/python/cpython/blob/edd0cb8e77d7b65f5a9c2c69dc81f9c4514878d5/Doc/whatsnew/3.12.rst but don't see
tp_basesbeing mentioned there.I also verified that the
tp_bases == nullptrpatch is still needed with current 3.12 HEAD (edd0cb8).Linked PRs