Repository navigation
[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type #83754
Description
Activity
Today, CPython is leaking too many implementation through its public C API. We cannot easily change the "default" C API, but we can enhance the "limited" C API (when Py_LIMITED_API macro is defined). Example of leaking implementation details: memory allocator, garbage collector, structure layouts, etc.
Making PyObject an opaque structure would allow in the long term of modify structures to implement more efficient types (ex: list specialized for small integers), and it can prepare CPython to experiment tagged pointers.
Longer rationale:
- https://pythoncapi.readthedocs.io/
- https://pythoncapi.readthedocs.io/bad_api.html
- https://pythoncapi.readthedocs.io/optimization_ideas.html
I propose to incremental evolve the existing limited C API towards opaque PyObject, by trying to reduce the risk of breakage.
We may test changes on PyQt which uses the limited C API.
Another idea would be to convert some C extensions of the standard library to the limited C API. It would ensure that the limited C API contains enough functions to be useful, but would also notify us directly if the API is broken.
Another idea would be to convert some C extensions of the standard library to the limited C API. It would ensure that the limited C API contains enough functions to be useful, but would also notify us directly if the API is broken.
First issues that I met when I tried that:
- C code generated by Argument Clinic is incompatible the limited C API: METH_FASTCALL, _PyArg_CheckPositional(), static _PyArg_Parser, etc. are excluded from the limited C API.
- PyTypeObject is opaque and so it's not possible to implement a deallocator function (tp_dealloc) which calls tp_free like:
Py_TYPE(self)->tp_free((PyObject*)self); - _Py_IDENTIFIER() is not part of the limited C API
it can prepare CPython to experiment tagged pointers
In September 2018, Neil Schemenauer did an experiment:
- https://mail.python.org/archives/list/capi-sig@python.org/thread/EGAY55ZWMF2WSEMP7VAZSFZCZ4VARU7L/
- https://github.com/nascheme/cpython/commits/tagged_int
More recent discussion on the capi-sig list:
https://mail.python.org/archives/list/capi-sig@python.org/thread/JPUNPN3AILGXOA3C2TTSLMOFNSWJE3QX/
See also my notes:
https://pythoncapi.readthedocs.io/optimization_ideas.html#tagged-pointers-doableWikipedia article: https://en.wikipedia.org/wiki/Tagged_pointer
In the limited C API, Py_REFCNT() should be converted to:
static inline Py_ssize_t _Py_REFCNT(const PyObject *ob) { return ob->ob_refcnt; } #define Py_REFCNT(ob) _Py_REFCNT(_PyObject_CAST(ob))
It would enforce the usage of newly added Py_SET_REFCNT() (PR 18389) and advertise that the object is not modified (const).
That would only be the first step towards a really opaque Py_REFCNT() function.
TODO: Add Py_IS_TYPE() macro:
#define Py_IS_TYPE(ob, tp) (Py_TYPE(ob) == (tp))
For example, replace:
#define PyBool_Check(x) (Py_TYPE(x) == &PyBool_Type)
with:
#define PyBool_Check(x) Py_IS_TYPE(x, &PyBool_Type)
IMHO it makes the code more readable.
Py_TYPE() is commonly used to render the type name in an error message. Example:
PyErr_Format(PyExc_TypeError, "cannot convert '%.200s' object to bytearray", Py_TYPE(arg)->tp_name);
This code has multiple issues:
- It truncates type name to 200 characters: there is no Python exception, not even a marker to indicate that the string has been truncated
- It's only the short name: the qualified name (tp_qualname) would be more helpful. The best would be to generate the fully qualified name: module + qualname.
- Py_TYPE() returns a borrowed reference which is causing multiple issues: https://pythoncapi.readthedocs.io/bad_api.html#borrowed-references
In September 2018, I created bpo-34595: "PyUnicode_FromFormat(): add %T format for an object type name". But there was disagreement, so I rejected my change.
I started "bpo-34595: How to format a type name?" thread on python-dev:
I didn't continue this work (until now), since it wasn't my priority.
Make PyObject an opaque structure is also a first step towards the more ambitious project "HPy" project which is fully opaque:
https://github.com/pyhandle/hpyThis API is written from scratch and currently implemented on top on the existing C API.
The following article is a nice introduction to the overall idea:
https://morepypy.blogspot.com/2019/12/hpy-kick-off-sprint-report.htmlFrom my point of view, the long term goal would be to get better performance on PyPy and having a single API for C extension which would be efficient on all Python implementations (not only CPython).
Currently, the C API is not only a performance issue to run C extensions on PyPy. It's also an issue in CPython. Because the C API leaks too many implementation details, we cannot experiment optimizations.
81 remaining items
I changed the issue title to restrict its scope: "[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type".
Making PyObject and PyVarObject structures opaque is a broader topic which should be splited into sub-issues.
"Py_TYPE(obj)=type;" is now disallowed. I consider that the work of this issue is now completed and I close the issue.
Thanks everyone who help to fix these tedious issues!
You can continue to use this issue if you need my help to adapt your C extensions to Py_SET_TYPE()/Py_SET_SIZE().
See also the upgrade_pythoncapi.py script of the pythoncapi_compat project which helps to port your C extensions without losing support for old Python versions:
https://github.com/pythoncapi/pythoncapi_compatSee also the Py_TYPE() change announcement on the capi-sig list:
https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC32DQTACPPX36TPR2GLJAFRB/- changed the title
[-][C API] Make PyObject an opaque structure in the limited C API[/-][+][C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type[/+]on Sep 8, 2021 I wrote an article about these changes:
https://vstinner.github.io/c-api-abstract-pyobject.htmlIt elaborates the rationale for making these changes.
@victor, git bisect tells me the change f3fa63e caused test_exceptions.ExceptionTests.test_recursion_in_except_handler to stack overflow only on windows debug builds.
FYI this regression was handled last year in bpo-44348 "test_exceptions.ExceptionTests.test_recursion_in_except_handler stack overflow on Windows debug builds" and fixed at 2021-09-07 by using the trashcan mecanism in the BaseException deallocator function:
New changeset fb30509 by Victor Stinner in branch 'main':
bpo-44348: BaseException deallocator uses trashcan (GH-28190)
fb30509- added a commit that references this issue
on Apr 19, 2023
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs