[C API] Make Py_TYPE() opaque in limited C API 3.14

@colesbury

In the limited C API 3.14 and newer, I propose to change Py_TYPE() and Py_SET_TYPE() implementation to opaque function calls to hide implementation details. I made a similar change for Py_REFCNT() and Py_SET_REFCNT() in Python 3.12.

The problem is that with Free Threading (PEP 703), the implementation of these functions become less trivial than just getting/setting an object member:

static inline PyTypeObject* Py_TYPE(PyObject *ob) {
    return (PyTypeObject *)_Py_atomic_load_ptr_relaxed(&ob->ob_type);
}

static inline void Py_SET_TYPE(PyObject *ob, PyTypeObject *type) {
    _Py_atomic_store_ptr(&ob->ob_type, type);
}

_Py_atomic_load_ptr_relaxed() and _Py_atomic_store_ptr() must now be called. But I would prefer to not "leak" such implementation detail into the limited C API.

cc @colesbury @Fidget-Spinner

Linked PRs

gh-120600: Make Py_TYPE() opaque in limited C API 3.14 #120601

pythongh-120600: Make Py_TYPE() opaque in limited C API 3.14

I'd prefer if Py_TYPE was a public exported function. If it remains a macro, non-C languages will still need to access ob_type directly (or start using private API).

As for Py_SET_TYPE, you need knowledge of the internals to use it correctly. I support changing it to call an opaque function, but unlike Py_TYPE I'd prefer not exposing any more than it is (so: keep it as a macro).
The proper way to change a type is currently setting the __class__ attribute. We could add a new API function, like PyObject_SetType, that does the required checks and updates. And then deprecate Py_SET_TYPE.

Please add tests for the old implementation (getting/setting ob_type directly), to ensure existing extensions aren't unintentionally broken.

Please add tests for the old implementation (getting/setting ob_type directly), to ensure existing extensions aren't unintentionally broken.

_testcapi has test_set_type_size() which tests Py_TYPE() and Py_SET_TYPE():

    // Ensure that following tests don't modify the object,
    // to ensure that Py_DECREF() will not crash.
    assert(Py_TYPE(obj) == &PyList_Type);
    assert(Py_SIZE(obj) == 0);

    // bpo-39573: Test Py_SET_TYPE() and Py_SET_SIZE() functions.
    Py_SET_TYPE(obj, &PyList_Type);
    Py_SET_SIZE(obj, 0);

pythongh-120600: Make Py_TYPE() opaque in limited C API 3.14

I modified my PR to only reuse Py_TYPE() name (instead of adding a _Py_TYPE() function) and to leave Py_SET_TYPE() unchanged.

Sounds good, thanks!

I don't think it'll solve thread safety -- Py_TYPE returns a borrowed reference, and something could invalidate that reference while you're using it. But for the use cases where Py_TYPE is already safe, it's nice to have functions rather than macros in the limited API.

I don't think it'll solve thread safety -- Py_TYPE returns a borrowed reference

PyObject_Type(obj) already exists to get a strong reference.

I also wrote PEP 737 "C API to format a type fully qualified name" to avoid a common usage of Py_TYPE() borrowed reference. (The old usage can lead to a crash, see the PEP!)

Related change done in Python 3.12: Limited C API: implement Py_INCREF() and Py_DECREF() as function calls.

gh-120600: Make Py_TYPE() opaque in limited C API 3.14 (#120601)

pythongh-120600: Make Py_TYPE() opaque in limited C API 3.14 (python#…

Is there something left to do here?

Fixed by change 16f8e22.

There's still Py_SET_TYPE, which IMO could use a private opaque function.
Should that be done with a new issue?

There's still Py_SET_TYPE, which IMO could use a private opaque function. Should that be done with a new issue?

I suggest to open a new issue. If I understood correctly, Py_SET_TYPE() should not exist in the limited C API, since static types cannot be defined with the limited C API.

pythongh-120600: Make Py_TYPE() opaque in limited C API 3.14 (python#…

I created an issue for Py_REFCNT(): #124127

Repository navigation

[C API] Make Py_TYPE() opaque in limited C API 3.14 #120600

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[C API] Make Py_TYPE() opaque in limited C API 3.14 #120600

Description

Linked PRs

Activity

encukou commented on Jun 17, 2024

vstinner commented on Jun 17, 2024

vstinner commented on Jun 17, 2024

encukou commented on Jun 17, 2024

vstinner commented on Jun 17, 2024

vstinner commented on Jun 17, 2024

nineteendo commented on Jun 19, 2024

vstinner commented on Jun 19, 2024

encukou commented on Jun 20, 2024

vstinner commented on Jun 20, 2024

vstinner commented on Sep 16, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions