Skip to content

[C API] Prepare the C API to make PyThreadState opaque: add getter functions #84128

Closed
Closed
[C API] Prepare the C API to make PyThreadState opaque: add getter functions#84128
Labels
3.11only security fixestopic-C-API

Description

@vstinner
vstinner
opened on Mar 12, 2020
BPO 39947
Nosy @vstinner, @fabioz, @markshannon, @ericsnowcurrently, @iritkatriel
PRs
  • bpo-39947: Hide implementation detail of trashcan macros #18971
  • bpo-39947: Move Py_EnterRecursiveCall() to internal C API #18972
  • bpo-39947: Move get_recursion_depth() to _testinternalcapi #18974
  • bpo-39947: Use _PyInterpreterState_GET_UNSAFE() #18978
  • bpo-39947: Add PyInterpreterState_Get() function #18979
  • bpo-39947: Add PyThreadState_GetInterpreter() #18981
  • bpo-39947: Add PyThreadState_GetFrame() function #19092
  • bpo-39947: Use PyThreadState_GetFrame() #19159
  • bpo-39947: Add _PyThreadState_GetDict() #19160
  • bpo-39947: Add PyThreadState_GetID() function #19163
  • Add symbols of the stable ABI to python3dll.c #23598
  • [3.9] bpo-42415: Add symbols of the stable ABI to python3dll.c (GH-23598) #23801
  • bpo-39947: revert incorrect change to a comment #26788
  • bpo-39947: Remove old private trashcan C API functions #26869
  • bpo-39947: Add PyThreadState_SetTrace() function #29121
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2020-03-12.17:46:54.454>
labels = ['expert-C-API', '3.11']
title = '[C API] Make the PyThreadState structure opaque (move it to the internal C API)'
updated_at = <Date 2022-04-06.13:20:11.746>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2022-04-06.13:20:11.746>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API']
creation = <Date 2020-03-12.17:46:54.454>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 39947
keywords = ['patch']
message_count = 25.0
messages = ['364034', '364078', '364080', '364083', '364084', '364103', '364108', '364111', '364113', '364124', '364591', '364674', '365011', '365021', '365022', '370589', '372319', '382551', '383169', '383210', '390054', '396421', '402368', '403816', '416868']
nosy_count = 5.0
nosy_names = ['vstinner', 'fabioz', 'Mark.Shannon', 'eric.snow', 'iritkatriel']
pr_nums = ['18971', '18972', '18974', '18978', '18979', '18981', '19092', '19159', '19160', '19163', '23598', '23801', '26788', '26869', '29121']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue39947'
versions = ['Python 3.11']

    Activity

    1. vstinner commented on Mar 12, 2020

      @vstinner
      vstinner
      on Mar 12, 2020
      MemberAuthor

      Python 3.8 moved PyInterpreterState to the internal C API (commit be3b295 of bpo-35886)... which caused bpo-38500 issue.

      In Python 3.9, I provided Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() as regular functions for the limited API: commit f4b1e3d of bpo-38644. Previously, there were defined as macros, but these macros didn’t compile with the limited C API which cannot access PyThreadState.recursion_depth field (the structure is opaque in the limited C API).

      That's an enhancement for the limited C API, but PyThreadState is still exposed to the "cpython" C API (Include/cpython/).

      We should prepare the C API to make the PyThreadState structure opaque. It cannot be done at once, there are different consumers of the PyThreadState structure. In CPython code base, I found:

      • Py_TRASHCAN_BEGIN_CONDITION and Py_TRASHCAN_END macros access tstate->trash_delete_nesting field. Maybe we can hide these implementation details into new private function calls.

      • faulthandler.c: faulthandler_py_enable() reads tstate->interp. We should maybe provide a getter function.

      • _tracemalloc.c: traceback_get_frames() reads tstate->frame. We should maybe provide a getter function.

      • Private _Py_EnterRecursiveCall() and _Py_LeaveRecursiveCall() access tstate->recursion_depth. One solution is to move these functions to the internal C API.

      faulthandler and _tracemalloc are low-level debugging C extensions. Maybe it's ok for them to use the internal C API. But they are examples of C extensions accessing directly the PyThreadState structure.

      See also bpo-39946 "Is it time to remove _PyThreadState_GetFrame() hook?" about PyThreadState.frame.

    2. vstinner
      added
      3.9 (EOL)end of life
      topic-C-API
      on Mar 12, 2020
    3. vstinner
      changed the title [-]Move PyThreadState structure to the internal C API[/-] [+]Make the PyThreadState structure opaque (move it to the internal C API)[/+] on Mar 12, 2020
    4. vstinner
      changed the title [-]Move PyThreadState structure to the internal C API[/-] [+]Make the PyThreadState structure opaque (move it to the internal C API)[/+] on Mar 12, 2020

    5. vstinner commented on Mar 13, 2020

      @vstinner
      vstinner
      on Mar 13, 2020
      MemberAuthor

      New changeset 224481a by Victor Stinner in branch 'master':
      bpo-39947: Move Py_EnterRecursiveCall() to internal C API (GH-18972)
      224481a

    6. fabioz commented on Mar 13, 2020

      @fabioz
      fabiozmannequin
      on Mar 13, 2020
      Mannequin

      As a note, externally I have to use it in pydevd to set the tracing for different threads -- i.e.: https://bugs.python.org/issue35370

      Will that still be possible?

    7. vstinner commented on Mar 13, 2020

      @vstinner
      vstinner
      on Mar 13, 2020
      MemberAuthor

      As a note, externally I have to use it in pydevd to set the tracing for different threads -- i.e.: https://bugs.python.org/issue35370 Will that still be possible?

      My intent is not to prevent third-party C extension modules to modify PyThreadState, but to make the structure opaque. I mean that we should add getter and setter function for the most commonly used PyThreadState fields.

    8. vstinner commented on Mar 13, 2020

      @vstinner
      vstinner
      on Mar 13, 2020
      MemberAuthor

      New changeset 3f2f4fe by Victor Stinner in branch 'master':
      bpo-39947: Move get_recursion_depth() to _testinternalcapi (GH-18974)
      3f2f4fe

    9. vstinner commented on Mar 13, 2020

      @vstinner
      vstinner
      on Mar 13, 2020
      MemberAuthor

      New changeset 38965ec by Victor Stinner in branch 'master':
      bpo-39947: Hide implementation detail of trashcan macros (GH-18971)
      38965ec

    10. vstinner commented on Mar 13, 2020

      @vstinner
      vstinner
      on Mar 13, 2020
      MemberAuthor

      I tested to build numpy with an opaque PyThreadState. First issue, Plex gets the current interpreter using PyThreadState.interp:

      /tmp/pip-install-aq60p8w2/Cython/Cython/Plex/Scanners.c:7447:73: erreur: déréférencement d'un pointeur du type incomplet « PyThreadState » {alias « struct _ts »}
 7447 |     PY_INT64_T current_id = PyInterpreterState_GetID(PyThreadState_Get()->interp);

      We should add a PyThreadState_GetInterpreter(tstate) getter. faulthandler_py_enable() would use it for example.

      Maybe _PyInterpreterState_Get() can be used, but it's a private function. There are also _PyThreadState_UncheckedGet() and _PyGILState_GetInterpreterStateUnsafe() which are worse: don't check for NULL pointers.

    11. vstinner commented on Mar 13, 2020

      @vstinner
      vstinner
      on Mar 13, 2020
      MemberAuthor

      New changeset ff4584c by Victor Stinner in branch 'master':
      bpo-39947: Use _PyInterpreterState_GET_UNSAFE() (GH-18978)
      ff4584c

    12. vstinner commented on Mar 13, 2020

      @vstinner
      vstinner
      on Mar 13, 2020
      MemberAuthor

      New changeset be79373 by Victor Stinner in branch 'master':
      bpo-39947: Add PyInterpreterState_Get() function (GH-18979)
      be79373

    13. 11 remaining items

    14. vstinner commented on Dec 16, 2020

      @vstinner
      vstinner
      on Dec 16, 2020
      MemberAuthor

      New changeset fcc6935 by Victor Stinner in branch 'master':
      Add symbols of the stable ABI to python3dll.c (GH-23598)
      fcc6935

    15. vstinner commented on Dec 16, 2020

      @vstinner
      vstinner
      on Dec 16, 2020
      MemberAuthor

      New changeset 1662868 by Victor Stinner in branch '3.9':
      Add symbols of the stable ABI to python3dll.c (GH-23598) (GH-23801)
      1662868

    16. vstinner commented on Apr 2, 2021

      @vstinner
      vstinner
      on Apr 2, 2021
      MemberAuthor

      About setting frame local variables, see:

    17. vstinner commented on Jun 23, 2021

      @vstinner
      vstinner
      on Jun 23, 2021
      MemberAuthor

      New changeset db532a0 by Victor Stinner in branch 'main':
      bpo-39947: Remove old private trashcan C API functions (GH-26869)
      db532a0

    18. vstinner commented on Sep 21, 2021

      @vstinner
      vstinner
      on Sep 21, 2021
      MemberAuthor

      See bpo-43760 and PR 28498 for a discussion about the PyThreadState.use_tracing member.

    19. vstinner commented on Oct 13, 2021

      @vstinner
      vstinner
      on Oct 13, 2021
      MemberAuthor

      See also bpo-45247: [C API] Add explicit support for Cython to the C API.

    20. vstinner commented on Apr 6, 2022

      @vstinner
      vstinner
      on Apr 6, 2022
      MemberAuthor

      I would be nice to make the PyThreadState opaque in Python 3.12. IMO it's too late for Python 3.11. Hopefully, Cython should be prepared for such change. At the beginning, maybe Cython can just use the internal C API, as it does to access the internal PyFrameObject structure.

    21. vstinner
      added
      3.11only security fixes
      and removed
      3.9 (EOL)end of life
      on Apr 6, 2022
    22. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    23. vstinner commented on Sep 15, 2022

      @vstinner
      vstinner
      on Sep 15, 2022
      MemberAuthor

      Many PyThreadState structure changes were discussed recently in the issue #87926.

      I created PR #29121 to add PyThreadState_SetProfile() and PyThreadState_SetTrace() functions, but I wasn't sure my implementation would fit the use cases of debuggers and profilers which need these. I abandoned this PR. @pablogsal added PyEval_SetProfileAllThreads() and PyEval_SetTraceAllThreads() functions to Python 3.12 (commit e34c82a) which should fit the use case, with a different design.

    24. vstinner
      closed this as completedon Dec 13, 2022
    25. vstinner
      changed the title [-][C API] Make the PyThreadState structure opaque (move it to the internal C API)[/-] [+][C API] Prepare the C API to make PyThreadState opaque: add getter functions[/+] on Jun 21, 2023
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.11only security fixestopic-C-API

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions