Skip to content

[C API] The internal C API <pycore_dict.h> cannot be used in C++ #108216

Closed as not planned
Closed as not planned
[C API] The internal C API <pycore_dict.h> cannot be used in C++#108216
Labels
topic-C-APItype-bugAn unexpected behavior, bug, or error

Description

@P403n1x87
P403n1x87
opened on Aug 21, 2023

Bug report

Checklist

  • I am confident this is a bug in CPython, not a bug in a third-party project
  • I have searched the CPython issue tracker,
    and am confident this bug has not been reported before

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

3.12.0rc1

A clear and concise description of the bug:

This looks like a new instance of #67832.

@vstinner This seems to occur in 3.12 when importing e.g. pycore_dict.h in a C++ module https://github.com/P403n1x87/echion/actions/runs/5775912109/job/16065738879

In file included from /opt/hostedtoolcache/Python/3.12.0-rc.1/x64/include/python3.12/internal/pycore_atomic.h:15,
                       from /opt/hostedtoolcache/Python/3.12.0-rc.1/x64/include/python3.12/internal/pycore_runtime.h:12,
                       from /opt/hostedtoolcache/Python/3.12.0-rc.1/x64/include/python3.12/internal/pycore_dict.h:13,
                       from ./echion/mirrors.h:14,
                       from echion/coremodule.cc:29:
      /usr/lib/gcc/x86_64-linux-gnu/9/include/stdatomic.h: At global scope:
      /usr/lib/gcc/x86_64-linux-gnu/9/include/stdatomic.h:40:9: error: ‘_Atomic’ does not name a type
         40 | typedef _Atomic _Bool atomic_bool;
            |         ^~~~~~~

https://github.com/P403n1x87/echion/blob/29275a13aea67a8d008471523f38bc4477499a98/echion/mirrors.h#L14

The same job with clang on OSX doesn't show the same issue: https://github.com/P403n1x87/echion/actions/runs/5775912109/job/16065739714.

Linked PRs

Activity

  1. P403n1x87
    added
    type-bugAn unexpected behavior, bug, or error
    on Aug 21, 2023
  2. P403n1x87
    changed the title [-]Cannot compile C++ extensions with GCC when importing from core[/-] [+]Cannot compile C++ extensions with GCC when including core headers[/+] on Aug 21, 2023

  3. vstinner commented on Aug 21, 2023

    @vstinner
    vstinner
    on Aug 21, 2023
    Member

    This looks like a new instance of #67832.

    This old issue was about including the public C API <Python.h>. This issue is about using the internal C API in C++.

  4. vstinner
    changed the title [-]Cannot compile C++ extensions with GCC when including core headers[/-] [+][C API] The internal C API <pycore_dict.h> cannot be used in C++[/+] on Aug 21, 2023

  5. vstinner commented on Aug 21, 2023

    @vstinner
    vstinner
    on Aug 21, 2023
    Member

    What is your use case for using the internal C API?

    https://github.com/P403n1x87/echion/ description:

    Near-zero-overhead, in-process CPython frame stack sampler with async support

    It looks like a Python profiler.

  6. vstinner
    added
    topic-C-API
    on Aug 21, 2023

  7. vstinner commented on Aug 21, 2023

    @vstinner
    vstinner
    on Aug 21, 2023
    Member

    In Python 3.11, I added test_cppext to check that the Python C API is usable in C++, but only the public C API.

  8. P403n1x87 commented on Aug 21, 2023

    @P403n1x87
    P403n1x87
    on Aug 21, 2023
    ContributorAuthor

    It looks like a Python profiler.

    Indeed it is a profiler (actually just a frame stack sample fwiw). The internal API is used to access internal fields of structures from copies of Python objects so that it can all be done without the GIL

  9. vstinner
    added 2 commits that reference this issue on Aug 21, 2023

    pythongh-108216: Cleanup #include in internal header files

    dff2218

    pythongh-108216: Cleanup #include in internal header files

    b001304

  10. vstinner commented on Aug 21, 2023

    @vstinner
    vstinner
    on Aug 21, 2023
    Member

    With Python 3.11, it's possible to include <pycore_dict.h> in C++? In 3.11, there is no include.

    Python 3.12 added two includes:

    #include "pycore_dict_state.h"
#include "pycore_runtime.h"         // _PyRuntime

    PEP 684: A Per-Interpreter GIL modified DICT_NEXT_VERSION() to access interp->dict_state.global_version: the counter is now "per-interpreter". The problem is that accessing PyInterprepter members requires to include pycore_interp.h is a giant beast: pycore_interp.h includes the problematic pycore_atomic.h.

  11. vstinner
    added 4 commits that reference this issue on Aug 21, 2023

    gh-108216: Cleanup #include in internal header files (#108228)

    0dd3fc2

    pythongh-108216: Add pycore_dict_struct.h internal header file

    7d79868

    pythongh-108216: Add pycore_dict_struct.h internal header file

    18556aa

    pythongh-108216: Add pycore_dict_struct.h internal header file

    bc92115

  12. 2 remaining items

  13. P403n1x87 commented on Aug 21, 2023

    @P403n1x87
    P403n1x87
    on Aug 21, 2023
    ContributorAuthor

    With Python 3.11, it's possible to include <pycore_dict.h> in C++? In 3.11, there is no include.

    Yep I add the import of pycore_dict if PY_VERSION_HEX >= 0x030b0000 and it compiles fine

    https://github.com/P403n1x87/echion/blob/29275a13aea67a8d008471523f38bc4477499a98/echion/mirrors.h#L12-L15

  14. methane commented on Aug 22, 2023

    @methane
    methane
    on Aug 22, 2023
    Member

    Maybe workaround:

    #ifdef HAVE_STD_ATOMIC
#undef HAVE_STD_ATOMIC
#endif
#include <pycore_dict.h>

  15. vstinner commented on Aug 27, 2023

    @vstinner
    vstinner
    on Aug 27, 2023
    Member

    @P403n1x87: Does @methane's workaround work for you? If yes, there is no need to change the header file, right?

  16. P403n1x87 commented on Aug 28, 2023

    @P403n1x87
    P403n1x87
    on Aug 28, 2023
    ContributorAuthor

    @vstinner will try to test today and report

  17. P403n1x87 commented on Aug 28, 2023

    @P403n1x87
    P403n1x87
    on Aug 28, 2023
    ContributorAuthor

    @vstinner @methane's workaround seems to work as it makes all the atomic-related errors disappear

    GH job: https://github.com/P403n1x87/echion/actions/runs/5997079612/job/16262783675?pr=8

    PR: P403n1x87/echion#8

    With that "noise" reduced, I spotted this other seemingly GCC-only issue

          In file included from /opt/hostedtoolcache/Python/3.12.0-rc.1/x64/include/python3.12/objimpl.h:227,
                       from /opt/hostedtoolcache/Python/3.12.0-rc.1/x64/include/python3.12/Python.h:45,
                       from echion/coremodule.cc:6:
      /opt/hostedtoolcache/Python/3.12.0-rc.1/x64/include/python3.12/internal/pycore_gc.h: At global scope:
      /opt/hostedtoolcache/Python/3.12.0-rc.1/x64/include/python3.12/cpython/objimpl.h:85:30: error: ‘int PyObject_GC_IsFinalized(PyObject*)’ was declared ‘extern’ and later ‘static’ [-fpermissive]
         85 | #  define _PyGC_FINALIZED(o) PyObject_GC_IsFinalized(o)
            |                              ^~~~~~~~~~~~~~~~~~~~~~~
      /opt/hostedtoolcache/Python/3.12.0-rc.1/x64/include/python3.12/internal/pycore_gc.h:84:19: note: in expansion of macro ‘_PyGC_FINALIZED’
         84 | static inline int _PyGC_FINALIZED(PyObject *op) {
            |                   ^~~~~~~~~~~~~~~
      In file included from /opt/hostedtoolcache/Python/3.12.0-rc.1/x64/include/python3.12/Python.h:45,
                       from echion/coremodule.cc:6:
      /opt/hostedtoolcache/Python/3.12.0-rc.1/x64/include/python3.12/objimpl.h:209:17: note: previous declaration of ‘int PyObject_GC_IsFinalized(PyObject*)’
        209 | PyAPI_FUNC(int) PyObject_GC_IsFinalized(PyObject *);
            |                 ^~~~~~~~~~~~~~~~~~~~~~~

    but this likely a different issue.

  18. vstinner commented on Aug 28, 2023

    @vstinner
    vstinner
    on Aug 28, 2023
    Member

    I removed # define _PyGC_FINALIZED(o) in the main branch, but my backport to 3.12 was rejected. Just undefine the macro manually.

  19. vstinner commented on Aug 28, 2023

    @vstinner
    vstinner
    on Aug 28, 2023
    Member

    @vstinner @methane's workaround seems to work as it makes all the atomic-related errors disappear

    In that case, I propose to not add pycore_dict_struct.h for now. I suggest you using:

    #include <Python.h>

// Workaround pycore_atomic.h build error:
// https://github.com/python/cpython/issues/108216
#ifdef HAVE_STD_ATOMIC
#  undef HAVE_STD_ATOMIC
#endif

// Workaround _PyGC_FINALIZED() build error on Python <= 3.12:
// https://github.com/python/cpython/issues/105268
#undef _PyGC_FINALIZED

#include <pycore_dict.h>

    Thanks for the feedback. I close the issue.

  20. vstinner
    closed this as completedon Aug 28, 2023
  21. erlend-aasland
    closed this as not plannedon Aug 29, 2023
  22. erlend-aasland
    closed this as completedon Aug 29, 2023

  23. erlend-aasland commented on Aug 29, 2023

    @erlend-aasland
    erlend-aasland
    on Aug 29, 2023
    Contributor

    Ops, sorry for the noise, Victor! I didn't see there was a related PR that fixed some of the issues.

  24. vstinner commented on Aug 29, 2023

    @vstinner
    vstinner
    on Aug 29, 2023
    Member

    This issue was not fixed. Only workaround is offered as a way to avoid the issue 😁

  25. erlend-aasland commented on Aug 29, 2023

    @erlend-aasland
    erlend-aasland
    on Aug 29, 2023
    Contributor

    Aha, so we close it as not planned then! :)

  26. erlend-aasland
    closed this as not plannedon Aug 29, 2023

  27. vstinner commented on Aug 29, 2023

    @vstinner
    vstinner
    on Aug 29, 2023
    Member

    Oh. I tried but apparently failed to do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic-C-APItype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions