Skip to content

[C API] Support the limited C API in debug mode (Py_INCREF and Py_DECREF) #87854

Closed
Closed
[C API] Support the limited C API in debug mode (Py_INCREF and Py_DECREF)#87854
Labels
3.10 (EOL)end of lifetopic-C-API

Description

@vstinner
vstinner
opened on Apr 1, 2021
BPO 43688
Nosy @vstinner, @encukou
PRs
  • bpo-43688: Support the limited C API in debug mode #25131
  • bpo-43688: Support "make regen-limited-abi" in debug mode #25133
  • bpo-43688: Run make regen-limited-abi #25134
  • bpo-43688: Fix Py_LIMITED_API version of xxlimited #25135
    		•
    Files
  • bench_limited.py
  • bench_testcapi.py
  • bench.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-05-26.22:20:07.057>
created_at = <Date 2021-04-01.06:47:31.015>
labels = ['expert-C-API', '3.10']
title = '[C API] Support the limited C API in debug mode (Py_INCREF and Py_DECREF)'
updated_at = <Date 2021-05-26.22:20:07.056>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2021-05-26.22:20:07.056>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-26.22:20:07.057>
closer = 'vstinner'
components = ['C API']
creation = <Date 2021-04-01.06:47:31.015>
creator = 'vstinner'
dependencies = []
files = ['49925', '49926', '49927']
hgrepos = []
issue_num = 43688
keywords = ['patch']
message_count = 12.0
messages = ['389956', '389960', '389962', '389971', '389982', '389983', '390002', '390057', '390062', '390063', '390067', '394478']
nosy_count = 2.0
nosy_names = ['vstinner', 'petr.viktorin']
pr_nums = ['25131', '25133', '25134', '25135']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43688'
versions = ['Python 3.10']

    Activity

    1. vstinner commented on Apr 1, 2021

      @vstinner
      vstinner
      on Apr 1, 2021
      MemberAuthor

      Currently, setup.py doesn't build xxlimited and xxlimited_35 extension modules with the limited C API if Python is built in debug mode. I only found two functions affected by Py_DEBUG macro in the limited C API: Py_INCREF() and Py_DECREF().

      Example:

      #if defined(Py_DEBUG) && !defined(Py_REF_DEBUG)
#define Py_REF_DEBUG
#endif

static inline void _Py_INCREF(PyObject *op)
{
#ifdef Py_REF_DEBUG
    _Py_RefTotal++;
#endif
    op->ob_refcnt++;
}
#define Py_INCREF(op) _Py_INCREF(_PyObject_CAST(op))

      If Py_DEBUG is defined (Python built in debug mode), Py_INCREF() increments the private _Py_RefTotal variable.

      The limited C API is supposed to provide a stable ABI, but Py_INCREF() leaks _Py_RefTotal implementation if Python is built in debug mode.

      I propose to modify Py_INCREF() and Py_DECREF() of the limited C API to always declare them as opaque function calls. The regular (non limited) C API will continue to use the same static inline functions.

      See also #25115 and bpo-41111 "[C API] Convert a few stdlib extensions to the limited C API (PEP-384)".

      Note: Py_XINCREF() and Py_XDECREF() should be fine.

    2. vstinner
      added
      3.10 (EOL)end of life
      topic-C-API
      on Apr 1, 2021
    3. vstinner
      changed the title [-][C API] Fix Py_INCREF and Py_DECREF in the limited C API for Python built in debug mode[/-] [+][C API] Support the limited C API in debug build (Py_INCREF and Py_DECREF)[/+] on Apr 1, 2021
    4. vstinner
      changed the title [-][C API] Fix Py_INCREF and Py_DECREF in the limited C API for Python built in debug mode[/-] [+][C API] Support the limited C API in debug build (Py_INCREF and Py_DECREF)[/+] on Apr 1, 2021
    5. vstinner
      changed the title [-][C API] Support the limited C API in debug build (Py_INCREF and Py_DECREF)[/-] [+][C API] Support the limited C API in debug mode (Py_INCREF and Py_DECREF)[/+] on Apr 1, 2021
    6. vstinner
      changed the title [-][C API] Support the limited C API in debug build (Py_INCREF and Py_DECREF)[/-] [+][C API] Support the limited C API in debug mode (Py_INCREF and Py_DECREF)[/+] on Apr 1, 2021

    7. vstinner commented on Apr 1, 2021

      @vstinner
      vstinner
      on Apr 1, 2021
      MemberAuthor

      One nice advantage of this change is that it becomes possible to check for reference leaks in C extensions written with the limited C API!

      $ ./python -m test test_xxlimited -R 3:3
0:00:00 load avg: 2.99 Run tests sequentially
0:00:00 load avg: 2.99 [1/1] test_xxlimited
beginning 6 repetitions
123456
......

      == Tests result: SUCCESS ==

      1 test OK.

      Total duration: 367 ms
      Tests result: SUCCESS

    8. vstinner commented on Apr 1, 2021

      @vstinner
      vstinner
      on Apr 1, 2021
      MemberAuthor

      New changeset baf10da by Victor Stinner in branch 'master':
      bpo-43688: Run make regen-limited-abi (GH-25134)
      baf10da

    9. encukou commented on Apr 1, 2021

      @encukou
      encukou
      on Apr 1, 2021
      Member

      If you do this, please check the performance impact. Py_INCREF/Py_DECREF are very common.

    10. vstinner commented on Apr 1, 2021

      @vstinner
      vstinner
      on Apr 1, 2021
      MemberAuthor

      New changeset 2ac0515 by Victor Stinner in branch 'master':
      bpo-43688: Fix Py_LIMITED_API version of xxlimited (GH-25135)
      2ac0515

    11. vstinner commented on Apr 1, 2021

      @vstinner
      vstinner
      on Apr 1, 2021
      MemberAuthor

      New changeset baf10da by Victor Stinner in branch 'master':
      bpo-43688: Run make regen-limited-abi (GH-25134)

      With this change, "Tests / Check if generated files are up to date" job started fails (on PR 25135): "Some symbols from the limited API are missing: PyType_HasFeature".

      I fixed this issue in bpo-43690 which removes PyType_HasFeature from Doc/data/stable_abi.dat.

    12. vstinner commented on Apr 1, 2021

      @vstinner
      vstinner
      on Apr 1, 2021
      MemberAuthor

      I wrote a microbenchmark on Py_INCREF()+Py_DECREF():

      $ python3 -m pyperf compare_to ref.json limited.json 
Mean +- std dev: [ref] 3.45 ns +- 0.17 ns -> [limited] 6.03 ns +- 0.21 ns: 1.75x slower

      If a function is only made of Py_INCREF() and Py_DECREF(), it can be up to 1.8x slower in the worst case. But in practice, I don't think that functions are only made of Py_INCREF() and Py_DECREF(). They do a few other things.

      I'm not sure how to run a "macro benchmark" on my PR 25131, since I don't know any C extension doing anything useful. There is xxlimited, but it does almost nothing.

      What would be a fair benchmark for this change?

      --

      To run my microbenchmark:

      git apply bench.patch
      ./configure --with-lto --enable-optimizations
      make
      ./python -m venv env
      ./env/bin/python -m pip install pyperf
      ./env/bin/python ../bench_limited.py -o ../limited.json -v
      ./env/bin/python ../bench_testcapi.py -o ../ref.json -v

    13. vstinner commented on Apr 2, 2021

      @vstinner
      vstinner
      on Apr 2, 2021
      MemberAuthor

      I modified my PR 25131 to only implement Py_INCREF/Py_DECREF as opaque function calls under two conditions:

      • Python is built in debug mode
      • Py_LIMITED_API macro targets Python 3.10 or newer

      So this PR 25131 now only has an impact on performance if Python is built in debug mode. Performance on Python built in release mode is not affected.

    14. vstinner commented on Apr 2, 2021

      @vstinner
      vstinner
      on Apr 2, 2021
      MemberAuthor

      In release mode, I get the libraries:

      • xxlimited.cpython-310-x86_64-linux-gnu.so
      • xxlimited_35.cpython-310-x86_64-linux-gnu.so

      In debug mode, I get the libraries:

      • xxlimited.cpython-310d-x86_64-linux-gnu.so
      • xxlimited_35.cpython-310d-x86_64-linux-gnu.so

      It's still a different ABI: "cpython-310" vs "cpython-310d" ("D" for debug). So there is risk to be confused between the stable ABI in release mode and the stable ABI in debug mode :-) I don't how how to get the ".abi3.so" suffix.

    15. vstinner commented on Apr 2, 2021

      @vstinner
      vstinner
      on Apr 2, 2021
      MemberAuthor

      New changeset 3359cab by Victor Stinner in branch 'master':
      bpo-43688: Support the limited C API in debug mode (GH-25131)
      3359cab

    16. vstinner commented on Apr 2, 2021

      @vstinner
      vstinner
      on Apr 2, 2021
      MemberAuthor

      New changeset 9bb5658 by Victor Stinner in branch 'master':
      bpo-43688: Support "make regen-limited-abi" in debug mode (GH-25133)
      9bb5658

    17. vstinner commented on May 26, 2021

      @vstinner
      vstinner
      on May 26, 2021
      MemberAuthor

      Thanks for the help Petr. I close the issue, it's now fixed!

    18. vstinner
      closed this as completedon May 26, 2021
    19. vstinner
      closed this as completedon May 26, 2021
    20. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of lifetopic-C-API

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions