Skip to content

C API: Consider adding a public PyList_Extend() function #111138

Closed
Closed
C API: Consider adding a public PyList_Extend() function#111138
Labels
topic-C-APItype-featureA feature request or enhancement

Description

@vstinner
vstinner
opened on Oct 20, 2023

Feature or enhancement

The private _PyList_Extend() function has been removed in Python 3.13: see PR #108451.

@scoder asked what is the replacement for this removed function.

The obvious replacement is PyObject_CallMethod(list, "extend", "O", arg): call the list.extend() method. But it's slower, PyObject_CallMethod() has to get the method and decode the "O" format string.

Another replacement is PyList_SetSlice(L, PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, arg) which is less straightforward, but it's efficient.

I propose adding a public PyList_Extend() function. The list type is commonly used in C extensions, it's a convenient API to create a collection when the length is not known is advance.

I don't think that performance is really the problem here since PyList_SetSlice(L, PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, arg) is available. The problem is more to more the function easier to discover and make the API easier to use. Also, it should help people to migrate away from the removed function.

There is already a public PyDict_Update() API for the dict type, it is part of the limited C API.

If we add a function for the list type, should we also add PySet_Update() "for completeness"? The private _PySet_Update() was removed in Python 3.13.

cc @serhiy-storchaka @encukou

A code search on _PyList_Extend in PyPI top 5,000 projects (2023-07-04) found 4 projects using this function:

  • Cython (0.29.36)
  • catboost (1.2)
  • mypy (1.4.1)
  • pyrsistent (0.19.3)

Logs:

Details 
PYPI-2023-07-04/catboost-1.2.tar.gz: catboost-1.2/catboost_all_src/contrib/tools/cython/Cython/Compiler/Builtin.py: BuiltinMethod("extend",  "TO",   "r", "__Pyx_PyList_Extend",
PYPI-2023-07-04/catboost-1.2.tar.gz: catboost-1.2/catboost_all_src/contrib/tools/cython/Cython/Compiler/ExprNodes.py: extend_func = "__Pyx_PyList_Extend"
PYPI-2023-07-04/catboost-1.2.tar.gz: catboost-1.2/catboost_all_src/contrib/tools/cython/Cython/Utility/Optimize.c: static CYTHON_INLINE int __Pyx_PyList_Extend(PyObject* L, PyObject* v) {
PYPI-2023-07-04/catboost-1.2.tar.gz: catboost-1.2/catboost_all_src/contrib/tools/cython/Cython/Utility/Optimize.c: PyObject* none = _PyList_Extend((PyListObject*)L, v);

PYPI-2023-07-04/Cython-0.29.36.tar.gz: Cython-0.29.36/Cython/Compiler/Builtin.py: BuiltinMethod("extend",  "TO",   "r", "__Pyx_PyList_Extend",
PYPI-2023-07-04/Cython-0.29.36.tar.gz: Cython-0.29.36/Cython/Compiler/ExprNodes.py: extend_func = "__Pyx_PyList_Extend"
PYPI-2023-07-04/Cython-0.29.36.tar.gz: Cython-0.29.36/Cython/Utility/Optimize.c: static CYTHON_INLINE int __Pyx_PyList_Extend(PyObject* L, PyObject* v) {
PYPI-2023-07-04/Cython-0.29.36.tar.gz: Cython-0.29.36/Cython/Utility/Optimize.c: PyObject* none = _PyList_Extend((PyListObject*)L, v);

PYPI-2023-07-04/mypy-1.4.1.tar.gz: mypy-1.4.1/mypyc/lib-rt/dict_ops.c: PyObject *res = _PyList_Extend((PyListObject *)list, view);
PYPI-2023-07-04/mypy-1.4.1.tar.gz: mypy-1.4.1/mypyc/lib-rt/dict_ops.c: PyObject *res = _PyList_Extend((PyListObject *)list, view);
PYPI-2023-07-04/mypy-1.4.1.tar.gz: mypy-1.4.1/mypyc/lib-rt/dict_ops.c: PyObject *res = _PyList_Extend((PyListObject *)list, view);
PYPI-2023-07-04/mypy-1.4.1.tar.gz: mypy-1.4.1/mypyc/lib-rt/list_ops.c: return _PyList_Extend((PyListObject *)o1, o2);

PYPI-2023-07-04/pyrsistent-0.19.3.tar.gz: pyrsistent-0.19.3/pvectorcmodule.c: PyObject *retVal = _PyList_Extend((PyListObject *)self->appendList, args);

Cython uses the function to implement its own __Pyx_PyList_Extend() API:

static CYTHON_INLINE int __Pyx_PyList_Extend(PyObject* L, PyObject* v) {
#if CYTHON_COMPILING_IN_CPYTHON
    PyObject* none = _PyList_Extend((PyListObject*)L, v);
    if (unlikely(!none))
        return -1;
    Py_DECREF(none);
    return 0;
#else
    return PyList_SetSlice(L, PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, v);
#endif
}

Linked PRs

Activity

  1. vstinner
    added
    type-featureA feature request or enhancement
    topic-C-API
    on Oct 20, 2023
  2. vstinner
    changed the title [-]C API: Consider adding a public PyList_Extend() function to replace removed private _PyList_Extend()[/-] [+]C API: Consider adding a public PyList_Extend() function[/+] on Oct 20, 2023

  3. serhiy-storchaka commented on Oct 21, 2023

    @serhiy-storchaka
    serhiy-storchaka
    on Oct 21, 2023
    Member

    PyDict_Update() is not trivial. PyList_Extend() is trivial, literally one-liner.

    #define PyList_Extend(list, arg) PyList_SetSlice((list), PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, (arg))

    I don't mind adding it as a macro. Together with PyList_Clear().

    #define PyList_Clear(list) PyList_SetSlice((list), 0, PY_SSIZE_T_MAX, NULL)
  4. musicinmybrain
    added a commit that references this issue on Oct 26, 2023

    Replace _PyList_Extend with PyList_SetSlice

    5272878

  5. musicinmybrain commented on Oct 26, 2023

    @musicinmybrain
    musicinmybrain
    on Oct 26, 2023

    Just noting that the proposed PyList_Extend macro is not an exact drop-in replacement for the removed _PyList_Extend() function, since the necessary error-checking is a bit different: _PyList_Extend() returns PyObject *, but PyList_SetSlice() returns int.

  6. musicinmybrain
    added a commit that references this issue on Oct 26, 2023

    Replace _PyList_Extend with PyList_SetSlice

    c876adc

  7. vstinner commented on Oct 27, 2023

    @vstinner
    vstinner
    on Oct 27, 2023
    MemberAuthor

    #define PyList_Extend(list, arg) ...

    I dislike macros: they cannot be used in programming languages other than C, and in projects which only load libpython symbols (ex: vim text editor). See for example PEP 670 – Convert macros to functions in the Python C API.

  8. vstinner commented on Oct 27, 2023

    @vstinner
    vstinner
    on Oct 27, 2023
    MemberAuthor

    _PyList_Extend() returns PyObject *

    For a method used in Python, it makes sense. For a C API, it looks inefficient to return None on success.

  9. musicinmybrain commented on Oct 27, 2023

    @musicinmybrain
    musicinmybrain
    on Oct 27, 2023

    For a method used in Python, it makes sense. For a C API, it looks inefficient to return None on success.

    I agree, and I think returning int is the correct signature for a C API.

    It’s just worth pointing out that extension authors porting to Python 3.13 will need to be aware that they cannot simply search-and-replace _PyList_Extend with the proposed PyList_Extend, despite the similar names. tobgu/pyrsistent#284 is a good example.

  10. vstinner
    added 4 commits that reference this issue on Oct 31, 2023

    pythongh-111138: Add test_capi.list tests

    56f91fe

    pythongh-111138: Add test_capi.list tests

    d0356f2

    pythongh-111138: Add test_capi.list tests

    eaf882f

    pythongh-111138: Test the public PyList C API

    89aef64

  11. vstinner commented on Nov 1, 2023

    @vstinner
    vstinner
    on Nov 1, 2023
    MemberAuthor

    First step: I wrote PR #111582 to test the existing public PyList C API.

  12. vstinner
    added a commit that references this issue on Nov 1, 2023

    pythongh-111138: Test the public PyList C API

    d5fd400
  13. vstinner
    added 4 commits that reference this issue on Nov 8, 2023

    pythongh-111138: Test the public PyList C API

    31036e7

    pythongh-111138: Add PyList_Extend() and PyList_Clear() functions

    c9f8d2b

    pythongh-111138: Add PyList_Extend() and PyList_Clear() functions

    e8f1dc7

    pythongh-111138: Add PyList_Extend() and PyList_Clear() functions

    bb0d5a8

  14. vstinner commented on Nov 8, 2023

    @vstinner
    vstinner
    on Nov 8, 2023
    MemberAuthor

    PR gh-111562 added tests on the PyList C API.

    I wrote PR gh-111862 to add PyList_Extend() and PyList_Clear() to the public C API.

  15. vstinner
    added 3 commits that reference this issue on Nov 9, 2023

    pythongh-111138: Add PyList_Extend() and PyList_Clear() functions

    218fe30

    pythongh-111138: Add PyList_Extend() and PyList_Clear() functions

    751158a

    gh-111138: Add PyList_Extend() and PyList_Clear() functions (#111862)

    babb787

  16. vstinner commented on Nov 13, 2023

    @vstinner
    vstinner
    on Nov 13, 2023
    MemberAuthor

    I close the issue.

    Commit babb787 adds PyList_Extend() and PyList_Clear(). I wrote python/pythoncapi-compat#80 to add these functions to pythoncapi-compat.

    @serhiy-storchaka's macros can also be used on Python 3.12 and older:

    #define PyList_Extend(list, arg) PyList_SetSlice((list), PY_SSIZE_T_MAX, PY_SSIZE_T_MAX, (arg))
#define PyList_Clear(list) PyList_SetSlice((list), 0, PY_SSIZE_T_MAX, NULL)

    If we add a function for the list type, should we also add PySet_Update() "for completeness"? The private _PySet_Update() was removed in Python 3.13.

    I didn't add this function.

  17. vstinner
    closed this as completedon Nov 13, 2023
  18. aisk
    added a commit that references this issue on Feb 11, 2024

    pythongh-111138: Add PyList_Extend() and PyList_Clear() functions (py…

    a5bb657
  19. hauntsaninja
    added a commit that references this issue on Jul 7, 2024

    Fix `_PyList_Extend` for Python 3.13 (#17503)

    9175ce5
  20. Glyphack
    added a commit that references this issue on Sep 2, 2024

    pythongh-111138: Add PyList_Extend() and PyList_Clear() functions (py…

    2b3832f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic-C-APItype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions