Skip to content

test_threading.test_recursion_limit() does crash (segfault) on Alpine Linux (musl) #148260

Closed
Closed
test_threading.test_recursion_limit() does crash (segfault) on Alpine Linux (musl)#148260
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)testsTests in the Lib/test dirtopic-free-threading

Description

@vstinner
vstinner
opened on Apr 8, 2026

Python built with ./configure --with-pydebug --disable-gil and gcc -Og on Alpine Linux (Python built with musl C library).

Example of crash on "AMD64 Alpine Linux NoGIL 3.x" buildbot: https://buildbot.python.org/#/builders/1819/builds/1699

Script:

import threading

def recurse():
    return recurse()

def outer():
    try:
        recurse()
    except RecursionError:
        pass

w = threading.Thread(target=outer)
w.start()
w.join()
print('end of main thread')

gdb traceback:

(gdb) where
#0  0x0000560e9bedb871 in _Py_Dealloc (op=0x20004156010) at Objects/object.c:3277
#1  0x0000560e9bedb9ef in _Py_MergeZeroLocalRefcount (op=op@entry=0x20004156010) at Objects/object.c:444
#2  0x0000560e9c088aaf in Py_DECREF (filename=filename@entry=0x560e9c12425b "./Include/refcount.h", 
    lineno=lineno@entry=520, op=0x20004156010) at ./Include/refcount.h:359
#3  0x0000560e9c088af7 in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:520
#4  0x0000560e9c088e30 in tb_dealloc (op=0x20004156070) at Python/traceback.c:245
#5  0x0000560e9bedb8e0 in _Py_Dealloc (op=0x20004156070) at Objects/object.c:3300
#6  0x0000560e9bedb9ef in _Py_MergeZeroLocalRefcount (op=op@entry=0x20004156070) at Objects/object.c:444
#7  0x0000560e9c088aaf in Py_DECREF (filename=filename@entry=0x560e9c12425b "./Include/refcount.h", 
    lineno=lineno@entry=520, op=0x20004156070) at ./Include/refcount.h:359
#8  0x0000560e9c088af7 in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:520
#9  0x0000560e9c088e30 in tb_dealloc (op=0x200041560d0) at Python/traceback.c:245
#10 0x0000560e9bedb8e0 in _Py_Dealloc (op=0x200041560d0) at Objects/object.c:3300
#11 0x0000560e9bedb9ef in _Py_MergeZeroLocalRefcount (op=op@entry=0x200041560d0) at Objects/object.c:444
#12 0x0000560e9c088aaf in Py_DECREF (filename=filename@entry=0x560e9c12425b "./Include/refcount.h", 
    lineno=lineno@entry=520, op=0x200041560d0) at ./Include/refcount.h:359
#13 0x0000560e9c088af7 in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:520
#14 0x0000560e9c088e30 in tb_dealloc (op=0x20004156130) at Python/traceback.c:245
#15 0x0000560e9bedb8e0 in _Py_Dealloc (op=0x20004156130) at Objects/object.c:3300
#16 0x0000560e9bedb9ef in _Py_MergeZeroLocalRefcount (op=op@entry=0x20004156130) at Objects/object.c:444
#17 0x0000560e9c088aaf in Py_DECREF (filename=filename@entry=0x560e9c12425b "./Include/refcount.h", 
    lineno=lineno@entry=520, op=0x20004156130) at ./Include/refcount.h:359
#18 0x0000560e9c088af7 in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:520
#19 0x0000560e9c088e30 in tb_dealloc (op=0x20004156190) at Python/traceback.c:245
#20 0x0000560e9bedb8e0 in _Py_Dealloc (op=0x20004156190) at Objects/object.c:3300
#21 0x0000560e9bedb9ef in _Py_MergeZeroLocalRefcount (op=op@entry=0x20004156190) at Objects/object.c:444
#22 0x0000560e9c088aaf in Py_DECREF (filename=filename@entry=0x560e9c12425b "./Include/refcount.h", 
    lineno=lineno@entry=520, op=0x20004156190) at ./Include/refcount.h:359
#23 0x0000560e9c088af7 in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:520
#24 0x0000560e9c088e30 in tb_dealloc (op=0x200041561f0) at Python/traceback.c:245
#25 0x0000560e9bedb8e0 in _Py_Dealloc (op=0x200041561f0) at Objects/object.c:3300
#26 0x0000560e9bedb9ef in _Py_MergeZeroLocalRefcount (op=op@entry=0x200041561f0) at Objects/object.c:444
#27 0x0000560e9c088aaf in Py_DECREF (filename=filename@entry=0x560e9c12425b "./Include/refcount.h", 
    lineno=lineno@entry=520, op=0x200041561f0) at ./Include/refcount.h:359
#28 0x0000560e9c088af7 in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:520
#29 0x0000560e9c088e30 in tb_dealloc (op=0x20004156250) at Python/traceback.c:245
#30 0x0000560e9bedb8e0 in _Py_Dealloc (op=0x20004156250) at Objects/object.c:3300
#31 0x0000560e9bedb9ef in _Py_MergeZeroLocalRefcount (op=op@entry=0x20004156250) at Objects/object.c:444
#32 0x0000560e9c088aaf in Py_DECREF (filename=filename@entry=0x560e9c12425b "./Include/refcount.h", 
(...)
#3751 0x0000560e9be774fb in PyObject_Call (callable=<optimized out>, args=<optimized out>, kwargs=<optimized out>)
    at Objects/call.c:373
#3752 0x0000560e9c10f6ef in thread_run (boot_raw=boot_raw@entry=0x20000030920) at ./Modules/_threadmodule.c:388
#3753 0x0000560e9c0874c0 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:234
#3754 0x00007f484b327573 in start (p=<optimized out>) at src/thread/pthread_create.c:207
#3755 0x00007f484b328ec1 in __clone () at src/thread/x86_64/clone.s:22
Backtrace stopped: frame did not save the PC

(gdb) info threads
  Id   Target Id                  Frame 
  1    LWP 9804 "python"          __cp_end () at src/thread/x86_64/syscall_cp.s:29
* 2    LWP 9807 "Thread-1 (outer" 0x0000560e9bedb871 in _Py_Dealloc (op=0x20004156010) at Objects/object.c:3277

_Py_Dealloc() has the Trashcan mechanism which is supposed to avoid long chain of _Py_Dealloc() calls to reduce the stack memory consumption. But it seems like it doesn't work on Alpine Linux (Python built with musl C library).

gdb variables in the _Py_Dealloc() frame:

(gdb) p type->tp_name
$6 = 0x555ba66f5c7f "traceback"

(gdb) p gc_flag
$7 = 16384

In the parent _Py_Dealloc() frame:

(gdb) p type->tp_name
$9 = 0x555ba66f5c7f "traceback"
(gdb) p gc_flag
$10 = <optimized out>
(gdb) p margin
$11 = 115

In the first _Py_Dealloc() frame (outer):

(gdb) p margin
$12 = 119
(gdb) p type->tp_name
$13 = 0x555ba66e9dcd "RecursionError"
(gdb) p gc_flag
$14 = <optimized out>
(gdb) p type->tp_flags & (1UL << 14)  # gc_flag
$15 = 16384

On a regular build (without Free Threading), the script works as expected, it displays:

end of main thread

Linked PRs

Activity

  1. vstinner
    added
    testsTests in the Lib/test dir
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    topic-free-threading
    on Apr 8, 2026

  2. vstinner commented on Apr 8, 2026

    @vstinner
    vstinner
    on Apr 8, 2026
    MemberAuthor

    Ah, I can also trigger the bug on a regular build (not Free Threaded) on Alpine Linux, Python built with: ./configure --with-pydebug and gcc -Og. I just have to increase the recursion limit to 2 000 frames:

    import threading
import sys

def recurse():
    return recurse()

def outer():
    try:
        recurse()
    except RecursionError:
        pass

sys.setrecursionlimit(2_000)
w = threading.Thread(target=outer)
w.start()
w.join()
print('end of main thread')

    I get a similar gdb traceback:

    #0  0x0000560e899e7155 in _PyObject_GC_UNTRACK (filename=filename@entry=0x560e89af12ee "Python/gc.c", 
    lineno=lineno@entry=2382, op=op@entry=0x7f5d79f06170) at ./Include/internal/pycore_gc.h:279
#1  0x0000560e899e9442 in PyObject_GC_UnTrack (op_raw=op_raw@entry=0x7f5d79f06170) at Python/gc.c:2382
#2  0x0000560e89a49d11 in tb_dealloc (op=0x7f5d79f06170) at Python/traceback.c:244
#3  0x0000560e898a60b6 in _Py_Dealloc (op=op@entry=0x7f5d79f06170) at Objects/object.c:3300
#4  0x0000560e89a49b28 in Py_DECREF (filename=filename@entry=0x560e89ade25b "./Include/refcount.h", 
    lineno=lineno@entry=520, op=0x7f5d79f06170) at ./Include/refcount.h:410
#5  0x0000560e89a49b47 in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:520
#6  0x0000560e89a49d1a in tb_dealloc (op=0x7f5d79f061c0) at Python/traceback.c:245
#7  0x0000560e898a60b6 in _Py_Dealloc (op=op@entry=0x7f5d79f061c0) at Objects/object.c:3300
#8  0x0000560e89a49b28 in Py_DECREF (filename=filename@entry=0x560e89ade25b "./Include/refcount.h", 
    lineno=lineno@entry=520, op=0x7f5d79f061c0) at ./Include/refcount.h:410
#9  0x0000560e89a49b47 in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:520
#10 0x0000560e89a49d1a in tb_dealloc (op=0x7f5d79f06210) at Python/traceback.c:245
#11 0x0000560e898a60b6 in _Py_Dealloc (op=op@entry=0x7f5d79f06210) at Objects/object.c:3300
#12 0x0000560e89a49b28 in Py_DECREF (filename=filename@entry=0x560e89ade25b "./Include/refcount.h", 
    lineno=lineno@entry=520, op=0x7f5d79f06210) at ./Include/refcount.h:410
#13 0x0000560e89a49b47 in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:520
#14 0x0000560e89a49d1a in tb_dealloc (op=0x7f5d79f06260) at Python/traceback.c:245
#15 0x0000560e898a60b6 in _Py_Dealloc (op=op@entry=0x7f5d79f06260) at Objects/object.c:3300
#16 0x0000560e89a49b28 in Py_DECREF (filename=filename@entry=0x560e89ade25b "./Include/refcount.h", 
    lineno=lineno@entry=520, op=0x7f5d79f06260) at ./Include/refcount.h:410
#17 0x0000560e89a49b47 in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:520
#18 0x0000560e89a49d1a in tb_dealloc (op=0x7f5d79f062b0) at Python/traceback.c:245
#19 0x0000560e898a60b6 in _Py_Dealloc (op=op@entry=0x7f5d79f062b0) at Objects/object.c:3300
#20 0x0000560e89a49b28 in Py_DECREF (filename=filename@entry=0x560e89ade25b "./Include/refcount.h", 
    lineno=lineno@entry=520, op=0x7f5d79f062b0) at ./Include/refcount.h:410
#21 0x0000560e89a49b47 in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:520
#22 0x0000560e89a49d1a in tb_dealloc (op=0x7f5d79f06300) at Python/traceback.c:245
#23 0x0000560e898a60b6 in _Py_Dealloc (op=op@entry=0x7f5d79f06300) at Objects/object.c:3300
#24 0x0000560e89a49b28 in Py_DECREF (filename=filename@entry=0x560e89ade25b "./Include/refcount.h", 
    lineno=lineno@entry=520, op=0x7f5d79f06300) at ./Include/refcount.h:410
#25 0x0000560e89a49b47 in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:520
#26 0x0000560e89a49d1a in tb_dealloc (op=0x7f5d79f06350) at Python/traceback.c:245
#27 0x0000560e898a60b6 in _Py_Dealloc (op=op@entry=0x7f5d79f06350) at Objects/object.c:3300
...
#4128 0x0000560e8984887b in PyObject_Call (callable=<optimized out>, args=<optimized out>, kwargs=<optimized out>)
    at Objects/call.c:373
#4129 0x0000560e89aca755 in thread_run (boot_raw=boot_raw@entry=0x7f5d7a1cdd50) at ./Modules/_threadmodule.c:388
#4130 0x0000560e89a485c6 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:234
#4131 0x00007f5d7a3e9573 in start (p=<optimized out>) at src/thread/pthread_create.c:207
#4132 0x00007f5d7a3eaec1 in __clone () at src/thread/x86_64/clone.s:22
Backtrace stopped: frame did not save the PC

    First _Py_Dealloc() call:

    (gdb) p type->tp_name
$1 = 0x560e89ae4d03 "RecursionError"
(gdb) p type->tp_flags & (1UL << 14)  # gc_flag
$3 = 16384
(gdb) p margin
$4 = 119

    Last _Py_Dealloc() call:

    (gdb) p type->tp_name
$5 = 0x560e89af0a40 "traceback"
(gdb) p type->tp_flags & (1UL << 14)  # gc_flag
$7 = 16384
(gdb) p margin
$8 = 115
  3. vstinner
    changed the title [-]test_threading.test_recursion_limit() does crash (segfault) on Alpine with Free Threaded build[/-] [+]test_threading.test_recursion_limit() does crash (segfault) on Alpine Linux (musl)[/+] on Apr 8, 2026

  4. vstinner commented on Apr 8, 2026

    @vstinner
    vstinner
    on Apr 8, 2026
    MemberAuthor

    https://wiki.musl-libc.org/functional-differences-from-glibc.html says:

    musl provides a default thread stack size of 128k

    On my Linux x86-64 (Fedora 43) with glibc, I get a default stack size of 8 MiB.

    We can add -Wl,-z,stack-size=1048576 flag to LDFLAGS_NODIST or LINKFORSHARED to increase the thread stack size to 1 MiB.

    See also commit 1e9b8f2 (PR gh-134336) which modified Python/ceval.c to no longer use pthread_attr_getstack() on musl.

  5. vstinner commented on Apr 8, 2026

    @vstinner
    vstinner
    on Apr 8, 2026
    MemberAuthor

    I tried using $ac_cv_libc variable in configure.ac, but this variable is unset: #143730 (comment).

  6. vstinner commented on May 7, 2026

    @vstinner
    vstinner
    on May 7, 2026
    MemberAuthor

    It doesn't seem possible to detect that Python will be linked to musl C library. Maybe another option would be to query the thread stack size to decide to pass -Wl,-z,stack-size=1048576 option to the linker or not.

    Example of C program which returns exit code 1 if the thread stack size is smaller than 1 MiB:

    #include <pthread.h>
#include <assert.h>

size_t get(void)
{
    pthread_attr_t attrs;
    size_t size;
    int rc = pthread_attr_init(&attrs);
    assert(rc == 0);
    rc = pthread_attr_getstacksize(&attrs, &size);
    assert(rc == 0);
    return size;
}

int main()
{
    size_t size = get();
    if (size < 1024 * 1024) {
        return 1;
    }
    return 0;
}

  7. vstinner commented on May 18, 2026

    @vstinner
    vstinner
    on May 18, 2026
    MemberAuthor

    I wrote PR gh-149993 which adds -Wl,-z,stack-size=1048576 linker flag when Python is linked to the musl C library.

  8. vstinner
    added a commit that references this issue on May 20, 2026

    gh-148260: Use at least 1 MiB stack size on musl (#149993)

    df6c157
  9. vstinner
    closed this as completedon May 20, 2026
  10. vstinner
    added a commit that references this issue on May 20, 2026

    [3.15] gh-148260: Use at least 1 MiB stack size on musl (GH-149993) (#…

    c555e26

  11. vstinner commented on May 20, 2026

    @vstinner
    vstinner
    on May 20, 2026
    MemberAuthor

    I increased the minimum thread stack size to 1 MiB, whereas musl uses 128 kiB by default. If 1 MiB is not used (glibc uses 8 MiB on Linux!), we can increase the stack size later.

  12. vstinner
    added a commit that references this issue on Jun 17, 2026

    [3.14] gh-148260: Use at least 1 MiB stack size on musl (GH-149993) (#…

    5516672
  13. nvuillam
    added a commit that references this issue on Aug 22, 2026

    Fix random Segmentation fault crashes on Alpine images (#8733) (#8779)

    403219a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    interpreter-core(Objects, Python, Grammar, and Parser dirs)testsTests in the Lib/test dirtopic-free-threading

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions