Segfault/abort from calling BytesIO unshare_buffer in threads on a free-threaded build

@vstinner

Crash report

What happened?

This seems to be almost the same issue as #111174, but for free-threaded builds.

In a free-threaded build it's possible to segfault (rare on debug build) or abort (common on debug build) the interpreter with the following code (which might not be minimal, I'll try to reduce it further later today):

from io import BytesIO
from threading import Thread
from time import sleep


def call_getbuffer(obj: BytesIO) -> None:
    obj.getvalue()
    obj.getbuffer()
    obj.getbuffer()
    sleep(0.001)
    obj.getbuffer()
    obj.getbuffer()
    obj.getbuffer()
    sleep(0.006)
    obj.getbuffer()
    obj.getvalue()

for x in range(100):
    alive = []

    obj = BytesIO()
    for x in range(50):
        alive.append(Thread(target=call_getbuffer, args=(obj,)))
        alive.append(Thread(target=call_getbuffer, args=(obj,)))
        alive.append(Thread(target=call_getbuffer, args=(obj,)))

    alive.append(Thread(target=obj.__exit__, args=(None, None, None)))
    alive.append(Thread(target=call_getbuffer, args=(obj,)))
    alive.append(Thread(target=call_getbuffer, args=(obj,)))
    alive.append(Thread(target=call_getbuffer, args=(obj,)))

    for t in alive:
        t.start()
    for t in alive:
        t.join()

Segfault backtrace 1 (gcc, release):

Thread 613 "Thread-612 (cal" received signal SIGSEGV, Segmentation fault.

bytesiobuf_getbuffer (op=0x48ec60f00f0, view=0x48ec61101b0, flags=284) at ./Modules/_io/bytesio.c:1090
1090        if (b->exports == 0 && SHARED_BUF(b)) {

#0  bytesiobuf_getbuffer (op=0x48ec60f00f0, view=0x48ec61101b0, flags=284) at ./Modules/_io/bytesio.c:1090
#1  0x00005555556c6519 in _PyManagedBuffer_FromObject (flags=284, base=0x48ec60f00f0)
    at Objects/memoryobject.c:97
#2  PyMemoryView_FromObjectAndFlags (flags=284, v=0x48ec60f00f0) at Objects/memoryobject.c:813
#3  PyMemoryView_FromObject (v=v@entry=0x48ec60f00f0) at Objects/memoryobject.c:856
#4  0x00005555558b98bf in _io_BytesIO_getbuffer_impl (cls=<optimized out>, self=0x48ebe9f8020)
    at ./Modules/_io/bytesio.c:337
#5  _io_BytesIO_getbuffer (self=0x48ebe9f8020, cls=<optimized out>, args=<optimized out>,
    nargs=<optimized out>, kwnames=<optimized out>) at ./Modules/_io/clinic/bytesio.c.h:103
#6  0x0000555555652cb7 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>,
    args=<optimized out>, callable=0x48ebe3e6ae0, tstate=0x555555c0bb70)
    at ./Include/internal/pycore_call.h:169
#7  PyObject_Vectorcall (callable=0x48ebe3e6ae0, args=<optimized out>, nargsf=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:327
#8  0x00005555555e9486 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>,
    throwflag=<optimized out>) at Python/generated_cases.c.h:3850
#9  0x00005555557d01de in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0x555555c0bb70)
    at ./Include/internal/pycore_ceval.h:119
#10 _PyEval_Vector (tstate=0x555555c0bb70, func=0x48ebe713f00, locals=0x0, args=0x7fff85791958,
    argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1913
#11 0x0000555555656b23 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7fff85791958,
    callable=0x48ebe713f00, tstate=0x555555c0bb70) at ./Include/internal/pycore_call.h:169
#12 method_vectorcall (method=<optimized out>, args=0x7fff85791c68, nargsf=<optimized out>, kwnames=0x0)
    at Objects/classobject.c:72
#13 0x00005555557eeac6 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=0, args=0x7fff85791c68,
    callable=0x48ec60d0100, tstate=0x555555c0bb70) at ./Include/internal/pycore_call.h:169
#14 context_run (self=0x48ebea12fc0, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>)
    at Python/context.c:728
#15 0x00005555555ee959 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>,
    throwflag=<optimized out>) at Python/generated_cases.c.h:3521
#16 0x00005555557d01de in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0x555555c0bb70)
    at ./Include/internal/pycore_ceval.h:119
#17 _PyEval_Vector (tstate=0x555555c0bb70, func=0x48ebe713fc0, locals=0x0, args=0x7fff85791da8,
    argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1913
#18 0x0000555555656b23 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7fff85791da8,
    callable=0x48ebe713fc0, tstate=0x555555c0bb70) at ./Include/internal/pycore_call.h:169
#19 method_vectorcall (method=<optimized out>, args=0x555555b41890 <_PyRuntime+114512>,
    nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:72
#20 0x00005555558f0221 in thread_run (boot_raw=0x555555c07040) at ./Modules/_threadmodule.c:353
#21 0x000055555586c03b in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:242
#22 0x00007ffff7d32ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#23 0x00007ffff7dc4850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Segfault backtrace 2 (clang, debug, ASAN):

Thread 457 "Thread-456 (cal" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff5f1af640 (LWP 1109867)]
PyBytes_AS_STRING (op=0x0) at ./Include/cpython/bytesobject.h:25
25          return _PyBytes_CAST(op)->ob_sval;

#0  PyBytes_AS_STRING (op=0x0) at ./Include/cpython/bytesobject.h:25
#1  bytesiobuf_getbuffer (op=op@entry=<_io._BytesIOBuffer at remote 0x7fffba0e01f0>, view=<optimized out>,
    flags=284) at ./Modules/_io/bytesio.c:1097
#2  0x00005555559dfed1 in PyObject_GetBuffer (obj=obj@entry=<_io._BytesIOBuffer at remote 0x7fffba0e01f0>,
    view=view@entry=0x7fffba170400, flags=0, flags@entry=284) at Objects/abstract.c:445
#3  0x0000555555b69e59 in _PyManagedBuffer_FromObject (
    base=base@entry=<_io._BytesIOBuffer at remote 0x7fffba0e01f0>, flags=flags@entry=284)
    at Objects/memoryobject.c:97
#4  0x0000555555b65759 in PyMemoryView_FromObjectAndFlags (
    v=<_io._BytesIOBuffer at remote 0x7fffba0e01f0>, flags=284) at Objects/memoryobject.c:813
#5  0x0000555556091f8d in _io_BytesIO_getbuffer_impl (self=0x7fffb4afccd0, cls=<optimized out>)
    at ./Modules/_io/bytesio.c:337
#6  _io_BytesIO_getbuffer (self=<_io.BytesIO at remote 0x7fffb4afccd0>, cls=<optimized out>,
    args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>)
    at ./Modules/_io/clinic/bytesio.c.h:103
#7  0x0000555555a4316b in _PyObject_VectorcallTstate (tstate=0x5290011b2210,
    callable=<method_descriptor at remote 0x7fffb45560c0>, args=0x7fffba170400, nargsf=0,
    kwnames=<unknown at remote 0xffff684f6ed>) at ./Include/internal/pycore_call.h:169
#8  0x0000555555dfb5aa in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>,
    throwflag=<optimized out>) at Python/generated_cases.c.h:3850
#9  0x0000555555ddcb03 in _PyEval_EvalFrame (tstate=0x5290011b2210, frame=0x529001176328, throwflag=0)
    at ./Include/internal/pycore_ceval.h:119
#10 _PyEval_Vector (tstate=0x5290011b2210, func=0x7fffb4a9a110, locals=0x0, args=<optimized out>,
    argcount=1, kwnames=0x0) at Python/ceval.c:1913
#11 0x0000555555a4f30b in _PyObject_VectorcallTstate (tstate=0x5290011b2210,
    callable=<function at remote 0x7fffb4a9a110>, args=0x7fffba170400, nargsf=0, nargsf@entry=1,
    kwnames=<unknown at remote 0xffff684f6ed>, kwnames@entry=0x0) at ./Include/internal/pycore_call.h:169
#12 0x0000555555a4cb9f in method_vectorcall (method=<optimized out>, args=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:72
#13 0x0000555555e7f6cb in _PyObject_VectorcallTstate (tstate=0x5290011b2210,
    callable=<method at remote 0x7fffba0c00d0>, args=0x7fff5f1ae6d8, nargsf=0, kwnames=0x0)
    at ./Include/internal/pycore_call.h:169
#14 context_run (self=<_contextvars.Context at remote 0x7fffb4c89070>, args=<optimized out>,
    nargs=<optimized out>, kwnames=0x0) at Python/context.c:728
#15 0x0000555555e09a25 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>,
    throwflag=<optimized out>) at Python/generated_cases.c.h:3521
#16 0x0000555555ddcb03 in _PyEval_EvalFrame (tstate=0x5290011b2210, frame=0x529001176220, throwflag=0)
    at ./Include/internal/pycore_ceval.h:119
#17 _PyEval_Vector (tstate=0x5290011b2210, func=0x7fffb4a9a1f0, locals=0x0, args=<optimized out>,
    argcount=1, kwnames=0x0) at Python/ceval.c:1913
#18 0x0000555555a4f30b in _PyObject_VectorcallTstate (tstate=0x5290011b2210,
    callable=<function at remote 0x7fffb4a9a1f0>, args=0x7fffba170400, nargsf=0, nargsf@entry=1,
    kwnames=<unknown at remote 0xffff684f6ed>, kwnames@entry=0x0) at ./Include/internal/pycore_call.h:169
#19 0x0000555555a4cb9f in method_vectorcall (method=<optimized out>, args=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:72
#20 0x000055555612d7ae in thread_run (boot_raw=boot_raw@entry=0x507000010b80)
    at ./Modules/_threadmodule.c:353
#21 0x0000555555fe705d in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:242
#22 0x000055555585cd47 in asan_thread_start(void*) ()
#23 0x00007ffff7cfeac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#24 0x00007ffff7d90850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Segfault backtrace 3 (this one only happened with code similar to the MRE, not from this exact code):

Thread 562 "Thread-561 (get" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xfffebe06f100 (LWP 2600253)]
_Py_REFCNT (ob=0x0) at ./Include/refcount.h:109
109             uint32_t local = _Py_atomic_load_uint32_relaxed(&ob->ob_ref_local);

Received signal 11, stopping
(gdb) bt
#0  _Py_REFCNT (ob=0x0) at ./Include/refcount.h:109
#1  bytesiobuf_getbuffer (op=<_io._BytesIOBuffer at remote 0x200100a0060>, view=0x200101200e0, flags=284) at ./Modules/_io/bytesio.c:1090
#2  0x0000000000489e80 in PyObject_GetBuffer (obj=obj@entry=<_io._BytesIOBuffer at remote 0x200100a0060>, view=view@entry=0x200101200e0, flags=flags@entry=284)
    at Objects/abstract.c:445
#3  0x00000000005499bc in _PyManagedBuffer_FromObject (base=base@entry=<_io._BytesIOBuffer at remote 0x200100a0060>, flags=flags@entry=284)
    at Objects/memoryobject.c:97
#4  0x0000000000549abc in PyMemoryView_FromObjectAndFlags (v=v@entry=<_io._BytesIOBuffer at remote 0x200100a0060>, flags=flags@entry=284)
    at Objects/memoryobject.c:813
#5  0x000000000054bf1c in PyMemoryView_FromObject (v=v@entry=<_io._BytesIOBuffer at remote 0x200100a0060>) at Objects/memoryobject.c:856
#6  0x00000000007f3794 in _io_BytesIO_getbuffer_impl (self=0x2000691c790, cls=<optimized out>) at ./Modules/_io/bytesio.c:337
#7  0x00000000007f393c in _io_BytesIO_getbuffer (self=<optimized out>, cls=<optimized out>, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>)
    at ./Modules/_io/clinic/bytesio.c.h:103
#8  0x000000000054deac in cfunction_vectorcall_FASTCALL_KEYWORDS_METHOD (func=<builtin_method at remote 0x2000691e5c0>, args=0xb2d058 <_PyRuntime+130520>, 
    nargsf=<optimized out>, kwnames=0x0) at Objects/methodobject.c:486
#9  0x00000000004b994c in _PyVectorcall_Call (tstate=tstate@entry=0xc3d0d0, func=0x54dce0 <cfunction_vectorcall_FASTCALL_KEYWORDS_METHOD>, 
    callable=callable@entry=<builtin_method at remote 0x2000691e5c0>, tuple=tuple@entry=(), kwargs=kwargs@entry={}) at Objects/call.c:273
#10 0x00000000004b9d30 in _PyObject_Call (tstate=0xc3d0d0, callable=callable@entry=<builtin_method at remote 0x2000691e5c0>, args=(), kwargs={}) at Objects/call.c:348
#11 0x00000000004b9d8c in PyObject_Call (callable=callable@entry=<builtin_method at remote 0x2000691e5c0>, args=<optimized out>, kwargs=<optimized out>)
    at Objects/call.c:373
#12 0x00000000006a2f20 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0xc3d0d0, frame=frame@entry=0xffffea6fb128, throwflag=throwflag@entry=0)
    at Python/generated_cases.c.h:2449
#13 0x00000000006d13ec in _PyEval_EvalFrame (throwflag=0, frame=0xffffea6fb128, tstate=0xc3d0d0) at ./Include/internal/pycore_ceval.h:119
#14 _PyEval_Vector (tstate=0xc3d0d0, func=0x20002ada490, locals=locals@entry=0x0, args=0xfffebe06e268, argcount=1, kwnames=0x0) at Python/ceval.c:1913
#15 0x00000000004b6e24 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:413
#16 0x00000000004bc7d8 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0xfffebe06e268, callable=<function at remote 0x20002ada490>, tstate=0xc3d0d0)
    at ./Include/internal/pycore_call.h:169
#17 method_vectorcall (method=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:72
#18 0x00000000006f5c70 in _PyObject_VectorcallTstate (tstate=tstate@entry=0xc3d0d0, callable=<method at remote 0x200100300d0>, args=args@entry=0xfffebe06e538, 
    nargsf=nargsf@entry=0, kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:169
#19 0x00000000006f739c in context_run (self=<_contextvars.Context at remote 0x200054c61f0>, args=0xfffebe06e530, nargs=1, kwnames=0x0) at Python/context.c:728
#20 0x00000000006a7b30 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0xc3d0d0, frame=0xffffea6fb098, frame@entry=0xffffea6fb020, throwflag=throwflag@entry=0)
    at Python/generated_cases.c.h:3509
#21 0x00000000006d13ec in _PyEval_EvalFrame (throwflag=0, frame=0xffffea6fb020, tstate=0xc3d0d0) at ./Include/internal/pycore_ceval.h:119
#22 _PyEval_Vector (tstate=0xc3d0d0, func=0x20002ada570, locals=locals@entry=0x0, args=0xfffebe06e718, argcount=1, kwnames=0x0) at Python/ceval.c:1913
#23 0x00000000004b6e24 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:413
#24 0x00000000004bc7d8 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0xfffebe06e718, callable=<function at remote 0x20002ada570>, tstate=0xc3d0d0)
    at ./Include/internal/pycore_call.h:169

Segfault backtrace 4:

Thread 612 "Thread-611 (cal" received signal SIGSEGV, Segmentation fault.

0x0000555556093f78 in _Py_atomic_load_uint32_relaxed (obj=0xc) at ./Include/cpython/pyatomic_gcc.h:367
367     { return __atomic_load_n(obj, __ATOMIC_RELAXED); }

#0  0x0000555556093f78 in _Py_atomic_load_uint32_relaxed (obj=0xc) at ./Include/cpython/pyatomic_gcc.h:367
#1  _Py_REFCNT (ob=0x0) at ./Include/refcount.h:109
#2  unshare_buffer (self=self@entry=0x7fffb4afc640, size=0) at ./Modules/_io/bytesio.c:115
#3  0x0000555556094b0a in bytesiobuf_getbuffer (op=<_io._BytesIOBuffer at remote 0x7fffce3601f0>,
    view=0x7fffce2c0400, flags=284) at ./Modules/_io/bytesio.c:1091
#4  0x00005555559dfed1 in PyObject_GetBuffer (obj=obj@entry=<_io._BytesIOBuffer at remote 0x7fffce3601f0>,
    view=0x0, view@entry=0x7fffce2c0400, flags=flags@entry=284) at Objects/abstract.c:445
#5  0x0000555555b69e59 in _PyManagedBuffer_FromObject (
    base=base@entry=<_io._BytesIOBuffer at remote 0x7fffce3601f0>, flags=flags@entry=284)
    at Objects/memoryobject.c:97
#6  0x0000555555b65759 in PyMemoryView_FromObjectAndFlags (
    v=<_io._BytesIOBuffer at remote 0x7fffce3601f0>, flags=284) at Objects/memoryobject.c:813
#7  0x0000555556091f8d in _io_BytesIO_getbuffer_impl (self=0x7fffb4afc640, cls=<optimized out>)
    at ./Modules/_io/bytesio.c:337
#8  _io_BytesIO_getbuffer (self=<_io.BytesIO at remote 0x7fffb4afc640>, cls=<optimized out>,
    args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>)
    at ./Modules/_io/clinic/bytesio.c.h:103
#9  0x0000555555a4316b in _PyObject_VectorcallTstate (tstate=0x5290017ca210,
    callable=<method_descriptor at remote 0x7fffb45560c0>, args=0x0, nargsf=284,
    kwnames=<unknown at remote 0xffff684f6ed>) at ./Include/internal/pycore_call.h:169
#10 0x0000555555dfb5aa in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>,
    throwflag=<optimized out>) at Python/generated_cases.c.h:3850
#11 0x0000555555ddcb03 in _PyEval_EvalFrame (tstate=0x5290017ca210, frame=0x529001810328, throwflag=0)
    at ./Include/internal/pycore_ceval.h:119
#12 _PyEval_Vector (tstate=0x5290017ca210, func=0x7fffb4a9a110, locals=0x0, args=<optimized out>,
    argcount=1, kwnames=0x0) at Python/ceval.c:1913
#13 0x0000555555a4f30b in _PyObject_VectorcallTstate (tstate=0x5290017ca210,
    callable=<function at remote 0x7fffb4a9a110>, args=0x0, nargsf=284, nargsf@entry=1,
    kwnames=<unknown at remote 0xffff684f6ed>, kwnames@entry=0x0) at ./Include/internal/pycore_call.h:169
#14 0x0000555555a4cb9f in method_vectorcall (method=<optimized out>, args=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:72
#15 0x0000555555e7f6cb in _PyObject_VectorcallTstate (tstate=0x5290017ca210,
    callable=<method at remote 0x7fffce042410>, args=0x7fff5ecb56d8, nargsf=0, kwnames=0x0)
    at ./Include/internal/pycore_call.h:169
#16 context_run (self=<_contextvars.Context at remote 0x7fffb4c78dd0>, args=<optimized out>,
    nargs=<optimized out>, kwnames=0x0) at Python/context.c:728
#17 0x0000555555e09a25 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>,
    throwflag=<optimized out>) at Python/generated_cases.c.h:3521
#18 0x0000555555ddcb03 in _PyEval_EvalFrame (tstate=0x5290017ca210, frame=0x529001810220, throwflag=0)
    at ./Include/internal/pycore_ceval.h:119
#19 _PyEval_Vector (tstate=0x5290017ca210, func=0x7fffb4a9a1f0, locals=0x0, args=<optimized out>,
    argcount=1, kwnames=0x0) at Python/ceval.c:1913
#20 0x0000555555a4f30b in _PyObject_VectorcallTstate (tstate=0x5290017ca210,
    callable=<function at remote 0x7fffb4a9a1f0>, args=0x0, nargsf=284, nargsf@entry=1,
    kwnames=<unknown at remote 0xffff684f6ed>, kwnames@entry=0x0) at ./Include/internal/pycore_call.h:169
#21 0x0000555555a4cb9f in method_vectorcall (method=<optimized out>, args=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:72
#22 0x000055555612d7ae in thread_run (boot_raw=boot_raw@entry=0x507000014f50)
    at ./Modules/_threadmodule.c:353
#23 0x0000555555fe705d in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:242
#24 0x000055555585cd47 in asan_thread_start(void*) ()
#25 0x00007ffff7cfeac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#26 0x00007ffff7d90850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Abort backtrace:

python: ./Modules/_io/bytesio.c:116: int unshare_buffer(bytesio *, size_t): Assertion `self->exports == 0' failed.
[New Thread 0x7fff6e740640 (LWP 999886)]
[New Thread 0x7fff70553640 (LWP 999887)]
[Thread 0x7fff7347f640 (LWP 999882) exited]

Thread 1349 "Thread-1348 (ca" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fff72c7e640 (LWP 999883)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140735119091264) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140735119091264) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140735119091264) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140735119091264, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7cac476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c927f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c9271b in __assert_fail_base (
    fmt=0x7ffff7e47130 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=0x555556388a20 <str> "self->exports == 0",
    file=0x555556387300 <str> "./Modules/_io/bytesio.c", line=116, function=<optimized out>)
    at ./assert/assert.c:94
#6  0x00007ffff7ca3e96 in __GI___assert_fail (assertion=0x555556388a20 <str> "self->exports == 0",
    file=0x555556387300 <str> "./Modules/_io/bytesio.c", line=line@entry=116,
    function=0x555556388ae0 <__PRETTY_FUNCTION__.unshare_buffer> "int unshare_buffer(bytesio *, size_t)")
    at ./assert/assert.c:103
#7  0x00005555560941df in unshare_buffer (self=self@entry=0x7fffb46ca340, size=<optimized out>)
    at ./Modules/_io/bytesio.c:116
#8  0x0000555556094b0a in bytesiobuf_getbuffer (op=<_io._BytesIOBuffer at remote 0x7fffbc0f0240>,
    view=0x7fffbc1804a0, flags=284) at ./Modules/_io/bytesio.c:1091
#9  0x00005555559dfed1 in PyObject_GetBuffer (obj=obj@entry=<_io._BytesIOBuffer at remote 0x7fffbc0f0240>,
    view=0xf41cb, view@entry=0x7fffbc1804a0, flags=6, flags@entry=284) at Objects/abstract.c:445
#10 0x0000555555b69e59 in _PyManagedBuffer_FromObject (
    base=base@entry=<_io._BytesIOBuffer at remote 0x7fffbc0f0240>, flags=flags@entry=284)
    at Objects/memoryobject.c:97
#11 0x0000555555b65759 in PyMemoryView_FromObjectAndFlags (
    v=<_io._BytesIOBuffer at remote 0x7fffbc0f0240>, flags=284) at Objects/memoryobject.c:813
#12 0x0000555556091f8d in _io_BytesIO_getbuffer_impl (self=0x7fffb46ca340, cls=<optimized out>)
    at ./Modules/_io/bytesio.c:337
#13 _io_BytesIO_getbuffer (self=<_io.BytesIO at remote 0x7fffb46ca340>, cls=<optimized out>,
    args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>)
    at ./Modules/_io/clinic/bytesio.c.h:103
#14 0x0000555555a4316b in _PyObject_VectorcallTstate (tstate=0x5290034c6210,
    callable=<method_descriptor at remote 0x7fffb45560c0>, args=0xf41cb, nargsf=6,
    kwnames=<unknown at remote 0x7fff72c7bd30>) at ./Include/internal/pycore_call.h:169
#15 0x0000555555dfb5aa in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>,
    throwflag=<optimized out>) at Python/generated_cases.c.h:3850
#16 0x0000555555ddcb03 in _PyEval_EvalFrame (tstate=0x5290034c6210, frame=0x52900348f328, throwflag=0)
    at ./Include/internal/pycore_ceval.h:119
#17 _PyEval_Vector (tstate=0x5290034c6210, func=0x7fffb4a9a110, locals=0x0, args=<optimized out>,
    argcount=1, kwnames=0x0) at Python/ceval.c:1913
#18 0x0000555555a4f30b in _PyObject_VectorcallTstate (tstate=0x5290034c6210,
    callable=<function at remote 0x7fffb4a9a110>, args=0xf41cb, nargsf=6, nargsf@entry=1,
    kwnames=<unknown at remote 0x7fff72c7bd30>, kwnames@entry=0x0) at ./Include/internal/pycore_call.h:169
#19 0x0000555555a4cb9f in method_vectorcall (method=<optimized out>, args=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:72
#20 0x0000555555e7f6cb in _PyObject_VectorcallTstate (tstate=0x5290034c6210,
    callable=<method at remote 0x7fffbc0600d0>, args=0x7fff72c7d6d8, nargsf=0, kwnames=0x0)
    at ./Include/internal/pycore_call.h:169
#21 context_run (self=<_contextvars.Context at remote 0x7fffb46773f0>, args=<optimized out>,
    nargs=<optimized out>, kwnames=0x0) at Python/context.c:728
#22 0x0000555555e09a25 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>,
    throwflag=<optimized out>) at Python/generated_cases.c.h:3521
#23 0x0000555555ddcb03 in _PyEval_EvalFrame (tstate=0x5290034c6210, frame=0x52900348f220, throwflag=0)
    at ./Include/internal/pycore_ceval.h:119
#24 _PyEval_Vector (tstate=0x5290034c6210, func=0x7fffb4a9a1f0, locals=0x0, args=<optimized out>,
    argcount=1, kwnames=0x0) at Python/ceval.c:1913
#25 0x0000555555a4f30b in _PyObject_VectorcallTstate (tstate=0x5290034c6210,
    callable=<function at remote 0x7fffb4a9a1f0>, args=0xf41cb, nargsf=6, nargsf@entry=1,
    kwnames=<unknown at remote 0x7fff72c7bd30>, kwnames@entry=0x0) at ./Include/internal/pycore_call.h:169
#26 0x0000555555a4cb9f in method_vectorcall (method=<optimized out>, args=<optimized out>,
    nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:72
#27 0x000055555612d7ae in thread_run (boot_raw=boot_raw@entry=0x5070000291c0)
    at ./Modules/_threadmodule.c:353
#28 0x0000555555fe705d in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:242
#29 0x000055555585cd47 in asan_thread_start(void*) ()
#30 0x00007ffff7cfeac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#31 0x00007ffff7d90850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Rare error message:

Exception ignored while finalizing file <_io.BytesIO object at 0x7fffb46b7f60>:
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored in the internal traceback machinery:
ImportError: sys.meta_path is None, Python is likely shutting down
SystemError: deallocated BytesIO object has exported buffers

Found using fusil by @vstinner.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a7+ experimental free-threading build (heads/main:c7f6535e4a3, Apr 15 2025, 09:09:58) [GCC 11.4.0]

Linked PRs

_io module appears to have not been made free-thread safe yet. I can get this to behave much better (not perfect) by adding @critical_section to _io_BytesIO_getbuffer_impl and _io_BytesIO_getvalue_impl. Unless someone else is actively working on this or has objections I can take this on to fix.

_io is pretty hot, so I'm pretty sure that would do horrid things to performance.

@serhiy-storchaka

cc @serhiy-storchaka

Edit:

_io is pretty hot, so I'm pretty sure that would do horrid things to performance.

But is get_buffer in any hot path?

Ah, wait, unshare_buffer is called in resize_buffer and write_bytes, which I guess are pretty hot indeed.

FYI _BufferedIOBase, TextIOWrapper and StringIO all have critical sections already, and BufferedIOBase is base for BytesIO. So crits here left out on purpose because BytesIO is more super-duper-specialer than the other kids? Or oversight because _BufferedIOBase was supposed to handle it? See #111965. I went ahead and slapped critical sections on everything in this and did some very unscientific timing.

CURRENT: (note empty buffer so just timing overhead)

$ ./python -m timeit --setup "from io import BytesIO; o = BytesIO()" "o.getvalue()"
50000000 loops, best of 5: 9.39 nsec per loop

$ ./python -m timeit --setup "from io import BytesIO; o = BytesIO()" "o.getbuffer()"
5000000 loops, best of 5: 63.3 nsec per loop

CRITS:

$ ./python -m timeit --setup "from io import BytesIO; o = BytesIO()" "o.getvalue()"
20000000 loops, best of 5: 13.4 nsec per loop

./python -m timeit --setup "from io import BytesIO; o = BytesIO()" "o.getbuffer()"
5000000 loops, best of 5: 64 nsec per loop

Some pyperformace benchmarks which MIGHT use BytesIO? Any ideas for better benchmark?

                     CURRENT1   CURRENT2      CRIT1      CRIT2    norm diff

django_template          23.7       23.6       23.9       23.8      -0.0084
docutils                 1.69       1.65       1.73       1.72      -0.0319
json_dumps               7.25       7.35       7.26        7.4      -0.0041
logging_format           4.52       4.42        4.4       4.39       0.0171
logging_simple           4.12       3.97       3.96          4       0.0163
pickle                   6.13       6.14       6.13       6.14       0.0000
pprint_safe_repr          473        472        466        473       0.0064
pprint_pyformat           974        966        954        973       0.0067
telco                     5.2       5.15       5.16       5.18       0.0010
xml_etree_generate       56.3       55.5       55.1       55.3       0.0127

AVERAGE                                                              0.0016

This is a cost of using many little locks instead of one global lock. If this does not affect performance for non free-threaded builds, we should add critical sections for all methods. Correctness first, performance second.

If this does not affect performance for non free-threaded builds, we should add critical sections for all methods. Correctness first, performance second.

Shouldn't, all the critical stuff disappears in GIL build:

$ ./python -m timeit --setup "from io import BytesIO; o = BytesIO()" "o.getvalue()"

CURRENT:  50000000 loops, best of 5: 9.32 nsec per loop
CRITICAL: 50000000 loops, best of 5: 9.28 nsec per loop

$ ./python -m timeit --setup "from io import BytesIO; o = BytesIO()" "o.getbuffer()"

CURRENT:  5000000 loops, best of 5: 62.5 nsec per loop
CRITICAL: 5000000 loops, best of 5: 62.3 nsec per loop

pythongh-132551: make BytesIO object free-thread safe

gh-132551: make `io.BytesIO` thread safe (#132616)

Thanks for the PR!

pythongh-132551: make `io.BytesIO` thread safe (python#132616)

gh-132551: add missing critical sections on BytesIO methods (#137073)

pythongh-132551: make `io.BytesIO` thread safe (python#132616)

pythongh-132551: add missing critical sections on BytesIO methods (py…

[3.14] pythongh-132551: make `io.BytesIO` thread safe (pythonGH-132616)

pythongh-132551: add missing critical sections on BytesIO methods (py…

[3.14] gh-132551: make `io.BytesIO` thread safe (GH-132616) (#138551)

Navigation Menu

Repository navigation

Segfault/abort from calling `BytesIO` `unshare_buffer` in threads on a free-threaded build #132551

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Segfault/abort from calling BytesIO unshare_buffer in threads on a free-threaded build #132551

Description

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Activity

tom-pytel commented on Apr 15, 2025

ZeroIntensity commented on Apr 16, 2025

devdanzin commented on Apr 16, 2025

tom-pytel commented on Apr 16, 2025

serhiy-storchaka commented on Apr 16, 2025

tom-pytel commented on Apr 16, 2025

kumaraditya303 commented on May 8, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Segfault/abort from calling `BytesIO` `unshare_buffer` in threads on a free-threaded build #132551