Skip to content

Crash Due to Exception in threading._shutdown() #113148

Closed
Closed
Crash Due to Exception in threading._shutdown()#113148
Labels
3.12only security fixes3.13only security fixes3.14bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-subinterpreterstype-crashA hard crash of the interpreter, possibly with a core dump

Description

@ericsnowcurrently
ericsnowcurrently
opened on Dec 14, 2023

One of the first thing that happens during interpreter finalization is waiting for all non-daemon threads to finish. This is implemented by calling threading._shutdown(). If an exception is raised there (e.g. in a threading "atexit" function), it gets ignored and we end up with non-daemon threads still running. In subinterpreters that results in a fatal error almost immediately after, since we make sure there's only one thread state left at that point.

Reproducer: 
$ cat > /tmp/crash-interp-lingering-thread.py << EOF
import threading
from test.support import interpreters

interp = interpreters.create()
interp.exec_sync(f"""if True:
    import threading
    import time

    done = False

    def notify_fini():
        global done
        done = True
        raise Exception  # <-------
        t.join()
    threading._register_atexit(notify_fini)

    def task():
        while not done:
            time.sleep(0.1)
    t = threading.Thread(target=task)
    t.start()
""")
interp.close()


EOF
$ ./python /tmp/crash-interp-lingering-thread.py
Exception ignored on threading shutdown:
Traceback (most recent call last):
  File "./cpython/Lib/threading.py", line 1623, in _shutdown
    atexit_call()
  File "<string>", line 10, in notify_fini
Exception:
Fatal Python error: Py_EndInterpreter: not the last thread
Python runtime state: initialized

Current thread 0x00007fc8bf76a100 (most recent call first):
  <no Python frame>

Thread 0x00007fc8bd3d5700 (most recent call first):
  File "<string>", line 16 in task
  File "./cpython/Lib/threading.py", line 1020 in run
  File "./cpython/Lib/threading.py", line 1083 in _bootstrap_inner
  File "./cpython/Lib/threading.py", line 1040 in _bootstrap
Aborted (core dumped)

The solution? Make sure threading._shutdown() finishes its fundamental job: stop all non-daemon threads and wait for them to finish. At the very least, this means ignoring exceptions from functions registered with threading._register_atexit().

Linked PRs

Activity

  1. ericsnowcurrently
    added
    type-bugAn unexpected behavior, bug, or error
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    topic-subinterpreters
    3.12only security fixes
    3.13only security fixes
    on Dec 14, 2023
  2. github-project-automation
    added this to Subinterpreterson Dec 14, 2023
  3. github-project-automation
    moved this to Todo in Subinterpreterson Dec 14, 2023
  4. ericsnowcurrently
    added
    type-crashA hard crash of the interpreter, possibly with a core dump
    and removed
    type-bugAn unexpected behavior, bug, or error
    on Feb 27, 2024
  5. serhiy-storchaka
    added this to Threading issues 🧵on Mar 5, 2024

  6. devdanzin commented on Feb 23, 2025

    @devdanzin
    devdanzin
    on Feb 23, 2025
    Member

    On code generated by fuzzing, I get the same error as above 99.9% of the time, but there's a rare segfault coming from the same code (in a GILfull build), here's the backtrace:

    Thread 49 "Thread-43 (init" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffb6c006c0 (LWP 1768452)]
tstate_delete_common (tstate=tstate@entry=0xdddddddddddddddd, release_gil=release_gil@entry=0) at Python/pystate.c:1769
1769        assert(tstate->_status.cleared && !tstate->_status.finalized);
(gdb) bt
#0  tstate_delete_common (tstate=tstate@entry=0xdddddddddddddddd, release_gil=release_gil@entry=0) at Python/pystate.c:1769
#1  0x0000555555827626 in zapthreads (interp=interp@entry=0x7fffcc001b80) at Python/pystate.c:1840
#2  0x0000555555827de7 in PyInterpreterState_Delete (interp=interp@entry=0x7fffcc001b80) at Python/pystate.c:978
#3  0x0000555555822b03 in finalize_interp_delete (interp=0x7fffcc001b80) at Python/pylifecycle.c:1905
#4  0x00005555558248e6 in Py_EndInterpreter (tstate=tstate@entry=0x7fffd01b26c0) at Python/pylifecycle.c:2414
#5  0x00005555558281e2 in _PyInterpreterState_IDDecref (interp=<optimized out>) at Python/pystate.c:1287
#6  0x00007ffff6f308e7 in interp_decref (self=self@entry=0x7ffff6419130, args=args@entry=0x7ffff64d8370, kwds=kwds@entry=0x0) at ./Modules/_interpretersmodule.c:1365
#7  0x00005555556bc106 in cfunction_call (func=func@entry=0x7ffff6451190, args=args@entry=0x7ffff64d8370, kwargs=kwargs@entry=0x0) at Objects/methodobject.c:551
#8  0x0000555555666499 in _PyObject_MakeTpCall (tstate=tstate@entry=0x555556cfe740, callable=callable@entry=0x7ffff6451190, args=args@entry=0x7ffff6dfb2c0, 
    nargs=<optimized out>, keywords=keywords@entry=0x0) at Objects/call.c:242
#9  0x00005555556666d9 in _PyObject_VectorcallTstate (tstate=0x555556cfe740, callable=callable@entry=0x7ffff6451190, args=args@entry=0x7ffff6dfb2c0, 
    nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:165
#10 0x000055555566672f in PyObject_Vectorcall (callable=callable@entry=0x7ffff6451190, args=args@entry=0x7ffff6dfb2c0, nargsf=<optimized out>, 
    kwnames=kwnames@entry=0x0) at Objects/call.c:327
#11 0x00005555557a4239 in _PyEval_EvalFrameDefault (tstate=0x555556cfe740, frame=0x7ffff6dfb248, throwflag=0) at Python/generated_cases.c.h:3573
#12 0x00005555557b99d7 in _PyEval_EvalFrame (tstate=tstate@entry=0x555556cfe740, frame=frame@entry=0x7ffff6dfb1b0, throwflag=throwflag@entry=0)
    at ./Include/internal/pycore_ceval.h:116
#13 0x00005555557b9bdb in _PyEval_Vector (tstate=0x555556cfe740, func=0x7ffff64559d0, locals=locals@entry=0x0, args=0x7fffb6bff928, argcount=1, kwnames=0x0)
    at Python/ceval.c:1746
#14 0x00005555556662ea in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:413
#15 0x000055555566967f in _PyObject_VectorcallTstate (tstate=tstate@entry=0x555556cfe740, callable=callable@entry=0x7ffff64559d0, args=args@entry=0x7fffb6bff928, 
    nargsf=nargsf@entry=1, kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:167
#16 0x00005555556698e4 in method_vectorcall (method=<optimized out>, args=0x555555b4dee0 <_PyRuntime+104992>, nargsf=0, kwnames=0x0) at Objects/classobject.c:72
#17 0x0000555555667f3b in _PyVectorcall_Call (tstate=tstate@entry=0x555556cfe740, func=0x555555669707 <method_vectorcall>, callable=callable@entry=0x7ffff64d0d10, 
    tuple=tuple@entry=0x555555b4dec8 <_PyRuntime+104968>, kwargs=kwargs@entry=0x7ffff64d1730) at Objects/call.c:273
#18 0x000055555566825b in _PyObject_Call (tstate=0x555556cfe740, callable=callable@entry=0x7ffff64d0d10, args=args@entry=0x555555b4dec8 <_PyRuntime+104968>, 
    kwargs=kwargs@entry=0x7ffff64d1730) at Objects/call.c:348
#19 0x0000555555668297 in PyObject_Call (callable=callable@entry=0x7ffff64d0d10, args=args@entry=0x555555b4dec8 <_PyRuntime+104968>, 
    kwargs=kwargs@entry=0x7ffff64d1730) at Objects/call.c:373
#20 0x00005555557a056d in _PyEval_EvalFrameDefault (tstate=0x555556cfe740, frame=0x7ffff6dfb128, throwflag=0) at Python/generated_cases.c.h:2277
#21 0x00005555557b99d7 in _PyEval_EvalFrame (tstate=tstate@entry=0x555556cfe740, frame=frame@entry=0x7ffff6dfb020, throwflag=throwflag@entry=0)
    at ./Include/internal/pycore_ceval.h:116
#22 0x00005555557b9bdb in _PyEval_Vector (tstate=0x555556cfe740, func=0x7ffff7682450, locals=locals@entry=0x0, args=0x7fffb6bffd48, argcount=1, kwnames=0x0)
    at Python/ceval.c:1746
#23 0x00005555556662ea in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:413
#24 0x000055555566967f in _PyObject_VectorcallTstate (tstate=tstate@entry=0x555556cfe740, callable=callable@entry=0x7ffff7682450, args=args@entry=0x7fffb6bffd48, 
    nargsf=nargsf@entry=1, kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:167
#25 0x00005555556698e4 in method_vectorcall (method=<optimized out>, args=0x555555b4dee0 <_PyRuntime+104992>, nargsf=0, kwnames=0x0) at Objects/classobject.c:72
#26 0x0000555555667f3b in _PyVectorcall_Call (tstate=tstate@entry=0x555556cfe740, func=0x555555669707 <method_vectorcall>, callable=callable@entry=0x7ffff6472210, 
    tuple=tuple@entry=0x555555b4dec8 <_PyRuntime+104968>, kwargs=kwargs@entry=0x0) at Objects/call.c:273
#27 0x000055555566825b in _PyObject_Call (tstate=0x555556cfe740, callable=0x7ffff6472210, args=0x555555b4dec8 <_PyRuntime+104968>, kwargs=0x0) at Objects/call.c:348
#28 0x0000555555668297 in PyObject_Call (callable=<optimized out>, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:373
#29 0x00005555558b0832 in thread_run (boot_raw=boot_raw@entry=0x555556c99640) at ./Modules/_threadmodule.c:354
#30 0x000055555584077a in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:242
#31 0x00007ffff7c9caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#32 0x00007ffff7d29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

    Do you think it's the same issue, or should it be filed as a new one?

    Since the segfault is very rare, it's hard to reduce the test case :(

  7. picnixz
    added
    3.14bugs and security fixes
    on Feb 23, 2025
  8. sergey-miryanov
    added 2 commits that reference this issue on Feb 26, 2025

    pythongh-113148: Handle exceptions from threading.atexit functions

    854dfab

    Merge branch 'main' into pythongh-113148-handle-exceptions-at-thread-…

    433bb6c

  9. devdanzin commented on Mar 25, 2025

    @devdanzin
    devdanzin
    on Mar 25, 2025
    Member

    If this is the same issue, the reproducer can be simplified to:

    from concurrent.futures import interpreter
from threading import Thread

instance = interpreter.WorkerContext((lambda: None, "function"))

for x in range(10):
    try:
        Thread(target=instance.initialize, args=()).start()
    except Exception:
        pass

  10. devdanzin commented on Mar 26, 2025

    @devdanzin
    devdanzin
    on Mar 26, 2025
    Member

    Here's the backtrace for an even harder to trigger (1 in 1500 runs, as opposed to 1 in 600 runs for the other one) segfault from the repro above:

    Thread 10 "Thread-9 (initi" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffde7fc640 (LWP 322341)]
tstate_delete_common (tstate=tstate@entry=0xdddddddddddddddd, release_gil=release_gil@entry=0) at Python/pystate.c:1800
1800        assert(tstate->_status.cleared && !tstate->_status.finalized);
Received signal 11, stopping
(gdb) bt
#0  tstate_delete_common (tstate=tstate@entry=0xdddddddddddddddd, release_gil=release_gil@entry=0) at Python/pystate.c:1800
#1  0x00005555558e2b6d in zapthreads (interp=interp@entry=0x7fffdd542020) at Python/pystate.c:1871
#2  0x00005555558e33f2 in PyInterpreterState_Delete (interp=interp@entry=0x7fffdd542020) at Python/pystate.c:997
#3  0x00005555558d1ca6 in finalize_interp_delete (interp=0x7fffdd542020) at Python/pylifecycle.c:1919
#4  0x00005555558dfeef in Py_EndInterpreter (tstate=tstate@entry=0x7fffc819a420) at Python/pylifecycle.c:2433
#5  0x00005555558e5cf7 in _PyInterpreterState_IDDecref (interp=<optimized out>) at Python/pystate.c:1306
#6  0x00007ffff765d36e in interp_decref (self=<optimized out>, args=args@entry=(10,), kwds=kwds@entry=0x0)
    at ./Modules/_interpretersmodule.c:1369
#7  0x00005555556f79d3 in cfunction_call (func=<built-in method decref of module object at remote 0x7ffff78a2f30>, args=(10,),
    kwargs=0x0) at Objects/methodobject.c:551
#8  0x000055555567bbaa in _PyObject_MakeTpCall (tstate=tstate@entry=0x555555e55c20,
    callable=callable@entry=<built-in method decref of module object at remote 0x7ffff78a2f30>, args=args@entry=0x7ffff40502c0,
    nargs=<optimized out>, keywords=keywords@entry=0x0) at Objects/call.c:242
#9  0x000055555567bfcb in _PyObject_VectorcallTstate (tstate=0x555555e55c20,
    callable=<built-in method decref of module object at remote 0x7ffff78a2f30>, args=0x7ffff40502c0, nargsf=<optimized out>,
    kwnames=0x0) at ./Include/internal/pycore_call.h:165
#10 0x000055555567bff7 in PyObject_Vectorcall (callable=<optimized out>, args=args@entry=0x7ffff40502c0, nargsf=<optimized out>,
    kwnames=kwnames@entry=0x0) at Objects/call.c:327
#11 0x000055555582b924 in _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555e55c20, frame=0x7ffff4050248,
    frame@entry=0x7ffff40501b0, throwflag=throwflag@entry=0) at Python/generated_cases.c.h:1373
#12 0x000055555585ad8a in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff40501b0, tstate=0x555555e55c20)
    at ./Include/internal/pycore_ceval.h:116
#13 _PyEval_Vector (tstate=0x555555e55c20, func=0x7ffff76890d0, locals=locals@entry=0x0, args=0x7fffde7fb918, argcount=1,
    kwnames=0x0) at Python/ceval.c:1820
#14 0x000055555567b97f in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>,
    kwnames=<optimized out>) at Objects/call.c:413

  11. ZeroIntensity commented on Mar 26, 2025

    @ZeroIntensity
    ZeroIntensity
    on Mar 26, 2025
    Member

    That looks like a duplicate of #128639.

  12. ZeroIntensity commented on Sep 19, 2025

    @ZeroIntensity
    ZeroIntensity
    on Sep 19, 2025
    Member

    I can no longer reproduce this, so I think we finished this one.

  13. ZeroIntensity
    closed this as completedon Sep 19, 2025
  14. github-project-automation
    moved this from Todo to Done in Subinterpreterson Sep 19, 2025
  15. github-project-automation
    moved this to Done in Threading issues 🧵on Sep 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.12only security fixes3.13only security fixes3.14bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-subinterpreterstype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions