Skip to content

SEGV in faulthandler.dump_traceback_later #140815

Closed
Closed
SEGV in faulthandler.dump_traceback_later#140815
Assignees
sergey-miryanov
Labels
interpreter-core(Objects, Python, Grammar, and Parser dirs)triagedThe issue has been accepted as valid by a triager.type-crashA hard crash of the interpreter, possibly with a core dump

Description

@YuanchengJiang
YuanchengJiang
opened on Oct 31, 2025

Crash report

What happened?

import faulthandler
import sys
import unittest

class Test(unittest.TestCase):
    def setUp(self):
        faulthandler.dump_traceback_later(10 * 1e-308, exit=True, file=sys.__stderr__)
    def test_sendall(self):
        os.mkfifo(filename)

if __name__ == '__main__':
    unittest.main()
FAILED (errors=1)
    #0 0x5646ef225cff in dump_traceback ../Python/traceback.c:1111
    #1 0x5646ef229ae2 in _Py_DumpTracebackThreads ../Python/traceback.c:1302
    #2 0x5646ef2709d4 in faulthandler_thread ../Modules/faulthandler.c:706
    #3 0x5646ef2220af in pythread_wrapper ../Python/thread_pthread.h:234
    #4 0x76c294e77a41 in asan_thread_start ../../../../src/libsanitizer/asan/asan_interceptors.cpp:234
    #5 0x76c294bbaaa3  (/lib/x86_64-linux-gnu/libc.so.6+0x9caa3) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
    #6 0x76c294c47a33 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x129a33) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)

Address 0x76c292fb39da is located in stack of thread T0 at offset 2522 in frame
    #0 0x5646eeb38bcf in _PyEval_EvalFrameDefault ../Python/ceval.c:1032

  This frame has 37 object(s):
    [32, 36) 'method_found'
    [48, 52) 'level'
    [64, 68) 'handler'
    [80, 84) 'lasti'
    [96, 104) 'temp'
    [128, 136) 'res_o'
    [160, 168) 'match_o'
    [192, 200) 'rest_o'
    [224, 232) 'null_or_index'
    [256, 264) 'null_or_index'
    [288, 296) 'executor'
    [320, 328) 'bc_o'
    [352, 360) 'value_o'
    [384, 392) 'v_o'
    [416, 424) 'retval_o'
    [448, 456) 'ann_dict'
    [480, 488) 'kwnames' (line 1906)
    [512, 528) 'stack'
    [544, 560) 'stack'
    [576, 592) 'args'
    [608, 648) 'stack'
    [688, 752) 'stack_array' (line 1909)
    [784, 872) 'values_o_temp'
    [912, 1000) 'pieces_o_temp'
    [1040, 1128) 'args_o_temp'
    [1168, 1256) 'args_o_temp'
    [1296, 1384) 'args_o_temp'
    [1424, 1512) 'args_o_temp'
    [1552, 1640) 'args_o_temp'
    [1680, 1768) 'args_o_temp'
    [1808, 1896) 'args_o_temp'
    [1936, 2024) 'args_o_temp'
    [2064, 2152) 'args_o_temp'
    [2192, 2280) 'args_o_temp'
    [2320, 2408) 'args_o_temp'
    [2448, 2544) 'entry' (line 1051) <== Memory access at offset 2522 is inside this variable
    [2576, 2656) 'buffer' (line 518)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-return ../Python/traceback.c:1111 in dump_traceback
Shadow bytes around the buggy address:
  0x76c292fb3700: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3780: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3800: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3880: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3900: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
=>0x76c292fb3980: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5[f5]f5 f5 f5 f5
  0x76c292fb3a00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3a80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3b00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3b80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x76c292fb3c00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
Thread T1 created by T0 here:
    #0 0x76c294f0e1f9 in pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:245
    #1 0x5646ef22228b in do_start_joinable_thread ../Python/thread_pthread.h:281
    #2 0x5646ef2229a1 in PyThread_start_new_thread ../Python/thread_pthread.h:336
    #3 0x5646ef274706 in faulthandler_dump_traceback_later_impl ../Modules/faulthandler.c:868
    #4 0x5646ef274706 in faulthandler_dump_traceback_later ../Modules/clinic/faulthandler.c.h:366
    #5 0x5646eec9f677 in _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:169
    #6 0x5646eec9f677 in PyObject_Vectorcall ../Objects/call.c:327
    #7 0x5646eeb42bba in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:2920
    #8 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #9 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #10 0x5646eeca90f0 in _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:169
    #11 0x5646eeca90f0 in method_vectorcall ../Objects/classobject.c:95
    #12 0x5646eeca478e in _PyVectorcall_Call ../Objects/call.c:273
    #13 0x5646eeca478e in _PyObject_Call ../Objects/call.c:348
    #14 0x5646eeca478e in PyObject_Call ../Objects/call.c:373
    #15 0x5646eeb43e9c in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:2616
    #16 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #17 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #18 0x5646eeca2db3 in _PyObject_VectorcallDictTstate ../Objects/call.c:135
    #19 0x5646eeca346c in _PyObject_Call_Prepend ../Objects/call.c:504
    #20 0x5646eee5ff64 in call_method ../Objects/typeobject.c:3077
    #21 0x5646eee5ff64 in slot_tp_call ../Objects/typeobject.c:10606
    #22 0x5646eec9dc5d in _PyObject_MakeTpCall ../Objects/call.c:242
    #23 0x5646eeb417ac in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:4021
    #24 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #25 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #26 0x5646eeca90f0 in _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:169
    #27 0x5646eeca90f0 in method_vectorcall ../Objects/classobject.c:95
    #28 0x5646eeca478e in _PyVectorcall_Call ../Objects/call.c:273
    #29 0x5646eeca478e in _PyObject_Call ../Objects/call.c:348
    #30 0x5646eeca478e in PyObject_Call ../Objects/call.c:373
    #31 0x5646eeb43e9c in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:2616
    #32 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #33 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #34 0x5646eeca2db3 in _PyObject_VectorcallDictTstate ../Objects/call.c:135
    #35 0x5646eeca346c in _PyObject_Call_Prepend ../Objects/call.c:504
    #36 0x5646eee5ff64 in call_method ../Objects/typeobject.c:3077
    #37 0x5646eee5ff64 in slot_tp_call ../Objects/typeobject.c:10606
    #38 0x5646eec9dc5d in _PyObject_MakeTpCall ../Objects/call.c:242
    #39 0x5646eeb40ad2 in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:1620
    #40 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #41 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #42 0x5646eeca90f0 in _PyObject_VectorcallTstate ../Include/internal/pycore_call.h:169
    #43 0x5646eeca90f0 in method_vectorcall ../Objects/classobject.c:95
    #44 0x5646eeca478e in _PyVectorcall_Call ../Objects/call.c:273
    #45 0x5646eeca478e in _PyObject_Call ../Objects/call.c:348
    #46 0x5646eeca478e in PyObject_Call ../Objects/call.c:373
    #47 0x5646eeb43e9c in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:2616
    #48 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #49 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #50 0x5646eeca2db3 in _PyObject_VectorcallDictTstate ../Objects/call.c:135
    #51 0x5646eeca346c in _PyObject_Call_Prepend ../Objects/call.c:504
    #52 0x5646eee5ff64 in call_method ../Objects/typeobject.c:3077
    #53 0x5646eee5ff64 in slot_tp_call ../Objects/typeobject.c:10606
    #54 0x5646eec9dc5d in _PyObject_MakeTpCall ../Objects/call.c:242
    #55 0x5646eeb40ad2 in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:1620
    #56 0x5646ef020785 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #57 0x5646ef020785 in _PyEval_Vector ../Python/ceval.c:2005
    #58 0x5646eeca2db3 in _PyObject_VectorcallDictTstate ../Objects/call.c:135
    #59 0x5646eeca346c in _PyObject_Call_Prepend ../Objects/call.c:504
    #60 0x5646eee4d2e0 in call_method ../Objects/typeobject.c:3077
    #61 0x5646eee4d2e0 in slot_tp_init ../Objects/typeobject.c:10835
    #62 0x5646eee3f457 in type_call ../Objects/typeobject.c:2461
    #63 0x5646eec9dc5d in _PyObject_MakeTpCall ../Objects/call.c:242
    #64 0x5646eeb5c9f8 in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:1620
    #65 0x5646ef01ffb6 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #66 0x5646ef01ffb6 in _PyEval_Vector ../Python/ceval.c:2005
    #67 0x5646ef01ffb6 in PyEval_EvalCode ../Python/ceval.c:888
    #68 0x5646ef1de3fe in run_eval_code_obj ../Python/pythonrun.c:1365
    #69 0x5646ef1de3fe in run_mod ../Python/pythonrun.c:1459
    #70 0x5646ef1e30a7 in pyrun_file ../Python/pythonrun.c:1293
    #71 0x5646ef1e30a7 in _PyRun_SimpleFileObject ../Python/pythonrun.c:521
    #72 0x5646ef1e3bcc in _PyRun_AnyFileObject ../Python/pythonrun.c:81
    #73 0x5646ef2563cc in pymain_run_file_obj ../Modules/main.c:410
    #74 0x5646ef2563cc in pymain_run_file ../Modules/main.c:429
    #75 0x5646ef2563cc in pymain_run_python ../Modules/main.c:691
    #76 0x5646ef257cae in Py_RunMain ../Modules/main.c:772
    #77 0x5646ef257cae in pymain_main ../Modules/main.c:802
    #78 0x5646ef257cae in Py_BytesMain ../Modules/main.c:826
    #79 0x76c294b481c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
    #80 0x76c294b4828a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)

==2938816==ABORTING

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

Activity

  1. YuanchengJiang
    added
    type-crashA hard crash of the interpreter, possibly with a core dump
    on Oct 31, 2025
  2. picnixz
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    on Oct 31, 2025
  3. picnixz
    changed the title [-]SEGV in faulthandler[/-] [+]SEGV in `faulthandler.dump_traceback_later`[/+] on Oct 31, 2025

  4. picnixz commented on Oct 31, 2025

    @picnixz
    picnixz
    on Oct 31, 2025
    Member

    Is

    this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork

    relevant? cc @vstinner

  5. sergey-miryanov commented on Oct 31, 2025

    @sergey-miryanov
    sergey-miryanov
    on Oct 31, 2025
    Contributor

    I have following repro:

    import faulthandler
import sys


faulthandler.dump_traceback_later(10 * 1e-308, exit=True, file=sys.__stderr__)
assert False

    It fails with assert in the C code:

    Running Debug|x64 interpreter...
Traceback (most recent call last):
Timeout (0:00:00.000001)!
Thread 0x00006e38 (most recent call first):
  File "X:\Work\cpython\main\Lib\traceback.py"  File "X:\Work\cpython\issues\b140815.py", line 8, in <module>
    assert False
           ^^^^^
Assertion failed: !PyStackRef_IsNull(f->f_executable), file X:\Work\cpython\main\Include\internal\pycore_interpframe.h, line 21
AssertionError

    If no one minds, I'd like to take a look.

  6. picnixz
    added
    triagedThe issue has been accepted as valid by a triager.
    on Oct 31, 2025
  7. sergey-miryanov
    self-assigned this
    on Oct 31, 2025

  8. sergey-miryanov commented on Nov 1, 2025

    @sergey-miryanov
    sergey-miryanov
    on Nov 1, 2025
    Contributor

    This segfault occurs because the faulthandler_thread runs concurrently with interpreter finalization.

    This happens because the timeout value of the dump_traceback_later is 10 * 1e-308 which is cropped to 1us. Therefore, the faulthandler_thread runs as soon as possible.

    faulthandler_thread doesn't use any synchronization primitives, so it is not a thread-safe, even in the GIL-enabled build.

    I found an issue #128400 and a PR #128422 where the call to the StopTheWorld function was added to the faulthandler_dump_traceback_py_impl to prevent multithreading issues in the no-GIL build. Therefore, I believe this approach is valid for this issue as well. For the GIL-enabled build, I added the GIL acquisition.

  9. vstinner
    added 4 commits that reference this issue on Nov 2, 2025

    pythongh-140815: Fix faulthandler for invalid/freed frame

    2712e72

    pythongh-140815: Fix faulthandler for invalid/freed frame

    0c04631

    pythongh-140815: Fix faulthandler for invalid/freed frame

    a4e3209

    pythongh-140815: Fix faulthandler for invalid/freed frame

    1cab1e5

  10. vstinner commented on Nov 2, 2025

    @vstinner
    vstinner
    on Nov 2, 2025
    Member

    @sergey-miryanov:

    This segfault occurs because the faulthandler_thread runs concurrently with interpreter finalization.

    _Py_DumpTracebackThreads() is called by the faulthandler.dump_traceback_later() thread while the main thread is freeing its frame.

    _Py_DumpTracebackThreads() reads freed memory and does crash randomly. It already calls _PyMem_IsPtrFreed() in a few places to detect freed memory, but it's not enough.

    faulthandler_thread doesn't use any synchronization primitives, so it is not a thread-safe, even in the GIL-enabled build.

    This is by design. faulthandler.dump_traceback_later(exit=True) dumps the traceback and exits the process to implement a software watchdog. It is used by buildbots and other CIs to exit after a timeout.

    faulthandler.dump_traceback_later() should not attempt to acquire the GIL or any other lock since it's used to dump the traceback in case in other deadlocks.

  11. vstinner commented on Nov 2, 2025

    @vstinner
    vstinner
    on Nov 2, 2025
    Member

    I wrote #140921 to fix the issue: detect if a frame object or a code object is invalid or freed.

    With my fix, I can no longer reproduce the crash: I ran the script in a loop for 1 minute.

  12. sergey-miryanov commented on Nov 2, 2025

    @sergey-miryanov
    sergey-miryanov
    on Nov 2, 2025
    Contributor

    faulthandler.dump_traceback_later() should not attempt to acquire the GIL or any other lock since it's used to dump the traceback in case in other deadlocks.

    Thanks to point me this! It was not clear to me after reading the documentation (it is only one reference to the deadlock on the page and I just missed it).

  13. vstinner commented on Nov 2, 2025

    @vstinner
    vstinner
    on Nov 2, 2025
    Member

    Dummy example of deadlock:

    import faulthandler
import threading

def work():
    lock = threading.Lock()
    lock.acquire()
    lock.acquire()
    lock.release()
    lock.release()

faulthandler.dump_traceback_later(5, exit=True)
work()
faulthandler.cancel_dump_traceback_later()

    Output:

    Timeout (0:00:05)!
Thread 0x00007fc30261ebc0 [python] (most recent call first):
  File "/home/vstinner/python/main/x.py", line 7 in work
  File "/home/vstinner/python/main/x.py", line 12 in <module>

    faulthandler shows where the deadlock occurs.

  14. sergey-miryanov commented on Nov 2, 2025

    @sergey-miryanov
    sergey-miryanov
    on Nov 2, 2025
    Contributor

    faulthandler shows where the deadlock occurs.

    I can use this at work somehow 🤔 ... Thanks!

  15. vstinner
    added a commit that references this issue on Nov 4, 2025

    gh-140815: Fix faulthandler for invalid/freed frame (#140921)

    a84181c
  16. miss-islington
    added a commit that references this issue on Nov 4, 2025

    pythongh-140815: Fix faulthandler for invalid/freed frame (pythonGH-1…

    ed39713
  17. vstinner
    added 3 commits that reference this issue on Nov 4, 2025

    pythongh-140815: Fix faulthandler for invalid/freed frame (python#140921

    1aee44b

    [3.14] gh-140815: Fix faulthandler for invalid/freed frame (GH-140921) (

    331b4b8

    [3.13] gh-140815: Fix faulthandler for invalid/freed frame (#140921) (#…

    43882c7

  18. vstinner commented on Nov 5, 2025

    @vstinner
    vstinner
    on Nov 5, 2025
    Member

    Instead of crashing, the initial reproducer now completes with incomplete traceback like:

    $ ./python x.py
Timeout (0:00:00.000001)!
Thread 0x00007f107d86bbc0 [python] (most recent call first):
  File "/home/vstinner/python/main/Lib/tokenize.py", line ??? in find_cookie
  <invalid frame>

    Even if, as I explained in other comments, it's not 100% reliable. I consider that the initial issue has been fixed and I close the issue. Thanks for your bug report.

  19. vstinner
    closed this as completedon Nov 5, 2025
  20. StanFromIreland
    added a commit that references this issue on Dec 6, 2025

    pythongh-140815: Fix faulthandler for invalid/freed frame (python#140921

    592b099
  21. allanrmartins
    added a commit that references this issue on Sep 11, 2026

    fix(idlewatch): sentinela em thread Python, sem o watchdog em C do fa…

    b309c93
  22. meta-codesync
    added a commit that references this issue on Sep 24, 2026

    Backport gh-140815 faulthandler trampoline-frame fix

    247b988
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

interpreter-core(Objects, Python, Grammar, and Parser dirs)triagedThe issue has been accepted as valid by a triager.type-crashA hard crash of the interpreter, possibly with a core dump

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions