Skip to content

JIT executors are not properly freed #139540

Closed
1 of 1 issue completed
Closed
JIT executors are not properly freed#139540
1 of 1 issue completed
Labels
3.14bugs and security fixes3.15bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-JITtype-bugAn unexpected behavior, bug, or error

Description

@YuanchengJiang
YuanchengJiang
opened on Oct 3, 2025

Bug report

Bug description:

PoC:

import bdb
code = 'a = 1\nb = 2\nc = a + b'
globals_dict = {}
locals_dict = {}
debugger = bdb.Bdb()
debugger.runctx(code, globals_dict, locals_dict)
for i in range(10):
    pass
with tempfile.NamedTemporaryFile() as tmpfile:
    tmpfile.seek(0)

Config: --enable-experimental-jit=yes --with-address-sanitizer

ASan:

Traceback (most recent call last):
  File "min.py", line 9, in <module>
    with tempfile.NamedTemporaryFile() as tmpfile:
         ^^^^^^^^
NameError: name 'tempfile' is not defined. Did you mean: 'compile'? Or did you forget to import 'tempfile'?

==3445760==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 600 byte(s) in 1 object(s) allocated from:
    #0 0x7ab0b95889c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x5cc52547f6b7 in _PyObject_MallocWithType ../Include/internal/pycore_object_alloc.h:46
    #2 0x5cc52547f6b7 in gc_alloc ../Python/gc.c:2327
    #3 0x5cc52547f6b7 in _PyObject_GC_NewVar ../Python/gc.c:2369
    #4 0x5cc52555de54 in make_executor_from_uops ../Python/optimizer.c:1120
    #5 0x5cc52555de54 in uop_optimize ../Python/optimizer.c:1341
    #6 0x5cc52555de54 in _PyOptimizer_Optimize ../Python/optimizer.c:136
    #7 0x5cc524f27940 in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:7656
    #8 0x5cc5253e9686 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #9 0x5cc5253e9686 in _PyEval_Vector ../Python/ceval.c:1997
    #10 0x5cc5253e9686 in PyEval_EvalCode ../Python/ceval.c:880
    #11 0x5cc5255a6b0e in run_eval_code_obj ../Python/pythonrun.c:1365
    #12 0x5cc5255a6b0e in run_mod ../Python/pythonrun.c:1459
    #13 0x5cc5255ab7b7 in pyrun_file ../Python/pythonrun.c:1293
    #14 0x5cc5255ab7b7 in _PyRun_SimpleFileObject ../Python/pythonrun.c:521
    #15 0x5cc5255ac2dc in _PyRun_AnyFileObject ../Python/pythonrun.c:81
    #16 0x5cc525628bdc in pymain_run_file_obj ../Modules/main.c:410
    #17 0x5cc525628bdc in pymain_run_file ../Modules/main.c:429
    #18 0x5cc525628bdc in pymain_run_python ../Modules/main.c:691
    #19 0x5cc52562a4be in Py_RunMain ../Modules/main.c:772
    #20 0x5cc52562a4be in pymain_main ../Modules/main.c:802
    #21 0x5cc52562a4be in Py_BytesMain ../Modules/main.c:826
    #22 0x7ab0b91ba1c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
    #23 0x7ab0b91ba28a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
    #24 0x5cc524f4cf54 in _start (cpython/build/python+0x218f54) (BuildId: 3087b1f6c97d85c049f8eaa36e3ac5b15eccf317)

SUMMARY: AddressSanitizer: 600 byte(s) leaked in 1 allocation(s).

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

Activity

  1. YuanchengJiang
    added
    type-bugAn unexpected behavior, bug, or error
    on Oct 3, 2025
  2. picnixz
    added
    interpreter-core(Objects, Python, Grammar, and Parser dirs)
    on Oct 3, 2025

  3. picnixz commented on Oct 3, 2025

    @picnixz
    picnixz
    on Oct 3, 2025
    Member

    Is there a smaller reproducer? maybe it's just when we have a NameError that this happens (or when bdb is up).

    @ZeroIntensity This looks like something we already had but I don't know if it's the same issue. Do you remember a similar issue?

    cc @brandtbucher for the JIT

  4. picnixz
    changed the title [-]memory leak tempfile[/-] [+]Possible memory leak with ASAN+JIT when combining `NameError` and BDB[/+] on Oct 3, 2025
  5. picnixz
    added
    topic-JIT
    on Oct 3, 2025

  6. ZeroIntensity commented on Oct 3, 2025

    @ZeroIntensity
    ZeroIntensity
    on Oct 3, 2025
    Member

    I don't remember any issues similar to this.

  7. YuanchengJiang commented on Oct 4, 2025

    @YuanchengJiang
    YuanchengJiang
    on Oct 4, 2025
    Author

    another poc:

    import base64
empty_encoded = base64.a85encode(b'')
formatted_non_utc = email.utils.format_datetime(dt_non_utc)

    asan:

    Traceback (most recent call last):
  File "/home/fuzz/WorkSpace/flowfusion-cpython/my_leak_pocs/912/./test.py", line 35, in <module>
    formatted_non_utc = email.utils.format_datetime(dt_non_utc)
                                                    ^^^^^^^^^^
NameError: name 'dt_non_utc' is not defined

=================================================================
==1994340==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 912 byte(s) in 1 object(s) allocated from:
    #0 0x753de56d59c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x5e2822c466b7 in _PyObject_MallocWithType ../Include/internal/pycore_object_alloc.h:46
    #2 0x5e2822c466b7 in gc_alloc ../Python/gc.c:2327
    #3 0x5e2822c466b7 in _PyObject_GC_NewVar ../Python/gc.c:2369
    #4 0x5e2822d24e54 in make_executor_from_uops ../Python/optimizer.c:1120
    #5 0x5e2822d24e54 in uop_optimize ../Python/optimizer.c:1341
    #6 0x5e2822d24e54 in _PyOptimizer_Optimize ../Python/optimizer.c:136
    #7 0x5e28226ee940 in _PyEval_EvalFrameDefault ../Python/generated_cases.c.h:7656
    #8 0x5e2822bb0686 in _PyEval_EvalFrame ../Include/internal/pycore_ceval.h:121
    #9 0x5e2822bb0686 in _PyEval_Vector ../Python/ceval.c:1997
    #10 0x5e2822bb0686 in PyEval_EvalCode ../Python/ceval.c:880
    #11 0x5e2822d6db0e in run_eval_code_obj ../Python/pythonrun.c:1365
    #12 0x5e2822d6db0e in run_mod ../Python/pythonrun.c:1459
    #13 0x5e2822d727b7 in pyrun_file ../Python/pythonrun.c:1293
    #14 0x5e2822d727b7 in _PyRun_SimpleFileObject ../Python/pythonrun.c:521
    #15 0x5e2822d732dc in _PyRun_AnyFileObject ../Python/pythonrun.c:81
    #16 0x5e2822defbdc in pymain_run_file_obj ../Modules/main.c:410
    #17 0x5e2822defbdc in pymain_run_file ../Modules/main.c:429
    #18 0x5e2822defbdc in pymain_run_python ../Modules/main.c:691
    #19 0x5e2822df14be in Py_RunMain ../Modules/main.c:772
    #20 0x5e2822df14be in pymain_main ../Modules/main.c:802
    #21 0x5e2822df14be in Py_BytesMain ../Modules/main.c:826
    #22 0x753de53071c9  (/lib/x86_64-linux-gnu/libc.so.6+0x2a1c9) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
    #23 0x753de530728a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2a28a) (BuildId: 282c2c16e7b6600b0b22ea0c99010d2795752b5f)
    #24 0x5e2822713f54 in _start (/home/fuzz/WorkSpace/flowfusion-cpython/cpython/build/python+0x218f54) (BuildId: 3087b1f6c97d85c049f8eaa36e3ac5b15eccf317)

SUMMARY: AddressSanitizer: 912 byte(s) leaked in 1 allocation(s).

    the poc is not related; however, backtrace is almost the same. shall i regard this as the same or different one?

  8. picnixz commented on Oct 4, 2025

    @picnixz
    picnixz
    on Oct 4, 2025
    Member

    The relation lies in the NameError.

  9. picnixz
    changed the title [-]Possible memory leak with ASAN+JIT when combining `NameError` and BDB[/-] [+]Possible memory leak with ASAN+JIT after raising `NameError`[/+] on Oct 4, 2025

  10. picnixz commented on Oct 4, 2025

    @picnixz
    picnixz
    on Oct 4, 2025
    Member

    another poc:

    email.utils is not imported but there is no error on your side. Can you show me the entire file, as well as the command that you executed? (is it via python test.py or something else?) I cannot reproduce this on my side.

    Nevermind, I managed to reproduce it if run as a script instead the REPL:

    import base64
empty_encoded = base64.a85encode(b'')
empty_encded

    Changing a85encode to b64encode doesn't seem to work so the JIT-ification seems to be important here. However, changing it to b85encode also produces a crash. I'll investigate.

  11. picnixz
    self-assigned this
    on Oct 4, 2025
  12. picnixz
    added
    3.14bugs and security fixes
    3.15bugs and security fixes
    on Oct 4, 2025

  13. picnixz commented on Oct 4, 2025

    @picnixz
    picnixz
    on Oct 4, 2025
    Member

    To be clear, we also have a simple leak without address sanitizer:

    $ ./configure --with-pydebug -q CC=clang --enable-experimental-jit=yes
$ make -s -j12 
$ ./python -X dev -X showrefcount crash.py
Traceback (most recent call last):
  File "[...]/cpython/crash.py", line 3, in <module>
    empty_encded
NameError: name 'empty_encded' is not defined. Did you mean: 'empty_encoded'?
[2 refs, 1 blocks]

    So we're indeed having a leak somewhere. I think the executors are not properly cleaned so I'll investigate. The contents of crash.py are:

    import base64
empty_encoded = base64.b85encode(b'')
empty_encded

    The JIT traceback:

    Optimizing b85encode ([...]/cpython/Lib/base64.py:442) at byte offset 144
   1 ADD_TO_TRACE: _START_EXECUTOR (0, target=72, operand0=0x7f1e704e1820, operand1=0)
   2 ADD_TO_TRACE: _MAKE_WARM (0, target=0, operand0=0, operand1=0)
72: JUMP_BACKWARD_JIT(13)
   3 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=72, operand0=0, operand1=0)
   4 ADD_TO_TRACE: _SET_IP (0, target=72, operand0=0x7f1e704e1820, operand1=0)
   5 ADD_TO_TRACE: _CHECK_PERIODIC (0, target=72, operand0=0, operand1=0, error_target=0)
61: FOR_ITER_LIST(11)
   6 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=61, operand0=0, operand1=0)
   7 ADD_TO_TRACE: _SET_IP (0, target=61, operand0=0x7f1e704e180a, operand1=0)
   8 ADD_TO_TRACE: _ITER_CHECK_LIST (11, target=61, operand0=0, operand1=0)
   9 ADD_TO_TRACE: _GUARD_NOT_EXHAUSTED_LIST (11, target=61, operand0=0, operand1=0)
  10 ADD_TO_TRACE: _ITER_NEXT_LIST_TIER_TWO (11, target=61, operand0=0, operand1=0)
63: STORE_FAST_LOAD_FAST(3)
  11 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=63, operand0=0, operand1=0)
  12 ADD_TO_TRACE: _SET_IP (0, target=63, operand0=0x7f1e704e180e, operand1=0)
  13 ADD_TO_TRACE: _STORE_FAST (0, target=63, operand0=0, operand1=0)
  14 ADD_TO_TRACE: _LOAD_FAST (3, target=63, operand0=0, operand1=0)
64: LOAD_FAST_BORROW(0)
  15 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=64, operand0=0, operand1=0)
  16 ADD_TO_TRACE: _SET_IP (0, target=64, operand0=0x7f1e704e1810, operand1=0)
  17 ADD_TO_TRACE: _LOAD_FAST_BORROW (0, target=64, operand0=0, operand1=0)
65: BINARY_OP(0)
  18 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=65, operand0=0, operand1=0)
  19 ADD_TO_TRACE: _SET_IP (0, target=65, operand0=0x7f1e704e1812, operand1=0)
  20 ADD_TO_TRACE: _BINARY_OP (0, target=65, operand0=0, operand1=0, error_target=0)
71: LIST_APPEND(5)
  21 ADD_TO_TRACE: _CHECK_VALIDITY (0, target=71, operand0=0, operand1=0)
  22 ADD_TO_TRACE: _SET_IP (0, target=71, operand0=0x7f1e704e181e, operand1=0)
  23 ADD_TO_TRACE: _LIST_APPEND (5, target=71, operand0=0, operand1=0, error_target=0)
  24 ADD_TO_TRACE: _JUMP_TO_TOP (0, target=0, operand0=0, operand1=0)
Created a proto-trace for b85encode ([...]/cpython/Lib/base64.py:442) at byte offset 144 -- length 24
Optimized trace (length 31):
   0 OPTIMIZED: _START_EXECUTOR (0, jump_target=20, operand0=0x556ea0b9feb0, operand1=0)
   1 OPTIMIZED: _MAKE_WARM (0, target=0, operand0=0, operand1=0)
   2 OPTIMIZED: _SET_IP (0, target=72, operand0=0x7f1e704e1820, operand1=0)
   3 OPTIMIZED: _CHECK_PERIODIC (0, jump_target=0, operand0=0, operand1=0, error_target=21)
   4 OPTIMIZED: _CHECK_VALIDITY (0, jump_target=22, operand0=0, operand1=0)
   5 OPTIMIZED: _SET_IP (0, target=61, operand0=0x7f1e704e180a, operand1=0)
   6 OPTIMIZED: _ITER_CHECK_LIST (11, jump_target=23, operand0=0, operand1=0)
   7 OPTIMIZED: _GUARD_NOT_EXHAUSTED_LIST (11, jump_target=24, operand0=0, operand1=0)
   8 OPTIMIZED: _ITER_NEXT_LIST_TIER_TWO (11, jump_target=25, operand0=0, operand1=0)
   9 OPTIMIZED: _CHECK_VALIDITY (0, jump_target=26, operand0=0, operand1=0)
  10 OPTIMIZED: _SET_IP (0, target=63, operand0=0x7f1e704e180e, operand1=0)
  11 OPTIMIZED: _STORE_FAST_0 (0, target=63, operand0=0, operand1=0)
  12 OPTIMIZED: _LOAD_FAST_3 (3, target=63, operand0=0, operand1=0)
  13 OPTIMIZED: _CHECK_VALIDITY (0, jump_target=27, operand0=0, operand1=0)
  14 OPTIMIZED: _LOAD_FAST_BORROW_0 (0, target=64, operand0=0, operand1=0)
  15 OPTIMIZED: _SET_IP (0, target=65, operand0=0x7f1e704e1812, operand1=0)
  16 OPTIMIZED: _BINARY_OP (0, jump_target=0, operand0=0, operand1=0, error_target=28)
  17 OPTIMIZED: _CHECK_VALIDITY (0, jump_target=29, operand0=0, operand1=0)
  18 OPTIMIZED: _LIST_APPEND (5, jump_target=0, operand0=0, operand1=0, error_target=30)
  19 OPTIMIZED: _JUMP_TO_TOP (0, jump_target=1, operand0=0, operand1=0)
  20 OPTIMIZED: _DEOPT (0, target=72, operand0=0, operand1=0)
  21 OPTIMIZED: _ERROR_POP_N (0, target=0, operand0=0x48, operand1=0)
  22 OPTIMIZED: _DEOPT (0, target=61, operand0=0, operand1=0)
  23 OPTIMIZED: _EXIT_TRACE (0, target=61, operand0=0x556ea0b9ff28, operand1=0)
  24 OPTIMIZED: _EXIT_TRACE (0, target=75, operand0=0x556ea0b9ff38, operand1=0)
  25 OPTIMIZED: _DEOPT (0, target=61, operand0=0, operand1=0)
  26 OPTIMIZED: _DEOPT (0, target=63, operand0=0, operand1=0)
  27 OPTIMIZED: _DEOPT (0, target=64, operand0=0, operand1=0)
  28 OPTIMIZED: _ERROR_POP_N (0, target=0, operand0=0x41, operand1=0)
  29 OPTIMIZED: _DEOPT (0, target=71, operand0=0, operand1=0)
  30 OPTIMIZED: _ERROR_POP_N (0, target=0, operand0=0x47, operand1=0)
  14. picnixz
    changed the title [-]Possible memory leak with ASAN+JIT after raising `NameError`[/-] [+]JIT sometimes leaks after raising `NameError`[/+] on Oct 4, 2025

  15. 17 remaining items

  16. picnixz commented on Oct 4, 2025

    @picnixz
    picnixz
    on Oct 4, 2025
    Member

    For now, I'm marking it as a release blocker because I don't know if such leaks are actually fine in 3.14.0 for now as it's an experimental feature. Otherwise, we can fix this in 3.14.1.
    cc @hugovk

  17. Zheaoli commented on Oct 4, 2025

    @Zheaoli
    Zheaoli
    on Oct 4, 2025
    Contributor

    @picnixz Actually, I prefer to release the patch in 3.14.1. JIT is still an experimental feature.

  18. picnixz commented on Oct 4, 2025

    @picnixz
    picnixz
    on Oct 4, 2025
    Member

    That's my idea as well, but I want the RM decision here.

  19. Fidget-Spinner commented on Oct 4, 2025

    @Fidget-Spinner
    Fidget-Spinner
    on Oct 4, 2025
    Member

    JIT bugs have never been a release blocker. Removing release blocker label.

  20. Fidget-Spinner
    removed
    release-blocker
    on Oct 4, 2025
  21. Fidget-Spinner
    removed this from Release and Deferred blockers 🚫on Oct 4, 2025
  22. picnixz
    marked memory leak with glob and mock in JIT #139750 as a duplicate of this issue on Oct 8, 2025
  23. picnixz
    marked memory leak after ValueError in JIT #139749 as a duplicate of this issue on Oct 8, 2025
  24. picnixz
    marked memory leak in email message make mixed with long boundary #139827 as a duplicate of this issue on Oct 9, 2025
  25. picnixz
    removed their assignment
    on Oct 11, 2025

  26. Fidget-Spinner commented on Oct 11, 2025

    @Fidget-Spinner
    Fidget-Spinner
    on Oct 11, 2025
    Member

    I think this is partially also reported in #137007. However, the repro here is way better, so I'm keeping this issue open.

  27. picnixz commented on Oct 11, 2025

    @picnixz
    picnixz
    on Oct 11, 2025
    Member

    Should we close 137007? because I don't like having multiple issues open (or we can mark this one as a parent issue of 137007). This issue was specifically for the cold executors used in side exits.

  28. picnixz
    added a sub-issue on Oct 11, 2025

  29. cocolato commented on Feb 13, 2026

    @cocolato
    cocolato
    on Feb 13, 2026
    Member

    It can't be reproduced on the main branch. Since #137007 has been closed, I believe this issue can be closed as well.

  30. Fidget-Spinner
    closed this as completedon Feb 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.14bugs and security fixes3.15bugs and security fixesinterpreter-core(Objects, Python, Grammar, and Parser dirs)topic-JITtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions