Skip to content

Race on _PyRuntime.signals.unhandled_keyboard_interrupt when calling eval() concurrently in free-threading mode #128130

Closed
Closed
Race on _PyRuntime.signals.unhandled_keyboard_interrupt when calling eval() concurrently in free-threading mode#128130
Labels
3.13only security fixes3.14bugs and security fixesextension-modulesC modules in the Modules dirtopic-free-threadingtype-bugAn unexpected behavior, bug, or error

Description

@hawkinsp
hawkinsp
opened on Dec 20, 2024

Bug report

Bug description:

Run the following code under Python 3.13.1t with thread-sanitizer enabled:

from collections import namedtuple
import concurrent.futures
import functools
import threading

num_threads = 8


def closure(b):
  b.wait()
  namedtuple('mytuple', ['f'])

with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
  for _ in range(100):
    b = threading.Barrier(num_threads)
    for _ in range(num_threads):
      executor.submit(functools.partial(closure, b))

Get:

WARNING: ThreadSanitizer: data race (pid=682761)
  Write of size 4 at 0x56312bb5f168 by thread T6:
    #0 run_eval_code_obj /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1318:53 (python3.13+0x4a02a6) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #1 run_mod /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1422:19 (python3.13+0x49fa95) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #2 _PyRun_StringFlagsWithName /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1221:15 (python3.13+0x49cfe4) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #3 PyRun_StringFlags /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1233:12 (python3.13+0x49cfe4)
    #4 builtin_eval_impl /usr/local/google/home/phawkins/p/cpython/Python/bltinmodule.c:1027:18 (python3.13+0x3d8189) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #5 builtin_eval /usr/local/google/home/phawkins/p/cpython/Python/clinic/bltinmodule.c.h:467:20 (python3.13+0x3d8189)
    #6 cfunction_vectorcall_FASTCALL_KEYWORDS /usr/local/google/home/phawkins/p/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a010) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #7 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafaa) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #8 PyObject_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c:327:12 (python3.13+0x1eafaa)
    #9 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e24fb) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #10 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #11 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #12 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #13 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x571bb2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #14 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x571bb2)
    #15 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #16 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #17 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #18 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e46e2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #19 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #20 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #21 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #22 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef5ef) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #23 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef5ef)
    #24 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #25 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #26 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #27 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564292) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #28 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bd637) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)

  Previous write of size 4 at 0x56312bb5f168 by thread T8:
    #0 run_eval_code_obj /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1318:53 (python3.13+0x4a02a6) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #1 run_mod /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1422:19 (python3.13+0x49fa95) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #2 _PyRun_StringFlagsWithName /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1221:15 (python3.13+0x49cfe4) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #3 PyRun_StringFlags /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1233:12 (python3.13+0x49cfe4)
    #4 builtin_eval_impl /usr/local/google/home/phawkins/p/cpython/Python/bltinmodule.c:1027:18 (python3.13+0x3d8189) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #5 builtin_eval /usr/local/google/home/phawkins/p/cpython/Python/clinic/bltinmodule.c.h:467:20 (python3.13+0x3d8189)
    #6 cfunction_vectorcall_FASTCALL_KEYWORDS /usr/local/google/home/phawkins/p/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a010) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #7 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafaa) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #8 PyObject_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c:327:12 (python3.13+0x1eafaa)
    #9 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e24fb) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #10 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #11 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #12 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #13 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x571bb2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #14 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x571bb2)
    #15 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #16 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #17 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #18 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e46e2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #19 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #20 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #21 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #22 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef5ef) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #23 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef5ef)
    #24 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #25 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #26 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #27 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564292) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #28 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bd637) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)

I think namedtuple calls eval(), which triggers a race here:

cpython/Python/pythonrun.c

Line 1318 in 0671451

_PyRuntime.signals.unhandled_keyboard_interrupt = 0;

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

Activity

  1. hawkinsp
    added
    type-bugAn unexpected behavior, bug, or error
    on Dec 20, 2024
  2. picnixz
    added
    extension-modulesC modules in the Modules dir
    3.13only security fixes
    topic-free-threading
    3.14bugs and security fixes
    on Dec 20, 2024
  3. colesbury
    changed the title [-]Race when calling collections.namedtuple() concurrently in free-threading mode[/-] [+]Race on `_PyRuntime.signals.unhandled_keyboard_interrupt` when calling `eval()` concurrently in free-threading mode[/+] on Dec 21, 2024

  4. colesbury commented on Dec 21, 2024

    @colesbury
    colesbury
    on Dec 21, 2024
    Contributor

    I think there's a race condition here even in the default build where a concurrent eval() may wipe out the unhandled_keyboard_interrupt from the main thread.

  5. colesbury commented on Feb 6, 2025

    @colesbury
    colesbury
    on Feb 6, 2025
    Contributor

    Also seen when running ./python -m test test_exceptions --parallel-threads=10 -v under Tsan in main (3.14). It's in a different function, but still on _PyRuntime.signals.unhandled_keyboard_interrupt.

    WARNING: ThreadSanitizer: data race (pid=990837)
  Write of size 4 at 0x555555cd2530 by thread T595:
    #0 _PyErr_Display Python/pythonrun.c:1127 (python+0x43bad3) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #1 thread_excepthook_file Modules/_threadmodule.c:2189 (python+0x52f068) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #2 thread_excepthook Modules/_threadmodule.c:2268 (python+0x52f068)
    #3 cfunction_vectorcall_O Objects/methodobject.c:523 (python+0x1fd56e) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #4 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x149161) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #5 PyObject_Vectorcall Objects/call.c:327 (python+0x149161)
    #6 _PyEval_EvalFrameDefault Python/generated_cases.c.h:1367 (python+0x8a1ff) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #7 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x37d7fc) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #8 _PyEval_Vector Python/ceval.c:1761 (python+0x37d7fc)
    #9 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14957b) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #10 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x14f138) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #11 method_vectorcall Objects/classobject.c:72 (python+0x14f138)
    #12 _PyVectorcall_Call Objects/call.c:273 (python+0x14b9b7) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #13 _PyObject_Call Objects/call.c:348 (python+0x14be80) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #14 PyObject_Call Objects/call.c:373 (python+0x14be80)
    #15 thread_run Modules/_threadmodule.c:354 (python+0x52e235) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #16 pythread_wrapper Python/thread_pthread.h:242 (python+0x466079) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)

  Previous write of size 4 at 0x555555cd2530 by thread T599:
    #0 _PyErr_Display Python/pythonrun.c:1127 (python+0x43bad3) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #1 thread_excepthook_file Modules/_threadmodule.c:2189 (python+0x52f068) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #2 thread_excepthook Modules/_threadmodule.c:2268 (python+0x52f068)
    #3 cfunction_vectorcall_O Objects/methodobject.c:523 (python+0x1fd56e) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #4 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x149161) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #5 PyObject_Vectorcall Objects/call.c:327 (python+0x149161)
    #6 _PyEval_EvalFrameDefault Python/generated_cases.c.h:1367 (python+0x8a1ff) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #7 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x37d7fc) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #8 _PyEval_Vector Python/ceval.c:1761 (python+0x37d7fc)
    #9 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14957b) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #10 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x14f138) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #11 method_vectorcall Objects/classobject.c:72 (python+0x14f138)
    #12 _PyVectorcall_Call Objects/call.c:273 (python+0x14b9b7) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #13 _PyObject_Call Objects/call.c:348 (python+0x14be80) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #14 PyObject_Call Objects/call.c:373 (python+0x14be80)
    #15 thread_run Modules/_threadmodule.c:354 (python+0x52e235) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #16 pythread_wrapper Python/thread_pthread.h:242 (python+0x466079) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
  6. colesbury
    added 3 commits that reference this issue on Feb 10, 2025

    pythongh-128130: Fix unhandled keyboard interrupt data race

    9bf5214

    pythongh-128130: Fix unhandled keyboard interrupt data race

    a5267ca

    gh-128130: Fix unhandled keyboard interrupt data race (gh-129975)

    451f291

  7. nascheme commented on Mar 12, 2025

    @nascheme
    nascheme
    on Mar 12, 2025
    Member

    It looks like this bug has been fixed by gh-128130 and can be closed.

    While testing with TSAN, I found another warning and made a PR to fix it: gh-131126.

  8. hawkinsp commented on Mar 21, 2025

    @hawkinsp
    hawkinsp
    on Mar 21, 2025
    ContributorAuthor

    Closing, on the assumption the PR linked above fixed this!

  9. hawkinsp
    closed this as completedon Mar 21, 2025
  10. hawkinsp
    reopened this on Mar 24, 2025

  11. hawkinsp commented on Mar 24, 2025

    @hawkinsp
    hawkinsp
    on Mar 24, 2025
    ContributorAuthor

    Not fixed on the 3.13 branch:

    WARNING: ThreadSanitizer: data race (pid=113280)
  Write of size 4 at 0x567f47699168 by thread T6 (mutexes: read M0):
    #0 run_eval_code_obj /__w/jax/jax/cpython/Python/pythonrun.c:1362:53 (python3.13+0x4a2ee6) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #1 run_mod /__w/jax/jax/cpython/Python/pythonrun.c:1466:19 (python3.13+0x4a26d5) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #2 _PyRun_StringFlagsWithName /__w/jax/jax/cpython/Python/pythonrun.c:1261:15 (python3.13+0x49fa64) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #3 PyRun_StringFlags /__w/jax/jax/cpython/Python/pythonrun.c:1273:12 (python3.13+0x49fa64)
    #4 builtin_exec_impl /__w/jax/jax/cpython/Python/bltinmodule.c (python3.13+0x3d99c2) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #5 builtin_exec /__w/jax/jax/cpython/Python/clinic/bltinmodule.c.h:556:20 (python3.13+0x3d99c2)
    #6 cfunction_vectorcall_FASTCALL_KEYWORDS /__w/jax/jax/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a5d0) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #7 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eb0ea) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #8 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb0ea)
    #9 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e385b) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #10 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df98a) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #11 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1814:12 (python3.13+0x3df98a)
    #12 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb75f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #13 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef7e0) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #14 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef7e0)
    #15 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb3d3) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #16 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb3d3)
    #17 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb455) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #18 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5a42) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #19 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df98a) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #20 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1814:12 (python3.13+0x3df98a)
    #21 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb75f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #22 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:135:15 (python3.13+0x1ea2dd) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #23 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1ebdd7) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #24 slot_tp_call /__w/jax/jax/cpython/Objects/typeobject.c:9556:15 (python3.13+0x2f9718) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #25 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea54c) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #26 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb1a8) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #27 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb1a8)
    #28 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e385b) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #29 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df98a) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #30 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1814:12 (python3.13+0x3df98a)
    #31 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb75f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #32 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef72f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #33 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef72f)
    #34 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb3d3) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #35 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb3d3)
    #36 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb455) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #37 thread_run /__w/jax/jax/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x568202) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #38 pythread_wrapper /__w/jax/jax/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4c1107) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)

  Previous write of size 4 at 0x567f47699168 by thread T7 (mutexes: read M0):
    #0 run_eval_code_obj /__w/jax/jax/cpython/Python/pythonrun.c:1362:53 (python3.13+0x4a2ee6) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #1 run_mod /__w/jax/jax/cpython/Python/pythonrun.c:1466:19 (python3.13+0x4a26d5) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #2 _PyRun_StringFlagsWithName /__w/jax/jax/cpython/Python/pythonrun.c:1261:15 (python3.13+0x49fa64) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #3 PyRun_StringFlags /__w/jax/jax/cpython/Python/pythonrun.c:1273:12 (python3.13+0x49fa64)
    #4 builtin_exec_impl /__w/jax/jax/cpython/Python/bltinmodule.c (python3.13+0x3d99c2) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #5 builtin_exec /__w/jax/jax/cpython/Python/clinic/bltinmodule.c.h:556:20 (python3.13+0x3d99c2)
    #6 cfunction_vectorcall_FASTCALL_KEYWORDS /__w/jax/jax/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a5d0) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #7 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eb0ea) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #8 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb0ea)
    #9 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e385b) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #10 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df98a) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #11 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1814:12 (python3.13+0x3df98a)
    #12 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb75f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #13 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef7e0) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #14 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef7e0)
    #15 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb3d3) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #16 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb3d3)
    #17 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb455) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #18 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5a42) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #19 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df98a) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #20 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1814:12 (python3.13+0x3df98a)
    #21 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb75f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #22 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:135:15 (python3.13+0x1ea2dd) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #23 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1ebdd7) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #24 slot_tp_call /__w/jax/jax/cpython/Objects/typeobject.c:9556:15 (python3.13+0x2f9718) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #25 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea54c) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #26 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb1a8) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #27 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb1a8)
    #28 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e385b) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #29 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df98a) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #30 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1814:12 (python3.13+0x3df98a)
    #31 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb75f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #32 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef72f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #33 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef72f)
    #34 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb3d3) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #35 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb3d3)
    #36 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb455) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #37 thread_run /__w/jax/jax/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x568202) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #38 pythread_wrapper /__w/jax/jax/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4c1107) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)

  12. colesbury commented on Mar 24, 2025

    @colesbury
    colesbury
    on Mar 24, 2025
    Contributor

    The PR was merged into main, but I didn't backport it to 3.13. I've been generally hesitant to backport PRs that primarily fix TSAN warnings as opposed to crashes or other user visible misbehavior.

    If it makes testing Jax easier, let me know and we can reconsider backporting the fix.

  13. hawkinsp commented on Mar 24, 2025

    @hawkinsp
    hawkinsp
    on Mar 24, 2025
    ContributorAuthor

    The PR was merged into main, but I didn't backport it to 3.13. I've been generally hesitant to backport PRs that primarily fix TSAN warnings as opposed to crashes or other user visible misbehavior.

    If it makes testing Jax easier, let me know and we can reconsider backporting the fix.

    It's not critical. It's just an entry in a tsan suppression file.
    https://github.com/jax-ml/jax/blob/d3836f160cc6270280ba75c0ae5b12cf1b99535f/.github/workflows/tsan-suppressions.txt#L27
    I'm just trying to be aggressive in making sure that these issues get fixed upstream since I want 3.14 to be race free for us.

    Actually, we're doing pretty well overall. We're down to only 2 unfixed races in our test suite (#129748, #128050), assuming the things tagged as fixed in 3.14 are indeed fixed.

  14. colesbury commented on Mar 24, 2025

    @colesbury
    colesbury
    on Mar 24, 2025
    Contributor

    Are you able to test with 3.14 as well? If not, what are the blockers?

  15. hawkinsp commented on Mar 24, 2025

    @hawkinsp
    hawkinsp
    on Mar 24, 2025
    ContributorAuthor

    Are you able to test with 3.14 as well? If not, what are the blockers?

    No blockers. We just don't have that test. We've done it manually a few times. Let me add that to our CI.

  16. hawkinsp commented on Apr 7, 2025

    @hawkinsp
    hawkinsp
    on Apr 7, 2025
    ContributorAuthor

    Closing, we haven't seen this race under 3.14.

  17. hawkinsp
    closed this as completedon Apr 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.13only security fixes3.14bugs and security fixesextension-modulesC modules in the Modules dirtopic-free-threadingtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions