Skip to content

test_concurrent_futures.test_interpreter_pool failing #125716

Closed
Closed
test_concurrent_futures.test_interpreter_pool failing#125716
Assignees
ericsnowcurrently
Labels
3.14bugs and security fixestopic-subinterpreterstype-bugAn unexpected behavior, bug, or error

Description

@ericsnowcurrently
ericsnowcurrently
opened on Oct 18, 2024

Bug report

Bug description:

I've seen 4 kinds of failure which I'm failure sure have the same cause:

  • segfault during WorkerContext.initialize() (line 137)
  • hanging
  • weird test failure
  • undefined behavior on USAN buildbot

The failures have happened in different test methods. Different failures have happened during the retry. Sometimes the retry passes. In all cases the architecture is AMD64, but across a variety of builders and non-Windows operating systems. The failures have all been on either refleaks buildbots or the USAN buildbot.

FWIW, it looks like InterpreterPoolExecutor has only exposed an underlying problem in the _interpqueues module, which means any fix would need to target 3.13 also (and maybe 3.12).

Here are the buildbots where I've seen failures:

  • AMD64 RHEL8 Refleaks 3.x
  • AMD64 FreeBSD Refleaks 3.x
  • AMD64 CentOS9 NoGIL Refleaks 3.x
  • AMD64 Arch Linux Usan Function 3.x

Here's the failure text:

segfault 
test_submit (test.test_concurrent_futures.test_interpreter_pool.InterpreterPoolExecutorTest.test_submit) ... Fatal Python error:

Segmentation fault
Current thread 0x00007fb4fbfff700 (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/concurrent/futures/interpreter.py", line 137 in initialize
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/concurrent/futures/thread.py", line 98 in _worker
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/threading.py", line 992 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/threading.py", line 1041 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/threading.py", line 1012 in _bootstrap

Thread 0x00007fb521d71240 (most recent call first):
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/threading.py", line 359 in wait
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/concurrent/futures/_base.py", line 443 in result
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/test_concurrent_futures/executor.py", line 31 in test_submit
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/unittest/case.py", line 660 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/unittest/case.py", line 716 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/unittest/suite.py", line 122 in run  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/unittest/runner.py", line 240 in run
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/libregrtest/single.py", line 57 in _run_suite
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/libregrtest/single.py", line 37 in run_unittest
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/libregrtest/single.py", line 135 in test_func
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/libregrtest/refleak.py", line 132 in runtest_refleak
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/libregrtest/single.py", line 87 in regrtest_runner
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/libregrtest/single.py", line 138 in _load_run_test
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/libregrtest/single.py", line 181 in _runtest_env_changed_exc
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/libregrtest/single.py", line 281 in _runtest
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/libregrtest/single.py", line 310 in run_single_test
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/libregrtest/worker.py", line 83 in worker_process
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/libregrtest/worker.py", line 118 in main
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/test/libregrtest/worker.py", line 122 in <module>
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/runpy.py", line 88 in _run_code
  File "/home/buildbot/buildarea/3.x.cstratak-RHEL8-x86_64.refleak/build/Lib/runpy.py", line 198 in _run_module_as_main
hang 1 
test_submit_exception_in_func (test.test_concurrent_futures.test_interpreter_pool.InterpreterPoolExecutorTest.test_submit_exception_in_func) ... Timeout (3:20:00)!

Thread 0x000000082e546e00 (most recent call first):  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/concurrent/futures/interpreter.py", line 190 in run
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/concurrent/futures/thread.py", line 85 in run
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/concurrent/futures/thread.py", line 118 in _worker
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/threading.py", line 992 in run
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/threading.py", line 1041 in _bootstrap_inner
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/threading.py", line 1012 in _bootstrap

Thread 0x0000000825a7c000 (most recent call first):
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/threading.py", line 359 in wait
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/concurrent/futures/_base.py", line 443 in result
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/test_concurrent_futures/test_interpreter_pool.py", line 251 in test_submit_exception_in_func
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/unittest/case.py", line 660 in run
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/unittest/case.py", line 716 in __call__
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/unittest/suite.py", line 122 in run
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/unittest/suite.py", line 84 in __call__
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/unittest/suite.py", line 122 in run
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/unittest/suite.py", line 84 in __call__
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/unittest/runner.py", line 240 in run
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/libregrtest/single.py", line 57 in _run_suite
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/libregrtest/single.py", line 37 in run_unittest
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/libregrtest/single.py", line 135 in test_func
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/libregrtest/refleak.py", line 132 in runtest_refleak
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/libregrtest/single.py", line 87 in regrtest_runner
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/libregrtest/single.py", line 138 in _load_run_test
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/libregrtest/single.py", line 181 in _runtest_env_changed_exc
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/libregrtest/single.py", line 281 in _runtest
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/libregrtest/single.py", line 310 in run_single_test
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/libregrtest/worker.py", line 83 in worker_process
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/libregrtest/worker.py", line 118 in main
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/test/libregrtest/worker.py", line 122 in <module>
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/runpy.py", line 88 in _run_code
  File "/buildbot/buildarea/3.x.ware-freebsd.refleak/build/Lib/runpy.py", line 198 in _run_module_as_main
hang 2 
test_shutdown_race_issue12456 (test.test_concurrent_futures.test_interpreter_pool.InterpreterPoolExecutorTest.test_shutdown_race_issue12456) ... Exception in initializer:
RuntimeError: Failed to import encodings module

During handling of the above exception, another exception occurred:

interpreters.Interpreter Error: sub-interpreter creation failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/concurrent/futures/thread.py", line 98, in _worker
    ctx.initialize()
    ~~~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/concurrent/futures/interpreter.py", line 131, in initialize
    self.interpid = _interpreters.create(reqrefs=True)
                    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
interpreters.InterpreterError: interpreter creation failed
Timeout (0:45:00)!

Thread 0x00007fe3febe2640 (most recent call first):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/concurrent/futures/thread.py", line 115 in _worker
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 992 in run
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 1041 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 1012 in _bootstrap

Thread 0x00007fe3fcbda640 (most recent call first):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/concurrent/futures/thread.py", line 115 in _worker
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 992 in run
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 1041 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 1012 in _bootstrap

Thread 0x00007fe3e77fe640 (most recent call first):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/concurrent/futures/thread.py", line 115 in _worker
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 992 in run
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 1041 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 1012 in _bootstrap

Thread 0x00007fe3e7fff640 (most recent call first):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/concurrent/futures/thread.py", line 115 in _worker
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 992 in run
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 1041 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 1012 in _bootstrap

Thread 0x00007fe3ffe6f740 (most recent call first):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/threading.py", line 1092 in join
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/concurrent/futures/thread.py", line 272 in shutdown
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/test_concurrent_futures/executor.py", line 79 in test_shutdown_race_issue12456
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/unittest/case.py", line 660 in run
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/unittest/case.py", line 716 in __call__
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/unittest/runner.py", line 240 in run
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/libregrtest/single.py", line 57 in _run_suite
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/libregrtest/single.py", line 37 in run_unittest
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/libregrtest/single.py", line 135 in test_func
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/libregrtest/refleak.py", line 132 in runtest_refleak
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/libregrtest/single.py", line 87 in regrtest_runner
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/libregrtest/single.py", line 138 in _load_run_test
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/libregrtest/single.py", line 181 in _runtest_env_changed_exc
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/libregrtest/single.py", line 281 in _runtest
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/libregrtest/single.py", line 310 in run_single_test
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/libregrtest/worker.py", line 83 in worker_process
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/libregrtest/worker.py", line 118 in main
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/libregrtest/worker.py", line 122 in <module>
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/runpy.py", line 88 in _run_code
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/runpy.py", line 198 in _run_module_as_main
test failed 
======================================================================
FAIL: test_free_reference (test.test_concurrent_futures.test_interpreter_pool.InterpreterPoolExecutorTest.test_free_reference)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.itamaro-centos-aws.refleak.nogil/build/Lib/test/test_concurrent_futures/executor.py", line 132, in test_free_reference
    self.assertIsNone(wr())
    ~~~~~~~~~~~~~~~~~^^^^^^
AssertionError: <test.test_concurrent_futures.executor.MyObject object at 0x200121a00a0> is not None
USAN 
test_map_exception (test.test_concurrent_futures.test_interpreter_pool.InterpreterPoolExecutorTest.test_map_exception) ... Python/thread_pthread.h:555:42: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/semaphore.h:55:36: note: nonnull attribute specified here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/thread_pthread.h:555:42 in 
Fatal Python error: Segmentation fault

Thread 0x00007f89aedfd6c0 (most recent call first):  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/concurrent/futures/interpreter.py", line 131 in initialize
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/concurrent/futures/thread.py", line 98 in _worker
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 992 in run
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 1041 in _bootstrap_inner
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 1012 in _bootstrap

Thread 0x00007f89adcfb6c0 (most recent call first):  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/concurrent/futures/interpreter.py", line 131 in initialize
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/concurrent/futures/thread.py", line 98 in _worker
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 992 in run
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 1041 in _bootstrap_inner
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 1012 in _bootstrap

Current thread 0x00007f89af7fe6c0 (most recent call first):  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/concurrent/futures/interpreter.py", line 137 in initialize
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/concurrent/futures/thread.py", line 98 in _worker
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 992 in run
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 1041 in _bootstrap_inner
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 1012 in _bootstrap

Thread 0x00007f89ad4fa6c0 (most recent call first):  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/concurrent/futures/interpreter.py", line 131 in initialize
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/concurrent/futures/thread.py", line 98 in _worker
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 992 in run
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 1041 in _bootstrap_inner
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 1012 in _bootstrap

Thread 0x00007f89b64500c0 (most recent call first):
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/threading.py", line 359 in wait
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/concurrent/futures/_base.py", line 443 in result
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/concurrent/futures/_base.py", line 309 in _result_or_cancel
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/concurrent/futures/_base.py", line 611 in result_iterator
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/test/test_concurrent_futures/executor.py", line 54 in test_map_exception
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/unittest/case.py", line 606 in _callTestMethod
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/unittest/case.py", line 660 in run
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/unittest/case.py", line 716 in __call__
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/unittest/suite.py", line 122 in run
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/unittest/suite.py", line 84 in __call__
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/unittest/suite.py", line 122 in run
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/unittest/suite.py", line 84 in __call__
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/unittest/runner.py", line 240 in run
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/test/libregrtest/single.py", line 57 in _run_suite
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/test/libregrtest/single.py", line 37 in run_unittest
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/test/libregrtest/single.py", line 135 in test_func
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/test/libregrtest/single.py", line 91 in regrtest_runner
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/test/libregrtest/single.py", line 138 in _load_run_test
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/test/libregrtest/single.py", line 181 in _runtest_env_changed_exc
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/test/libregrtest/single.py", line 281 in _runtest
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/test/libregrtest/single.py", line 310 in run_single_test
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/test/libregrtest/worker.py", line 83 in worker_process
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/test/libregrtest/worker.py", line 118 in main
  File "/buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Lib/test/libregrtest/worker.py", line 122 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Extension modules: _testinternalcapi (total: 1)
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==2260212==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x03e800227cf4 (pc 0x7f89b64e1194 bp 0x7f89af7fd370 sp 0x7f89af7fd330 T2260241)
==2260212==The signal is caused by a READ memory access.
    #0 0x7f89b64e1194  (/usr/lib/libc.so.6+0x90194) (BuildId: 915eeec6439cfded1125deefc44a8d73e57873d9)
    #1 0x7f89b648dd6f in raise (/usr/lib/libc.so.6+0x3cd6f) (BuildId: 915eeec6439cfded1125deefc44a8d73e57873d9)
    #2 0x555a15c75b3d in faulthandler_fatal_error /buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/./Modules/faulthandler.c:338:5
    #3 0x7f89b648de1f  (/usr/lib/libc.so.6+0x3ce1f) (BuildId: 915eeec6439cfded1125deefc44a8d73e57873d9)
    #4 0x7f89b64e7504 in sem_wait (/usr/lib/libc.so.6+0x96504) (BuildId: 915eeec6439cfded1125deefc44a8d73e57873d9)
    #5 0x555a15c3e86b in PyThread_acquire_lock_timed /buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Python/thread_pthread.h:555:33
    #6 0x7f89b47308c6 in _queues_add /buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/./Modules/_interpqueuesmodule.c:909:5
    #7 0x7f89b47308c6 in queue_create /buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/./Modules/_interpqueuesmodule.c:1103:19
    #8 0x7f89b47308c6 in queuesmod_create /buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/./Modules/_interpqueuesmodule.c:1487:19
    #9 0x555a157a3caa in cfunction_call /buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Objects/methodobject.c:551:18
    #10 0x555a15677105 in _PyObject_MakeTpCall /buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Objects/call.c:242:18
    #11 0x555a15a556ef in _PyEval_EvalFrameDefault /buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Python/generated_cases.c.h:2759:35
    #12 0x555a15680c7c in _PyObject_VectorcallTstate /buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/./Include/internal/pycore_call.h:167:11
    #13 0x555a1567e32f in method_vectorcall /buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Objects/classobject.c:71:20
    #14 0x555a15da9af5 in thread_run /buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/./Modules/_threadmodule.c:337:21
    #15 0x555a15c3f5f3 in pythread_wrapper /buildbot/buildarea/3.x.pablogsal-arch-x86_64.clang-ubsan-function/build/Python/thread_pthread.h:242:5
    #16 0x7f89b64df1ce  (/usr/lib/libc.so.6+0x8e1ce) (BuildId: 915eeec6439cfded1125deefc44a8d73e57873d9)
    #17 0x7f89b65606eb  (/usr/lib/libc.so.6+0x10f6eb) (BuildId: 915eeec6439cfded1125deefc44a8d73e57873d9)

UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: SEGV (/usr/lib/libc.so.6+0x90194) (BuildId: 915eeec6439cfded1125deefc44a8d73e57873d9)
==2260212==ABORTING

CC @encukou

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

Activity

  1. ericsnowcurrently
    added
    type-bugAn unexpected behavior, bug, or error
    topic-subinterpreters
    3.14bugs and security fixes
    on Oct 18, 2024
  2. ericsnowcurrently
    self-assigned this
    on Oct 18, 2024
  3. github-project-automation
    added this to Subinterpreterson Oct 18, 2024
  4. github-project-automation
    moved this to Todo in Subinterpreterson Oct 18, 2024

  5. ZeroIntensity commented on Oct 19, 2024

    @ZeroIntensity
    ZeroIntensity
    on Oct 19, 2024
    Member

    I've been looking into this for the past hour or so, and I wasn't able to reproduce a segfault, nor do any debuggers detect any foul play.

    However, I was able to get this assertion to fail after sending a CTRL+C to unittest. That, and running test_interpreter_pool with a TSan build causes it to absolutely explode with errors, but I wouldn't be surprised if they were all false positives--TSan is far from perfect (do we even support it on GIL-ful builds?)

  6. rruuaanng commented on Oct 19, 2024

    @rruuaanng
    rruuaanng
    on Oct 19, 2024
    Contributor

    Oh, it turns out this is a BUG. I triggered this while implementing the PR.

  7. ericsnowcurrently commented on Oct 21, 2024

    @ericsnowcurrently
    ericsnowcurrently
    on Oct 21, 2024
    MemberAuthor

    The biggest clue is what the USAN buildbot tells us:

    • a NULL pointer is being passed to sem_wait()
    • the pointer points to uninitialized/deallocated memory
    • the failure happens when the executor's worker context is initialized and calls _interpqueues.create()
    Python/thread_pthread.h:555:42: runtime error: null pointer passed as argument 1, which is declared to never be null
/usr/include/semaphore.h:55:36: note: nonnull attribute specified here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Python/thread_pthread.h:555:42 in 

Current thread 0x00007f89af7fe6c0 (most recent call first):
  File "Lib/concurrent/futures/interpreter.py", line 137 in initialize

==2260212==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x03e800227cf4
==2260212==The signal is caused by a READ memory access.
    #4 0x7f89b64e7504 in sem_wait (/usr/lib/libc.so.6+0x96504) (BuildId: 915eeec6439cfded1125deefc44a8d73e57873d9)
    #5 0x555a15c3e86b in PyThread_acquire_lock_timed Python/thread_pthread.h:555:33
    #6 0x7f89b47308c6 in _queues_add Modules/_interpqueuesmodule.c:909:5
    #7 0x7f89b47308c6 in queue_create Modules/_interpqueuesmodule.c:1103:19
    #8 0x7f89b47308c6 in queuesmod_create Modules/_interpqueuesmodule.c:1487:19

    The pointer in question is the mutex created in _globals_init(), which is called by the module exec function the first time the module is loaded. The mutex is cleared (in _globals_fini()) when the last copy of the module is cleared.

    There is a unlikely-but-possible race there in _globals_init() with the module count, which may play a part here. It would make sense to do the following:

    • make the lock a PyMutex
    • use atomic operations for the module count

    That may improve the situation. However, it would be best if we could definitively determine why the mutex is NULL when it shouldn't be.

  8. ZeroIntensity commented on Oct 21, 2024

    @ZeroIntensity
    ZeroIntensity
    on Oct 21, 2024
    Member

    Something that could possibly be related: I've noticed an issue with exceptions inside subinterpreters created by _interpreters that causes them to finalize earlier than they should--depending on what's going on, that could explain the NULL mutex. I'm investigating to see if that's an issue on my end, or something deeper.

  9. ericsnowcurrently
    added a commit that references this issue on Oct 21, 2024

    gh-125716: Raise an Exception If _globals_init() Fails In the _interp…

    44f841f
  10. miss-islington
    added a commit that references this issue on Oct 21, 2024

    pythongh-125716: Raise an Exception If _globals_init() Fails In the _…

    6a8081c
  11. ericsnowcurrently
    added 2 commits that reference this issue on Oct 21, 2024

    [3.13] gh-125716: Raise an Exception If _globals_init() Fails In the …

    b752764

    gh-125716: Use A Global Mutex When Initializing Global State For The …

    4848b0b
  12. miss-islington
    added a commit that references this issue on Oct 21, 2024

    pythongh-125716: Use A Global Mutex When Initializing Global State Fo…

    b739d2e
  13. ericsnowcurrently
    added a commit that references this issue on Oct 21, 2024

    [3.13] gh-125716: Use a Global Mutex When Initializing Global State F…

    0e350d4

  14. ericsnowcurrently commented on Oct 22, 2024

    @ericsnowcurrently
    ericsnowcurrently
    on Oct 22, 2024
    MemberAuthor

    It looks like the latest fix has helped. I'll reopen this if there are any intermittent failures.

  15. ericsnowcurrently
    closed this as completedon Oct 22, 2024
  16. github-project-automation
    moved this from Todo to Done in Subinterpreterson Oct 22, 2024
  17. ebonnal
    added 2 commits that reference this issue on Jan 12, 2025

    pythongh-125716: Raise an Exception If _globals_init() Fails In the _…

    c594724

    pythongh-125716: Use A Global Mutex When Initializing Global State Fo…

    d996135
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.14bugs and security fixestopic-subinterpreterstype-bugAn unexpected behavior, bug, or error

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions