Repository navigation
test_concurrent_futures.test_deadlock: test_crash_big_data() hangs randomly on Windows #107219
Description
Activity
By the way, test.regrtest doesn't work with test_concurrent_futures: when test_concurrent_futures is re-run in verbose mode, no tests is ran!
0:40:52 Re-running test_concurrent_futures in verbose mode ---------------------------------------------------------------------- Ran 0 tests in 0.001s NO TESTS RANThis second bug maybe hides the first bug (test_concurrent_futures hangs sometimes).
Windows x64 job, also blocked on
test_crash_big_data(): https://github.com/python/cpython/actions/runs/5652189009/job/15311447694Similar threads state.
0:22:04 load avg: 5.60 [447/447/2] test_concurrent_futures crashed (Exit code 1) Timeout (0:20:00)! Thread 0x00001be0 (most recent call first): File "D:\a\cpython\cpython\Lib\multiprocessing\connection.py", line 282 in _send_bytes File "D:\a\cpython\cpython\Lib\multiprocessing\connection.py", line 199 in send_bytes File "D:\a\cpython\cpython\Lib\multiprocessing\queues.py", line 246 in _feed (...) File "D:\a\cpython\cpython\Lib\threading.py", line 1009 in _bootstrap Thread 0x000002a0 (most recent call first): File "D:\a\cpython\cpython\Lib\threading.py", line 1146 in _wait_for_tstate_lock File "D:\a\cpython\cpython\Lib\threading.py", line 1126 in join (...) File "D:\a\cpython\cpython\Lib\threading.py", line 1009 in _bootstrap Thread 0x00000c18 (most recent call first): File "D:\a\cpython\cpython\Lib\threading.py", line 1146 in _wait_for_tstate_lock File "D:\a\cpython\cpython\Lib\threading.py", line 1126 in join File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 836 in shutdown File "D:\a\cpython\cpython\Lib\concurrent\futures\_base.py", line 647 in __exit__ File "D:\a\cpython\cpython\Lib\test\test_concurrent_futures.py", line 1386 in test_crash_big_data (...)Azure Pipelines: Windows PR Tests win32 hangs on
test_interpreter_shutdown(): https://dev.azure.com/Python/cpython/_build/results?buildId=133201&view=logs&j=d554cd63-f8f4-5b2d-871b-33e4ea76e915&t=5a14d0eb-dbd4-5b80-f5d0-7909f950a1cc- Main thread:
run_python_until_end()=>subprocess.Popen.communicate() - Thread 2 (stdout?): subprocess _readerthread()
- Thread 3 (stderr?): subprocess _readerthread()
In short, the main thread is waiting until the process completes and the process is killed after 20 minutes.
On the same CI, the win64 job ran test_concurrent_futures in 1 min 33 sec.
(...) test_hang_gh94440 (test.test_concurrent_futures.ProcessPoolSpawnProcessPoolShutdownTest.test_hang_gh94440) shutdown(wait=True) doesn't hang when a future was submitted and ... skipped 'Tested platform does not support the alarm signal' test_hang_issue12364 (test.test_concurrent_futures.ProcessPoolSpawnProcessPoolShutdownTest.test_hang_issue12364) ... ok Timeout (0:20:00)! Thread 0x00001184 (most recent call first): File "D:\a\1\s\Lib\subprocess.py", line 1597 in _readerthread File "D:\a\1\s\Lib\threading.py", line 989 in run File "D:\a\1\s\Lib\threading.py", line 1052 in _bootstrap_inner File "D:\a\1\s\Lib\threading.py", line 1009 in _bootstrap Thread 0x00001224 (most recent call first): File "D:\a\1\s\Lib\subprocess.py", line 1597 in _readerthread File "D:\a\1\s\Lib\threading.py", line 989 in run File "D:\a\1\s\Lib\threading.py", line 1052 in _bootstrap_inner File "D:\a\1\s\Lib\threading.py", line 1009 in _bootstrap Thread 0x00001560 (most recent call first): File "D:\a\1\s\Lib\threading.py", line 1146 in _wait_for_tstate_lock File "D:\a\1\s\Lib\threading.py", line 1126 in join File "D:\a\1\s\Lib\subprocess.py", line 1626 in _communicate File "D:\a\1\s\Lib\subprocess.py", line 1209 in communicate File "D:\a\1\s\Lib\test\support\script_helper.py", line 139 in run_python_until_end File "D:\a\1\s\Lib\test\support\script_helper.py", line 149 in _assert_python File "D:\a\1\s\Lib\test\support\script_helper.py", line 166 in assert_python_ok File "D:\a\1\s\Lib\test\test_concurrent_futures.py", line 302 in test_interpreter_shutdown (...) File "<frozen runpy>", line 198 in _run_module_as_main- Main thread:
I reported a similar issue in May 2022, but closed it as it seemed the issue stopped occurring in CI:
./python -m test -v test_concurrent_futures -m test_crash_big_data --foreverget me this:many lines.. 0:00:05 [ 9] test_concurrent_futures test_crash_big_data (test.test_concurrent_futures.ProcessPoolForkExecutorDeadlockTest.test_crash_big_data) ... skipped 'require un ix system' test_crash_big_data (test.test_concurrent_futures.ProcessPoolForkserverExecutorDeadlockTest.test_crash_big_data) ... skipped 'requ ire unix system' test_crash_big_data (test.test_concurrent_futures.ProcessPoolSpawnExecutorDeadlockTest.test_crash_big_data) ... Warning -- Uncaugh t thread exception: InvalidStateError Exception in thread Thread-9: Traceback (most recent call last): File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\threading.py", line 1059, in _bootstrap_inner self.run() File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\process.py", line 344, in run self.terminate_broken(cause) File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\process.py", line 492, in terminate_broken work_item.future.set_exception(bpe) File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\_base.py", line 559, in set_exception raise InvalidStateError('{}: {!r}'.format(self._state, self)) concurrent.futures._base.InvalidStateError: CANCELLED: <Future at 0x2cf561cf260 state=cancelled> 0.55s Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2) Warning -- Dangling thread: <_MainThread(MainThread, started 11880)> Warning -- Dangling thread: <Thread(QueueFeederThread, started daemon 4112)> ok Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2) Warning -- Dangling thread: <_MainThread(MainThread, started 11880)> Warning -- Dangling thread: <Thread(QueueFeederThread, started daemon 4112)>
Sadly, but it's hard to reproduce.
Sadly, but it's hard to reproduce.
You can stress the system to make the issue more likely. For example, open a second terminal and run:
python -m test -j2You can use
-j4or more depending on the number of CPUs and how much you want your machine to be stressed :-)Sadly, but it's hard to reproduce.
You can stress the system to make the issue more likely. For example, open a second terminal and run:
python -m test -j2You can use
-j4or more depending on the number of CPUs and how much you want your machine to be stressed :-)Oh, that's right! With -j8 (in a separate terminal) I can reproduce bug more easily.
(I'm also seeing this hang in the mingw fork after updating from 3.11.4 to 3.11.5)
Error on Linux, not sure if it's related.
aarch64 Fedora Stable LTO + PGO 3.x buildbot: https://buildbot.python.org/all/#/builders/524/builds/4310
Log (reformatted for readability):
FAIL: test_interpreter_shutdown (test.test_concurrent_futures.test_shutdown.ProcessPoolForkserverProcessPoolShutdownTest.test_interpreter_shutdown) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/test/test_concurrent_futures/test_shutdown.py", line 49, in test_interpreter_shutdown self.assertFalse(err) AssertionError: b'Exception in thread Thread-1: Traceback (most recent call last): File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/threading.py", line 1059, in _bootstrap_inner self.run() File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/concurrent/futures/process.py", line 339, in run self.add_call_item_to_queue() File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/concurrent/futures/process.py", line 394, in add_call_item_to_queue self.call_queue.put(_CallItem(work_id, File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/multiprocessing/queues.py", line 94, in put self._start_thread() File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/multiprocessing/queues.py", line 177, in _start_thread self._thread.start() File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/threading.py", line 978, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can\'t create new thread at interpreter shutdown Traceback (most recent call last): File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/multiprocessing/forkserver.py", line 274, in main code = _serve_one(child_r, fds, ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/multiprocessing/forkserver.py", line 313, in _serve_one code = spawn._main(child_r, parent_sentinel) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/multiprocessing/spawn.py", line 132, in _main self = reduction.pickle.load(from_parent) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.lto-pgo/build/Lib/multiprocessing/synchronize.py", line 115, in __setstate__ self._semlock = _multiprocessing.SemLock._rebuild(*state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory ' is not falseSimilar error on another Linux machine.
ARM Raspbian 3.x: https://buildbot.python.org/all/#/builders/424/builds/4736
Logs (reformatted):
FAIL: test_interpreter_shutdown (test.test_concurrent_futures.test_shutdown.ProcessPoolForkserverProcessPoolShutdownTest.test_interpreter_shutdown) ---------------------------------------------------------------------- Traceback (most recent call last): File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/test/test_concurrent_futures/test_shutdown.py", line 49, in test_interpreter_shutdown self.assertFalse(err) AssertionError: b'Exception in thread Thread-1: Traceback (most recent call last): File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/threading.py", line 1059, in _bootstrap_inner self.run() File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/concurrent/futures/process.py", line 339, in run self.add_call_item_to_queue() File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/concurrent/futures/process.py", line 394, in add_call_item_to_queue self.call_queue.put(_CallItem(work_id, File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/queues.py", line 94, in put self._start_thread() File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/queues.py", line 177, in _start_thread self._thread.start() File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/threading.py", line 978, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can\'t create new thread at interpreter shutdown Traceback (most recent call last): File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/forkserver.py", line 274, in main code = _serve_one(child_r, fds, ^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/forkserver.py", line 313, in _serve_one code = spawn._main(child_r, parent_sentinel) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/spawn.py", line 132, in _main self = reduction.pickle.load(from_parent) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/buildbot/workers/3.x.gps-raspbian.nondebug/build/Lib/multiprocessing/synchronize.py", line 115, in __setstate__ self._semlock = _multiprocessing.SemLock._rebuild(*state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory ' is not false- added 2 commits that reference this issue
on Aug 26, 2023 35 remaining items
- added 5 commits that reference this issue
on Feb 19, 2024
GHA Windows x86 job, test_crash_big_data() hangs on
ProcessPoolExecutor.shutdown(): https://github.com/python/cpython/actions/runs/5651960914/job/15310873235?pr=107217ProcessPoolExecutor.shutdown()Threading.join()_feed()=> connectionsend_bytes()Linked PRs