Repository navigation
test_asyncio environment changed during test execution due to empty threading._dangling without MainThread before some tests #91676
Description
We're seeing frequent flakiness on CI and buildbots (heavily loaded systems, triggering more race conditions?) with test_asyncio of late. #91260 has some discussion as that seems to have surfaced it more frequently, at least in the stable 3.10 branch.
$ python -m test.bisect_cmd test_asyncio --fail-env-changed
...
0:00:00 load avg: 0.93 [1/2/1] test_asyncio.test_threads failed (env changed)
Warning -- threading._dangling was modified by test_asyncio.test_threads
Before: {<weakref at 0x7f3b2e77fa10; to '_MainThread' at 0x7f3b2f107490>}
After: {<weakref at 0x7f3b2e77ffb0; to '_MainThread' at 0x7f3b2f107490>}
from a 3.10 branch with #91674 patched in.
0:09:20 load avg: 1.49 [427/427/1] test_asyncio failed (env changed) (1 min 22 sec)
Warning -- threading_cleanup() failed to cleanup -1 threads (count: 0, dangling: 1)
Warning -- Dangling thread: <_MainThread(MainThread, started 140300013757632)>
from https://github.com/python/cpython/runs/6069450575?check_suite_focus=true CI and from many stable buildbots such as https://buildbot.python.org/all/#/builders/608/builds/700/steps/6/logs/stdio
While "interesting" for _MainThread to wind up in threading._dangling... code wise it should always be in there. Yet somehow it is not before execution?!?
after a fork() from the non-main thread or from C, it won't be. danger - something is using fork with threads running, a guaranteed source of problems. Fixing it to always exist in _dangling beforehand would not be a bad thing, but only if we can figure out how it happens to not be in the first place.
Activity
[-]test environment chained due to empty `threading._dangling` without MainThread before some tests[/-][+]test_asyncio environment changed during test execution due to empty `threading._dangling` without MainThread before some tests[/+]gpshead commentedon Apr 18, 2022
I've also seen this one:
gpshead commentedon Apr 18, 2022
notice the specific weakref instance to the same address
_MainThreadchanged above and here:gpshead commentedon Apr 19, 2022
The code in
Lib/test/libregrtest/save_env.pyhas race conditions that could lead to this https://github.com/python/cpython/blob/main/Lib/test/libregrtest/save_env.py#L216 are invoked by enter and exit on thesaved_test_environmentcontext manager. seeresource_infofor how.gpshead commentedon Apr 19, 2022
the irony of code that is supposed to help buildbot instability also being one probable cause of buildbot instability...
gpshead commentedon Apr 19, 2022
The excess threads running on occasion after a test ends are asyncio concurrent.futures threadpool _worker threads named asyncio_0 or asyncio_1...
('<Thread(asyncio_0, started 140097331353152)>', (<function _worker at 0x7f6af3c3f5f0>, '_worker', 'concurrent.futures.thread', None))The change that exacerbated this race condition notably no longer does a
loop.run_until_complete(loop.shutdown_default_executor())which is what I believe would've seen the worker threads cleaned up? [edit: confirmed, PR coming][3.10] gh-91676 gh-91260 unittest.IsolatedAsyncioTestCase no longer l…
[3.10] pythongh-91676 pythongh-91260 unittest.IsolatedAsyncioTestCase…
[3.10] gh-91676 gh-91260 unittest.IsolatedAsyncioTestCase no longer l…
vstinner commentedon May 13, 2022
Thanks for the fix ;-)
[3.10] pythongh-91676 pythongh-91260 unittest.IsolatedAsyncioTestCase…