Repository navigation
test_interpreters Failing Intermittently Under Py_DISABLE_GIL #118731
Description
Bug report
Bug description:
On one of my recent PRs, gh-118157, I was getting consistent failures on CI in test_interpreters. Different tests would fail, but the error was always the same:
(expand)
0:08:24 load avg: 4.18 Re-running 1 failed tests in verbose mode in subprocesses
0:08:24 load avg: 4.18 Run 1 test in parallel using 1 worker process (timeout: 10 min, worker timeout: 15 min)
0:08:26 load avg: 3.93 [1/1/1] test_interpreters failed (6 failures)
Re-running test_interpreters in verbose mode (matching: test_display_preserved_exception, test_gh_109793, test_sys_path_0, test_sys_path_0, test_sys_path_0, test_sys_path_0)
test_display_preserved_exception (test.test_interpreters.test_api.TestInterpreterExec.test_display_preserved_exception) ... FAIL
test_gh_109793 (test.test_interpreters.test_lifecycle.FinalizationTests.test_gh_109793) ... FAIL
test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) ...
### start subtest debug ###
# python3 script.py failed:
--- stdout ---
{
"main": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4",
"sub": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4"
}
------
--- stderr ---
Exception ignored in: <function Interpreter.__del__ at 0x2000119aed0>
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x00005639aed8ea10)
------
### end subtest debug ###
test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 script.py] ... FAIL
### start subtest debug ###
# python3 -m script failed:
--- stdout ---
{
"main": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4",
"sub": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4"
}
------
--- stderr ---
Exception ignored in: <function Interpreter.__del__ at 0x2000117c0d0>
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x0000564591b58a10)
------
### end subtest debug ###
test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m script] ... FAIL
### start subtest debug ###
# python3 -m pkg failed:
--- stdout ---
{
"main": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4",
"sub": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4"
}
------
--- stderr ---
Exception ignored in: <function Interpreter.__del__ at 0x2000118c0d0>
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x000055aaa11eaa10)
------
### end subtest debug ###
test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m pkg] ... FAIL
### start subtest debug ###
# python3 -m pkg.script failed:
--- stdout ---
{
"main": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4",
"sub": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4"
}
------
--- stderr ---
Exception ignored in: <function Interpreter.__del__ at 0x2000118c0d0>
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x00005637ca7cfa10)
------
### end subtest debug ###
test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m pkg.script] ... FAIL
======================================================================
FAIL: test_display_preserved_exception (test.test_interpreters.test_api.TestInterpreterExec.test_display_preserved_exception)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/__init__.py", line 2603, in wrapper
return func(*args, **kwargs)
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_api.py", line 764, in test_display_preserved_exception
self.assertEqual(stderr, dedent(f"""\
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
Traceback (most recent call last):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<18 lines>...
RuntimeError: uh-oh!
^^^^^^^^^^^^^^^^^^^^
"""))
^^^^^
AssertionError: 'Trac[778 chars]oh!\nException ignored in: <function Interpret[519 chars]\n\n' != 'Trac[778 chars]oh!\n'
Traceback (most recent call last):
File "/tmp/test_python_tk7ha1xh/tmpkqxiakqz/script.py", line 9, in <module>
interp.exec(script)
~~~~~~~~~~~^^^^^^^^
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 227, in exec
raise ExecutionFailed(excinfo)
test.support.interpreters.ExecutionFailed: RuntimeError: uh-oh!
Uncaught in the interpreter:
Traceback (most recent call last):
File "/tmp/test_python_tk7ha1xh/tmpkqxiakqz/script.py", line 6, in script
spam.eggs()
~~~~~~~~~^^
File "/tmp/test_python_tk7ha1xh/tmpkqxiakqz/spam.py", line 6, in eggs
ham()
~~~^^
File "/tmp/test_python_tk7ha1xh/tmpkqxiakqz/spam.py", line 3, in ham
raise RuntimeError('uh-oh!')
RuntimeError: uh-oh!
- Exception ignored in: <function Interpreter.__del__ at 0x2000119aed0>
- Traceback (most recent call last):
- File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
- File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
- TypeError: catching classes that do not inherit from BaseException is not allowed
- Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
- Python runtime state: finalizing (tstate=0x000055b06f0a1a10)
-
======================================================================
FAIL: test_gh_10[979](https://github.com/python/cpython/actions/runs/8978599353/job/24660038142#step:19:980)3 (test.test_interpreters.test_lifecycle.FinalizationTests.test_gh_109793)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 185, in test_gh_109793
self.assertEqual(proc.returncode, 1)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 1
======================================================================
FAIL: test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 script.py]
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 29, in subTest
yield ctx
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 157, in test_sys_path_0
out = self.run_python(argv, cwd=cwd)
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 114, in run_python
self.assertEqual(proc.returncode, 0)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 0
======================================================================
FAIL: test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m script]
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 29, in subTest
yield ctx
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 157, in test_sys_path_0
out = self.run_python(argv, cwd=cwd)
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 114, in run_python
self.assertEqual(proc.returncode, 0)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 0
======================================================================
FAIL: test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m pkg]
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 29, in subTest
yield ctx
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 157, in test_sys_path_0
out = self.run_python(argv, cwd=cwd)
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 114, in run_python
self.assertEqual(proc.returncode, 0)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 0
======================================================================
FAIL: test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m pkg.script]
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 29, in subTest
yield ctx
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 157, in test_sys_path_0
out = self.run_python(argv, cwd=cwd)
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 114, in run_python
self.assertEqual(proc.returncode, 0)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 0
----------------------------------------------------------------------
Ran 3 tests in 1.720s
FAILED (failures=6)
test test_interpreters failed
1 test failed again:
test_interpreters
The error looks like this:
Exception ignored in: <function Interpreter.__del__ at 0x2000117c0d0>
Traceback (most recent call last):
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x0000564591b58a10)
I'm fairly sure that means one of the 3 calls in _interpreters._decref() is raising a bogus exception.
It isn't clear to me how gh-118157 would have caused this to happen. As a very temporary solution, I've skipped test_interpreters on free-threading builds.
CC @colesbury
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
Activity
colesbury commentedon May 7, 2024
I think what's happening is:
module.__dict__. This is not behavior that we want long term, but it's an important performance work around for 3.13. (See Temporarily immortalize objects that use deferred reference counting #117783)finalize_remove_modules, but here that's not happening because the module's dictionary was immortalized.finalize_modules_clear_weaklistwhen we start clearing the module's__dict__in_PyModule_Clear.By the time
Interpreter.__del__is called, a lot of global variables including_interpretershave already been deleted from the module's dictionary so you get a bunch of weird errors, and the interpreter is not deleted.ericsnowcurrently commentedon May 7, 2024
That makes sense. You have any thoughts already on how to resolve this?
colesbury commentedon May 7, 2024
I'm not sure. We could call the finalizers on all GC-enabled objects during shutdown in the free-threaded build (before we start clearing module's dictionaries). I think that makes logical sense: everything is going away, why not call their finalizers? That seems to fix
test_interpreters, but a few other test cases would need tweaking.kumaraditya303 commentedon Jul 14, 2025
On current main,
test_interpreterspasses under free-threading as deferred reference counting is implemented now, the tests can be enabled back now.