Repository navigation
Not all tests are installed. #115421
Description
Bug report
Bug description:
Not all of the tests in Lib/test are being installed, which means some tests are (silently) not run from an installed python:
% /tmp/testinstall/bin/python3 -m test test_multiprocessing_fork
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/tmp/testinstall/lib/python3.13/test/__main__.py", line 2, in <module>
main(_add_python_opts=True)
~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/testinstall/lib/python3.13/test/libregrtest/main.py", line 680, in main
Regrtest(ns, _add_python_opts=_add_python_opts).main(tests=tests)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/tmp/testinstall/lib/python3.13/test/libregrtest/main.py", line 662, in main
selected, tests = self.find_tests(tests)
~~~~~~~~~~~~~~~^^^^^^^
File "/tmp/testinstall/lib/python3.13/test/libregrtest/main.py", line 198, in find_tests
selected = split_test_packages(selected)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/tmp/testinstall/lib/python3.13/test/libregrtest/findtests.py", line 70, in split_test_packages
splitted.extend(findtests(testdir=subdir, exclude=exclude,
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
split_test_dirs=split_test_dirs,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
base_mod=name))
^^^^^^^^^^^^^^
File "/tmp/testinstall/lib/python3.13/test/libregrtest/findtests.py", line 43, in findtests
for name in os.listdir(testdir):
~~~~~~~~~~^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/testinstall/lib/python3.13/test/test_multiprocessing_fork'
From a quick look it seems we're missing test_concurrent_futures, test_interpreters, test_multiprocessing_fork, test_multiprocessing_forkserver and test_multiprocessing_spawn. This is a problem in 3.12 and earlier as well.
Besides tests it's also missing the test.support.interpreters package:
% ls Lib/test/support/interpreters/
channels.py __init__.py queues.py
% ls /tmp/testinstall/lib/python3.13/test/support/interpreters
ls: cannot access '/tmp/testinstall/lib/python3.13/test/support/interpreters': No such file or directory
... which has not caused issues because its main user, test/test_subinterpreters, isn't being installed, and the other uses of it in the testsuite import ignore the import error if it's not available.
This is a problem back to at least 3.10 (which is missing test_capi as well as various testdata directories).
We should either make the Makefile install all of the test subdirectories automatically, or have a CI check to make sure all subdirectories are listed in TESTSUBDIRS in the Makefile.
CPython versions tested on:
3.10, 3.11, 3.12, CPython main branch
Operating systems tested on:
No response
Linked PRs
- gh-115421: Update the list of installed test subdirectories. #115422
- gh-115421: List all test subdirs in
Makefilebymakesetup#115511 - gh-115421: Test that our Makefile has all needed test folders #115813
- [3.12] gh-115421: List all test subdirs in Makefile, and test them (GH-115813) #116462
- [3.11] gh-115421: List all test/ subdirs in Makefile, and test them (GH-115813) #116498
Activity
vstinner commentedon Feb 13, 2024
cc @sobolevn: oh, this bug is bad :-(
sobolevn commentedon Feb 13, 2024
I somehow dodged this bullet this time: none of the directories are mine.
But, I had this bug several times. I think that we need a better check for that.
I will send a PR with the proposed check that we can run in CI.
gh-115421: Update the list of installed test subdirectories. (#115422)
Yhg1s commentedon Feb 14, 2024
I've fixed main by updating the list so they're included in the upcoming 3.13.0a4 release. For main we should try and figure out how to avoid the explicit list of test subdirectories to install, so I'll leave the bug open for that. For 3.12-3.10, though, I'm not sure it's worth fixing (and it changes what's getting installed, which is uncomfortable in a patch release), and I don't really want to try to do any invasive changes to the Makefile to avoid listing the directories in them.
pythongh-115421: Update the list of installed test subdirectories. (p…
pythongh-115421: List all test subdirs in `Makefile` by `makesetup`
pythongh-115421: Test that our Makefile has all needed test folders
pythongh-115421: Test that our Makefile has all needed test folders
pythongh-115421: Test that our Makefile has all needed test folders
pythongh-115421: Test that our Makefile has all needed test folders
gh-115421: Test that our Makefile has all needed test folders (GH-115813
pythongh-115421: Test that our Makefile has all needed test folders (p…
pythongh-115421: Test that our Makefile has all needed test folders (p…
encukou commentedon Mar 7, 2024
The test is in -- thank you, @sobolevn!
The change to Makefile isn't that invasive, so you might want to reconsider if this is good for 3.12. PR here: #116462
Yhg1s commentedon Mar 7, 2024
Now it's down to @pablogsal to decide if it's worth backporting to 3.11 :) I believe the missing directories in 3.11, and thus tests that aren't being run from installed Python (like during the release process) are test_concurrent_futures and test_multiprocessing_*.
[3.12] gh-115421: List all test subdirs in Makefile, and test them (G…
pablogsal commentedon Mar 7, 2024
I am happy backporting it to 3.11 👍
[3.11] pythongh-115421: List all test subdirs in Makefile, and test t…
[3.11] gh-115421: List all test/ subdirs in Makefile, and test them (G…
encukou commentedon Mar 11, 2024
Thanks to everyone involved :)
pythongh-115421: Test that our Makefile has all needed test folders (p…
pythongh-115421: Test that our Makefile has all needed test folders (p…