ResourceWarnings in test_importlib in select buildbots

In #132947 (comment) and the subsequent investigation, we learned that the recent changes to importlib.metadata cause ResourceWarnings to be emitted, causing select buildbot runs to fail.

After extensive investigation and code bisection, the ultimate cause is still not known. The problem has been temporarily remedied by bypassing a test that triggers the failure, but a more permanent solution is needed.

Here's a summary of the situation:

The resource warning is caused by an open handle on the test_packages_distributions_example2

cpython/Lib/test/test_importlib/metadata/test_main.py

Lines 350 to 356 in 3706ef6

    
               def test_packages_distributions_example2(self): 
        
                   """ 
        
                   Test packages_distributions on a wheel built 
        
                   by trampolim. 
        
                   """ 
        
                   self._fixture_on_path('example2-1.0.0-py3-none-any.whl') 
        
                   assert packages_distributions()['example2'] == ['example2']

That seems to implicate something holding onto that zip file, which gets added to sys.path during fixture setup.

The resource warning isn't triggered until test_importlib completes.
The warning is only emitted in a few buildbots, but not on local tests, GHA tests, or other buidlbots.
The changes in importlib.metadata.__init__ seem to be implicated, but even after removing nearly every change in that file, the problem remained.

Testing using buildbot is slow and tedious. Probably the next best step is to find a way to replicate the issue in a development environment, either by getting access to a buildbot image or by creating a comparable image that replicates the issue.

Linked PRs

gh-138313: Suppress ResourceWarnings and restore skipped test #138597

only emitted in a few buildbots

Which buildbots are affected? Is it a consistent set?

Only some of the buildbots fail. Maybe there's something about the subset of implicated buildbots that can give a clue.

The failed buildbots are:

ARM64 Raspbian 3.x (tier-3)
s390x RHEL8 LTO + PGO 3.x (tier-3)
s390x RHEL9 LTO + PGO 3.x (tier-3)
s390x RHEL8 LTO 3.x (tier-3)
s390x Fedora Stable LTO + PGO 3.x (tier-3)
s390x RHEL9 LTO 3.x (tier-3)
s390x Fedora Stable LTO 3.x (tier-3)
AMD64 RHEL8 LTO + PGO 3.x (tier-1)
AMD64 RHEL8 LTO 3.x (tier-1)

Yes, the set is consistent. Except for the Raspbian build, the LTO (Link Time Optimization?) seems like it may be a factor.

I'm thinking trying to replicate the AMD64 RHEL8 LTO 3.x may be the easiest.

Gemini helped create a Dockerfile in an attempt to replicate the issue, but while the docker image builds, it doesn't replicate the issue, even if I'm sure that the architecture is AMD64:

docker build --platform linux/amd64 .

Oh, but maybe that's because the workaround is in place. I need to make sure the test is not skipped.

I adapted the dockerfile thus, but when I run it, it fails for unrelated reasons.

Running just the importlib tests succeeds... although test_packages_distributions_example2 is indicated without an "ok" so maybe that means it's being skipped.

Stan has graciously offered me access to the raspbian builder, and I've gained access to that buildbot, so I'll be testing in that environment.

I've been unsuccessful in reproducing the failure in the raspbian buildbot.

I cloned main, build Python thus:

./configure --with-pydebug; CFLAGS=-UNDEBUG make

Then ran the tests with:

GITHUB_ACTION=yes make buildbottests

I've proven to myself that setting GITHUB_ACTION=yes does in fact re-activate the afflicted test.

And yet the tests succeed.

I'm not completely confident that I've replicated the commands that this particular buildbot runs. I should ask CoPilot to help elicit the exact commands the buildbot would run... or maybe see if I can find those details in the logs of one of the runs. I'm now wishing I'd saved links to more of the failed buildbot runs.

I've created a PR to remove the workaround and gather a new baseline of failing buildbots.

Aha. ~~I'm now noticing that the Raspbian failure is different - with the issue being "env changed" and not a ResourceWarning. I see the resource warning in this build.~~

~~Perhaps the Raspbian failure is unrelated, or maybe it's a different manifestation triggered by the same test.~~

There are failures in two buildbots that seem to be entirely unrelated (don't event implicate test_importlib):

Here are the list of tests that failed most recently:

buildbot/AMD64 Fedora Stable LTO + PGO PR
buildbot/AMD64 RHEL8 LTO PR
buildbot/AMD64 Windows PGO NoGIL PR
buildbot/AMD64 Windows Server 2022 NoGIL PR
buildbot/ARM Raspbian PR
buildbot/ARM64 Raspbian PR
buildbot/s390x Fedora Stable LTO + PGO PR
buildbot/s390x Fedora Stable LTO PR
buildbot/s390x RHEL8 LTO + PGO PR
buildbot/s390x RHEL9 LTO + PGO PR

It's apparent the list is not exactly stable. It's fairly stable, but there are some differences (ignoring unrelated Windows failures):

A new failure on AMD64 Fedora Stable
s390x RHEL8/9 without PGO has passed.
Raspbian failures now includes two architectures adding ARM.
The "3.x" has gone away. Maybe that was a recent change across all buildbots.

So while the failures are fairly stable, they're also somewhat intermittent.

Aha. ~~I'm now noticing that the Raspbian failure is different - with the issue being "env changed" and not a ResourceWarning. I see the resource warning in this build.~~

~~Perhaps the Raspbian failure is unrelated, or maybe it's a different manifestation triggered by the same test.~~

Correction: I've gone back and looked at the Raspbian failures, and they are indeed the same. I see the ResourceWarning in those. Also, I've noticed that (env changed) indicated as the failure for test_importlib across all the failures.

On the Raspbian buildbot host, I've run the exact commands extracted from the buildbot logs during a failure. I'm also now using the same branch/commit as failed in the PR.

./configure --prefix '$(PWD)/target' CFLAGS=-UNDEBUG
make -j4 all
make buildbottest 'TESTOPTS=-j2 -u-cpu --junit-xml test-results.xml -j4 ${BUILDBOT_TESTOPTS}' TESTPYTHONOPTS= TESTTIMEOUT=2400

And yet, the tests still pass. Perhaps $BUILDBOT_TESTOPTS is relevant.

Success! I've replicated the issue in the buildbot host.

I noticed that the buildbot runners were also doing this command before running tests:

make pythoninfo

So I ran that and then ran the tests again, and this time the failure occurred. So either the failure is intermittent or make pythoninfo is required to trigger the failure.

I re-ran the tests and they failed. Then I cleaned the repo (git clean -fdx) and rebuilt without make pythoninfo and again the test failed (issue replicated), so it does seem to have been an intermittent issue. So now to try to pare down the tests to something that doesn't take many minutes to run. I'll start on that as early as tomorrow.

I've found I can replicate the failure more narrowly by adding test_importlib to the invocation.

make buildbottest 'TESTOPTS=-j2 -u-cpu --junit-xml test-results.xml -j4 ${BUILDBOT_TESTOPTS} test_importlib' TESTPYTHONOPTS= TESTTIMEOUT=2400

Another problem I've noticed: I can pass -v to the test runner and it will give a verbose output of the tests run, but if I pass --fast-ci or --slow-ci, the -v seems to be ignored:

stan@raspberrypi:~/cpython $ ./python -W error -m test --fast-ci test_importlib --dont-add-python-opts -v
== CPython 3.15.0a0 (heads/gh-138313:43357c2, Sep 7 2025, 18:10:29) [GCC 12.2.0]
== Linux-6.12.20+rpt-rpi-v8-aarch64-with-glibc2.36 little-endian
== Python build: release
== cwd: /home/stan/cpython/build/test_python_worker_3504242æ
== CPU count: 4
== encodings: locale=ISO-8859-1 FS=utf-8
== resources: all,-cpu

Using random seed: 27757186
0:00:00 load avg: 0.36 Run 1 test in parallel using 1 worker process (timeout: 10 min, worker timeout: 15 min)
0:00:10 load avg: 0.46 [1/1] test_importlib passed

== Tests result: SUCCESS ==

10 slowest tests:
- test_importlib: 10.5 sec

1 test OK.

Total duration: 10.5 sec
Total tests: run=1,349 skipped=19
Total test files: run=1/1
Result: SUCCESS
stan@raspberrypi:~/cpython $ ./python -W error -m test --slow-ci test_importlib --dont-add-python-opts -v
== CPython 3.15.0a0 (heads/gh-138313:43357c2, Sep 7 2025, 18:10:29) [GCC 12.2.0]
== Linux-6.12.20+rpt-rpi-v8-aarch64-with-glibc2.36 little-endian
== Python build: release
== cwd: /home/stan/cpython/build/test_python_worker_3504340æ
== CPU count: 4
== encodings: locale=ISO-8859-1 FS=utf-8
== resources: all

Using random seed: 958552036
0:00:00 load avg: 0.30 Run 1 test in parallel using 1 worker process (timeout: 20 min, worker timeout: 25 min)
0:00:10 load avg: 0.48 [1/1] test_importlib passed

== Tests result: SUCCESS ==

10 slowest tests:
- test_importlib: 10.4 sec

1 test OK.

Total duration: 10.4 sec
Total tests: run=1,349 skipped=19
Total test files: run=1/1
Result: SUCCESS

So I'm not at all confident on which tests are being run because the test runner is only reporting running one test.

If I ignore the "resources" tests, the tests also pass:

stan@raspberrypi:~/cpython $ ./python -W error -m test --fast-ci --junit-xml test-results.xml test_importlib --dont-add-python-opts --ignore '*test_importlib.resources*'
Using random seed: 1373926192
0:00:00 load avg: 0.29 Run 1 test in parallel using 1 worker process (timeout: 10 min, worker timeout: 15 min)
0:00:09 load avg: 0.40 [1/1] test_importlib passed

== Tests result: SUCCESS ==

10 slowest tests:
- test_importlib: 9.2 sec

1 test OK.

Total duration: 9.3 sec
Total tests: run=1,142 (filtered) skipped=18
Total test files: run=1/1 (filtered)
Result: SUCCESS

So now this indicates that all of these factors are necessary to trigger the ResourceWarning:

Be on a Raspberry Pi (or other affected environment).
Run tests with --fast-ci or --slow-ci.
Run tests with -W error.
Generate jUnit XML.
Include test_importlib.resources.

The -W error makes sense - as the ResourceWarning would be silently ignored if not for that flag. I don't yet understand how the other factors are implicated.

Tests also pass when excluding just test_importlib.test_api.

stan@raspberrypi:~/cpython $ ./python -W error -m test --fast-ci --junit-xml test-results.xml test_importlib --dont-add-python-opts --ignore '*test_importlib.test_api*'
Using random seed: 3678508461
0:00:00 load avg: 0.08 Run 1 test in parallel using 1 worker process (timeout: 10 min, worker timeout: 15 min)
0:00:10 load avg: 0.21 [1/1] test_importlib passed

== Tests result: SUCCESS ==

10 slowest tests:
- test_importlib: 10.6 sec

1 test OK.

Total duration: 10.6 sec
Total tests: run=1,293 (filtered) skipped=19
Total test files: run=1/1 (filtered)
Result: SUCCESS

This seems to suggest that any little tweak to which tests are run seems to suppress the failure. I suspect all of these factors are conspiring to create a perfect storm and timing that triggers the failure.

I believe --fast-ci --dont-add-python-opts decomposes into -j0 --randomize --fail-env-changed --rerun --slowest --verbose3 -u "all,-cpu" --timeout=600, which could allow for more granularity/binary search in reproducing the failure.

A

I tried simply adding a sleep to an arbitrary test in the suite, but that didn't seem to affect the outcome:

stan@raspberrypi:~/cpython $ git diff
diff --git a/Lib/test/test_importlib/test_util.py b/Lib/test/test_importlib/test_util.py
index a77ce23..dc9a2c2 100644
--- a/Lib/test/test_importlib/test_util.py
+++ b/Lib/test/test_importlib/test_util.py
@@ -789,6 +789,10 @@ def test_complete_multi_phase_init_module(self):
 
 
 class MiscTests(unittest.TestCase):
+    def tearDown(self):
+        import time
+        time.sleep(1)
+
     def test_atomic_write_should_notice_incomplete_writes(self):
         import _pyio

Interestingly, adding --single-process consistently causes the first test to fail and the second to succeed.

stan@raspberrypi:~/cpython $ ./python -W error -m test --fast-ci --junit-xml test-results.xml test_importlib --dont-add-python-opts --single-process 2>&1 | tail -n 20
----------------------------------------------------------------------
Ran 1349 tests in 9.030s

OK (skipped=19)
0:00:20 load avg: 0.54 [1/1] test_importlib passed

== Tests result: ENV CHANGED then SUCCESS ==

10 slowest tests:
- test_importlib: 10.0 sec

1 re-run test:
    test_importlib

1 test OK.

Total duration: 20.2 sec
Total tests: run=2,698 skipped=38
Total test files: run=2/1 rerun=1
Result: ENV CHANGED then SUCCESS

I believe --fast-ci --dont-add-python-opts decomposes into -j0 --randomize --fail-env-changed --rerun --slowest --verbose3 -u "all,-cpu" --timeout=600, which could allow for more granularity/binary search in reproducing the failure.

It appears as if only --fail-env-changed is pertinent to the failure.

stan@raspberrypi:~/cpython $ ./python -W error -m test --fail-env-changed --junit-xml test-results.xml test_importlib --dont-add-python-opts
Using random seed: 1088677188
0:00:00 load avg: 0.40 Run 1 test sequentially in a single process
0:00:00 load avg: 0.40 [1/1] test_importlib
Warning -- Unraisable exception
Exception ignored while finalizing file <_io.FileIO name='/home/stan/cpython/Lib/test/test_importlib/metadata/data/example2-1.0.0-py3-none-any.whl' mode='rb' closefd=True>:
Traceback (most recent call last):
  File "/home/stan/cpython/Lib/test/support/__init__.py", line 837, in gc_collect
    gc.collect()
    ~~~~~~~~~~^^
ResourceWarning: unclosed file <_io.BufferedReader name='/home/stan/cpython/Lib/test/test_importlib/metadata/data/example2-1.0.0-py3-none-any.whl'>
0:00:10 load avg: 0.49 [1/1/1] test_importlib failed (env changed)

== Tests result: ENV CHANGED ==

1 test altered the execution environment (env changed):
    test_importlib

Total duration: 10.0 sec
Total tests: run=1,349 skipped=19
Total test files: run=1/1 env_changed=1
Result: ENV CHANGED

I guess that makes sense, as the failure is the env change.

Now I'm uncertain which failure is most pertinent - the ResourceWarning that occurs or the env changed warning. Too bad the ENV CHANGED error doesn't indicate what about the env changed.

Maybe what's happening - when the ResourceWarning occurs, it happens during teardown, causing other teardown not to take effect, and thereby leaving other fixtures not cleaned up, thereby leaving the environment changed.

That still doesn't explain why ignoring unrelated tests bypasses the issue or why it doesn't happen on other environments.

In an attempt to try to determine what env is changing, I added this diff:

diff --git a/Lib/test/test_importlib/metadata/fixtures.py b/Lib/test/test_importlib/metadata/fixtures.py
index 494047d..d29df7d 100644
--- a/Lib/test/test_importlib/metadata/fixtures.py
+++ b/Lib/test/test_importlib/metadata/fixtures.py
@@ -367,8 +367,13 @@ def _fixture_on_path(self, filename):
         pkg_file = resources.files(self.root).joinpath(filename)
         file = self.resources.enter_context(resources.as_file(pkg_file))
         assert file.name.startswith('example'), file.name
+        print('before', sys.path)
         sys.path.insert(0, str(file))
         self.resources.callback(sys.path.pop, 0)
+        self.resources.callback(self.show)
+
+    def show(self):
+        print('after', sys.path)
 
     def setUp(self):
         # Add self.zip_name to the front of sys.path.

But with this diff, the issue no longer occurs. The failure is sensitive to the smallest of changes.

Even this diff suppresses the error:

diff --git a/Lib/test/test_importlib/metadata/fixtures.py b/Lib/test/test_importlib/metadata/fixtures.py
index 494047d..f2e6779 100644
--- a/Lib/test/test_importlib/metadata/fixtures.py
+++ b/Lib/test/test_importlib/metadata/fixtures.py
@@ -6,6 +6,7 @@
 import shutil
 import sys
 import textwrap
+import time
 
 from test.support import import_helper
 from test.support import os_helper
@@ -369,6 +370,7 @@ def _fixture_on_path(self, filename):
         assert file.name.startswith('example'), file.name
         sys.path.insert(0, str(file))
         self.resources.callback(sys.path.pop, 0)
+        self.resources.callback(time.sleep, .001)
 
     def setUp(self):
         # Add self.zip_name to the front of sys.path.

An even smaller diff suppresses the error.

diff --git a/Lib/test/test_importlib/metadata/fixtures.py b/Lib/test/test_importlib/metadata/fixtures.py
index 494047d..ac24c0e 100644
--- a/Lib/test/test_importlib/metadata/fixtures.py
+++ b/Lib/test/test_importlib/metadata/fixtures.py
@@ -369,6 +369,7 @@ def _fixture_on_path(self, filename):
         assert file.name.startswith('example'), file.name
         sys.path.insert(0, str(file))
         self.resources.callback(sys.path.pop, 0)
+        self.resources.callback(lambda: None)
 
     def setUp(self):
         # Add self.zip_name to the front of sys.path.

Just adding a single no-op to the cleanup of the ZipFixtures seems to suppress the failure.

And even having the no-op happen before the sys.path.pop suppresses the failure.

diff --git a/Lib/test/test_importlib/metadata/fixtures.py b/Lib/test/test_importlib/metadata/fixtures.py
index 494047d..91bfe57 100644
--- a/Lib/test/test_importlib/metadata/fixtures.py
+++ b/Lib/test/test_importlib/metadata/fixtures.py
@@ -368,6 +368,7 @@ def _fixture_on_path(self, filename):
         file = self.resources.enter_context(resources.as_file(pkg_file))
         assert file.name.startswith('example'), file.name
         sys.path.insert(0, str(file))
+        self.resources.callback(lambda: None)
         self.resources.callback(sys.path.pop, 0)
 
     def setUp(self):

Same if I move the callback ahead of the enter_context for the pkg_file.

I've pushed a change that merely adds a no-op callback and now the buildbots are passing. Given how elusive this issue is, this janky workaround may be the best option for now.

Merge branch 'main' into pythongh-138313

I can see the refleaks PR builders are failing in test_capi since 3 days ago, so unrelated to these changes. I'm going to proceed with this change. I welcome others to come in and find an even more robust solution.

gh-138313: Suppress ResourceWarnings and restore skipped test (#138597)

Navigation Menu

Repository navigation

ResourceWarnings in test_importlib in select buildbots #138313

Linked PRs

9 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	def test_packages_distributions_example2(self):
	"""
	Test packages_distributions on a wheel built
	by trampolim.
	"""
	self._fixture_on_path('example2-1.0.0-py3-none-any.whl')
	assert packages_distributions()['example2'] == ['example2']

ResourceWarnings in test_importlib in select buildbots #138313

Description

Linked PRs

Activity

AA-Turner commented on Sep 1, 2025

jaraco commented on Sep 1, 2025

jaraco commented on Sep 1, 2025

jaraco commented on Sep 5, 2025

jaraco commented on Sep 6, 2025

jaraco commented on Sep 6, 2025

jaraco commented on Sep 6, 2025

jaraco commented on Sep 6, 2025

jaraco commented on Sep 6, 2025

jaraco commented on Sep 7, 2025

jaraco commented on Sep 7, 2025

jaraco commented on Sep 7, 2025

9 remaining items

jaraco commented on Sep 21, 2025

jaraco commented on Sep 21, 2025

jaraco commented on Sep 21, 2025

AA-Turner commented on Sep 21, 2025

jaraco commented on Sep 21, 2025

jaraco commented on Sep 21, 2025

jaraco commented on Sep 21, 2025

jaraco commented on Sep 21, 2025

jaraco commented on Sep 21, 2025

jaraco commented on Sep 21, 2025

jaraco commented on Sep 21, 2025

jaraco commented on Sep 22, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions