importlib.resources.files() doesn't work correctly when importlib library is compiled

Bug report

Bug description:

If importlib library is compiled (in particular _common.py), then any package, that uses bare files() to get its resources, breaks, because _infer_caller returns wrong frame (as mentioned in #123037 (comment), __file__ in

cpython/Lib/importlib/resources/_common.py

Line 96 in 35d8ac7

return frame_info.filename == __file__

is not valid)

Reproducer:

import importlib
import os
import pathlib
import py_compile
import shutil
import sys
import tempfile
import textwrap


def compile(tempdir):
    target_dir = pathlib.Path(tempdir) / 'cimportlib'
    souce_dir = pathlib.Path(importlib.__file__).parent
    shutil.copytree(souce_dir, target_dir, ignore=lambda *_: ['__pycache__'])

    for dirpath, _, filenames in os.walk(target_dir):
        for filename in filenames:
            if filename != "_common.py":
                continue
            source_path = pathlib.Path(dirpath) / filename
            cfile = source_path.with_suffix('.pyc')
            py_compile.compile(source_path, cfile)
            pathlib.Path.unlink(source_path)


def create_package(tempdir):
    package_dir = pathlib.Path(tempdir) / 'somepkg'
    package_dir.mkdir()
    contents = {
        "__init__.py": textwrap.dedent(
            """
            import cimportlib.resources as res
            val = res.files().joinpath('resource.txt').read_text(encoding='utf-8')
            """
        ),
        "resource.txt": "data",
    }

    for file, content in contents.items():
        path = pathlib.Path(package_dir) / file
        path.write_text(content)


def main():
    with tempfile.TemporaryDirectory() as tempdir:
        compile(tempdir)
        create_package(tempdir)
        sys.path.insert(0, str(tempdir))
        print(importlib.import_module('somepkg').val)


if __name__ == "__main__":
    raise SystemExit(main())

Expectation:

data

Actual outcome:

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpngnuw441/cimportlib/resources/resource.txt'

It may be important for frozen python applications, as they don't include source code

CPython versions tested on:

3.12, CPython main branch

Operating systems tested on:

Linux, Windows

Linked PRs

It may be important for frozen python applications

Did you encounter this issue in practice, or did you theorize the concern it while reviewing the code?

pythongh-123085: Fix issue in inferred caller when resources package …

Did you encounter this issue in practice, or did you theorize the concern it while reviewing the code?

I encountered it in practice

chore(deps): Bump importlib-resources from 6.4.0 to 6.4.3 (#1971)

Bump importlib-resources from 6.1.1 to 6.4.3 (#8806)

Bump importlib-resources from 6.4.0 to 6.4.3 (#398)

Bump importlib-resources from 6.1.1 to 6.4.4 (#8915)

Bump importlib-resources from 5.13.0 to 6.4.4 (#91)

pythongh-123085: Fix issue in inferred caller when resources package …

[3.13] pythongh-123085: Fix issue in inferred caller when resource pa…

Some buildbots started failing when #123102 was merged, see:

I'll probably not have to investigate this week, but, my first guess would be that shutil.copytree copied some security metadata that says “this shouldn't be in /tmp”.

In the first log, I see:

0:01:29 load avg: 11.57 [1/1/1] test_importlib failed (2 errors)
Re-running test_importlib in verbose mode (matching: test_implicit_files_with_compiled_importlib, test_implicit_files_with_compiled_importlib)
test_implicit_files_with_compiled_importlib (test.test_importlib.resources.test_files.ImplicitContextFilesDiskTests.test_implicit_files_with_compiled_importlib)
Caller detection works for compiled-only resources module. ... ERROR
test_implicit_files_with_compiled_importlib (test.test_importlib.resources.test_files.ImplicitContextFilesZipTests.test_implicit_files_with_compiled_importlib)
Caller detection works for compiled-only resources module. ... ERROR
======================================================================
ERROR: test_implicit_files_with_compiled_importlib (test.test_importlib.resources.test_files.ImplicitContextFilesDiskTests.test_implicit_files_with_compiled_importlib)
Caller detection works for compiled-only resources module.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/test/test_importlib/resources/test_files.py", line 157, in test_implicit_files_with_compiled_importlib
    self._compile_importlib()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/test/test_importlib/resources/test_files.py", line 147, in _compile_importlib
    py_compile.compile(source_path, cfile)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/py_compile.py", line 172, in compile
    importlib._bootstrap_external._write_atomic(cfile, bytecode, mode)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 206, in _write_atomic
PermissionError: [Errno 13] Permission denied: '/tmp/test_python_ytm0gb4y/tmpw7y1yd3q/c_resources/abc.pyc.281473105247840'
======================================================================
ERROR: test_implicit_files_with_compiled_importlib (test.test_importlib.resources.test_files.ImplicitContextFilesZipTests.test_implicit_files_with_compiled_importlib)
Caller detection works for compiled-only resources module.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/test/test_importlib/resources/test_files.py", line 157, in test_implicit_files_with_compiled_importlib
    self._compile_importlib()
    ~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/test/test_importlib/resources/test_files.py", line 147, in _compile_importlib
    py_compile.compile(source_path, cfile)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang-installed/build/target/lib/python3.14/py_compile.py", line 172, in compile
    importlib._bootstrap_external._write_atomic(cfile, bytecode, mode)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 206, in _write_atomic
PermissionError: [Errno 13] Permission denied: '/tmp/test_python_ytm0gb4y/tmp52oghnez/c_resources/abc.pyc.281473106346832'
----------------------------------------------------------------------
Ran 2 tests in 0.020s

The error is the same on the other buildbot. Both are Fedora, so perhaps that's a factor.

It's failing on this open call:

cpython/Lib/importlib/_bootstrap_external.py

Lines 206 to 207 in bbb36c0

    
           fd = _os.open(path_tmp, 
        
                         _os.O_EXCL | _os.O_CREAT | _os.O_WRONLY, mode & 0o666)

It's conceivable the copy,

cpython/Lib/test/test_importlib/resources/test_files.py

Line 141 in bbb36c0

shutil.copytree(sources, c_resources, ignore=lambda *_: ['__pycache__'])

while it excludes __pycache__ could still be copying extant .pyc files and those are failing to write atomically. Although, that's probably not it, because the failure is happening when writing out the temp file.

my first guess would be that shutil.copytree copied some security metadata that says “this shouldn't be in /tmp”.

I'm not familiar with any OS that has such behaviors. I'd not have considered it. I did read up a bit on the noexec mount option. At first blush, I wouldn't expect such an option to conflict with _os.O_EXCL | _os.O_CREAT | _os.O_WRONLY.

I don't think I'll make much more progress on this without a local reproducer. Do you think this error can be reproduced in a Docker container? In the meantime, is the buildbot breakage worth putting in a check to allow these tests to skip or xfail when they cannot write out the compiled files?

Do you think this error can be reproduced in a Docker container?

I couldn't reproduce it locally. It might need some specific VM settings :(

In the meantime, is the buildbot breakage worth putting in a check to allow these tests to skip or xfail when they cannot write out the compiled files?

First let me try an implementation that avoids the initial copy, which should be a bit faster too: #124131

gh-123085: _compile_importlib: Avoid copying sources before compilati…

pythongh-123085: _compile_importlib: Avoid copying sources before com…

[3.12] gh-123085: Fix issue in inferred caller when resources package…

pythongh-123085: _compile_importlib: Avoid copying sources before com…

[3.12] gh-123085: _compile_importlib: Avoid copying sources before co…

pythongh-123085: Remove double 'import os' added by PR python#124021

[3.12] gh-123085: Remove double 'import os' added by PR #124021 (#128600

I've also encountered this issue in practice. It causes a failure when the stdlib is zipped, and

(importlib.resources.files() / "datafile.txt").read_text()

fails with

FileNotFoundError: /opt/python/lib/python313.zip/importlib/resources/datafile.txt

[3.13] gh-123085: Fix issue in inferred caller when resource package …

pythongh-123085: _compile_importlib: Avoid copying sources before com…

[3.13] gh-123085: _compile_importlib: Avoid copying sources before co…

Navigation Menu

Repository navigation

importlib.resources.files() doesn't work correctly when importlib library is compiled #123085

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

5 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	fd = _os.open(path_tmp,
	_os.O_EXCL \| _os.O_CREAT \| _os.O_WRONLY, mode & 0o666)

importlib.resources.files() doesn't work correctly when importlib library is compiled #123085

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

jaraco commented on Aug 17, 2024

Gatsik commented on Aug 17, 2024

5 remaining items

encukou commented on Sep 13, 2024

jaraco commented on Sep 13, 2024

encukou commented on Sep 17, 2024

godlygeek commented on Aug 14, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions