Repository navigation
Can't open orphan path on bare importlib.resources.files() under zipapp #121735
Description
Activity
- added a commit that references this issue
on Aug 4, 2024 The code that handles the module inference is here:
cpython/Lib/importlib/resources/_common.py
Lines 75 to 104 in 6ae942f
Next step is to determine what expectation is missed when running under zipapp.
- changed the title
[-]Cannot read package data file after opening with importlib.resources.files(anchor) and running with zipapp[/-][+]Can't open orphan path on bare importlib.resources.files() under zipapp[/+]on Aug 14, 2024 I was able to replicate the issue without going through the wheel build process:
draft 🐚 mkdir -p buildtemp/mypkg draft 🐚 cat > buildtemp/mypkg/a.py import importlib.resources def main(): x = importlib.resources.files() / "data.txt" print(x.read_text()) draft 🐚 echo datadata > buildtemp/mypkg/data.txt draft 🐚 py -m zipapp buildtemp --main mypkg.a:main -p '/usr/bin/env py' -o mypkg.pyz draft 🐚 ./mypkg.pyz Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/Users/jaraco/draft/./mypkg.pyz/__main__.py", line 3, in <module> mypkg.a.main() ~~~~~~~~~~~~^^ File "/Users/jaraco/draft/./mypkg.pyz/mypkg/a.py", line 6, in main print(x.read_text()) ~~~~~~~~~~~^^ File "/opt/python/lib/python3.13/importlib/resources/abc.py", line 89, in read_text with self.open(encoding=encoding) as strm: ~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/opt/python/lib/python3.13/importlib/resources/_adapters.py", line 139, in open raise FileNotFoundError("Can't open orphan path") FileNotFoundError: Can't open orphan path
The issue is that
CompatibilityFilesis being activated for loading resources.draft 🐚 py -m pdb mypkg.pyz > /Users/jaraco/draft/mypkg.pyz/__main__.py(2)<module>() -> import mypkg.a (Pdb) c Traceback (most recent call last): File "/opt/python/lib/python3.13/pdb.py", line 2435, in main pdb._run(target) ~~~~~~~~^^^^^^^^ File "/opt/python/lib/python3.13/pdb.py", line 2192, in _run self.run(target.code) ~~~~~~~~^^^^^^^^^^^^^ File "/opt/python/lib/python3.13/bdb.py", line 651, in run exec(cmd, globals, locals) ~~~~^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jaraco/draft/mypkg.pyz/__main__.py", line 3, in <module> mypkg.a.main() ~~~~~~~~~~~~^^ File "/Users/jaraco/draft/mypkg.pyz/mypkg/a.py", line 6, in main print(x.read_text()) ~~~~~~~~~~~^^ File "/opt/python/lib/python3.13/importlib/resources/abc.py", line 89, in read_text with self.open(encoding=encoding) as strm: ~~~~~~~~~^^^^^^^^^^^^^^^^^^^ File "/opt/python/lib/python3.13/importlib/resources/_adapters.py", line 139, in open raise FileNotFoundError("Can't open orphan path") FileNotFoundError: Can't open orphan path Uncaught exception. Entering post mortem debugging Running 'cont' or 'step' will restart the program > /opt/python/lib/python3.13/importlib/resources/_adapters.py(139)open() -> raise FileNotFoundError("Can't open orphan path") (Pdb) u > /opt/python/lib/python3.13/importlib/resources/abc.py(89)read_text() -> with self.open(encoding=encoding) as strm: (Pdb) u > /Users/jaraco/draft/mypkg.pyz/mypkg/a.py(6)main() -> print(x.read_text()) (Pdb) x <importlib.resources._adapters.CompatibilityFiles.OrphanPath object at 0x41f4c584f90>CompatibilityFilesgets employed when there's no proper resource provider found. I'll figure out why.It's clear that the bare call to
files()is gettingCompatibiltyFileswhen the call with an anchor returns a properMultiplexedPath:(Pdb) importlib.resources.files() <importlib.resources._adapters.CompatibilityFiles.SpecPath object at 0x41f4c595a10> (Pdb) importlib.resources.files('mypkg') MultiplexedPath('/Users/jaraco/draft/mypkg.pyz/mypkg/')By changing the
main()function to emit the inferred caller's name,def main(): print(importlib.resources._common._infer_caller().f_globals['__name__'])
I can confirm that the inferred name is correct when it emits
mypkg.a.Interestingly, if I then replace the main function with a static 'mypkg.a', the reported error occurs.
def main(): x = importlib.resources.files('mypkg.a') / "data.txt" print(x.read_text())
So the issue seems to be that
files('mypkg')works as expected butfiles('mypkg.a')does not, and it's only coincidental thatfiles()callsfiles('mypkg.a').importlib.resourceshas a behavior that when given a module will resolve resources in the package of that module.It seems that the
zipimporteras attached to the module is not resolving a resource reader:(Pdb) importlib.import_module('mypkg.a').__spec__.loader.get_resource_reader('mypkg.a') (Pdb) importlib.import_module('mypkg.a').__spec__.loader.get_resource_reader('mypkg') (Pdb) importlib.import_module('mypkg').__spec__.loader.get_resource_reader('mypkg') <importlib.resources.readers.NamespaceReader object at 0x3df2e585410>
I'm slightly surprised to see
NamespaceReaderand not aZipReader.It seems the issue traces to the logic in get_resource_reader:
Lines 258 to 270 in 6ae942f
When checking
loader.is_packageformypkg.a, it returns False:(Pdb) loader = importlib.import_module('mypkg.a').__spec__.loader (Pdb) loader.is_package('mypkg.a') False
In the early days of importlib.resources, all resources needed to be relative to a package (and not a module). Later, that restriction was relaxed to allow resources to be loaded relative to a module.
But it seems that
get_resource_readerdidn't get the corresponding treatment.I've confirmed the same issue occurs when using
importlib_resources:draft [1] 🐚 cat buildtemp/mypkg/a.py import importlib_resources def main(): x = importlib_resources.files('mypkg.a') / "data.txt" print(x.read_text()) draft 🐚 pip install -t buildtemp importlib_resources Collecting importlib_resources Using cached importlib_resources-6.4.0-py3-none-any.whl.metadata (3.9 kB) Using cached importlib_resources-6.4.0-py3-none-any.whl (38 kB) Installing collected packages: importlib_resources Successfully installed importlib_resources-6.4.0 draft 🐚 py -m zipapp buildtemp --main mypkg.a:main -p '/usr/bin/env py' -o mypkg.pyz draft 🐚 ./mypkg.pyz Traceback (most recent call last): ... FileNotFoundError: /Users/jaraco/draft/./mypkg.pyz/mypkg/a/data.txtI suspect the resolution is going to be update
get_resource_readerto accept a module.But first things first, let's figure out why this use-case isn't covered by the tests.
As a check, if I remove the
is_packagecheck:cpython main 🐚 git diff diff --git a/Lib/zipimport.py b/Lib/zipimport.py index f2724dd0268..c0a4abcd95d 100644 --- a/Lib/zipimport.py +++ b/Lib/zipimport.py @@ -256,16 +256,8 @@ def load_module(self, fullname): def get_resource_reader(self, fullname): - """Return the ResourceReader for a package in a zip file. - - If 'fullname' is a package within the zip file, return the - 'ResourceReader' object for the package. Otherwise return None. + """Return the ResourceReader for a module in a zip file. """ - try: - if not self.is_package(fullname): - return None - except ZipImportError: - return None from importlib.readers import ZipReader return ZipReader(self, fullname)
The routine then fails, but with a different error:
draft 🐚 ./mypkg.pyz Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/Users/jaraco/draft/./mypkg.pyz/__main__.py", line 3, in <module> mypkg.a.main() ~~~~~~~~~~~~^^ File "/Users/jaraco/draft/./mypkg.pyz/mypkg/a.py", line 6, in main print(x.read_text()) ~~~~~~~~~~~^^ File "/Users/jaraco/code/python/cpython/Lib/zipfile/_path/__init__.py", line 426, in read_text with self.open('r', encoding, *args, **kwargs) as strm: ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/jaraco/code/python/cpython/Lib/zipfile/_path/__init__.py", line 391, in open raise FileNotFoundError(self) FileNotFoundError: /Users/jaraco/draft/./mypkg.pyz/mypkg/a/data.txt
That's encouraging, suggesting that the
ZipReaderis getting involved, but that it still doesn't honor resources adjacent to modules.I started working on a test, seeking to extend these tests to zip files, but realized that the current test structure isn't easily amenable to extension to zip. In python/importlib_resources@3faf336, I've done some refactoring in order to leverage the zip fixtures for this purpose.
16 remaining items
- added a commit that references this issue
on Aug 19, 2024 - added a commit that references this issue
on Aug 21, 2024 - added a commit that references this issue
on Aug 22, 2024 - added a commit that references this issue
on Aug 27, 2024 - added a commit that references this issue
on Aug 28, 2024 - added 4 commits that reference this issue
on Sep 12, 2024 PRs are merged; closing.
Reacted by enwakan
Bug report
Bug description:
Context: In Python 3.12 the function importlib.resources.files() was updated: https://docs.python.org/3/library/importlib.resources.html#importlib.resources.files
The Issue: When using
importlib.resources.files(), I am able to open and read a text file included as package data in my package. However, if I zip up my package with zipapp and run the .pyz file, it is unable to read the text file.If I use
importlib.resources.files("mypkg")it will work in both cases.Minimal example:
Create a project directory named
mypkgwith the src layout.Create the following files.
mypkg/src/mypkg/a.py
mypkg/src/mypkg/data.txt
In pyproject.toml make sure to set up the entry script and include the data file.
mypkg/pyproject.toml
Install the package into a fresh Python (virtual) environment with pip.
Run
myscriptand verify it printsdatadata.Now back in the mypkg project root folder, we will generate a .pyz file with zipapp and run the program from the .pyz file.
Run
pip install . --target buildtemp --upgrade && python3 -m zipapp buildtemp --main mypkg.a:main -p '/usr/bin/env python3' -o mypkg.pyzin the terminal.Run
./mypkg.pyz. This results in an error:Note that if you change
x = importlib.resources.files() / "data.txt"tox = importlib.resources.files("mypkg") / "data.txt", it will work when running the .pyz file.CPython versions tested on:
3.12
Operating systems tested on:
Linux, macOS
Linked PRs