files(...).joinpath doesn't accept variable number of arguments in Python 3.10+

This happens to me when I try to access resources in a native namespace package.

Tested on:

Windows, Python 3.10.4
Docker in WSL2, image python:3.10-slim, Python 3.10.5

The reason seems to be that importlib_resources.files() returns MultiplexedPath from stdlib (importlib.readers) rather than from importlib_resources.readers.

To recreate:

Install any namespace package, e.g. sphinxcontrib-htmlhelp;
```
pip install sphinxcontrib-htmlhelp
```

Use files with joinpath to get the path of a resource.

from importlib_resources import files

print(root:=files('sphinxcontrib'), 'from', root.__module__)
print(root.joinpath('htmlhelp', 'templates'))

Expected output (which I get with Pythons 3.7-3.9):

MultiplexedPath('/usr/local/lib/python3.9/site-packages/sphinxcontrib') from importlib_resources.readers
/usr/local/lib/python3.9/site-packages/sphinxcontrib/htmlhelp/templates

Bug with Pythons 3.10:

MultiplexedPath('/usr/local/lib/python3.10/site-packages/sphinxcontrib') from importlib.readers
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: MultiplexedPath.joinpath() takes 2 positional arguments but 3 were given

I also tested with the code in #254 but to no aval.

I see support for namespace packages was added in 8e1a218 released in importlib_resources 3.2.0 and Python 3.10. I'm able to replicate the error with this one-liner (requires pip-run):

$ pip-run -q sphinxcontrib-htmlhelp importlib_resources -- -c "import importlib_resources as resources; resources.files('sphinxcontrib').joinpath('htmlhelp', 'templates')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: MultiplexedPath.joinpath() takes 2 positional arguments but 3 were given

In that release, the MultiplexedPath.joinpath only accepted one parameter.

But that restriction was lifted in 6440323 (released in 5.8.0).

But somehow, it seems that the MultiplexedPath on Python 3.10 is being preferred even though importlib_resources 5.9 is being invoked.

I think I understand what's going on:

importlib_resources wraps the spec in order to supply a TraversableResources reader if the loader/spec doesn't already provide one.
Because Python 3.10 provides one, however, importlib_resources defers to that implementation, so on Python 3.10 and 3.11, the users get an experience based on objects in the stdlib. With Python 3.12, users will get the importlib_resources 5.8+ experience.

It's not obvious to me how importlib_resources could intercept these calls for later versions of Python that were previously adequate, and even less obvious that's what should happen. importlib_resources has to support loaders from a range of Python versions as well as a range of behaviors from custom loaders.

As a short-term workaround, I suggest to use .joinpath('htmlhelp').joinpath('templates'), which should work in all Python versions. It's possible that .joinpath('htmlhelp/templates') will also work, but that's also risky, because support for that was improved recently.

@FFY00

@FFY00 I'd like your opinion - do you think based on this report that importlib_resources should be more aggressive about supplying its own Readers and Path objects?

I think that would be something worth exploring. That behavior seems desirable, but like you mentioned, the implementation might be a bit tricky. If we can come up with a reasonable way of doing this, it would be nice to have it, otherwise, I wouldn't worry too much.

Disable native reader even if it supplies files, because it might not…

Here's a unit test that fails in Python 3.10 (and only in that one, AFAIK):

diff --git a/importlib_resources/tests/test_files.py b/importlib_resources/tests/test_files.py
index 9e45f70..3e86ec6 100644
--- a/importlib_resources/tests/test_files.py
+++ b/importlib_resources/tests/test_files.py
@@ -34,6 +34,11 @@ class FilesTests:
     def test_traversable(self):
         assert isinstance(resources.files(self.data), Traversable)

+    def test_joinpath_with_multiple_args(self):
+        files = resources.files(self.data)
+        binfile = files.joinpath('subdirectory', 'binary.file')
+        self.assertTrue(binfile.is_file())
+
     def test_old_parameter(self):
         """
         Files used to take a 'package' parameter. Make sure anyone

Add test capturing missed expectation. Ref #257.

Mark test as xfail on Python 3.10 where it fails. Ref #257.

The issue is with this logic:

importlib_resources/importlib_resources/future/adapters.py

Lines 26 to 28 in 9f5b437

    
           # Python 3.10+ 
        
           if reader.__class__.__module__.startswith('importlib.resources.'): 
        
               return

On Python 3.10, the module name for the loaders is "importlib.readers", not "importlib.resources.readers".

Thanks so much for the test, which (shame on me) should have been included in the original solution.

Ensure stdlib readers are excluded on Python 3.10 even when found in …

Navigation Menu

Repository navigation

files(...).joinpath doesn't accept variable number of arguments in Python 3.10+ #257

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	# Python 3.10+
	if reader.__class__.__module__.startswith('importlib.resources.'):
	return

files(...).joinpath doesn't accept variable number of arguments in Python 3.10+ #257

Description

Activity

jaraco commented on Jul 22, 2022

jaraco commented on Jul 22, 2022

jaraco commented on Jul 22, 2022

FFY00 commented on Jul 22, 2022

Jakub-CZ commented on Mar 15, 2024

jaraco commented on Mar 16, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions