Repository navigation
Allow resources to be in subdirectories #58
Description
Activity
In GitLab by @warsaw on May 16, 2018, 16:28
This was a deliberate choice, but I think you have a valid use case. @brettcannon what do you think? And if we allow this, should we make sure it gets into Python 3.7?
I'm hesitant because I don't know how much work it will take or what the semantic and implementation implications are.
In GitLab by @brettcannon on May 16, 2018, 16:33
I get the desire to use this, but I'm personally not convinced it's worth the headache of having to manage paths separate from the import system since the abstraction handles a lot of subtle details for us.
IOW I'm with Barry and I'm not comfortable in doing this until someone produces a PR that implements this to see what kind of complications this would introduce as for me some empty files is not a huge cost for cross-platform file loading when that's needed (I get why people who are controlling their deployment and not doing this in a library feel like not bothering).
In GitLab by @warsaw on May 21, 2018, 15:27
I actually have another possible use case. Over in importlib_metadata it occurs to me that I'd really like to use
importlib_resourcesto handle the "loading the metadata from a zip file" case, but because the{egg,dist}-infodirectory is not a Python package, I can't use Python's import system to get to it.I have been considering writing a custom loader for that purpose, so we could use the
ResourceReaderAPI, but that's really not a perfect fit. It would make more sense to add a low-level API to finders rather than loaders to find a distribution package's metadata, because you obviously don't want to be able to import from an{egg,dist}-infodirectory.In GitLab by @warsaw on Sep 11, 2018, 19:31
Note that we are using finders, not loaders for
importlib_metadataso we don't need this over there. I won't close the issue, but currently still have no plans to implement this. I'm with @brettcannon that we can re-evaluate this if someone feels strongly enough about it to submit a PR.In GitLab by @jaraco on Oct 31, 2018, 17:27
I recently encountered another project where I adapted the implementation to use importlib_resources. This particular project had a jarfile in a subdirectory of the package. Package was
batonand the resource was inbaton/netsuite/. I ended up adding an empty__init__.pytobaton/netsuite, making it a package, and then instead of loadingnetsuite/tool.jarfrom thebatonpackage, I loadedtool.jarfrom the (empty)baton.netsuitepackage. A little awkward, but not horrible.You know what might be better? I know it's a little late to re-envision the design, but what if instead of returning a context that when entered ensures there's a file on the file system, it instead returns a first-class object which is traversable. So the OP could implement his application thus:
ld = importlib_resources.path('myapp') / 'static' / 'ld' with ld.open() as strm: handle_lines(strm) other_files = ld.listdir()Such a solution would be much more elegant in that :
- It doesn't require a context to delete potentially copied files (files are read from the source, whether it's a local file, file in a zip file, or some other resource such as a network file system).
- Users get the same familiar interface they've come to expect from pathlib (though with a possibly limited interface).
- All importlib_resources needs to do is construct the appropriate wrapper, and that wrapper provides read_binary, read_text through the familiar
.open('b').read()and.open().read().
Such an approach would require some functionality that doesn't exist (pathlib-compatible wrappers for zip files and others).
In GitLab by @warsaw on Nov 1, 2018, 13:30
I'd love to see zipfile compatible pathlib-like API. As @jaraco knows, this would also come in handy for importlib_metadata.
I wouldn't be opposed to a traversable object, but also remember that a motivating use case for
.path()'s current API are dynamic libraries which, due to the limitations ofdlopen()require a physical file system location.Can you think of a way to support the existing
.path()API and also provide a traversable object? If not, let's think about a new function that could return such a thing. I think it's a very interesting idea and would nicely support the OP use case.In GitLab by @jaraco on Dec 17, 2018, 15:19
I've just stumbled on another project that stores "package data" in a subdirectory of the package. And as it turns out, that's what distutils indicates to do. I think it's inconsistent for importlib.resources to declare this an unsupported case but distutils (as deprecated as it is) to still be recommending this for packaging.
Can you think of a way to support the existing
.path()API and also provide a traversable object?At first blush, it feels incompatible with the implementation I have in mind. I can draft what I have in mind as a separate function and then we can analyze if there's a way to combine these into a unified implementation.
In GitLab by @gregory.szorc on Feb 26, 2019, 23:59
I was told to comment here after filing https://bugs.python.org/issue36128.
I can see both sides of the argument for whether the
ResourceReaderinterface should allow resource names to exist in "subdirectories."On one hand, preventing path separators keeps things simple and provides only 1 way to access a specific resource.
On the other, requiring resources exist within Python packages can be rather annoying (as the original reporter has stated). Just today I was refactoring Mercurial to convert various directories into Python packages and it was somewhat annoying :/ It is much more convenient to place a bunch of resource files in a friendly directory tree and access them using hierarchical addressing with path separators as the delimiter. And this conveniently maps to filesystem paths.
From my perspective as someone who has hacked together Python module importing using zero-copy (https://gregoryszorc.com/blog/2018/12/28/faster-in-memory-python-module-importing/), I love the
ResourceReaderinterface because it allows resources to be imported from something that isn't the filesystem. My read of theResourceReaderinterface is that it is supposed to be an abstraction that allows resources to exist outside of traditional filesystems. Unfortunately, it falls a bit short.For starters, the implementation in
importlibtoday assumes platform native path separators. If running on POSIX, it treats/specially and on Windows,\is treated specially. If we're really talking about an agnostic interface for resource loading, we need to treat both path separators the same or pick only one that is special. Otherwise you have a leaky abstraction that behaves differently depending on OS. If path separators are special, normalizing on/would be preferred. But this code has already shipped in Python 3.7 and I doubt the backwards compatibility break to ignore\could be stomached.I can make the argument that resource names should be nearly anything and it is up to the
ResourceReaderto resolve them however it sees fit. This is truly agnostic of storage. Unfortunately, there is the practical concern that in the common case of mapping resources to filenames, we'll have multiple ways of referencing a resource from N Python packages in the Python package hierarchy. Yes, there are ways to limit this. But it feels overly complicated for aResourceReaderto enforce matters by looking for the existence of parent/child packages. Then there are more esoteric concerns, such as the fact that not all filesystems can represent all filenames. What happens when someone tries to use a resource namedAUXand Windows/NTFS refuses to write that file? Or what if resources collide on case sensitivity? Or what if the filesystem can't store all Unicode sequences? There arguably needs to be restrictions on what is a valid resource name.That being said, what is the harm for
ResourceReaders allowing N ways to access a specific resource for someResourceReaders? There are some theoretical concerns. But they feel like edge cases to me. I expect >95% of consumers ofResourceReaderto be within the current package and if people are doing bad things through multiple addressing, the fault seems to be theirs. If we're really concerned about the multiple addressing problems, perhaps we could define a well-named variable on modules to control whether resources are exposed. For example, if thefoo.barmodule defines__no_resources__ = True(or something),foo.bar.__spec__.loader.get_resource_loader()will fail for that module and consumers will be forced to usefoo.__spec__.loader.get_resource_loader()for their resource needs. (I'm not sure if this is feasible - I'm mostly just thinking aloud.) (If I were defining things from scratch, I would consider making modules opt in to exposing resources because it does feel like a special case.)In GitLab by @warsaw on Feb 27, 2019, 14:09
@gregory.szorc I'm a little strapped for time right now, but I just want to thank you for your very valuable feedback. It's always great to have real-world use cases to base decisions on, and I think you give us a really important data point (and one that validates the basic idea, even with its warts).
I'll follow up again once I have time to digest your comment and think about its implications.
In GitLab by @davidism on Oct 18, 2019, 10:35
I was removing the dependency on pkg_resources from Werkzeug's
SharedDataMiddlewareand Jinja'sPackageLoader, and ended up not using importlib_resources or most of the loader and resource reader APIs due to these limitations. Nested directories of resources can be included in a package with MANIFEST.in, so it seemed reasonable that these would still be accessible with the resource API.- SharedDataMiddleware returns 404 for package directory access pallets/werkzeug#1647
- PackageLoader doesn't depend on setuptools pallets/jinja#1082
Due to backwards compatibility as well as ease of use, it wasn't possible for me to require users to add init files to every folder and subfolder. Additionally, even if init files were added, there didn't appear to be a way to access subdirectories, or even determine that a name was a directory. There were secondary requirements such as being able to get the mtime of a resource if possible as well.
I'm hesitant to bring this up, since it may have been a convenient oversight, but
reader.open_resourcedoesn't have the "is not a directory" check that the rest of the API has, so it's possible to open files under nested directories, if you know they're already there. This worked for both filesystem and zip packages. That was enough for serving files, but not listing them.It would be great to see support for nested resources without adding init files.
17 remaining items
In GitLab by @brettcannon on Apr 22, 2020, 15:11
@ankostis because
pkgutil.get_data()is using an under-defined API.importlib_resourcesis much more rigorously defined. Honestly,pkgutilwill eventually be deprecated entirely once the final holes in support inimportlibare filled.In GitLab by @hinakuroori on Jun 22, 2020, 14:33
mentioned in commit hinakuroori/ament_package@0c80a10a6f7de541cc8b09c16846df5ffc9ab82f
In GitLab by @hinakuroori on Jun 22, 2020, 15:35
mentioned in commit hinakuroori/sros2@f1173fa1914b7ad59d1d2b46b854a86c7bba81c0
In GitLab by @hinakuroori on Jun 23, 2020, 13:33
mentioned in commit hinakuroori/sros2@dbec91d1c9efafadb9b7eeb183cc5f2663feb136
In GitLab by @hinakuroori on Jun 23, 2020, 14:01
mentioned in commit hinakuroori/ament_package@4db28d437e28dfdc0972fa65d92098647691bd9a
In GitLab by @brettcannon on Jun 25, 2020, 14:10
marked this issue as related to #104
In GitLab by @adam.hendry on Aug 19, 2020, 22:53
@brettcannon @warsaw I'm of the opinion we should keep
pkgutiland steer away fromimportlib.resources, or just port the exact same functionality ofpkgutilintoimportlib.resources(referencing https://stackoverflow.com/questions/6028000/how-to-read-a-static-file-from-inside-a-python-package/58941536?noredirect=1#comment112280625_58941536):Benefits:
- From an initial
timeitassesment, appears to be 2x faster thanimportlib.resources - More pythonic (resources aren't packages, but required to be so by adding
__init__.pywithimportlib.resources) - Does not require massive rewrites of existing code (i.e. adding
__init__.py's everywhere in published packages)
Cons:
- No context manager interface, so if users needs a long-lived resource they'll want
importlib.resources(if I need a long-lived resource, I can make a context manager myself...really not worried about this) - API is underdefined (not sure what that means, having looked at both the
importlib.resourcesandpkgutildocs)
Honestly, the cons above aren't really cons to me. I can't see any benefits of
importlib.resourcesoverpkgutil. Rather,importlib.resourcesfeels unpythonic (nonintuitive) andpkgutildoes the same thing and faster. If the API being underdefined is the only issue...why not just update the API? Let's updatepkgutilor port it toimportlib.resourcesif we want it to live there rather than creating something new that confuses everyone and is problematic.- From an initial
In GitLab by @adam.hendry on Aug 22, 2020, 19:37
@brettcannon and @warsaw I'm actually changing my answer.
pkgutilalso requires non-namespace packages. Hence, it also requires__init__.pyfiles in the package directory structure anyway as well. Therefore,importlib.resourcesis more feature-rich, and hence the appropriate module to use. Apologies for my confusion.In GitLab by @warsaw on Aug 23, 2020, 17:51
Please keep in mind that we all want to support data files in non-package subdirectories, it's just that none of us has had the time to implement it. In the fine tradition of open source, "Contributions Welcome!"
In GitLab by @wimglenn on Aug 26, 2020, 13:20
@warsaw Seems to be already implemented! The issue is closed, and Jason says the feature is complete. I've had a look on v3.0.0, and got pathlib instances to site-packages - when the package was zipped I got
zipfile.Pathinstance which quacked in similar ways. Worked on both Py2 and Py3.Example:
>>> import importlib_resources >>> resource = importlib_resources.files("pytz") / "zoneinfo" / "America" / "Chicago" >>> resource.read_bytes()[:10] b'TZif2\x00\x00\x00\x00\x00'I've updated https://github.com/wimglenn/resources-example to demonstrate
importlib.resourcesseparately fromimportlib_resources.Looks pretty good now, just a waiting game for the new APIs to appear in stdlib.
In GitLab by @jaraco on Aug 30, 2020, 21:06
That's right. The functionality is present in importlib_resources 1.3 and the same functionality is available in Python 3.9. What remains is to add support for namespace packages and work toward an API for alternate loaders to supply resources.
In GitLab by @rob.speer on May 16, 2018, 01:23
Suppose I have a Web application,
myapp, that needs to serve a static file, which by convention needs to be in the pathmyapp/static/ld/context.ld.json. Suppose I also want to be able to access that file from Python code, because its contents are used in a test.As importlib_resources is currently defined, I would need to rewrite the path as if it were a Python submodule, even though it does not contain actual Python code:
path(myapp.static.ld, "context.ld.json"). I would also need to create empty files namedmyapp/static/__init__.pyandmyapp/static/ld/__init__.py, and hopefully exclude them from being served as static files.That would be enough for me to give up and use paths relative to
__file__instead. In general, I would heartily recommend importlib if I could reasonably promise that it was an improvement over using__file__or over existing uses ofpkg_resources, which it wouldn't be if it doesn't support subdirectories.The call I would like to be able to make in this situation is
path(myapp, "static/ld/context.ld.json").