Repository navigation
Better docs for importlib.resources #93610
Description
The current documentation of importlib.resources can be improved.
The main hindrance to usability are references to classes that are not linked correctly, leaving the docs very hard to follow for someone who doesn't already know the module. For example, Traversable in the docs of files is not a link.
Activity
encukou commentedon Jun 8, 2022
PR: #93611
warsaw commentedon Jun 10, 2022
Just ran across this one too!
importlib.resources.rstlinks toimportlib.resources.abc.Traversablebut it should beimportlib.abc.Traversable. PR coming for that.@encukou were there other problems you found?
Fix links to importlib.abc.Traversable
`importlib.abc.Traversable.name` is an attribute, not a method
pythongh-89896: `importlib.abc.Traversable.name` is an attribute, not…
warsaw commentedon Jun 11, 2022
I've opened two PRs which seem to fix the documentation problem, but I'm not actually sure this is the right thing to do. The deal is that in 3.11,
Traversable,ResourceReader, andTraversableResourceswere moved toimportlib.resources.abc, with this compatibility hack added toimportlib.abc:However, the documentation for
importlib.resources.abcis included inimportlib.rst(inside the:mod:importlib.abcsection):which is why (I think), the doc cross references to
importlib.resources.abc.Traversabledon't work (without the above PRs). I don't see any mention of this reorg in the What's New in Python 3.11 docs, nor in the importlib.abc documentation, so I'm not sure what the intended public API for these ABC's is.@jaraco can you weigh in?
encukou commentedon Jun 13, 2022
Yes, that confused me as well, so I went with the implementation here –
<class 'importlib.resources.abc.Traversable'>.encukou commentedon Jun 13, 2022
I sent a PR with improvements, it's probably better to look there, but:
importlib_resources._legacy– which uses a private module for all of the implementation. (Perhaps withTraversabledocumentation being discoverable, this would be easier to use?)__file__jaraco commentedon Jun 13, 2022
At the time of writing, I considered the ABCs to be primarily an implementation detail, maybe occasionally referenced in the source for clarity. I wasn't even sure there should be a concrete ABC to describe the interface, but as it developed, the value of the abcs became more apparent and started having references in the documentation.
My preference is for code to be self-documenting as much as possible, limiting the amount of duplication that has to happen in prose.
When I moved the module, I still considered the Traversable object to be primarily an implementation detail, although by moving it to the abcs module, I was preparing it to be supported first-class for type hints.
It looks like the Traversable class was moved in python/importlib_resources#87, quite some time ago. Oh, that's not when the content moved to its own module under resources. bpo-46118 is where that work happened. Perhaps this news entry could have been more explicit about the move of some interfaces from importlib.abc to importlib.resources.abc.
encukou commentedon Jun 13, 2022
So, should it be documented as
importlib.resources.abc.Traversableorimportlib.abc.Traversable?warsaw commentedon Jun 13, 2022
Except that link doesn't work.
If the latter, my PR fixes that link. If the former, perhaps it shouldn't be
:include:'d inimportlib.abcdocs?encukou commentedon Jun 14, 2022
If the former, my PR fixes the link, by moving
importlib.resources.abcto its own page.2 remaining items
encukou commentedon Jun 15, 2022
Everything added in 3.7 is now deprecated. The current API is from 3.9.
warsaw commentedon Jun 15, 2022
I guess that's a valid reason for keeping the standalone package around, but at some point (maybe 3.9 EOL?) it's likely not worth it. Then again, that's a @jaraco decision.
jaraco commentedon Jun 16, 2022
I'd not say the current API is from 3.9. A major change landed in 3.9, but more changes landed in 3.10 and some have yet to land (for example, python/importlib_resources#250 was inspired by recent feedback). You can see in the compatibility matrix which versions of _resources map to which versions of CPython, and you can use the history to see what changes occurred between those versions. So users who want features from importlib_resources 5.1 or later will need to use Python 3.11 or importlib_resources.
If at some point, you want not to support backporting any behavior, we should make that decision, but the status quo is to develop the changes in the backport, prove their viability, then port them into the next CPython. Same is true for importlib.metadata.
There's no roadmap per se. It's pretty stable, but if there's an improvement added in the latest version, it improves adoption to be able to make that feature available in the backport.
Yes, I'd prefer to keep abcs for resources in that package. Even in isolation, it's better to have abcs in the same package where they're used for logical encapsulation (for the same reason that you don't put abcs from collections.abc and importlib.abc in a top-level .abc).
That said, it also helps the backport immensely, since the code that exists in 'importlib_resources' has a direct mapping to the same module in importlib.resources.
I would like to retire the backport too, but I also find it 10x easier to develop in the backport (massively better test runner, faster tests, no compiler needed, fast iteration, coverage testing, ...). Let's not make that decision an an "improve docs" bug. Maybe file a bug with importlib-resources?
For all the reasons above, let's use
importlib.resources.abc. It's a better choice. I'll update the what's changed to include that move. Let's get the docs to match.encukou commentedon Jun 16, 2022
IMO, that's a good strategy. The downside is that since the docs aren't synced with the backport, they feel somewhat second-class – they're split between CPython docs (reference) and backport docs (guides), and references to the docs (or to source code) seem awkward.
I hope that doesn't read as too negative. I mean it as constructive criticism – pointing out what I see as weak spots so I can help make them better.
And I don't mean to imply you need to work on the docs – if it's a chore you'd rather delegate, just say so! IMO there are people who'd love to work on this but are somewhat wary of touching “someone else's” docs.
@warsaw, do you want to review #93611?
@jaraco, what's the plan for having
Traversableimportable fromimportlib.abc? Is that temporary, or should it be documented and supported?jaraco commentedon Jun 16, 2022
And this split is inconsistent across .metadata and .resources, because docs for .resources were developed primarily in CPython but docs for .metadata were developed primarily in the backport. I'm unsure which model is preferable, but I'd like to align them. I lean slightly toward the .resources model, where most of the docs (reference and guides) are in CPython and maintained there, with backport-specific details in the backport.
See python/importlib_resources#240 where I've invited others to contribute to the docs.
Ultimately, I want not to be a blocker or hindrance to contribution, but I do want to avoid increasing maintenance toil. I'd prefer to have a development partner who can share ownership of the design, and reserve drive-by contributions to those that are mostly uncontroversial.
I'd say the presence in
importlib.abcis deprecated. I'm unsure if it actually needs to go through a deprecation process, since it wasn't documented.encukou commentedon Jun 16, 2022
What do you mean by sharing ownership of the design?
It turns out it is documented in 3.10.
It's
importlib.resources.abc.Traversablethat isn't documented (and that's the name mentioned in the function docs).jaraco commentedon Jun 16, 2022
I just mean I'm looking for someone who cares about the product and the design largely, someone invested enough to understand the motivations behind the existing implementation.
Okay. Then a deprecation process is probably in order. Let's get the docs to point to the preferred location (resources.abc) and worry about the deprecation separately.
pythongh-93610: Improve docs for importlib.resources (pythonGH-93611)
gh-93610: Improve docs for importlib.resources (#93611)
gh-93610: Improve docs for importlib.resources (GH-93611) (GH-95247)
jaraco commentedon Nov 26, 2022
I appreciate all of work you've done, @encukou . Is there anything more you'd like to see here? I'm closing this as complete, but feel free to re-open or otherwise revive the conversation.
aiven-anton commentedon Dec 6, 2022
Hi, I was about to open a new issue but noticed this discussion seems somewhat recent.
Currently
importlib.metadata.distributions(link) is not documented. Is this covered by the PRs that are already merged?It seems like this full API reference from the importlib-metadata project should somehow make its way into the CPython docs?
For this specific case, is it safe to consider
distributionspublic?Please let me know if I should open a separate issue.
jaraco commentedon Dec 6, 2022
This issue is mainly about
importlib.resources, which is different fromimportlib.metadata, though I too get those mixed up sometimes, and this issue did touch onimportlib.metadata.Unfortunately, last I checked, there's no support in CPython for generating the API docs from the source (where the behavior is documented). See python/importlib_metadata#119 for some work in this area. The solution at the time was for the CPython docs to link back to the API docs in
importlib_metadata, which is still present.Yes. Any names exposed by
__all__can be relied upon with the highest confidence. Any non-private names (those without a leading_) may also be fair game, though I would advise against using implementation details likeSectioned. In the past, I've not marked these classes private, but to my chagrin, I've since learned that that's the recommendation.If that didn't answer your questions, feel free to open a new issue.
aiven-anton commentedon Dec 6, 2022
@jaraco Thanks for your very thorough response, and sorry for the mixup, this answers all my questions! I didn't notice the reference to the API reference when first reading the docs, now I see it's there, so all good 👍