Complete port to CPython

In GitLab by @jaraco on Feb 20, 2019, 15:00

bpo-34632 describes the addition of the importlib_metadata functionality to CPython.

In python 9327, an early port was attempted, but the underlying implementation has since changed dramatically, so another port is probably the best approach.

Here's what I suggest for a port: Either CPython or this package should be the canonical, authoritative implementation. Currently, this implementation is that one, but at some point in the future, perhaps I suspect the CPython one may be. Regardless, one repo or another should contain the changes the comprise the port, the commits that transform the code from the canonical code base to the other (such that one could merge a contribution made in the canonical codebase into the other).

@warsaw

In GitLab by @jaraco on Feb 23, 2019, 16:04

@warsaw We're working on a design to help maintain this backport and trying to figure out the best way to:

keep one canonical source that's applied to CPython and the backport
rely on Git functionality to maintain ancestry from that canonical source to the CPython repo and backport
avoid pulling in the entire history into CPython

Here's what I've proposed:

On the left is GitHub (CPython). On the right is this GitLab repo (importlib_metadata). In the middle is a squash of the port, so that commit would appear in the CPython repo and reference (in the commit message) the commit from which it was squashed. From there, that squash commit becomes the canonical source for both the 3.8 implementation and the backport.

One concern about this approach is that CPython doesn't allow for merge commits, only squash merges, meaning that ancestral commit wouldn't exist in the CPython repo after accepting a PR in the default workflow. Would it be possible to bypass the GitHub workflow controls to include that orphan commit so it's available to support the backport?

@stevenjma

In GitLab by @jaraco on Feb 23, 2019, 16:25

assigned to @stevenjma

@warsaw

In GitLab by @jaraco on Feb 23, 2019, 17:23

@warsaw Do you have any opinion on how the zipp dependency should be handled in the CPython implementation? I see a few possible options:

vendor it
supply it as a top-level package under the same name or maybe another name
merge its functionality into zipfile

In GitLab by @jaraco on Feb 23, 2019, 20:43

We're going to aim for the least-intrusive "vendor" approach for now.

In GitLab by @warsaw on Feb 23, 2019, 23:42

We have the same general problem with importlib_resources, but that library is stable enough that it hasn't been much problem to just reapply the diffs from importlib_resources (the canonical version) to CPython.

I expect that eventually importlib_metadata will be as stable, and won't need much work to port changes over to CPython. Maybe that's being too optimistic. The problem I see is that importlib_resources still has to support Python 2.7 (for now - that's part of its mission to obsolete pkg_resources). So somehow, we'll need a repo with 2.7 support and without it. Maybe that's one of the things the middle section will do.

I'm not sure that changing the GitHub workflow for CPython for just this package will be acceptable.

The main thing that I would want in any approach (other that manual, as with importlib_resources) is automation, or well-documented procedures for cross-pollination.

As for the zipp dependency; I'm not a huge fan of vendorizing to be honest. It leads to all kinds of downstream problems. Is zipp something that you would consider adding to Python 3.8, either as a separate library or as part of pathlib? It seems like it would be pretty useful functionality for core Python, and I think you've mentioned that approach before. That would be ideal IMHO.

@warsaw

In GitLab by @jaraco on Feb 23, 2019, 23:59

@warsaw I think you're right about the code flow in CPython. I'm leaning more toward just maintaining both forms of the code here in this repo and making normal pull requests to CPython by copying from the CPython branch here.

We have another problem - the tests are failing in CPython because neither pip nor importlib_metadata exist as packages. And in the general case, there aren't any third-party packages against which to test.

Do you have any advice on how to handle this based on your past work on the last port? I'm thinking we need to make fixtures that emulate egg-info and dist-info packages, independent of the implicit ones.

@jaraco

In GitLab by @warsaw on Feb 24, 2019, 24:14

@jaraco That's a good question. I wonder if we can cheat by using the packages vendored in ensurepip? This is probably a good question for python-dev.

In GitLab by @jaraco on Feb 24, 2019, 22:02

In #49, we've made excellent progress on eliminating dependence on 'pip', 'wheel', and 'importlib_metadata' being installed.

There are still a couple of tests that need to be addressed:

Some tests use 'packaging' to validate the dependencies. That won't be possible in CPython. I suggest just neutering those tests during the port (see 6c35d56).
The tests for the 'local' distribution still expect to be running on the local importlib_metadata. We should build a fixture that simulates a package in the current working directory to validate the behavior (see !48).