Repository navigation
Presence of 'egg-info' metadata directory + use of virtualenv results in duplicate entry points #410
Description
(I'll preface this by saying that I think this might be similar to #101. It also feels like something that must have been reported elsewhere but my search-fu isn't sufficient to find those previous report(s))
If a {package}.egg-info directory exists and you're using a virtualenv, the importlib_metadata.distributions() function returns a duplicate distribution. Since this function is used by importlib_metadata.entry_points(), you'll also see a duplicate set of entrypoints. For example, consider the following using the simplest package I could find:
❯ cd /tmp
❯ git clone https://github.com/stephenfin/clouds2env
❯ cd clouds2env
❯ virtualenv .venv
❯ source .venv/bin/activate
❯ pip install . importlib-metadataimport importlib_metadata
for dist in importlib_metadata.distributions():
if not dist.entry_points:
continue
print(dist._path)
for ep in dist.entry_points:
print(f'\t{ep}')
print('***')This returns something like the following:
clouds2env.egg-info
EntryPoint(name='clouds2env', value='clouds2env:main', group='console_scripts')
***
/tmp/clouds2env/.venv/lib/python3.10/site-packages/clouds2env-1.1.1.dev1+g6d0ecef.dist-info
EntryPoint(name='clouds2env', value='clouds2env:main', group='console_scripts')
***
/tmp/clouds2env/.venv/lib/python3.10/site-packages/setuptools-62.3.2.dist-info
... {truncated} ...
***
/tmp/clouds2env/.venv/lib/python3.10/site-packages/wheel-0.37.1.dist-info
... {truncated} ...
***
/tmp/clouds2env/.venv/lib/python3.10/site-packages/pip-22.1.1.dist-info
... {truncated} ...
We're not installing anything in editable mode so I think the distributions are actually the same and the info from egg-info should be ignore in favour of the info from the dist-info directory. I also see attempts to deduplicate the returned entrypoints in importlib_metadata.entrypoints() but I haven't dived into why this isn't working (or whether preventing this issue is even the goal of that code).
Additional info
❯ virtualenv --version
virtualenv 20.13.4 from /usr/lib/python3.10/site-packages/virtualenv/__init__.py
❯ pip freeze
clouds2env @ file:///tmp/clouds2env
importlib-metadata==5.0.0
PyYAML==6.0
zipp==3.8.1
❯ cat /etc/system-release
Fedora release 36 (Thirty Six)
Activity
stephenfin commentedon Oct 4, 2022
Actually, no. Looking at this more, I see that
importlib_metadata.entry_points()is actually working as expected when run from the shell.The failures I'm seeing elsewhere are when running tests via
tox.toxmust be doing something weird here. It does mean my reproducer above isn't actually reproducing anything. I'll close this while I try to figure out what's going on with tox.stephenfin commentedon Oct 4, 2022
Okay, as expected I was hitting an existing issue but it wasn't reported as an issue but rather a PR, #377. That PR was abandoned in favor of another fix, #379, which was released in v4.12.0 and backported to 4.11 as part of v4.11.4. I was seeing it on some environments (Ubuntu 20.04) and not others (Fedora 36) because
setuptools(version 65.3.0) is vendoringimportlib_metadataon the former and the vendored version doesn't contain this fix.stephenfin commentedon Oct 4, 2022
Some more info. The reason #379 is relevant is because it fixes the value returned by
PathDistribution._name_from_stem. This value is used to generate the_normalized_nameattribute. Without the fix in #379, the presence of a{package}.egg-infodirectory (note: no version number) would result in this function returning a normalized package name of{package}.eggfor that distribution rather than{package}as intended. As I noted in comment 0, the call toimportlib_metadata.distributions()returns two distributions and we attempt to deduplicate the list of packages when generating entry points inimportlib_metadata.entry_points(). That de-duplication relies on the value of_normalized_namehowever, so because that was wrong, our whole de-duplication attempt failed and we end up with a double set of entry points.Regarding how to fix this: it's tricky. I noted above that
setuptoolsis vendoringimportlib_metadata. However, on closer inspection I noticed that setuptools on my Fedora host is even older (v62.3.2) and therefore still missing this fix. What seems to be different though is the type of objectimportlib_metadat.distributions()returns in the two environments when usingtox. On the Ubuntu 20.04 host, I see<class 'setuptools._vendor.importlib_metadata.PathDistribution'>objects being returned. On the Fedora 35 host, I see<class 'importlib_metadata.PathDistribution'>objects. I have no idea why, though I suspect it's something to do with "resolvers"? In any case, in stevedore we're opting to always de-duplicate the results fromentry_points()rather than making this de-duplication dependent on the version of setuptools in use and whether and the type of objects returned fromdistributions()are from an older, vendored version ofimportlib_metadataor not.jaraco commentedon Nov 4, 2022
I believe the best fix is to use
importlib_metadata>=4.11.4on Python < 3.12. I think I'd intended to backport that fix in CPython versions allowing it, but it seems I didn't get to it and even Python 3.11.0 has the flawed implementation. Setuptools needs to refresh its vendored version.require importlib_metadata
require importlib_metadata
require importlib_metadata
require importlib_metadata
FFY00 commentedon Mar 10, 2023
Can you show us the content of the metadata directory for your package?
jaraco commentedon Mar 18, 2023
The fix is definitely in place. Please do some more investigation on your environment or put together an (ideally minimal) reproducer and open a new issue.
Since this issue implicates "virtualenv", double-check the Setuptools version that's present in the virtualenv (or upgrade it).