Skip to content

ensurepip: add configure --with-wheel-pkg-dir=PATH to get wheel packages from a system directory #87022

Closed
Closed
ensurepip: add configure --with-wheel-pkg-dir=PATH to get wheel packages from a system directory#87022
Labels
3.10 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directory

Description

@vstinner
vstinner
opened on Jan 7, 2021
BPO 42856
Nosy @ncoghlan, @vstinner, @tiran, @mcepl, @dstufft, @pradyunsg, @hroncok
PRs
  • [WIP] bpo-42856: Add --with-wheel-pkg-dir=PATH configure option #24151
  • bpo-42856: Add --with-wheel-pkg-dir=PATH configure option #24210
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-01-21.09:50:21.117>
created_at = <Date 2021-01-07.11:21:06.475>
labels = ['library', '3.10']
title = 'ensurepip: add configure --with-wheel-pkg-dir=PATH to get wheel packages from a system directory'
updated_at = <Date 2021-01-21.09:50:21.117>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2021-01-21.09:50:21.117>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-21.09:50:21.117>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2021-01-07.11:21:06.475>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 42856
keywords = ['patch']
message_count = 7.0
messages = ['384577', '384578', '384579', '385027', '385031', '385032', '385353']
nosy_count = 7.0
nosy_names = ['ncoghlan', 'vstinner', 'christian.heimes', 'mcepl', 'dstufft', 'pradyunsg', 'hroncok']
pr_nums = ['24151', '24210']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42856'
versions = ['Python 3.10']

    Activity

    1. vstinner commented on Jan 7, 2021

      @vstinner
      vstinner
      on Jan 7, 2021
      MemberAuthor

      The Fedora packaging policy recommends to "unbundle" bundled dependencies.

      "Fedora packages SHOULD make every effort to avoid having multiple, separate, upstream projects bundled together in a single package."
      https://docs.fedoraproject.org/en-US/packaging-guidelines/#bundling

      The main motivation is to ease updates when fix serious vulnerabilities (less packaging work).

      The ensurepip package contains bundled wheel packages of setuptools and pip:

      $ ls Lib/ensurepip/_bundled/
pip-20.2.3-py2.py3-none-any.whl
setuptools-47.1.0-py3-none-any.whl

      The Fedora python3 package doesn't contain the ensurepip._bundled package:

      $ python3
Python 3.9.1 (default, Dec  8 2020, 00:00:00) 
>>> import ensurepip._bundled
ModuleNotFoundError: No module named 'ensurepip._bundled'

      Instead, a separated RPM package python-pip-wheel provides wheel packages in /usr/share/python-wheels/ directory:

      $ ls /usr/share/python-wheels/
pip-20.2.2-py2.py3-none-any.whl*
setuptools-49.1.3-py3-none-any.whl*
wheel-0.34.2-py2.py3-none-any.whl*

      Fedora has a downstream patch on ensurepip (written by Miro Hrončok) to always use /usr/share/python-wheels/:

      https://src.fedoraproject.org/rpms/python3.10/blob/master/f/00189-use-rpm-wheels.patch

      Fedora has packages of 9 CPython versions: 2.6, 2.7, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10.

      https://developer.fedoraproject.org/tech/languages/python/multiple-pythons.html

      Having a separated package for wheel packages allows us to upgrade a single package (python-pip-wheel) for setuptools/pip bugfix or security vulnerability.

      --

      I propose to add a new --with-wheel-pkg-dir=PATH option to the ./configure script. If used, ensurepip will only use wheel packages from this directory. Otherwise, the existing code is unchanged. In short, the behavior is unchanged, unless the option is used explicitly.

      If a directory is specified but wheel packages are missing, ensurepip fails.

      If the directory contains multiple wheel packages of different versions, the most recent version is used of each package.

      Note: In practice, the Fedora package only provides a single wheel package of each Python module. But I propose to make the Python upstream code as generic as possible.

      I'm working on a pull request to implement this.

      Downstream Fedora issue: https://bugzilla.redhat.com/show_bug.cgi?id=1874803

      --

      Fedora (Miro) already contributed to ensurepip to make ensurepip less dependent on pip internals:

      commit 88f82b2
      Author: Miro Hrončok <miro@hroncok.cz>
      Date: Tue Mar 10 22:16:28 2020 +0100

      bpo-38662: ensurepip invokes pip via runpy (GH-18901)

The ensurepip module now invokes pip via the runpy module.
Hence it is no longer tightly coupled with the internal API of the bundled
pip version, allowing easier updates to a newer pip version both
internally and for distributors.

This way, any changes to the internal pip API won't mean ensurepip needs to be
changed as well. Also, distributors can update their pip wheels independent on
CPython release schedule.

Co-Authored-By: Pradyun Gedam <pradyunsg@gmail.com>
Co-Authored-By: Miro Hrončok <miro@hroncok.cz>

      This change was already related to Fedora downstream change to get wheel packages from a different directory. Fedora can use a different pip version (older or more recent) than ensurepip._bundled.

    2. vstinner
      added
      3.10 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      on Jan 7, 2021

    3. vstinner commented on Jan 7, 2021

      @vstinner
      vstinner
      on Jan 7, 2021
      MemberAuthor

      If used, ensurepip will only use wheel packages from this directory.

      An alternative is to find packages in all directories and pick the most recent version.

      Example with a specified directory *and* ensurepip._bundled is available:

      • wheel package directory:

        • pip 20.2.2
        • setuptools 49.1.3
        • wheel 0.34.2

      • ensurepip._bundled

        • pip 20.2.3
        • setuptools 47.1.0

      Most recent versions:

      • pip 20.2.3 from ensurepip._bundled
      • setuptools 49.1.3 from the wheel package directory

      Problem: I'm not sure that pip is fully compatible with any setuptools version.

    4. hroncok commented on Jan 7, 2021

      @hroncok
      hroncokmannequin
      on Jan 7, 2021
      Mannequin

      For simplicity, I would avoid mixing wheels from 2 different directories.

    5. vstinner commented on Jan 13, 2021

      @vstinner
      vstinner
      on Jan 13, 2021
      MemberAuthor

      For simplicity, I would avoid mixing wheels from 2 different directories.

      Right. I wrote PR 24210 which is simpler. It either uses bundled wheels, or wheels from the directory.

    6. mcepl commented on Jan 13, 2021

      @mcepl
      mceplmannequin
      on Jan 13, 2021
      Mannequin

      We (SUSE) have updated versions of the wheels as special Sources, and then this in the %prep stage of our SPEC file:

      # Replace bundled wheels with the updates ones
rm -v [Lib/ensurepip/_bundled/](https://github.com/python/cpython/blob/main/Lib/ensurepip/_bundled)*.whl
cp -v %{SOURCE20} %{SOURCE21} [Lib/ensurepip/_bundled/](https://github.com/python/cpython/blob/main/Lib/ensurepip/_bundled)
STVER=$(basename %{SOURCE20}|cut -d- -f2)
PIPVER=$(basename %{SOURCE21}|cut -d- -f2)
sed -E -i -e "s/^(\s*_SETUPTOOLS_VERSION\s+=\s+)\"[0-9.]+\"/\1\"${STVER}\"/" \
          -e "s/^(\s*_PIP_VERSION\s+=\s+)\"[0-9.]+\"/\1\"${PIPVER}\"/" \
    [Lib/ensurepip/__init__.py](https://github.com/python/cpython/blob/main/Lib/ensurepip/__init__.py)

      A bit of manual work required, but it doesn't lead to so incredible convoluted constructs as I see in Fedora (nothing against it, but our build system is already convoluted enough).

    7. hroncok commented on Jan 13, 2021

      @hroncok
      hroncokmannequin
      on Jan 13, 2021
      Mannequin

      In Fedora, we update the wheels independently without rebuilding Python.

      What incredible convoluted constructs do you have in mind in particular?

    8. vstinner commented on Jan 20, 2021

      @vstinner
      vstinner
      on Jan 20, 2021
      MemberAuthor

      New changeset 75e59a9 by Victor Stinner in branch 'master':
      bpo-42856: Add --with-wheel-pkg-dir=PATH configure option (GH-24210)
      75e59a9

    9. vstinner
      closed this as completedon Jan 21, 2021
    10. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directory

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions