Repository navigation
Guide request: gracefully dropping support for older Python versions #450
Description
Activity
It occurs to me that an even more conservative approach would be to start out by specifying
Requires-Pythonfor currently supported Python versions. If you addRequires-Python: >= 2.7, != 3.0, != 3.1, != 3.2" for example, then it's relatively straightforward to get hold of a Python 2.6 installation and check that that gets pinned at the last version missing theRequires-Python` metadata as expected.That way, the release with the warning and updated
Requires-Pythonmetadata would only be about testing how up to date user's installation tools and caching proxies are (and could potentially be skipped entirely if PyPI's download metadata indicates that folks are using up to date deployment tools even for older target environments)@tonybaloney has investigated this further (by way of https://github.com/tonybaloney/friendly-deprecation-test), and found that while pip 9 won't install the incompatible version on older Pythons, the mere availability of that version still prevents installation of maintenance updates for the older version - you need to explicitly amend the installation command to request a compatible version.
This means that for older versions of pip, the problem will need to be addressed at the
pip-tools/pipenvlevel, with the tools checking for thedata-requires-pythonmetadata and amending the pinned version accordingly.That behaviour also becomes a feature request for future versions of
pipitself: checking data-requires-python compatibility when choosing the version to download, rather than when attempting to install the chosen version.Use case here is that package maintainers will continually drop support for out-of-support versions of Python, e.g. 2.6, 2.7 (soon), 3.2, 3.3 etc.
Every 12-18 months this will need to be updated.Looking at the download stats of users on packages I maintain, most downloads are coming from 2.7. I'd like to drop support for 2.7-3.4 and start implementing async/await features.
But, the users of older distributions probably aren't even aware that they're installing my package, it's part of an automated script, a CI/CD process. If most of the pip install'ing happens in background processes then user warnings are going to be largely ignored and erroring completely on installation is going to break their system.
If there were instead a way to continually develop and evolve packages inline with the supported Python distributions, whilst having "legacy" releases where I can still apply security fixes.
this could be a "large version that supported Python 2.6" tag on the metadata in the package API. Which I can periodically update, following semver.
Imagining this timeline..
**July 2018: **
Release v2.0.0 of package "foobar", drops Python 2.7 and 3.3 support. Users of "foobar" installing from Python 2.7 will get a warning and pip will install the last supported version, 1.3.0 insteadAugust 2018:
CVE raised against package "foobar", versions v2.0.1 and v1.3.1 released. All users for Python 3.4+ will get 2.0.1 and Python 2.7 will now get v1.3.1.September 2018
Maintainers of "foobar" want to drop Python 3.4 support and leverage asyncio, v3.0.0 is released.
Python 3.5 - v3.0.0
Python 3.4 - v2.0.1
Python 2.7, 3.3 - v1.3.1I think the current metadata can already express that OK - v2.0+ would have
Requires-Python: >= 3.5, while v1.3.x would either continue without the setting, or would have it set toRequires-Python: >= 2.7.The problem is that
pipisn't gracefully handling that case: it's erroring out because the incompatible version exists, rather than just installing the latest compatible version.Take Django as an example: https://pypi.python.org/simple/django/ and https://pypi.org/simple/django/ both set
data-requires-pythonon Django 2.0+. Ifpipwere to simply filter out the versions with unmetRequires-Pythonconstraints, thenpip install djangoon Python 2.7 would automatically pick up the latest 1.x release as desired.cc'ing @pradyunsg and @dstufft here before I turn this into an RFE for
pip's handling ofdata-requires-python.If pip were to simply filter out the versions with unmet Requires-Python constraints, then pip install django on Python 2.7 would automatically pick up the latest 1.x release as desired.
I'm not sure if I'm misunderstanding but I think this happens already.
@pradyunsg Thanks for checking that - it seemed odd that it wasn't working that way, so I'm happy to hear the problem is somewhere else rather than in pip's handling of
data-requires-python.Your results made me realise I hadn't checked the link metadata at https://pypi.org/simple/friendly-deprecation-test/, and that shows that PyPI hasn't picked up the
Requires-Pythonmetadata correctly, which then meansdata-requires-pythonisn't getting set properly (similar to what happened withpytestin this issue report, where @ewdurbin was able to retroactively update the link metadata for the older releases in the PyPI database)Checking https://pypi.org/project/friendly-deprecation-test/#files shows @tonybaloney hit the case where PyPI can't correctly infer
data-requires-pythonwhen only an sdist is uploaded: it needs to see a wheel archive with fully rendered metadata, and it needs to see that archive before it sees the sdist for the release.Yeah. There's some discussion over at pypi/warehouse#474 IIRC.
Basically, certain metadata is only loaded by warehouse when a wheel is uploaded first, and it's loaded from only that wheel. This might be one of those cases.
The specific problem hit here was to only upload the sdist (which is an entirely reasonable thing to do, it just has the consequence that PyPI doesn't know how to pull metadata from it without running arbitrary user uploaded code). I filed pypi/warehouse#3138 to suggest making it possible for project admins to set the
Requires-Pythonmetadata directly in the PyPI database.Having two potential "sources of truth" for this info is far from ideal, but it also makes it much easier to mitigate the consequences of incorrect metadata in a release (or uploading that metadata in a way that means PyPI doesn't infer the right value for
data-requires-python).I don't think it has anything to do with an sdist, you just need a new enough setuptools and twine when you do the upload.
I was using setuptools 38.5.1I was calling setup from distutils
and twine 1.8.1.. that'll be the issue@ncoghlan it works when using setuptools 👍
(env2) anthonyshaw ~/repo/friendly-deprecation-test master $ pip2 install friendly_deprecation_test --no-cache-dir Collecting friendly_deprecation_test Downloading friendly_deprecation_test-1.6.0-py2-none-any.whl Installing collected packages: friendly-deprecation-test Successfully installed friendly-deprecation-test-1.6.0 (env2) anthonyshaw ~/repo/friendly-deprecation-test master $ pip3 install friendly_deprecation_test --no-cache-dir Collecting friendly_deprecation_test Downloading friendly-deprecation-test-2.0.0.tar.gz Installing collected packages: friendly-deprecation-test Running setup.py install for friendly-deprecation-test ... done Successfully installed friendly-deprecation-test-2.0.0Oh yea, distutils doesn't support it at all.
I did post an article about this, happy to write another explaining the process. But considering I've fumbled my way into making it work I might not be the best person to do so :-)
1 remaining item
@tonybaloney Thanks for the useful article!
By the way, there's a autocorrect typo in
pip install — upgrade setuptoolsand likewise with the smart quotes intar xvfz dist/my-package-1.0.0.tar.gz -O | grep “Requires-Python”.I wrote:
I'll turn your experience with this into a packaging.python.org guide this weekend
Well, that didn't happen, and I'm no longer sure when I'll make time to do this myself (we're starting to get more feedback on the interpreter start-up changes in the 3.7 beta, so I have a few problems like https://bugs.python.org/issue33042 to find and fix).
If anyone does want to take a go at writing this themselves:
- comment here to say you're starting a draft (to reduce the change of duplicated effort)
- if you stop working on a draft, post to say that, too!
- this will be a new entry in the "guides" section of the repo
- I suggest "Dropping Support for Older Python Versions" as the title
- I suggest "source/guides/dropping-older-python-versions.rst" as the filename
- I suggest listing it immediately after "supporting-multiple-python-versions" in https://github.com/pypa/python-packaging-user-guide/blob/master/source/guides/index.rst
- adding an explicit cross-reference from the end of the multiple version support guide may also make sense (but don't worry about it for now if there doesn't seem to be a suitable place to add one)
I'll do it today.
Raised PR.
#459 has been merged. Thanks again @tonybaloney, for both the original article and the guide! :)
Reacted by Hugo van Kemenade- added a commit that references this issue
on Aug 21, 2018 - added a commit that references this issue
on Jan 18, 2019 - added a commit that references this issue
on Oct 18, 2019
This guide has been written by @tonybaloney and can now be found here: https://packaging.python.org/guides/dropping-older-python-versions/
Talking to @tonybaloney about https://twitter.com/anthonypjshaw/status/970088412177776640, I discovered that our support for gracefully dropping support for older Python versions is currently better than I realised, but it's also far from obvious how to actually do it and the potential errors to avoid.
The key moving parts are:
Requires-Pythonfield in metadata 1.2+: https://packaging.python.org/specifications/core-metadata/#requires-pythondata-requires-pythonlink attribute in the index server release listing API: https://www.python.org/dev/peps/pep-0503/#specificationdata-requires-pythonaware installer (e.g.pip9+)Requires-Pythonfield in the project metadata (e.g. thepython_requiressetting insetuptools24.2.0+)data-requires-pythoncorrectly (e.g.pypi.python.orgsince pypi/legacy@3e1ce8f)The most conservative approach to implicitly pinning older clients to the last compatible release would be to release an alpha or development release that sets "Requires-Python: >= 3.6" (for example) in the metadata, and includes an import-time warning snippet like the one below, but doesn't actually require the new version yet:
(Using
FutureWarningensures that the message will be visible by default even on versions whereDeprecationWarningis hidden by default)This conservative approach has a few key benefits:
pip install --prewhile others on recentpipversions will ignore it even ifRequires-Pythondidn't get set correctly (since it's a pre-release)Requires-Pythonin the metadata, you only end up with a pre-release that emits a spurious warning rather than one that's fundamentally broken, and can easily push a new version that sets that metadata correctly (since older clients should be updated to versions that default to ignoring pre-releases).