PEP 632: Remove the distutils package (Python 3.12)

[vstinner]

The distutils module was deprecated in Python 3.10 and is scheduled for removal in Python 3.12.

Right now, Python still uses distutils for different purpose:

• Build C extensions which can not be built by Makefile yet. Python 3.11 can build way more C extensions with Makefile than Python 3.10.
• peg_generator uses distutils to build C extensions.
• test_cppext uses distutils to check that the Python C API can be used in C++.
• Tools/c-analyzer/ uses distutils to get a C preprocessor.

Until Python can be built and used without distutils, I propose to first rename the distutils package to _distutils: rename th Lib/distutils/ directory to Lib/_distutils/.

Right now, test_venv fails without distutils, because pip fails on importing the distutils module in pip/_internal/locations/_distutils.py: https://github.com/pypa/pip/blob/cb24fb4052ca8ab8009866b0de61980c81a7e13c/src/pip/_internal/locations/_distutils.py#L9-L12

I created the PR #92585 which renames distutils to _distutils.

• PR: gh-92584: Remove the distutils package #99061

• PR: gh-92584: Remove references to removed _bootsubprocess #99062

• PR: gh-100238: Vendor distutils for peg_generator #104744

• PR: GH-92584: Move installation schemes overview to sysconfig docs #108018

• PR: GH-92584: Remove Installing Python Modules (Distutils version) #108020

• PR: GH-92584: Remove distutils from the newtypes tutorial includes #108024

• PR: GH-92584: Remove distutils mentions from importlib.metadata docs #108026

• PR: GH-92584: Redirect to Setuptools' guidance on extension modules #108030

• PR: GH-92584: Drop reference to Distutils in site.USER_BASE #108031

• PR: [3.12] GH-92584: Drop reference to Distutils in site.USER_BASE (GH-108031) #108039

• PR: GH-92584: Remove references to Distutils in PYTHONUSERBASE #108040

• PR: GH-92584: Remove references to Distutils in configure.rst #108043

• PR: GH-92584: Redirect macOS package installation to the PPUG #108044

• PR: GH-92584: Remove reference to Distutils in cx_Freeze's description #108047

• PR: [3.12] GH-92584: Remove reference to Distutils in cx_Freeze's description (GH-108047) #108057

• PR: [3.12] GH-92584: Redirect macOS package installation to the PPUG (GH-108044) #108058

• PR: [3.11] GH-92584: Redirect macOS package installation to the PPUG (GH-108044) #108059

• PR: [3.12] GH-92584: Remove references to Distutils in PYTHONUSERBASE (GH-108040) #108060

• PR: [3.11] GH-92584: Remove reference to Distutils in cx_Freeze's description (GH-108047) #108061

• PR: [3.12] GH-92584: Remove Installing Python Modules (Distutils version) (GH-108020) #108062

• PR: [3.12] GH-92584: Remove references to Distutils in configure.rst (GH-108043) #108063

• PR: [3.12] GH-92584: Remove distutils from the newtypes tutorial includes (GH-108024) #108333

• PR: [3.12] GH-92584: Move installation schemes overview to sysconfig docs (GH-108018) #110213

• PR: [3.11] GH-92584: Move installation schemes overview to sysconfig docs (GH-108018) #110214

[vstinner]

Right now, test_venv fails without distutils, because pip fails on importing the distutils module in pip/_internal/locations/_distutils.py

I reported this issue to pip: pypa/pip#11103

[vstinner]

The distutils package was deprecated by the issue #85454.

[vstinner]

PEP 632 "Deprecate distutils module": https://peps.python.org/pep-0632/

[FFY00]

That seems like a good plan. Thanks.

[jaraco]

My fear with renaming it is that it lingers indefinitely, and diverging in subtle ways from pypa/distutils. Why not address the dependencies and fully remove the functionality?

Regardless, I agree renaming is preferable to not removing the module at all.

[vstinner]

Why not address the dependencies and fully remove the functionality?

Someone has to modify Python to no longer depend on distutils. The issue description lists code using it. It doesn't seem trivial.

[FFY00]

Would it be reasonable to keep it around in the repo while we don't drop the dependency but not ship it as part of Python installations?

[vstinner]

Would it be reasonable to keep it around in the repo while we don't drop the dependency but not ship it as part of Python installations?

With my PR, the Lib/_distutils/ directory is not installed by make install.

[jkloth]

Do note that we have buildbots that run from an installed Python. The tests would need to also be modified to fail gracefully
(or skip) when _distutils is not available.

[vstinner]

Do note that we have buildbots that run from an installed Python. The tests would need to also be modified to fail gracefully (or skip) when _distutils is not available.

I propose skipping tests relying on _distutils if _distutils is not available.

[vstinner]

I wrote two PRs to avoid the deprecated distutils module in two tests:

• gh-92584: test_cppext uses setuptools #92639: test_cppext uses setuptools
• gh-92584: test_decimal uses shutil.which() #92640: test_decimal uses shutil.which() -- IMO an uncontroversial change