distutils and setuptools._distutils stubs are different #10255

Activity

1. AlexWaygood commented on Jun 4, 2023

   

   AlexWaygood

   on Jun 4, 2023

   Member

   More actions

   Note that we deliberately removed much of our setuptools/_distutils stubs in #9795, on the grounds that setuptools appeared to view the directory as an implementation detail; it was a significant maintenance burden to keep our stubs for the directory up to date with the runtime package; and users should probably be importing things from setuptools instead of setuptools._distutils wherever possible.

   FWIW I'm open to revisiting that decision if there's a good reason to, but we should think carefully about it. Prior to that change, it felt like stubtest was breaking every month due to the rate of changes setuptools was making to setuptools._distutils.
2. hauntsaninja commented on Jun 4, 2023

   

   hauntsaninja

   on Jun 4, 2023

   Collaborator

   More actions

   My opinions:

   stdlib distutils is gone. Most users already do not get it when they import distutils. I have no interest in seeing further improvements made to its stubs.

   setuptools._distutils seems not infrequently used in the wild, and I imagine it will be used more by projects looking to support 3.12 with the fewest changes possible. Given this, I'm supportive of improvements to these stubs.

   Regarding maintenance burden from setuptools development of setuptools._distutils: I default to the position that users are worse in the world where we don't have stubs at all (certain false positive) than in the world where stubs exist + we stubtest allowlist them (possible incorrect behaviour, but likely only if depending on implementation detail)

   Of course, if there's no usage of the stubs then better off staying deleted, e.g. I don't see much point in bringing back the setuptools._distutils.command.* removed in #9795

   So all of this adds up to the following possible plan of action:

   • Never touch stdlib distutils again, unless some user specifically asks, until 2028 when rm -rf.
   • Improve setuptools._distutils, specifically focussing on the stub files that already exist
   • Try vaguely to avoid trying back things deleted in Remove most of setuptools._distutils #9795, unless needed by existing stubs or used by users (for instance, we should probably bring back setuptools._distutils.ccompiler)
   • If additions cause maintenance burden on setuptools updates, we can allowlist more aggressively. We can also just let the stubsabot upgrade PR languish if we don't have time for it (e.g. the stripe one has been sitting around for most of a year and I'm fine with that)
   • And I don't view any of this as high priority, I think fine to do other stuff and wait for user issues

   (re maintenance: I think part of the problem setuptools has is that they themselves are not sure what is implementation detail or not. Leaving typing aside, I've been broken by setuptools a dozen times in the last couple years)

   👍React with 👍2Reacted by Alex Waygood and Sebastian Rittau
3. AlexWaygood commented on Jun 4, 2023

   

   AlexWaygood

   on Jun 4, 2023

   Member

   More actions

   That all sounds good; happy to cosign @hauntsaninja's plan of action (...in particular the last bullet 😉)
4. JukkaL commented on Jul 3, 2023

   

   JukkaL

   on Jul 3, 2023

   Contributor

   More actions

   It looks like distutils is importable if setuptools is installed, even on Python 3.12:

   $ python3
   Python 3.12.0b3 (main, Jul  3 2023, 17:56:25) [GCC 9.4.0] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import distutils
   >>> distutils.__file__
   '/home/jukka/venv/test/lib/python3.12/site-packages/setuptools/_distutils/__init__.py'
   

   I wonder if distutils isn't going away after all, and maybe setuptools stubs should ship with distutils stubs that just expose setuptools._distutils?
5. hauntsaninja

   added a commit that references this issue on Jul 3, 2023

   Bring back a few setuptools._distutils files

   ...

   1ece6e6
6. link2xt

   added 2 commits that reference this issue on Oct 4, 2023

   ci(mypy): ignore distutils

   ...

   32db62e

   ci(mypy): ignore distutils

   ...

   7fc2b06
7. abravalheri commented on Oct 23, 2023

   

   abravalheri

   on Oct 23, 2023

   Contributor

   More actions

   stdlib distutils is gone. Most users already do not get it when they import distutils. I have no interest in seeing further improvements made to its stubs.

   It looks like distutils is importable if setuptools is installed, even on Python 3.12

   As noted by the lead setuptools developer in pypa/setuptools#2806 (comment) and pypa/setuptools#2806 (comment), setuptools does allow users importing distutils directly via a MetaPathFinder. For all typing effects setuptools provides the implementation of distutils even for Python 3.12+ (with a "deprecated" status).

   setuptools._distutils seems not infrequently used in the wild, and I imagine it will be used more by projects looking to support 3.12 with the fewest changes possible. Given this, I'm supportive of improvements to these stubs.

   (re maintenance: I think part of the problem setuptools has is that they themselves are not sure what is implementation detail or not.

   setuptools._distutils is merely an implementation detail and should not be imported directly. Importing it directly causes well-known runtime errors (e.g. pypa/setuptools#3743). The only supported way of importing stuff from distutils by setuptools is via import distutils/from distutils... statements.

   If typeshed wants to support the stubs for the distribution of distutils that gets installed together with setuptools, that would need to be done via the distutils name not setuptools._distutils.
8. Avasam commented on Oct 28, 2023

   

   Avasam

   on Oct 28, 2023

   Last edited by Avasam

   SponsorCollaborator

   More actions

   I know we could ship distutils-stubs as part of types-setuptools. The main issue would still be the amount of code duplication and maintainability. (and whether all of typeshed's tooling would recognize it correctly)
   We could try something with partial once the typeshed-internal/stub_uploader#109 fix is merged.
   Or re-importing everything from stdlib distutils with star imports everywhere.
   Either way only manually typing/updating any issue raised by stubtest.

   ---

   Personally I'd prefer putting the efforts into typing setuptools itself and reducing false-positives (even if it means more false-negatives) in typeshed for distutils users in the short-to-mid term.
9. AlexWaygood

   closed this as completedin #10948on Mar 16, 2024