Deprecate and remove distutils

@brettcannon

BPO	41282
Nosy	@brettcannon, @doko42, @pfmoore, @jaraco, @ncoghlan, @tiran, @ned-deily, @merwok, @encukou, @methane, @ambv, @zooba, @dstufft, @pganssle, @hroncok, @frenzymadness, @pablogsal, @hugovk, @miss-islington, @tirkarthi, @FFY00
PRs	bpo-41282: (PEP 632) Deprecate distutils.sysconfig (partial implementation of the PEP) #23142 bpo-41282: Add deprecation warning and docs for distutils (PEP 632) #24355 bpo-41282: (PEP 632) Load install schemes from sysconfig #24549 bpo-41282: distutils: Fix stacklevel for DeprecationWarning #24657 bpo-41282: setup.py ignores distutils DeprecationWarning #25405 bpo-41282: Fix distutils.utils.byte_compile() DeprecationWarning #25406 bpo-41282: Consistent message and filter warning in setup.py (GH-25571) #25571 bpo-43976: add vendor config #25718 bpo-41282: Fix broken `make install` #26327 bpo-41282: Fix broken `make install` #26329 [3.10] bpo-41282: Fix broken `make install` (GH-26329) #26336

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-07-12.08:30:25.278>
labels = ['library', '3.10', '3.11']
title = 'Deprecate and remove distutils'
updated_at = <Date 2021-10-13.14:03:04.437>
user = 'https://github.com/jaraco'

bugs.python.org fields:

activity = <Date 2021-10-13.14:03:04.437>
actor = 'hroncok'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Distutils']
creation = <Date 2020-07-12.08:30:25.278>
creator = 'jaraco'
dependencies = []
files = []
hgrepos = []
issue_num = 41282
keywords = ['patch']
message_count = 44.0
messages = ['373548', '373549', '373586', '373612', '373613', '373614', '373615', '373622', '373629', '373630', '373631', '373633', '373649', '373651', '373652', '373653', '376390', '385812', '385946', '386453', '387897', '388224', '391082', '391087', '391175', '391683', '391768', '392322', '392327', '392329', '392331', '393017', '393018', '393020', '393021', '393023', '393026', '394241', '394242', '394248', '394275', '394277', '394302', '403835']
nosy_count = 23.0
nosy_names = ['brett.cannon', 'doko', 'paul.moore', 'jaraco', 'ncoghlan', 'christian.heimes', 'ned.deily', 'eric.araujo', 'Arfrever', 'ionelmc', 'petr.viktorin', 'methane', 'lukasz.langa', 'steve.dower', 'dstufft', 'p-ganssle', 'hroncok', 'frenzy', 'pablogsal', 'hugovk', 'miss-islington', 'xtreak', 'FFY00']
pr_nums = ['23142', '24355', '24549', '24657', '25405', '25406', '25571', '25718', '26327', '26329', '26336']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue41282'
versions = ['Python 3.10', 'Python 3.11']

Setuptools has adopted distutils as outlined in pypa/packaging-problems#127. Although there are some straggling issues, the current release of Setuptools fully obviates distutils if a certain environment variable is set. Soon, that behavior will be default.

Additionally, the distutils codebase remains maintained at pypa/distutils in a form suitable for releasing as a third-party package, should the need arise (i.e. pip install distutils).

The plan now is to freeze, deprecate, and in Python N + 0.1, remove distutils.

Already, Setuptools is identifying emergent bugs and other defects in distutils and providing fixes for them (bpo-41207, pypa/setuptools#2212). Keeping these changes in sync across three repos and different supported versions is tedious, so I'd like to move forward with the deprecation process as soon as possible.

Łukasz, would it be possible to add the deprecation warning and documented deprecation to Python 3.9?

So what is the plan to continue to support building cpython itself which depends on Distutils? Currently the build bootstraps itself without the aid of an existing Python interpreter instance. There would also be major impacts across the whole cpython development process. For example, there are many open Distutils issues in the bugs.python.org bug tracker. We need a plan on how those are to be handled (and that should take into account the expected transition from b.p.o to GitHub issues). People will continue to submit issues agains Distutils there so triage team members and core developers need to know how to handle such issues. What if an issue applies also or only to a previous release branch (i.e. where Distutils is still in the repo)? What about Distutils documentation in the Python docset? THose are just some off the top of my head.

I don't think any of these issues are necessarily blockers but they need to be planned for and reviewed. I think a PEP is definitely in order for a change of this magnitude.

It's too late to add a new deprecation in the Python 3.9 cycle. Next week is the *last* beta release. Most beta testing already took place.

So what is the plan to continue to support building cpython itself which depends on Distutils? Currently the build bootstraps itself without the aid of an existing Python interpreter instance. There would also be major impacts across the whole cpython development process.

My understanding was that the plan was to move the standard library distutils into a private module somewhere in the standard library and presumably to slim it down to only the bare minimum required for what is necessary to build Python itself. We're really only concerned with the use of distutils to build packages.

For example, there are many open Distutils issues in the bugs.python.org bug tracker. We need a plan on how those are to be handled (and that should take into account the expected transition from b.p.o to GitHub issues). People will continue to submit issues agains Distutils there so triage team members and core developers need to know how to handle such issues. What if an issue applies also or only to a previous release branch (i.e. where Distutils is still in the repo)?

As far as I can tell we've already been telling people that issues in distutils should be fixed in setuptools instead for a few years. I don't think anything needs to be done about the currently open distutils tickets before we *deprecate* distutils, though during the deprecation period we'll probably want to decide whether we want to migrate them, do a mass closure or just leave them to be ad hoc closed as people stumble upon them later. Mass closure may be complicated because tickets affecting CPython itself will still need to be addressed.

What about Distutils documentation in the Python docset? THose are just some off the top of my head.

The distutils documentation is already basically just a warning page that says "stop using distutils": https://docs.python.org/3/library/distutils.html#module-distutils

Before these reference materials are removed from the docs we'll need to make sure that all the stuff that's still supported is documented on the setuptools side.

I don't think any of these issues are necessarily blockers but they need to be planned for and reviewed. I think a PEP is definitely in order for a change of this magnitude.

A PEP may be a good idea, but I do think the change doesn't have a particularly large magnitude. Anyone using setuptools or pip has already been getting setuptools' monkey-patched version of distutils for ages now, and soon they will be getting setuptools' vendored version. The documentation already indicates that distutils is at least soft-deprecated in favor of setuptools and we've already been directing issues and PRs to setuptools instead of distutils. This last piece is really formalizing something we've been incrementally working towards for a long time now. Doesn't mean we shouldn't do it carefully and with a lot of notice, but it's also not a sudden and massive shift.

Deprecating in 3.10 is fine - everyone who needs to know about it releases whenever they like anyway, so we just need to make _some_ announcement.

I'd propose either moving it to Tools/distutils, or renaming it to _distutils. The point is that we're saying it's only fit for use for the core build now, and nobody else should ever import it (or complain about it ;) ).

Maybe it would make sense to remove distutils from the name completely, _buildutils or something. Dunno, seems like it might be reasonable just to further separate it from the concept of "distutils" the public library.

FYI PEP-387 (which I expect will be accepted once I catch up from vacation) specified deprecations are to be public for two releases before removal or approval from the SC for a shorter cycle.

So if distutils is deprecated in 3.10 then it can be removed in 3.12 or you can ask the SC for an exemption to do it in 3.11.

It's too late to add a new deprecation in the Python 3.9 cycle

Please can we add a note in 3.9, that it will be deprecated in 3.10?

gh-85454: Remove links from historical mentions of distutils (GH-95192)

pythongh-85454: Remove distutils documentation

gh-85454: Remove distutils documentation (#95239)

gh-85454: Remove distutils.ccompiler from Tools/c-analyzer (GH-95171)

pythongh-85454: Extract and inject distutils from setuptools whl

FWIW, test_cppext uses venv + setuptools already, but it's not exactly a shining example of how to do this well. (Helping to improve it would be appreciated in #92906 & #94751.)

Using setuptools on a newly built Python is not the most common case (run Python from its source directory, not on an "installed" Python). It causes multiple issues in sysconfig and setuptools. Hopefully, it seems like these issues are being fixed one by one, but it still seems to be bumpy work-in-progress (sysconfig+setuptools issues come back time to time).

As you wrote, it doesn't work on Windows yet.

#95254 is a different approach. It unpacks setuptools wheel and injects it into sys.path.

We like to remove distutils completely. [...] There are still some test cases and tools that have to be ported to virtual environments with setuptools or use different approaches. From the top of my head test_cppext, peg generator tests, and c-analyzer have to be changed first.

After gh-103316 vendored setuptools into Lib/test, should we consider this plan outdated and close the issue as done?

By the way, gh-92584 and gh-95254 are still open so I'm not sure.

The distutils package was removed in Python 3.12 by issue #92584. Remaining issues related to this removal are tracked by issue #92584. I close this one.

Debian advises at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1080851 that this breaks builds of Samba, as we use distutils.sysconfig.get_python_lib. Can distutils.sysconfig be made an alias for sysconfig so that software like ours that just needs this detail doesn't get unnecessarily broken?

It is very helpful for Samba to be able to build historical versions on modern operating systems, and changes like this continue to make our development process more painful.

Thanks for your understanding,

Andrew Bartlett

sysconfig has existed for 14 years already (including in Python 2) - is that not long enough to switch to it?

Installing setuptools should bring in the shim that you need. If we were to keep shims in the standard library, it would be impossible for them to properly migrate the old distutils interface.

Related - CPython has given this namespace over to Setuptools, so it no longer has the ability to present that name without breaking migration plans for distutils in Setuptools. Currently, Setuptools does present distutils, including distutils.sysconfig, but it's deprecated from there as well, and we're working to provide the necessary interfaces to migrate users away from that (including distutils.sysconfig -> sysconfig), so the best bet would be not to rely on Setuptools for this behavior if you can help it.

please can you check, if things like python3-config can solve your issues? this is also available for some time

Navigation Menu

Repository navigation

Deprecate and remove distutils #85454

52 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Deprecate and remove distutils #85454

Description

Activity

jaraco commented on Jul 12, 2020

jaraco commented on Jul 12, 2020

ned-deily commented on Jul 13, 2020

ambv commented on Jul 13, 2020

pganssle commented on Jul 13, 2020

zooba commented on Jul 13, 2020

dstufft commented on Jul 13, 2020

brettcannon commented on Jul 14, 2020

doko42 commented on Jul 14, 2020

52 remaining items

vstinner commented on Aug 3, 2022

tiran commented on Aug 4, 2022

arhadthedev commented on Jun 19, 2023

arhadthedev commented on Jun 19, 2023

vstinner commented on Jun 19, 2023

abartlet commented on Sep 5, 2024

zooba commented on Sep 5, 2024

jaraco commented on Sep 5, 2024

doko42 commented on Sep 5, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions