Repository navigation
Allow Python distributors to add custom site install schemes #88142
Description
Activity
As part of the distutils migration we plan to add a mechanism to let Python distributors to add site install schemes.
Currently, Python distributors are patching distutils to add custom install schemes for their packages. I think most of the reasoning boils down to them wanting to stop Python installers, such as pip, to modify/interfere with their packages.
With the distutils deprecation, and it becoming a 3rd party module, Python distributors can no longer patch it. Because of this, we made distutils use the sysconfig module instead, which fixes the issue at the moment -- Python distributors can now patch sysconfig itself -- but is not a long term solution.
To prevent Python distributors from having to patch implementation details, and have things break unexpectedly, we aim to introduce a system that distributors can use for this purpose.The idea is that they have a config file, which they can pass to configure, and in that config file they can specify some extra install schemes. These install schemes will get added in sysconfig, and will be loaded in the site module initialization.
In practice, it will look something like this:
config.py
EXTRA_SITE_INSTALL_SCHEMES = { 'posix_prefix': { 'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}', 'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}', 'purelib': '{base}/lib/python{py_version_short}/vendor-packages', 'platlib': '{platbase}/{platlibdir}/python{py_version_short}/vendor-packages', 'include': '{installed_base}/include/python{py_version_short}{abiflags}', 'platinclude': '{installed_platbase}/include/python{py_version_short}{abiflags}', 'scripts': '{base}/bin', 'data': '{base}', }, }./configure --with-vendor-config=config.py
Any reason this couldn't be in sitecustomize.py? Either by poking values into sysconfig directly (for back-compat) or we train sysconfig to look inside sitecustomize for a well-known name.
Making sysconfig look at sitecustomize seems like the wrong approach. It is behavior I would never expect, and there are use-cases where I still want the schemes to be present when the site module initialization is disabled.
I would also argue that having this mechanism available will be useful for other things.
hroncok commented
on Apr 30, 2021 hroncokmannequinMannequinCross referencing the discussion: https://discuss.python.org/t/mechanism-for-distributors-to-add-site-install-schemes-to-python-installations/8467
- changed the title
[-]Introduce mechanism to allow Python distributors to add custom site install schemes[/-][+]Allow Python distributors to add custom site install schemes[/+]on Apr 30, 2021 - changed the title
[-]Introduce mechanism to allow Python distributors to add custom site install schemes[/-][+]Allow Python distributors to add custom site install schemes[/+]on Apr 30, 2021 Making sysconfig look at sitecustomize seems like the wrong approach.
I mean, you're literally customizing the site, so having it be done from sitecustomize doesn't seem terribly wrong. But I agree, I'd rather see the code in sitecustomize poke paths into sysconfig, rather than the other way around.
The problem then would be that -S bypasses the path configuration entirely, which is likely going to point at non-existent paths. So yeah, for this case you need an override that isn't tied to the site module. Having a similar-but-different mechanism in sysconfig seems fine. I have a *slight* preference for non-executable code, mostly to avoid the risk of import hijacking, but it's only slight.
FYI, I have change the implementation to split the extra install schemes and extra schemes activated on site. This still makes sense over sitecustomize because we want the packages to be included in site.getsitepackages -- we want the vendor packages to essentially be the same as site-packages.
I have also moved sysconfig._get_preferred_schemes to the vendor config, instead of asking distributors to patch sysconfig -- this is why I prefer having it as executable code, we customize using functions, etc.
https://docs.python.org/3.10/library/sysconfig.html#sysconfig.\_get_preferred_schemesA config taking advantage of all these mechanisms should look like this:
EXTRA_INSTALL_SCHEMES = { 'vendor': { 'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}', 'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}', 'purelib': '{base}/lib/python{py_version_short}/vendor-packages', 'platlib': '{platbase}/{platlibdir}/python{py_version_short}/vendor-packages', 'include': '{installed_base}/include/python{py_version_short}{abiflags}', 'platinclude': '{installed_platbase}/include/python{py_version_short}{abiflags}', 'scripts': '{base}/bin', 'data': '{base}', }, } EXTRA_SITE_INSTALL_SCHEMES = [ 'vendor', ] def get_preferred_schemes(...): ...Do you have any thoughts on this?
Yes, I saw some of the latest changes in the PR.
My biggest concern is with the bare "import _vendor_config", which I'd prefer to have restricted to a fixed location, rather than being influenced by environment variables and other options. We already have an issue with readline being imported from anywhere it can be found.
A native flag to suppress it (i.e. something in sys.flags) could also become important for embedders, though it may matter more at a higher level (i.e. should an embedded CPython *ever* be using sysconfig? Probably not...). I wouldn't add a new flag for it right now, but I feel like sys.flags.isolated should probably imply that this should be ignored.
Though then we hit the issue again that these patches are about changing the "safe default" behaviour, which is what you want to get back when you run with -S or -I. And I'm not totally sure how to resolve this.
So basically, my concerns are:
- don't import arbitrary files
- ensure -S/-I options remain useful (or become even more useful)
Sorry for not getting to this sooner, but 5 days is really tight for such a change.
With -S/-I, It would be great if sys.path only included packages installed as part of the OS, and not those installed by
sudo pip. (Orpip --user, but that's covered).It seems that with the current patch, pip will install into site-packages and there's no way to disable/change site-packages. Is that the case?
My biggest concern is with the bare "import _vendor_config", which I'd prefer to have restricted to a fixed location, rather than being influenced by environment variables and other options. We already have an issue with readline being imported from anywhere it can be found.
Oh, I share the same concern! Though users could already mess up Python pretty badly by shadowing/overwriting parts of it, so I didn't thought it would be that big of an issue. Is there a way to achieve this while still allowing us to do everything we want?
Sorry for not getting to this sooner, but 5 days is really tight for such a change.
No worries. It was my fault, I should have been more attentive to the Python release timeline.
With -S/-I, It would be great if sys.path only included packages installed as part of the OS, and not those installed by
sudo pip. (Orpip --user, but that's covered).Perhaps we could add an option to enable only vendor site schemes?
It seems that with the current patch, pip will install into site-packages and there's no way to disable/change site-packages. Is that the case?
I mean, there is, though not as straightforward as -S/-I. I was planning on using it to build the distro entrypoint scripts, so that they only include the distro packages.
$ python -S > site.addsitedir(sysconfig.get_path('purelib', 'vendor')) > site.addsitedir(sysconfig.get_path('platlib', 'vendor'))
As I mentioned above, we could add a cli flag to do essentially the same.
The best option for restricting the import while still having it be a Python import is to find the file (if it's present in the expected location under sys.whatever), and then use importlib to import it: https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
I'd rather not have a new option here, I would much prefer "-S" in this context to mean "run Python with only core libraries" and "-s" to mean "run Python with only core and distro libraries" (and neither to mean "run Python with core, distro and user libraries").
That may be a bigger change, but there's enough angst around this issue that we would be better off getting it right this time, even if it changes things, than continuing to preserve the system that people dislike so much.
26 remaining items
Matthias, can you check if bpo-44982 solves your issues related to the conflicts you were experiencing for the Debian patching?
If so, it would unblock this issue.
I am probably gonna discuss this proposal with the conda-forge folks next week, to get some feedback from the Conda perspective.
I would like to have this unblocked so that we can move forward, as far as I know the only issues are the ones Debian is experiencing about this proposal alone not being able to completely replace the downstream patching.In the short term, and possible for the long term, Debian can continue to patch the install routine...
The problem with this approach is Setuptools is attempting to adopt distutils by exposing its own vendored copy of distutils as
distutils(top-level name). By doing this, it bypasses the Debian's patching of distutils as found in CPython. Because this bypass behavior breaks distutils for Debian users, the functionality has been disabled (opt-in).Setuptools would like to be able to present a version of distutils that, unpatched, runs on all the major platforms, and thus make it default.
That won't be possible until Debian can stop relying on its patching of distutils.
Here's what I propose:
- In pypa/distutils, add support for honoring the proposed install schemes (based on PR 25718). Merge with Setuptools.
- Add whatever ugly hacks are needed to pypa/distutils to honor other Debian-specific behaviors (revive Integrate Debian patches pypa/distutils#4 but properly match Debian expectations). Mark these ugly hacks as deprecated.
- In Debian, Fedora, etc, provide patches that configure the install schemes. Test with latest Setuptools and SETUPTOOLS_USE_DISTUTILS=local.
- Formalize the install schemes support in CPython (as in PR 25718).
- Ask Debian to propose more a cleaner interface for Debian-specific needs.
hroncok commented
on Jan 4, 2022 hroncokmannequinMannequinIn Fedora 36+ / Python 3.10+ we now use an install_scheme that looks like this:
'purelib': '{base}/local/lib/python{py_version_short}/site-packages', 'platlib': '{platbase}/local/{platlibdir}/python{py_version_short}/site-packages', 'scripts': '{base}/local/bin', 'data': '{base}/local', ...We got a user report [1] saying that
pip install --root ... --prefix /usrthe prefix is not respected at all.That is, users expect that /usr/local is the prefix, and when they explicitly set it to /usr, the /local/ bit will not be there, while in reality, /local/ is not a part of the prefix, but it is a part of the installation scheme.
I can somehow relate to that assumption.
Now I wonder whether we should have adapted prefix instead of the installation scheme :/
Any ideas on how to approach this problem? I am quite clueless.
I don't have a good answer, but given the title of this issue (which is specifically scoped to site install schemes), I'm tempted to say we should deal with prefixes in a separate, perhaps broader issue, and there address the reported issue (that a user's prefix override isn't honored by the scheme) and maybe more broadly the issue that there's not a design/spec for python installations (and probably there should be).
FRidh commented
on Jan 25, 2022 FRidhmannequinMannequinIn Nixpkgs we install every Python package under a unique prefix, a so-called Nix store path. If we were to use sysconfig for installing packages, then we'd need to be able to dynamically set the paths. This was also discussed as part of the Installer project. https://github.com/pradyunsg/installer/issues/98
We could use a custom scheme, however, we do need to be able to dynamically set a certain variable, e.g.
base.variables = {"installed_base": "$out", "base": "$out", "platbase": "$out", "installed_platbase": "$out"} # Note there is no `sysconfig.get_default_scheme()` sysconfig._expand_vars("posix_prefix", variables)I could imagine we do something like
# check whether we're in a nix build and want to install in a prefix if "IN_NIX_BUILD" in os.environ: base = os.environ["out"] scheme = {...}We'd then need to update the base variable in sysconfig or partially expand our own scheme using this variable.
I don't have a good answer, but given the title of this issue (which is specifically scoped to site install schemes), I'm tempted to say we should deal with prefixes in a separate, perhaps broader issue, and there address the reported issue (that a user's prefix override isn't honored by the scheme) and maybe more broadly the issue that there's not a design/spec for python installations (and probably there should be).
Technically, I guess we could (instead of redefining the default installation scheme) redefine the default
{base}and{platbase}insysconfig.get_config_vars(). However, I suspect that would require further changes todistutilsandpypa/distutilsto respect that :/I'll start a discussion on https://discuss.python.org/ trying to sum up what went wrong with our custom installation scheme and what we want to achieve instead.
From the feedback I have gathered so far, I think this suggestion is a hard sell for some people due to the performance impact, so I think that's the first thing we have to work on if we want to implement this. There is some duplication in
getpathandsysconfigthat could be removed, which seems like a good starting point.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: