Rewrite getpath.c in Python

@ncoghlan

BPO	45582
Nosy	@ncoghlan, @tiran, @benjaminp, @ambv, @ericsnowcurrently, @zooba, @miss-islington, @neonene, @FFY00
PRs	bpo-45582: Port getpath[p].c to Python #29041 bpo-45582: Fix out-of-tree build issues with new getpath (GH-29902) #29902 bpo-45582: Add conditions to isdir/isfile/isxfile in getpath.c #29906 bpo-45582 Fix prototype of _Py_Get_Getpath_CodeObject. #29907 bpo-45582: Fix signature of _Py_Get_Getpath_CodeObject (GH-29921) #29921 bpo-45582: Fix getpath_isxfile() and test_embed on Windows #29930 bpo-45582: Change the folder location of PGO-instrumented builds on Windows #29941 bpo-45582: framework build: modPath must not be const (GH-29944) #29944 bpo-45582: Ensure PYTHONHOME still overrides detected build prefixes #29948 bpo-45582: Fix framework path and bootstrap build #29954 bpo-45582: Write empty pybuilddir.txt on Windows to allow relocatable build directories #29979 bpo-45582: Fix test_embed failure during a PGO build on Windows #30014 bpo-45582: Don't fail if ENV_PATH is None in getpath.py (GH-31699) #31699

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2021-12-03.00:10:47.332>
created_at = <Date 2021-10-22.23:26:55.306>
labels = ['interpreter-core', 'type-feature', '3.11']
title = 'Rewrite getpath.c in Python'
updated_at = <Date 2022-03-06.19:49:35.401>
user = 'https://github.com/zooba'

bugs.python.org fields:

activity = <Date 2022-03-06.19:49:35.401>
actor = 'christian.heimes'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2021-12-03.00:10:47.332>
closer = 'steve.dower'
components = ['Interpreter Core']
creation = <Date 2021-10-22.23:26:55.306>
creator = 'steve.dower'
dependencies = []
files = []
hgrepos = []
issue_num = 45582
keywords = ['patch']
message_count = 30.0
messages = ['404841', '404843', '405348', '405640', '405641', '405653', '405658', '405731', '405742', '406183', '406184', '406384', '407113', '407322', '407326', '407561', '407562', '407565', '407587', '407604', '407609', '407616', '407720', '407846', '407852', '407881', '407960', '407994', '408226', '414622']
nosy_count = 9.0
nosy_names = ['ncoghlan', 'christian.heimes', 'benjamin.peterson', 'lukasz.langa', 'eric.snow', 'steve.dower', 'miss-islington', 'neonene', 'FFY00']
pr_nums = ['29041', '29902', '29906', '29907', '29921', '29930', '29941', '29944', '29948', '29954', '29979', '30014', '31699']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue45582'
versions = ['Python 3.11']

Linked PRs

As discussed in bpo-42260, combining the two getpath implementations into a single Python implementation would make it more maintainable and modifiable (particularly where distros need to patch to support alternative layouts).

The PR has more work to do, but the overall layout/changes are more or less there, so happy to discuss feedback/etc.

Obviously there are a lot of edge cases here, but they seem to be mostly tested already. And I think we're early enough in alpha to find any major issues (or absorb any necessary minor changes - seems like trailing slashes might change on some paths).

There are also some changes/hacks into the new frozen module support, so that I can freeze getpath.py without turning it into a module. I really just want to execute the bytecode - no reason for any of its contents to stick around - and this works out pretty neatly. But if the changes to frozen modules seem off then maybe we can split it into totally separate freezing support?

So I think I've found my first completely unavoidable API break: PyConfig_Read(config) has to work before initialisation, but is also supposed to fill out all the fields (including the search path). But because we need at least an interpreter state, we now can't calculate everything.

The only test that seems to be affected here is test_embed.test_init_read_set(), which does a PyConfig_Read() and then inserts new paths into module_search_paths before initialising. With that one skipped, I think everything else can be handled.

Last remaining test failure is one that I can't figure out on my own - the freeze test is rerunning a CPython build (on Linux) and is apparently building getpath.c with the ".c.o" rule rather than the "Modules/getpath.o" rule.

Any tips as to what I should be looking at to figure this one out?

On Wed, Nov 3, 2021 at 1:21 PM Steve Dower <report@bugs.python.org> wrote:

Last remaining test failure is one that I can't figure out on my own - the freeze test is rerunning a CPython build (on Linux) and is apparently building getpath.c with the ".c.o" rule rather than the "Modules/getpath.o" rule.

Any tips as to what I should be looking at to figure this one out?

That test does an out-of-tree build. Might that be related?

I'm betting the out-of-tree (actually just deeper within the same tree) bit is related, but I just can't see how. Modules/getbuildinfo.c takes extra parameters and they seem to be being used, so I can't tell why getpath.c's are not (those rules are listed right next to each other, but well above the .c.o rule).

Unsurprisingly, it was a bad edit that I made to the Makefile myself. The commit that undoes it is aedebcc but since I made the initial change as part of this PR, it was never merged in.

Now to find out why the old getpath could somehow locate the stdlib but new getpath cannot... (I'm guessing it is finding the "original" stdlib rather than the fresh clone, since AFAICT there's no reference at all to the original source dir)

New changeset f16f93e by Christian Heimes in branch 'main':
bpo-45582: framework build: modPath must not be const (GH-29944)
f16f93e

New changeset b7ef27b by Steve Dower in branch 'main':
bpo-45582: Ensure PYTHONHOME still overrides detected build prefixes (GH-29948)
b7ef27b

New changeset 06c4ae8 by Christian Heimes in branch 'main':
bpo-45582: Fix framework path and bootstrap build (GH-29954)
06c4ae8

New changeset b0b3086 by Steve Dower in branch 'main':
bpo-45582: Write empty pybuilddir.txt on Windows to allow relocatable build directories (GH-29979)
b0b3086

New changeset 3f398a7 by neonene in branch 'main':
bpo-45582: Fix test_embed failure during a PGO build on Windows (GH-30014)
3f398a7

New changeset 55a5e17 by Christian Heimes in branch 'main':
bpo-45582: Don't fail if ENV_PATH is None in getpath.py (GH-31699)
55a5e17

gh-89745: Avoid exact match when comparing program_name in test_embed…

pythongh-89745: Avoid exact match when comparing program_name in test…

gh-89745: Avoid exact match when comparing program_name in test_embed…

Revert 30935 defer bpo45162 to 312 (#94285)

pythongh-89745: Remove commented code in getpath.c

gh-89745: Remove commented code in getpath.c (#108307)

pythongh-89745: Remove test_embed.test_init_read_set()

gh-89745: Remove test_embed.test_init_read_set() (#139500)

Navigation Menu

Repository navigation

Rewrite getpath.c in Python #89745

Linked PRs

19 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Rewrite getpath.c in Python #89745

Description

Linked PRs

Activity

zooba commented on Oct 22, 2021

zooba commented on Oct 22, 2021

zooba commented on Oct 29, 2021

zooba commented on Nov 3, 2021

ericsnowcurrently commented on Nov 3, 2021

zooba commented on Nov 3, 2021

zooba commented on Nov 4, 2021

19 remaining items

tiran commented on Dec 6, 2021

zooba commented on Dec 7, 2021

ambv commented on Dec 7, 2021

zooba commented on Dec 8, 2021

zooba commented on Dec 10, 2021

tiran commented on Mar 6, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions