Building and linking C extensions in a post-distutils world

@vstinner

With the removal of distutils from 3.12 alphas, I've recently taken a new look at the use of distutils within https://mesonbuild.com in the hopes that we were finally unblocked and could migrate to sysconfig.

install schemes turn out to finally be viable since 3.10.3, when the deb_system scheme patch on Debian operating systems for distutils was patched into sysconfig. This hard blocker is gone, thank G-d.
linking to libpython itself may or may not be an issue, originally I thought it was, but now I think it may not be.

How can a C-extension supporting PEP 517 build backend know the best way to compile and link?

python <3.12

Meson has some code: https://github.com/mesonbuild/meson/blob/master/mesonbuild/modules/python.py#L338-L407

Here's the interesting bit:

def links_against_libpython():
    from distutils.core import Distribution, Extension
    cmd = Distribution().get_command_obj('build_ext')
    cmd.ensure_finalized()
    return bool(cmd.get_libraries(Extension('dummy', [])))

python >=3.8

In bpo-36721, bpo-21536 etc the above code changes from returning True to False, on many Unix platforms.

New additional methods suitable for getting C extension arguments are available. Well, mostly suitable.

sysconfig.get_config_var('LIBPYTHON')

this is configured into:

pkg-config --cflags --libs python3
python-config --embed

There's a couple problems with this:

on Cygwin and Android, LIBPYTHON and thus pkg-config is hardcoded to link to libpython, but distutils only does so when Py_ENABLE_SHARED
none of it works on Windows, which since it is built with PCBuild, doesn't have Makefile config vars (and also / consequently? distributes neither of the latter two)
pkg-config may not always be installed, so we want a fallback for that
python-config has tons of CPython build-time junk so we cannot use it

...

It feels uncomfortably like there's still way too much undocumented magic here.

@vstinner, I assume the Cygwin/Android inconsistency is probably a configure.ac bug that should behave the same way distutils does/did? I can provide a patch to fix it but would like confirmation of which side to fix.

@FFY00, I think in the long term, sysconfig should expose a function that returns cflags / libs required to build an extension (and works on Windows without _generate_posix_vars, and doesn't include lots of -O3 -pipe -fstack-protector-strong -fdiagnostics-color=always and more -- i.e. works like pkg-config, not like python-config). Even without the question of "whether to link to libpython", there's a 140-line class for figuring out what the name of the library is, which directory to find it in, and what the include directory is too. Of course, this is specific to the case where pkg-config is not available, and most of it is for the Windows case.

Linked PRs

pythongh-99942: fix regression in python.pc that broke linking on and…

pythongh-99942: correct the pkg-config/python-config flags for cygwin…

The Windows case really isn't complicated because the linking is all set up to isolate the two sides of the ABI. The only thing that's going to bleed through is C runtime state when you mix a debug build with a release build (which use two different CRT instances). Virtually nobody actually has a debug build of CPython, and those who do will know because the filenames are different, so there's nothing really to pick up from the interpreter.

Now, the complicated bit is finding the compiler, in effect, this bit:

cpython/Lib/distutils/_msvccompiler.py

Lines 66 to 80 in 7571764

    
           root = os.environ.get("ProgramFiles(x86)") or os.environ.get("ProgramFiles") 
        
           if not root: 
        
               return None, None 
        
           try: 
        
               path = subprocess.check_output([ 
        
                   os.path.join(root, "Microsoft Visual Studio", "Installer", "vswhere.exe"), 
        
                   "-latest", 
        
                   "-prerelease", 
        
                   "-requires", "Microsoft.VisualStudio.Component.VC.Tools.x86.x64", 
        
                   "-property", "installationPath", 
        
                   "-products", "*", 
        
               ], encoding="mbcs", errors="strict").strip() 
        
           except (subprocess.CalledProcessError, OSError, UnicodeDecodeError): 
        
               return None, None

This is assuming you want to compile with MSVC. If you want to compile an extension with a different compiler (many of which will work, since the ABI is well defined), you'll need different code. Again, this is best left to the tool figuring out how to run the build - one of the main reasons for dropping distutils is so that compiler options don't get trapped by the CPython release cycle.

Otherwise, the location of Python's header files (typically {prefix}\Include) and import libraries (typically {prefix}\libs) are all you need. They should both be in sysconfig somewhere already, but since you can't use that when cross-compiling, we either need a new data file containing the paths (which I'm in favour of, but it wasn't popular last time we discussed it) or to just stick with the assumptions tools use today.

They should both be in sysconfig somewhere already, but since you can't use that when cross-compiling, we either need a new data file containing the paths (which I'm in favour of, but it wasn't popular last time we discussed it) or to just stick with the assumptions tools use today.

There's also the undocumented $_PYTHON_SYSCONFIGDATA_NAME environment variable, which is unfortunately only available on posix. But POSIX systems do often make the assumption that they can convince e.g. distutils to cross compile by setting that environment variable before running any build tool (pip, python -m build, python setup.py ...).

gh-99942: correct the pkg-config/python-config flags for cygwin/android

gh-99942: python.pc on android/cygwin should link to libpython per co…

Merge branch 'main' into lockfreecp

pythongh-99942: python.pc on android/cygwin should link to libpython …

@sethmlarson

I welcome any technical critique or discussion this may prompt. The goal is clarity and resilience.

My technical critique is that this report appears to be AI generated. It contains numerous hallucinations which can be trivially disproven. It is also 100% unrelated to the current ticket, which has nothing to do with gcc (15 or otherwise), nor with C standard versions.

Please avoid posting distractingly incorrect info.

@sethmlarson we have another case of https://sethmlarson.dev/slop-security-reports I think

@eli-schwartz

@eli-schwartz Indeed, we received the same report to PSRT and have so far disregarded it entirely as being AI-generated nonsense.

Thanks. For the record, pandas / numpy / cython / matplotlib were also targeted (I linked them to your article and the reports were closed).

Navigation Menu

Repository navigation

Building and linking C extensions in a post-distutils world #99942

How can a C-extension supporting PEP 517 build backend know the best way to compile and link?

python <3.12

python >=3.8

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	root = os.environ.get("ProgramFiles(x86)") or os.environ.get("ProgramFiles")
	if not root:
	return None, None

	try:
	path = subprocess.check_output([
	os.path.join(root, "Microsoft Visual Studio", "Installer", "vswhere.exe"),
	"-latest",
	"-prerelease",
	"-requires", "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
	"-property", "installationPath",
	"-products", "*",
	], encoding="mbcs", errors="strict").strip()
	except (subprocess.CalledProcessError, OSError, UnicodeDecodeError):
	return None, None

Building and linking C extensions in a post-distutils world #99942

Description

How can a C-extension supporting PEP 517 build backend know the best way to compile and link?

python <3.12

python >=3.8

Linked PRs

Activity

zooba commented on Jan 12, 2023

eli-schwartz commented on Jan 12, 2023

BryteLite commented on May 17, 2025

eli-schwartz commented on May 18, 2025

sethmlarson commented on May 19, 2025

eli-schwartz commented on May 19, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions