Allow stable ABI extensions to include multiarch tuples in the filename

Feature or enhancement

Proposal:

Currently C extensions can include the multarch tuple of the architecture the support in their filename, e.g. zlib.cpython-314-x86_64-linux-gnu.so. This extensions for multiple different architectures to be installed on the same filesystem, next to each other, without clashing. This is used by Debian to allow packages for multiple architectures to be co-installed simultaneously.

The same should apply to stable ABI C extensions, but this was missed when the stable ABI was implemented.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

This will not be so easy to introduce. Currently build systems/backends get the cpython-314-x86_64-linux-gnu.so extension name from a value like sysconfig.get_config_var('EXT_SUFFIX') while they hardcode abi3.so because it's the only option.

I assume you want all build tools to do the new thing and produce abi3-x86_64-linux-gnu.so always instead of abi3.so? There are a lot of build tools to update then.

This will not be so easy to introduce.

The plan was: Start supporting this as an alternative, until we reach a point where we don't have to support older versions that don't support it any more. Then it can become the default.

We have supported this in Debian since 3.13, and were hoping to get support upstream into 3.14.0, but moved a little too late for that :(

I assume you want all build tools to do the new thing and produce abi3-x86_64-linux-gnu.so always instead of abi3.so? There are a lot of build tools to update then.

If build tools are asking the interpreter what to prefer, then they should do the right thing. Which is to not build anything with this, for now.

We handle renaming extensions to include this path in our package tooling. Nothing else really needs it, at this point.

We handle renaming extensions to include this path in our package tooling. Nothing else really needs it, at this point.

Ah great, thanks for the context. No concerns from me then.

This will not be so easy to introduce. Currently build systems/backends get the cpython-314-x86_64-linux-gnu.so extension name from a value like sysconfig.get_config_var('EXT_SUFFIX') while they hardcode abi3.so because it's the only option.

This is of course not true. Meson uses importlib.machinery.EXTENSION_SUFFIXES[1] on the theory that element 0 is the the EXT_SUFFIX (because it is most specific and thus most preferred) while element 1 is the second-most preferred which ought to be the stable ABI suffix.

Inserting another abi3 suffix so that it looks like

['.cpython-314-x86_64-linux-gnu.so', '.abi3-x86_64-linux-gnu.so', '.abi3.so', '.so']

is:

intrinsically necessary (to include in the list) to get the importlib machinery to work correctly
intrinsically necessary (to use that specific order) because an architecture specific abi3 extension needs to take priority over an abi3 extension of unknown compatibility with the current interpreter
automatically going to work for meson

The plan was: Start supporting this as an alternative, until we reach a point where we don't have to support older versions that don't support it any more. Then it can become the default.

This is an unnecessary complication since both names have to be supported for compatibility, so the default can immediately become the "better" one. Tools that don't know how to use the better one don't have to, because for compatibility they both work! Tools that do.know how to use the better one... should do so?

Older python versions don't matter, because the official word from on high is that if you compile an abi3 extension on python 3.13, you're not allowed to use it with a python 3.12 interpreter. Rationale is that the minimum limited API setting being lower than the current headers version, "doesn't actually guarantee you're avoiding stable ABIs from the most recent version", and shall not be supported, therefore it's useless to say you'd like to be able to do it.

Meson uses importlib.machinery.EXTENSION_SUFFIXES[1]

That does sound a little fragile. It select take the first one with .abi in it, and be a little more resilient to change. setuptools does that:
https://github.com/pypa/setuptools/blob/d198e86f57231e83de87975c5c82bc40c196da79/setuptools/command/build_ext.py#L79-L86

intrinsically necessary (to use that specific order) because an architecture specific abi3 extension needs to take priority over an abi3 extension of unknown compatibility with the current interpreter

I don't think we want that behaviour until we can assume that any interpreter that is able to get this extension in a wheel can import this filename.

This is an unnecessary complication since both names have to be supported for compatibility, so the default can immediately become the "better" one.

Same issue. If you're building any binaries that other people are going to use with other interpreters, you need to ensure that they can import them.

Older python versions don't matter, because the official word from on high is that if you compile an abi3 extension on python 3.13, you're not allowed to use it with a python 3.12 interpreter.

Meson uses importlib.machinery.EXTENSION_SUFFIXES[1]

That does sound a little fragile.

It is! It doesn't work in cross compilation! Changing this would be lovely, but playing games with modifying the heuristics about how to parse details of a runtime import of the importlib machinery are very uninteresting.

In fact it's already been changed -- if you use a PEP 739 build_details.json, the abi.stable_abi_suffix is mandatory for CPython (and any other python implementations that have a stable ABI) and is/can be used in preference to guessing anything.

One would presume therefore that that build_details.json would list the intended ABI suffix (the one that is most specific to the current interpreter, i.e. the one that lists the CPU arch).

Older python versions don't matter, because the official word from on high is that if you compile an abi3 extension on python 3.13, you're not allowed to use it with a python 3.12 interpreter.

IIRC that's a new thing, since we made this proposal. And there's nothing to declare the compatible versions in the data...

Also, right now if you compile on a Debian Python 3.13, the multiarch extension is there...

@zooba

And there's nothing to declare the compatible versions in the data...

@zooba specifically wrote PEP 809 to address this very point -- the stable ABI would permit neither additions nor removals except when revbumping the ABI tag (abi4, abi2026, depending).

Also, right now if you compile on a Debian Python 3.13, the multiarch extension is there...

If the idea is to add this change to the official unpatched interpreter, this problem doesn't exist.

This is of course not true. Meson uses importlib.machinery.EXTENSION_SUFFIXES[1]

Thanks for the correction. That makes it more fragile overall; meson-python does do what I described. sysconfig.get_config_var('EXT_SUFFIX') is only a single string value - hard to modify in a backwards compatible way.

Older python versions don't matter, because the official word from on high is that if you compile an abi3 extension on python 3.13, you're not allowed to use it with a python 3.12 interpreter.

I don't think I've ever seen that in writing. Using a 3.13 interpreter with -DPy_LIMITED_API=<3.12 hex version> in compile flags is common and should work. I'm guessing you meant -DPy_LIMITED_API=<3.13 hex version>?

I don't think I've ever seen that in writing. Using a 3.13 interpreter with -DPy_LIMITED_API=<3.12 hex version> in compile flags is common and should work. I'm guessing you meant -DPy_LIMITED_API=<3.13 hex version>?

I'm saying that I am pretty sure I've seen CPython core developers say all the time that you may use whatever Py_LIMITED_API you like, and it has the effect of fixing a lower bound, but it doesn't set an upper bound and it doesn't guarantee that the CPython headers won't silently replace functions with macros that use newer ABI than what you expect.

Therefore the only guarantee we have as users is that if you compile an extension using a Py_LIMITED_API specifying version X, it will:

error out if CPython is X-1
for CPython >= X, produces a compiled extension that requires at least $CPYTHON_VERSION and pays zero attention to Py_LIMITED_API

Or in simpler terms, specifying a particular version for Py_LIMITED_API does nothing except provide partial linting for cases where your source code is incompatible with the claim that Py_LIMITED_API is making. And lint tools do not an ABI make.

It is unfortunate that no available mechanisms are made use of in the CPython codebase for offering better compile time or runtime guarantees (runtime guarantees for this exact scenario were pioneered decades ago and heavily used by almost every Linux binary on the planet because glibc injects those guarantees whether you know it does or not, every time a compiled binary uses functions from glibc, which is ~100% of them, -+ 0%).

@eli-schwartz

@eli-schwartz: I don't disagree that it's defined that stable ABI doesn't work backords, or that PEP 809 proposes a future that makes this stuff clearer.

But right now, it's very hard to build tooling that respects those rules, because the tools don't have the data. So we have to be conservative in our approach to change, to not publish binaries that break existing systems. Tools like auditwheel are a great help here, but I'm still trying to be careful.

.abi3 should no longer conflict

pythongh-122931: Allow stable abi3 API extensions to include a multia…

This is now completely implemented and can be closed, right? gh-152461 was the last part of this, and that was merged and backported before 3.15.0rc1.

Patch setup.py to name abi3 extensions for the host platform

Fix abi3 extension suffix on cross-compiled py3.15 builds

Little late to the game but importlib.machinery.EXTENSION_SUFFIXES is hardcoded into the interpreter.

There are build systems that cross compile wheels by using the host interpreter and overriding the _PYTHON_SYSCONFIGDATA_* variables to influence the build environment since you can't always run a foreign python on the target (something we've hit before #115382). Since the source of truth is hardcoded into the interpreter and isn't influenced by the SYSCONFIG data, cross compiling wheels for arm on x86_64 will always return:

>>> import importlib.machinery
>>> importlib.machinery.EXTENSION_SUFFIXES
['.cpython-315-x86_64-linux-gnu.so', '.abi3-x86_64-linux-gnu.so', '.abi3.so', '.abi3t-x86_64-linux-gnu.so', '.abi3t.so', '.so']

This currently affects Buildroot + setuptools pypa/setuptools#5351

As previously discussed above, EXTENSION_SUFFIXES is already badly broken for cross in python 3.10 - 3.14 and this PR doesn't materially change that. I don't think you want cross compiling from Linux arm to windows amd64, for example, to build "abi3.so" windows files.

There is an approved path forward -- build_details.json. Building ever more elaborate hacks to get "same platform different CPU" specialized in order to work isn't a long term solution (although it might solve buildroot's needs, I guess).

@eli-schwartz

@eli-schwartz thaanks for takwthe time to respond! I appreciate it.

I'll review pep 739, I missed it while scrolling through the PR so my apologies. I expect build tools like setuptools have to find their own way to determine when to parse this or take it as an input at build time? Since we're building the target python for deployment into the root file system, we should have this file available.

Buildroot doesn't have any better options at the moment. We only target different cpu architectures for Linux so we'd only cross from say x86_64 to Armv7 or sparc or some other x86_64 target that may have instructions that aren't supported by the host or by emulation (plus the emulation overhead for builds would be extremely high and difficult to work in).

I'm interested in being involved with any discussions around solutions. Being able to natively cross compile c, go, rust, etc is extremely useful and would like to keep Python in this list. For us I think we can certainly patch our issue in setuptools for this specific case but we would need to solve this for all supported build backends of which setuptools is one.

I guess the other option is that we could patch all python libraries we have to disable the limited ABI which is useful for building publicly published wheels but for a fixed distribution may not make sense but since the sources are likely only tested with those builds, I imagine there may be some subsequent issues.

Sorry for all the words, thinking out loud, and for adding noise to this issue.

I'll review pep 739, I missed it while scrolling through the PR so my apologies. I expect build tools like setuptools have to find their own way to determine when to parse this or take it as an input at build time? Since we're building the target python for deployment into the root file system, we should have this file available.

Meson's implementation landed in mesonbuild/meson#14657 for comparison purposes.

I guess the other option is that we could patch all python libraries we have to disable the limited ABI which is useful for building publicly published wheels but for a fixed distribution may not make sense but since the sources are likely only tested with those builds, I imagine there may be some subsequent issues.

For meson-using packages, this is as simple as passing --config-settings="setup-args=-Dpython.allow_limited_api=false"

As previously discussed above, EXTENSION_SUFFIXES is already badly broken for cross in python 3.10 - 3.14 and this PR doesn't materially change that. I don't think you want cross compiling from Linux arm to windows amd64, for example, to build "abi3.so" windows files.

I guess the issue is mostly just that setuptools is using EXTENSION_SUFFIXES when it should be using a pure sysconfig approach instead? I went ahead and implemented a pure sysconfig based approach in #5352

No, that's a severe regression over the previous version of your PR that breaks it for native builds as well, as soon as cpython makes any changes. I can't think how you concluded that was a good idea based on anything I said, so I'm not sure how to correct the information gap either.

I'm not strongly motivated to worry about what setuptools does at this time -- my experience is that setuptools is always trouble, it cannot even reliably compile multiple extensions in a single setup.py without scribbling all over intermediate objects with junk data -- but I definitely don't want to be cited as a reason for incorrect data models, so please forget I said anything about setuptools or the stable ABI.

No, that's a severe regression over the previous version of your PR that breaks it for native builds as well, as soon as cpython makes any changes.

I mean, it would depend on what those changes are right? That could be the case with all sorts of changes AFAIU.

I can't think how you concluded that was a good idea based on anything I said, so I'm not sure how to correct the information gap either.

I concluded that since you stated EXTENSION_SUFFIXES is broken for cross that setuptools should instead use sysconfig instead which is the typical setuptools conventions for determining target specific variables and such from my understanding as it should be compatible with both cross and native builds. It's not like sysconfig isn't already heavily used by setuptools already.

I'm not strongly motivated to worry about what setuptools does at this time -- my experience is that setuptools is always trouble, it cannot even reliably compile multiple extensions in a single setup.py without scribbling all over intermediate objects with junk data -- but I definitely don't want to be cited as a reason for incorrect data models, so please forget I said anything about setuptools or the stable ABI.

Yeah, I'm aware setuptools has lots of issues.

I'm just confused how my change to setuptools is an incorrect approach, if EXTENSION_SUFFIXES is only for the running host interpreter and not the target interpreter it shouldn't be used by setuptools in the first place right? That EXTENSION_SUFFIXES worked previously was essentially by accident AFAIU.

#136267 (comment)

The json may not be as reliable as we'd like for cross builds

Navigation Menu

Repository navigation

Allow stable ABI extensions to include multiarch tuples in the filename #122931

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

7 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Allow stable ABI extensions to include multiarch tuples in the filename #122931

Description

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Activity

rgommers commented on Dec 3, 2025

stefanor commented on Dec 3, 2025

rgommers commented on Dec 3, 2025

eli-schwartz commented on Dec 4, 2025

stefanor commented on Dec 4, 2025

eli-schwartz commented on Dec 4, 2025

eli-schwartz commented on Dec 4, 2025

stefanor commented on Dec 4, 2025

eli-schwartz commented on Dec 4, 2025

merwok commented on Dec 4, 2025

rgommers commented on Dec 4, 2025

eli-schwartz commented on Dec 4, 2025

stefanor commented on Dec 4, 2025

7 remaining items

rgommers commented on Aug 25, 2026

vfazio commented on Oct 16, 2026

eli-schwartz commented on Oct 16, 2026

vfazio commented on Oct 16, 2026

eli-schwartz commented on Oct 16, 2026

jameshilliard commented on Oct 16, 2026

eli-schwartz commented on Oct 16, 2026

jameshilliard commented on Oct 16, 2026

jameshilliard commented on Oct 16, 2026

vfazio commented on Oct 17, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions