Repository navigation
3.15.0a1 rewrites sysconfig-saved flags on make install #140454
Description
Bug report
Bug description:
In Fedora, we have built the Python 3.15.0a1 release as we normally do. It results in broken sysconfig flags that prevent building 3rd-party extension modules.
There are some RPM-specific CFLAGS and LDFLAGS that we use to build Python, but we don't want them exposed to our users via sysconfig because it would break their build. One of such flags is -specs=/usr/lib/rpm/redhat/redhat-package-notes, but there are others. When such flags are used outside of RPM build, it fails:
$ git clone https://github.com/fedora-python/crypt_r.git && cd crypt_r
$ tox -e py315
...
building '_crypt_r' extension
creating build/temp.linux-x86_64-cpython-315/src
gcc -Werror=incompatible-pointer-types -fPIC -I/home/churchyard/.cache/uv/builds-v0/.tmpPUr6Gz/include -I/usr/include/python3.15 -c src/_crypt_r.c -o build/temp.linux-x86_64-cpython-315/src/_crypt_r.o
gcc -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1
-Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -Werror=incompatible-pointer-types build/temp.linux-x86_64-cpython-315/src/_crypt_r.o -L/usr/lib64 -lcrypt -o
build/lib.linux-x86_64-cpython-315/_crypt_r.cpython-315-x86_64-linux-gnu.so
...
gcc: fatal error: environment variable ‘RPM_ARCH’ not defined
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1
That's why we set those flags in the CFLAGS_NODIST, LDFLAGS_NODIST only. However:
$ python3.15
...
>>> for k, v in sysconfig.get_config_vars().items():
... if '-specs=/usr/lib/rpm/redhat/redhat-package-notes' in str(v):
... print(k, v)
...
BLDSHARED gcc -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes
CONFIGURE_LDFLAGS_NODIST -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -g -fno-semantic-interposition -flto -fuse-linker-plugin -ffat-lto-objects -g
LDCXXSHARED g++ -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes
LDFLAGS -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes
LDSHARED gcc -shared -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes
PY_CORE_LDFLAGS -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -g -fno-semantic-interposition -flto -fuse-linker-plugin -ffat-lto-objects -g
PY_LDFLAGS -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes
PY_LDFLAGS_NODIST -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -g -fno-semantic-interposition -flto -fuse-linker-plugin -ffat-lto-objects -g
PY_LDFLAGS_NOLTO -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -fno-lto
Compare with:
$ python3.14
...
>>> for k, v in sysconfig.get_config_vars().items():
... if '-specs=/usr/lib/rpm/redhat/redhat-package-notes' in str(v):
... print(k, v)
...
CONFIGURE_LDFLAGS_NODIST -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -g -fno-semantic-interposition -flto -fuse-linker-plugin -ffat-lto-objects -g
PY_CORE_LDFLAGS -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -g -fno-semantic-interposition -flto -fuse-linker-plugin -ffat-lto-objects -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -g
PY_LDFLAGS_NODIST -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -g -fno-semantic-interposition -flto -fuse-linker-plugin -ffat-lto-objects -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -g
PY_LDFLAGS_NOLTO -fno-lto -Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -g
I was not yet able to reproduce this behavior outside of our RPM build of Python, which makes bisecting this harder. We set the flags like this:
+ CFLAGS='-fcf-protection -fexceptions '
+ export 'CFLAGS_NODIST= -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -D_GNU_SOURCE -fPIC -fwrapv'
+ CFLAGS_NODIST=' -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -D_GNU_SOURCE -fPIC -fwrapv'
+ export 'CXXFLAGS=-fcf-protection -fexceptions '
+ CXXFLAGS='-fcf-protection -fexceptions '
++ pkg-config --cflags-only-I libffi
+ export CPPFLAGS=
+ CPPFLAGS=
+ export 'OPT=-fcf-protection -fexceptions '
+ OPT='-fcf-protection -fexceptions '
++ pkg-config --cflags openssl
+ export 'CFLAGS=-fcf-protection -fexceptions '
+ CFLAGS='-fcf-protection -fexceptions '
++ pkg-config --libs-only-L openssl
+ export 'LDFLAGS= '
+ LDFLAGS=' '
++ pkg-config --libs-only-L openssl
+ export 'LDFLAGS_NODIST=-Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -g '
+ LDFLAGS_NODIST='-Wl,-z,relro -Wl,--as-needed -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes -g '
...
+ /builddir/build/BUILD/python3.15-3.15.0_a1-build/Python-3.15.0a1/configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --runstatedir=/run --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-platlibdir=lib64 --enable-ipv6 --enable-shared --with-computed-gotos=yes --with-dbmliborder=gdbm:ndbm:bdb --with-system-expat --with-system-libmpdec --enable-loadable-sqlite-extensions --with-dtrace --with-lto --with-ssl-default-suites=openssl --without-static-libpython --with-wheel-pkg-dir=/usr/share/python-wheels --with-valgrind --without-ensurepip --enable-experimental-jit=yes-off --enable-optimizations
...
+ /usr/bin/make -O -j32 V=1 VERBOSE=1 EXTRA_CFLAGS=-O3 'CFLAGS_NODIST= -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -D_GNU_SOURCE -fPIC -fwrapv -O3' regen-all PYTHON_FOR_REGEN=python3.15
...
+ /usr/bin/make -O -j32 V=1 VERBOSE=1 EXTRA_CFLAGS=-O3 'CFLAGS_NODIST= -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -D_GNU_SOURCE -fPIC -fwrapv -O3'
Now I'm trying to see if reverting 9e3542a makes a difference.
CPython versions tested on:
3.15
Operating systems tested on:
Linux
Activity
hroncok commentedon Oct 22, 2025
No difference.
hroncok commentedon Oct 22, 2025
The installed
/usr/lib64/python3.15/_sysconfigdata__linux_x86_64-linux-gnu.pyis already busted.hroncok commentedon Oct 22, 2025
The
build/lib.linux-x86_64-3.15/_sysconfigdata__linux_x86_64-linux-gnu.pyis created as expected (e.g. emptyPY_LDFLAGS) and later duringmake installor shortly after that, the files is updated (e.g.PY_LDFLAGSis populated) :/hroncok commentedon Oct 22, 2025
The difefrence between 3.14 and 3.15 is that
Written build/lib.linux-x86_64-3.15/_sysconfigdata__linux_x86_64-linux-gnu.pyhappens inmake installas well. And here we did not bother setting the flags. I still am unable to reproduce this locally somehow.hroncok commentedon Oct 22, 2025
Indeed, when exporting the same set of flags for
make install, the leak does not happen. However, I was not yet able to determine what exactly triggers the re-generation of_sysconfigdata__linux_x86_64-linux-gnu.pyduringmake install.[-]3.15.0a1 leaks NODIST flags to other sysconfig-saved flags[/-][+]3.15.0a1 rewrites sysconfig-saved flags on make install[/+]hroncok commentedon Oct 22, 2025
OK, I isolated the problem to enabling the JIT.
Then, in a fresh shell:
I did not expect
make installto recompile the JIT. It did not do that on 3.14 -- I'll try bisecting that.hroncok commentedon Oct 22, 2025
git bisect says 85c1ef6
The nature of that change makes me believe that's very much the likely cause.
cc @savannahostrowski
hroncok commentedon Oct 22, 2025
Note that in Fedora, we have solved this by ensuring the same CFLAGS/LDFLAGS environment variables are set during
makeandmake install. Yet I believe this might be considered problematic anyway.savannahostrowski commentedon Oct 23, 2025
Okay, so after much spelunking, I think I've found the issue...
In #137265, I introduced a dual-target rule to prevent stale platform-specific stencil files. This exposed an issue with including
pyconfig.hin JIT_DEPS. With the dual-target rule, Make checks whether both stencil files are newer than all dependencies in JIT_DEPS:If
pyconfig.hhas a newer timestamp than either stencil file duringmake install, Make rebuilds the JIT stencils. This triggers rebuilds that ultimately cause_sysconfigdatato regenerate. When_sysconfigdataregenerates during install, it picks up build-only flags likeLDFLAGS_NODISTthat shouldn't be in the installed configuration.I think we can remove
pyconfig.hfromJIT_DEPS. We don't need it there sincePython/jit.calready includesPython.h, which pulls inpyconfig.h.savannahostrowski commentedon Oct 23, 2025
#140491 (comment)
We might need a person with more expertise in Makefiles/our build process to help debug here, since I am unable to repro.
4 remaining items
hroncok commentedon Oct 25, 2025
Thanks, I'll try your test as soon as I get to my computer.
I am reproducing this consistently outside of the RPM build environment, on my x86_64 Fedora 41 workstation. I haven't checked if this also happened in the RPM build for aarch64, will do to ensure this isn't x86_64 specific.
hroncok commentedon Oct 25, 2025
This also happened on aarch64.
hroncok commentedon Oct 25, 2025
Building from https://github.com/savannahostrowski/cpython/tree/jit-make-test-fix does not make a difference for me.
[-]3.15.0a1 rewrites sysconfig-saved flags on make install[/-][+]3.15.0a1 rewrites sysconfig-saved flags on `make install`[/+]hroncok commentedon Oct 25, 2025
@befeleme Could you please check if you are able to reproduce this locally (maybe my instructions omit some important setup I have)?
brandtbucher commentedon Oct 26, 2025
Catching up on this thread. Just thought I'd emphasize that we shouldn't remove
pyconfig.hfrom the JIT dependencies in the build system, since the JIT stencils themselves depend onpyconfig.h. If that file changes, the stencils must be recompiled.befeleme commentedon Oct 30, 2025
I have successfully repeated steps described in #140454 (comment) on my Fedora 41 x86_64 machine and the resulting diff looks exactly the same as posted there. I used CPython's v3.15.0a1 tag for testing.
hroncok commentedon Oct 31, 2025
Running
make installover and over again keeps compiling the JIT.It says so immediately when it starts:
I looked into it -- my configure-generated Makefile has:
Notice, namely, the jit_stencils-x86_64-pc-linux-gnu.h filename in the dependency.
I don't have that file. I only have jit_stencils-x86_64-unknown-linux-gnu.h. The
python3.15 ./Tools/jit/build.pycommand saysx86_64-pc-linux-gnu.If I do
cp -a jit_stencils-x86_64-unknown-linux-gnu.h jit_stencils-x86_64-pc-linux-gnu.h, the JIT is not compiled again.I believe the filename used in the Makefile dependency is from
JIT_STENCILS_H="jit_stencils-$host.h"in configure.ac. The argument forTools/jit/build.pyis${ARCH_TRIPLES:-$host}which expands tox86_64-pc-linux-gnuas well. But it outputs jit_stencils-x86_64-unknown-linux-gnu.h.hroncok commentedon Oct 31, 2025
I believe this is a mismatch with
cpython/Tools/jit/_targets.py
Lines 602 to 603 in e733dc9
pythongh-140454: Generate jit_stencils-*-pc-linux-gnu.h instead of ..…
pythongh-140454: Generate jit_stencils-*-pc-linux-gnu.h instead of ..…
gh-140454: Normalize the JIT stencils filename on Linux to avoid mism…
pythongh-140454: Normalize the JIT stencils filename on Linux to avoi…