Repository navigation
Python 3.13a4 ships with wrong pyconfig.h? #115582
Description
Bug report
Bug description:
Hello,
I tried to build a C-extension locally with python 3.13a4 and got the error LINK : fatal error LNK1104: cannot open file 'python313t.lib'.
(I got Python 3.13a4 from the python.org installer, tested Windows 32 and 64 bit).
After trying to install the free threaded builds using the new installer option, trying to build using those, and then installing everything fresh, I found that in include/pyconfig.h, there is a line #define Py_GIL_DISABLED 1. This line causes the compiler to look for python313t.lib instead of python313.lib, and commenting it out fixed the build.
It seems to me that the non-freethreaded build is shipping out with the wrong pyconfig.h setup.
This was reported by Pillow here: python-pillow/Pillow#7805
They linked it to #115545, but I think that may be a different issue? The interpreters launch fine from the python.org install, they just don't seem to work for building C-extensions with.
Thank you.
CPython versions tested on:
3.13
Operating systems tested on:
Windows
Activity
hugovk commentedon Feb 17, 2024
I'm guessing these might both have the same root cause, but cc @zooba
zooba commentedon Feb 19, 2024
Yeah, entirely possible that things have gotten mixed up. I already have #115545 which is going to be the same root cause.
zooba commentedon Feb 19, 2024
So fundamentally the problem is that I tried to share the
Includedirectory between both free-threaded and normal builds, but that obviously doesn't work.I'm most tempted to just drop the free-threaded binaries from the installer and only release them as a zip (essentially, the Nuget package, but a bit more discoverable).
The alternatives are to:
Includedirectory so it's not shared, which requires some contortions insysconfigand for anyone doing cross-compilationPy_DISABLE_GILthemselvesThe only one of those I kind of like is 3, but I suspect we'll run into more and more issues with trying to keep them together, and just splitting it out into a totally separate package is going to be safest.
I'm thinking out loud right now - going to sleep on it before making a decision, so if anyone has thoughts, please speak up!
encukou commentedon Feb 20, 2024
For reference/prior art: Fedora uses the same include directory for 64-bit and 32-bit builds.
Python has all the configuration in a single header file -- others use
#ifdefwith values defined inpyconfig.h. So, there ispyconfig-64.handpyconfig-32.h, and the shared one includes one of them:Ubuntu takes the idea much further -- for all architectures.
pyconfig.h from an Ubuntu container
Of course, free-threaded builds depend on
Py_DISABLE_GILbeing defined rather than on the target architecture, so the situation is different. But I don't think it's a bad idea to make all the config settings build-time (not configure-time) and base them on#ifdef Py_DISABLE_GIL. Let's make option 3 safe for the whole multi-year experiment.If the the nogil crowd agrees with the constraint, of course.
zooba commentedon Feb 20, 2024
Yeah, Windows also uses the same Include directory (including pyconfig.h) for all platforms. This is the first CPython build option other than debug (
_d.exe) that needs to be matched by an extension build.I believe everything in the headers is already based on
#ifdef Py_GIL_DISABLED(my typo above), so option 3 is safe. The issue is that the build backends need to know when to define it, and if we start telling them to do it then we may run into challenges later (e.g. if it becomes the only option, do we fail builds without it?).And defining it in your extension doesn't actually disable the GIL, it just says that your extension requires a build of Python that has it defined, which very much implies that you should be able to get it from somewhere. We have no precedent of a
CFLAGSsetting on Windows, so perhaps that needs to be added? The static config file will help, though that may not support distinguishing between different runtimes in the same directory either, which means we'll need to just have totally separate installs.The simplest immediate fix would be to make sure that defining
Py_GIL_DISABLEDexternally works, and then ship headers that don't define it (currently it would be undef'd on Windows, and I'm not sure about a generated pyconfig.h, but probably both need changes to be consistent). Probably for experimentation, that's fine, and maybe backends will choose to detect that they're running in/targeting free-threaded and define it themselves, even if we later label it experimental and say to stop doing it?encukou commentedon Feb 21, 2024
The simplest immediate fix does sound fine for experimentation. AFAIK, in 3.13, building extensions for free-threaded builds won't be a smooth experience in general; it's OK to just make it possible.
I can see
Py_GIL_DISABLEDbecoming no-op in the future, but I don't think it needs to break builds when free-threaded becomes the only option. Same as withPY_SSIZE_T_CLEAN: you can still define it, but it doesn't do anything.zooba commentedon Feb 21, 2024
So then the question we have is whether our header files should be agnostic towards
Py_GIL_DISABLEDand always require the builder to specify it? Right now the config header has it in there to explicitly match how Python was built, but this doesn't allow for sharing the headers between two different builds.Alternatively, if
Py_GIL_DISABLEDis specified (either 0 or 1) before including us, should it always override what is in the config header? Sopyconfig.hlooks like this:Or do we just allow "upgrading" (we won't undef it, so if you def it then you get it).
Starbuck5 commentedon Feb 23, 2024
Maybe this wouldn't be the simplest in the short term, but it seems like totally separate install folders (mentioned by zooba) would be a good option.
(Artists rendition of what this would look like in directories)
It would be nicely symmetrical with the current permutations possible to be picked up by the
pylauncher (32/64 bit), which currently have separate install directories.It also allows for separate
pips and separatesite-packagesdirectories so people can easily coexist Python 3.13 installations like 32/64 bit installations can coexist today.And this saves any complexity needed to update the build backends to set
Py_GIL_DISABLED, theoretically.encukou commentedon Feb 23, 2024
I lean toward only allowing “upgrading”. When the experiment is done, “downgrading” will stop working; we probably don't want
-DPy_GIL_DISABLED=0to be the mechanism for enabling the GIL.zooba commentedon Feb 23, 2024
There are two main concerns with this:
I'm not sure what's needed for the make build, but I can easily remove the undef from the default Windows headers.
pythongh-115582: Make default PC/pyconfig.h work for free-threaded bu…
gh-115582: Make default PC/pyconfig.h work for free-threaded builds w…
zooba commentedon Feb 29, 2024
The fix to our release build will ensure that the non-freethreaded header is shipped next time, and #115850 will allow people to override the variable if they want to build for freethreaded. That's the best we can do right now, and it at least unblocks the use of 3.13.
Once build backends start figuring out how they want to be doing freethreaded builds we can look at ways to make it smoother to select the right setting. Or if having the builds install together becomes more of a pain than the length of the downloads list, we might switch it to its own parallel install and then the default header will be fine.
pythongh-115582: Make default PC/pyconfig.h work for free-threaded bu…
pythongh-115582: Make default PC/pyconfig.h work for free-threaded bu…
pythongh-115582: Make default PC/pyconfig.h work for free-threaded bu…
pythongh-115582: Make default PC/pyconfig.h work for free-threaded bu…