Repository navigation
Hosting jit_stencils.h #115869
Description
Activity
A couple of naive questions:
-
Is the part of LLVM needed to produce jit_stencils.h small enough to consider extracting it somehow into our repository? And is the algorithm to do so stable enough to not necessarily be a maintenance burden?
-
What is the risk of the status quo? If a change to LLVM broke our usage of it, would that necessarily be considered a breakage of LLVM itself? Do we only depend on documented (and hopefully tested) behavior?
-
Should all platforms be in one file (with each platform guarded by #ifdefs), or many files?
This one is easy enough for me to answer right now: many files. If we have to download all the platforms every time, it may as well just be checked into the main repo.
The advantage here is space saved by not including every platform in the main repo (and then not having to decide which ones we do include). If that's not an interesting saving, then this isn't a question about hosting, it's a question about caching the code.
(And just for context, my sensitivity to time taken by a new process that runs in every clean CI build is about 15 seconds. If it takes more than that, I want a workaround that takes less than that. And my sensitivity for local development is basically 0 more installers to run - time matters less because there'll be caching.)
I'd be interested to understand folks' goals for this and any additional rationale. Are there specific pain points we hope to resolve by eliminating the dependency?
Reacted by Brandt Bucher, Erlend E. Aasland, Diego Russo and Miro HrončokWe don't like having build-time dependencies that can fail due to networking issues or installation issues. Virtually all our network access is only to github.com (the exception being
apt installon Linux, which is the most flaky part of that build), and is only accessing our own repositories (where we implicitly trust the potential contributors).Docs builds are also a minor exception for now, but those are being separated out in official builds so that our supply chain is as clean and tight as we can make it.
So basically, the goal is build reliability, and our way of achieving that is to have all of our build-time dependencies somewhere under github.com/python.
A secondary goal is build time, which is why we have checked in generated files and regen them when their sources change. This is primarily for local dev builds (CI has already gotten way out of hand, I don't think we'll ever get PR builds down to a reasonably fast check anymore, but that used to be a goal). We also don't assume that our contributors have the ability/desire to install additional apps beyond their system compilers, or that they have sufficient internet capacity for anything non-essential or large (besides system compilers).
So for the sake of contribution, we don't want contributors to have to locate/download/install anything outside of their main compiler unless it's scripted as a normal part of build and highly reliable.1
Footnotes
-
Or at least "as reliable as the rest of our dependencies", which more or less means if GitHub is down, builds can fail, but we don't want that list getting longer - not even PyPI. ↩
-
I'd be interested to understand folks' goals for this and any additional rationale. Are there specific pain points we hope to resolve by eliminating the dependency?
Usually Linux distributions only include one LLVM version, like LLVM 17 and clang 17 (version used by Fedora 39). Before, the Python JIT compiler required clang 16 and so it didn't work. Now it requires clang 18 and so it still doesn't work.
If Python source code (ex: in the Git repository) contains code generated by LLVM, Python doesn't have to attempt to use the same LLVM version than the one used by Debian (stable / old-stable), Ubuntu (latest / LTS), Fedora (Rawhide / stable), etc. Spoiler: there is no single LLVM version available on all Linux distributions if you consider all flavors (especially development version vs stable version).
Reacted by Steve DowerSome distros actually do provide multiple LLVM versions, but those tend to be more "advanced" distros. Even there, llvm is generally a fairly hefty burden to install. Especially if it's the only software you have that uses llvm, because your system uses a GCC toolchain. In contrast, cpython is an extremely fundamental package that is used extensively by the system stack.
(llvm is currently needed by... okay, well, I do need llvm 17 for a) mesa and b) gnome gjs / cinnamon cjs, which use mozilla spidermonkey. I don't need any other version of llvm, and I wouldn't need either one of those either if I was running a server system.)
The real kicker is that llvm depends on cpython. If cpython also depends on llvm, then which one do you build first? Answer: you have to build cpython twice, once without the JIT and once with the JIT. Dependency cycles are dreary and depressing to deal with, and may not be able to be fully automated at all. They are best avoided if it is possible to do, and every package that has to be added into the bootstrap set for extra-special handling is an extra burden.
Hosting the stencils just like any other generated code cpython uses, would allow sidestepping this worry. No need to pull llvm into the bootstrap set or add special cases to build it twice.
FWIW, given my experience as a Fedora maintainer: for the system Python package, Fedora will want to avoid pre-generated files (checked in or downloaded), which also means trying rather hard to have a working system-packaged version of LLVM. (You can ping @hroncok if you want details/confirmation.)
Reacted by Savannah OstrowskiReacted by Miro HrončokWe also have multiple llvm versions available in Fedora.
If we wrote all the stencils by hand and had them checked in, would you insist on recreating them?
What if we generated them and then validated them by hand, and then checked them in. Would you recreate them and revalidate before accepting your regeneration?
At what point do you decide to trust the sources provided by the project vs. trusting your own generated/unvendored ones more?
At what point do you decide to trust the sources provided by the project vs. trusting your own generated/unvendored ones more?
At a point where it's more of a binary blob than "source". I understand that line is fuzzy.
Reacted by Steve DowerThat's a very reasonable line, I don't see how there's any way around it for this case (assuming a initialised C array is more of a binary blob, which I'd sure say it is).
Though in those circumstances, having to build CPython without the JIT in order to bootstrap clang so that you can rebuild CPython wit the JIT seems entirely reasonable to me. Yes, it's an annoying amount of work, but it happens once at the distro level (whose job is to do the annoying work) rather than happening every time at the user/contributor level.
Reacted by Miro Hrončok and Petr Viktorin2 remaining items
Rather than committing the generated files into a PR author's branch, the stencil-generating CI jobs would instead commit the generated files into the stencils repo, and provide the PR author with a new hash to update the manifest with on their branch.
That sounds almost the same as Git LFS - large files are stored somethere else (usually some S3 instance), then git tracks only file hashes. But unlike a separated repo, git clone could fetch files stored in LFS without any changes to CI process (git-lfs plugin is usually already installed to CI runners).
Reacted by Michael DroettboomAfter thinking more about, I don't think a manifest file is required. Here is a rough idea:
-
make the path to the stencil file depend on the hash of the inputs and the platform triple
-
use something similar to
Tools/jit/_targets.py_compute_digest()to generate a directory or file path, for example:
jit_stencils/3.14/x86_64-pc-linux-gnu/ba5e6e3182fa318dd7fdab91fa5d5600c1f9068798c1baa52e779fdcacfc0f7b.h-
the
jit_stencils/folder is just another git repo stored ongithub.com/python/..., similar to other external deps -
we can have a CI tool that detects when stencils are missing, generate them and commit them to the
jit_stencilsrepo. -
for end users, we can also provide a tool that works if you have a network connection, populating
jit_stencils/with only the stencil file(s) you need to build the source version you have. I'm not familiar with it but maybe a git sparse checkout would work. -
for cpython devs, we could have some tooling to make life easier, something like
pull-stencils.pycommit-stencils.py. It might be best to let only the CI system commit to the stencils repo. If you have LLVM, you could just generate updated stencils like you currently do and not commit them.
-
So, if we wanted to remove the LLVM requirement for debug builds by just re-using the release stencils, we would need to address these issues. The simpler, but more wasteful option would just be to generate separate debug stencils for every platform.
Maybe another reasonable answer to this is "you need LLVM installed to create a debug build", since that is somewhat "advanced" usage already.
If LLVM is a requirement for the debug build, all the benefits of not needing it would go away for Fedora, unless we only build JIT for the "optimized" build.
If LLVM is a requirement for the debug build, all the benefits of not needing it would go away for Fedora, unless we only build JIT for the "optimized" build.
I thought Fedora would be regenerating the stencils anyway (#115869 (comment))?
Yes, we would. But for example when building on older Fedoras, we now disable JIT as we only have the latest LLVM on the latest Fedoras. I'm trying to chnage that, but e.g. in RHEL this will be hard.
By having the ability to regenerate, but not a requirement, we could benefit from this as well (but not when the benefit is reduced to only one type of build).
Reacted by Michael Droettboom and Erlend E. Aasland- added 5 commits that reference this issue
on Jul 31, 2026
While this is probably desirable, I'm not quite sure if it's feasible. With that said, several people (@vstinner at the sprint and @zooba during PR review) both expressed a desire to remove the LLVM build-time dependency for JIT builds. Let's have that conversation here.
Background
When building CPython with the JIT enabled,
LLVM 16LLVM 18 is used to compileTools/jit/template.cmany times, and process the resulting object files into a file calledjit_stencils.hin the build directory.A useful analogy
Because this file depends on
Python.h(and thuspyconfig.hand many build-specific configuration options, including things like_DEBUG/NDEBUG/Py_DEBUG/etc.) and contains binary code, it is probably most useful to think ofjit_stencils.has a binary extension module.If we could build, host, and manage compiled versions of, say,
itertoolsmodule.csomewhere and have it work correctly for those who need it, then such a scheme would probably work forjit_stencils.h.Open questions
#ifdefs), or many files?Linked PRs
jit_stencils.hreproducible #127166