Repository navigation
Modify get_externals.py to download binary dependencies from GitHub Releases #137604
Description
Activity
Rather than the GitHub API, is it possible to use the raw URL of the released file? Just for simplicity (better to do more work up-front when it's going to get a PR review and keep things simple at runtime/build time)
Looking at the relative sizes of these releases, one of these things is not like the others. The
opensslbundle is "big" at ~15MB, but LLVM 19 is enormous at 830MB. I think rather than rejiggering everything to support LLVM in thecpython-bin-depsmechanism, we might want to rethink how we're handling LLVM on Windows. I don't have any suggestions for that off the top of my head; I haven't looked into where and how exactly we're currently using it.Reacted by Ned Deily, Steve Dower and Adam TurnerReacted by Savannah OstrowskiTo add to @zware's comment ("one of these things is not like the others"), I believe it is the case that LLVM is only needed to create the JIT stencils at build time. We're still using MSVC to compile cpython, right? If so, LLVM would be the only thing in
cpython-source-depsandcpython-binary-depsthat is only a build tool, i.e. contributes no binaries to the built cpython artifacts (shared or static libraries, etc). We've had discussions in the past (last year's core developer sprint?) about the possibility of checking the JIT stencils into the cpython repo to avoid this kind of build dependency. That has its own drawbacks but, from a distance, I would think it should be something we could handle in CI. But I don't want to necessarily reopen that discussion 😉Reacted by Steve Dower... but as @savannahostrowski points out (and I had forgotten!), there was a PEP about this that was deferred by the Steering Council.
Reacted by Savannah OstrowskiRather than the GitHub API, is it possible to use the raw URL of the released file?
Yep! I was mislead by a post on the internet (😱 ) that this wasn't possible, but looks like it is. So I can remove a lot of the extra code and just use urlretrieve.
I think rather than rejiggering everything to support LLVM in the cpython-bin-deps mechanism, we might want to rethink how we're handling LLVM on Windows.
LLVM is used for the JIT build. As Ned points out above, removing it as a build time dependency was considered but deferred. In the interim, downloading LLVM from GitHub releases seems like a reliable way to source the binaries for CI and local development. While LLVM is much larger than other dependencies, I don't think it is a good idea to fracture how dependencies are downloaded. The more ways we download dependencies, the harder it is to figure out the right way to update them, or fix issues.
I guess, I'll come back to my initial question - Does LLVM have to be in
cpython-bin-deps? I agree with @zware that LLVM is sort of an odd one. I think it's worth pursuing the broader dep redesign convo, but I'd still advocate for doing this the "easy" way for now, especially since we are just using this at build time for stencil generation.Reacted by Emma Smith and Zachary WareIdeally we could use the LLVM installable via the Visual Studio Installer for JIT purposes and just note that that workload needs to be installed along with the otherwise necessary MSVC bits. I don't know what version(s) is/are currently available that way, or if all of the parts the JIT build needs are included. CI is also an open question on that route.
Ideally we could use the LLVM installable via the Visual Studio Installer for JIT purposes and just note that that workload needs to be installed along with the otherwise necessary MSVC bits. I don't know what version(s) is/are currently available that way, or if all of the parts the JIT build needs are included.
The version can't be guaranteed, and I believe right now it's too old but at the next update should be right.
LLVM releases too frequently (almost as frequently as us...), so relying on any "latest available" type system is probably going to be fairly unstable. Visual Studio doesn't let you pin a particular version of LLVM.
Building the JIT stencils and storing those in the deps repository would probably make more sense. A build job that exists solely to do that can use LLVM from whatever source it likes - if that build fails, releases/CI can still use the stored stencils and won't fail.
Caching LLVM instead of the stencils seems to be against the spirit of deferring that PEP (though I don't recall the exact reasoning that was given).
Reacted by Zachary WareHm, building the stencils can change per-CPython commit (in practice it's less often than this, but it's much more often than other bin-deps). It doesn't seem like a natural fit for cpython-bin-deps where the binaries are built once and cached.
Reacted by Savannah OstrowskiThe version can't be guaranteed, and I believe right now it's too old but at the next update should be right.
The VS2022 installer is presenting LLVM 18, which I think is supported?
Hm, building the stencils can change per-CPython commit (in practice it's less often than this, but it's much more often than other bin-deps)
Fair, but that kind of leaves the only option to be to generate them and check them in to the main repo (which I'm not totally against, but understand why others might have concerns, especially since it'll make
git clonemuch bigger over time).Probably we're just going to have to leave JIT builds to only be used when you specifically request to build them, and make setting up the environment a manual step. Even if we still mirror LLVM somewhere, it probably shouldn't be the default to download 1GB worth of compiler.
FWIW, if we ever switched from MSVC to Clang/LLVM on Windows, or if MSVC made itself available as an extractable package, I'd still insist that we not download the compiler automatically. We should fit into a build environment as much as possible, not dictate it.
Reacted by Zachary Ware and Adam TurnerThe VS2022 installer is presenting LLVM 18, which I think is supported?
We're currently on LLVM 19 and plan to move to 20 soon, which positions us to adopt 21 when it's released (there are upcoming features we’ll want). We only support a single LLVM version for stencil builds.
So it sounds like the consensus is that, while the JIT is still experimental, LLVM doesn’t need to live in bin-deps or similar. We’re fine with relying on system LLVM and a more manual setup for now, and can revisit this if/when the JIT moves beyond the experimental stage?
@zooba Circling back on this, I think we’re a bit between a rock and a hard place, and I’d like to find a short-term solution as soon as possible so we can unblock bumping our LLVM version (I have a branch with the changes, this is the last remaining blocker).
The way I see it, we have a couple of options here:
- Have Windows release builds source LLVM from a package manager (similar to other platforms) until we work out the details of the larger unified proposal for source and binary dependencies
- Remove JIT from Windows release builds if checking the LLVM binary into
cpython-bin-depsis a non-negotiable. I don’t like this option, as this would be counterproductive to our testing goals and work to progress the JIT into a non-experimental state - Appeal to the Steering Council to reconsider the PEP 774 decision, given that this is now a blocking issue
I would argue that option 1 seems the most pragmatic approach here, especially given that the JIT is an experimental feature. We also don’t have to wait for the one design to rule them all to figure out a more robust solution. I’d advocate that we harden our dependency management approach before making the JIT officially supported.
cc: @brandtbucher since we discussed at the sprint
Reacted by Emma SmithHow about we (you) manually post the required build as a release artifact on a gh/python/??? repo (bin-deps is fine), so at least we can download it from GitHub, even if it's hard coded for now?
Then for the build definition in release-tools, I'd want:
- queue-time parameter to enable/disable the download step (possibly the same as enable/disable the JIT?)
- download step occurs as early as possible whenever needed so that failure doesn't take too long to present
- retries to deal with network issues, failure causes the entire build to stop (and RM can decide to release without the JIT or to halt the release)
Not sure if we discussed it, but x86 and ARM64 builds are cross-compiled, but I'm not sure whether LLVM can do that? If we need separate LLVM builds for each architecture, then make them separate downloads - each arch is built in its own job, so it should only need to get the one it's targeting. And for now we need the ARM64 one to run on amd64.
In fact, if it's possible to generate the stencils once in a single job (for all architectures) and publish as a pipeline artifact for all subsequent builds, that would be preferable. Much safer to add download steps for pipeline artifacts than to download entire toolsets all over the place.
If we are okay with using release artifacts for this, I think that's fine, I can give that a shot. I don't currently have permissions to create releases on cpython-bin-deps, though - would someone be able to either grant me those permissions or create the LLVM 20.1.8.0 release for me? I can provide the tarball.
For the build definition enhancements, I'd say that this should be tackled as a separate work item. There are a lot of improvements I'd like to make during the 3.15 cycle. I'll create an issue and note your wishlist 😃
Reacted by Steve DowerReacted by Zachary WareI've created https://github.com/python/cpython-bin-deps/releases/tag/llvm-20.1.8.0 with the .tar.xz from https://github.com/llvm/llvm-project/releases/tag/llvmorg-20.1.8.
Reacted by Steve DowerReacted by Savannah OstrowskiCircling back on this, I think we can close this ticket now that #136895 and #141002 have been merged in. The remaining work here is to improve the tarball extraction to handle platform-specific tarballs for LLVM (right now, we are just using x86_64 binary and it cross-compiles for aarch64). I'm going to track that in #141362
Agreed, we can handle follow up in #141362 and the dependencies PEP :)
Reacted by Savannah Ostrowski and Itamar Oren
Feature or enhancement
Currently, upgrading the JIT build to LLVM 20 (#136895) is blocked on Windows because a file is too large to be checked in to cpython-bin-deps as it exceeds GitHub per-file limits.
As a workaround, I propose we upload the binaries to the GitHub Releases of cpython-bin-deps, and update
get_externals.pyto download binary dependencies from release assets. Sources will continue to be downloaded via git.I have a proof of concept working in a branch main...emmatyping:cpython:windeps-via-releases, which uses the GitHub API to download the files. It downloads the assets from the releases I've uploaded here: https://github.com/emmatyping/cpython-bin-deps/releases. I'm going to wait to open a PR as I want to get some initial feedback on the idea and before CI builds will pass, the binaries would need to uploaded to the main cpython-bin-deps releases.
Note: this change is independent of https://discuss.python.org/t/pre-pep-redesigning-cpython-source-binary-dependencies/101824 and is a shorter term solution to unblock #136895