Skip to content

[SC-18919] Cut library CI time by dropping unused GPU packages and caching installs - #568

Merged
juanmleng merged 4 commits into
mainfrom
juan/sc-18919/cut-library-ci-time-by-dropping-unused-gpu-package
Oct 22, 2026
Merged

juanmleng merged 4 commits into
mainfrom
juan/sc-18919/cut-library-ci-time-by-dropping-unused-gpu-package

Conversation

@juanmleng

@juanmleng juanmleng commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Pull Request Description

What and why?

Engineers waited 10 to 17 minutes on the pull request check, and one run hung for six hours. Most of it was setup: every run downloaded about 3 GB of GPU libraries the test machine cannot use, reinstalled PyTorch a second time at an old version that the next step immediately undid, cleared disk space to fit it all, and, whenever the lock file changed, rebuilt two packages from source because the download cache was discarded.

The repository's own installs now use the CPU-only build of PyTorch on Linux, at the version CI already tested (2.8.0 on Python 3.9), so the GPU libraries, the second PyTorch install and the disk-space step are gone. Downloaded packages are kept between runs and survive lock-file-only updates such as dependency bumps, and the job stops after 30 minutes. Every check still runs: lint, notebook checks, unit tests and both docs builds. The integration workflow drops an install whose environment was deleted before anything used it.

The CPU-only setting applies only to installs inside this repository. The published package still depends on plain torch, so pip install validmind is unchanged, and the two checks that exist to test what users install, the integration [all] install and the dependency compatibility matrix, still install the standard PyTorch from PyPI.

How to test

Automated tests

CI on this PR runs the full set of checks. Expected: "Code Quality and Unit Testing" and all "Dependency Compatibility Testing" legs pass.

From the repository root, on this branch:

uv lock --check
grep -c '^name = "triton"' uv.lock
grep '^name = "nvidia-' uv.lock

Expected: the lock is consistent, 0 triton entries, and one NVIDIA package, nvidia-nccl-cu12, which xgboost needs on Linux.

uv build --wheel -o /tmp/vm-wheel
unzip -p /tmp/vm-wheel/validmind-*.whl '*.dist-info/METADATA' | grep '^Requires-Dist: torch'

Expected: torch>=2.0.0 for the all, llm and pytorch extras, the same as on main.

uv pip compile pyproject.toml --no-sources -p 3.10 --python-platform x86_64-manylinux_2_28 --all-extras --upgrade -o /tmp/max.txt
grep -E '^(torch|nvidia-cudnn)' /tmp/max.txt

Expected: a plain torch== pin with no +cpu, plus the CUDA packages, as the dependency matrix's "max" leg installs.

Manual testing

  1. Open the latest "Code Quality and Unit Testing" run on this PR and expand Install Dependencies. Expected: + torch==2.8.0+cpu, no triton, no NVIDIA package other than nvidia-nccl-cu12, and no torch change in the aequitas fairlearn vl-convert-python install.
  2. Expand Install uv. Expected: Cache hit for: setup-uv-1-…. A run without a cache hit is expected after pyproject.toml changes, and on the first run on main after this merges.

The integration workflow runs only on pushes to main and prod, so its change is verified on the first run after merge.

What needs special review?

  • pyproject.toml [tool.uv.index] / [tool.uv.sources]: Linux-only, uv-only routing of torch to the PyTorch CPU index; the built wheel's metadata is unchanged.
  • dependency-testing.yaml: --no-sources keeps the "max" leg resolving PyPI torch; without it the leg pins a +cpu build that pip cannot find.
  • python.yaml cache-dependency-glob: pyproject.toml: the cache now refreshes only when pyproject.toml changes, and lock-only changes download the changed packages on top of it.

Dependencies, breaking changes, and deployment notes

No breaking changes and nothing to deploy. Linux developers who run uv sync now get the CPU-only PyTorch; GPU work in a local environment needs a CUDA build installed by hand. On Python 3.10 and later, Linux environments also move from PyTorch 2.11.0 to 2.14.1, because changing where uv gets PyTorch makes it pick a version again, the newest one allowed. #556 also adds a [tool.uv.sources] table and regenerates uv.lock, so whichever merges second merges the two tables and re-runs uv lock.

Release notes

Internal change to CI and the development environment only; no release notes.

Checklist

  • What and why
  • Screenshots or videos (Frontend)
  • How to test
  • What needs special review
  • Dependencies, breaking changes, and deployment notes
  • Labels applied
  • PR linked to Shortcut
  • Unit tests added (Backend)
  • Tested locally
  • Documentation updated (if required)
  • Environment variable additions/changes documented (if required)

🤖 Generated with Claude Code

PyPI's Linux torch pulls ~3 GB of NVIDIA libraries the CI runner cannot
use. Route torch to the PyTorch CPU index for this repo's own installs
(uv-only setting; the published package still depends on plain torch),
and relock: the NVIDIA/CUDA packages and triton drop out, and CI keeps
testing torch 2.8.0.

- Drop the torch==2.0.1 reinstall: the next install step already put the
  locked version back, so the tests never ran on 2.0.1
- Drop the Free Disk Space step, which only made room for the GPU wheels
- Keep downloaded wheels in the uv cache between runs
- Stop the job after 30 minutes instead of the 6-hour default
- Integration: drop the dependency install whose venv was deleted unused
  before the [all] install

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juanmleng juanmleng added the internal Not to be externalized in the release notes label Oct 17, 2026
@juanmleng juanmleng self-assigned this Oct 17, 2026
juanmleng and others added 2 commits October 17, 2026 12:56
uv pip compile reads tool.uv.sources, so the max leg pinned the
CPU-only torch build (torch==2.x+cpu), which pip cannot find on PyPI.
Pass --no-sources so the leg resolves torch as users get it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cache key hashed uv.lock, so every lock-only change (dependency
bumps) missed the cache and recompiled blis and thinc, which have no
Python 3.9 wheels, for ~8 minutes. Keying on pyproject.toml lets those
runs reuse the cache and download only the packages that changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juanmleng
juanmleng marked this pull request as ready for review October 21, 2026 18:51
@juanmleng
juanmleng requested a review from cachafla October 21, 2026 18:52
…rch index

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juanmleng
juanmleng merged commit 48a7c5a into main Oct 22, 2026
69 checks passed
@juanmleng
juanmleng deleted the juan/sc-18919/cut-library-ci-time-by-dropping-unused-gpu-package branch October 22, 2026 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal Not to be externalized in the release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants