Skip to content

CI: harden build-wheel.yml for the eMudhra/emSign CA incident (13.4.x backport) - #3305

Draft
juenglin wants to merge 1 commit into
NVIDIA:13.4.xfrom
juenglin:emudhra-build-wheel-13.4.x
Draft

juenglin wants to merge 1 commit into
NVIDIA:13.4.xfrom
juenglin:emudhra-build-wheel-13.4.x

Conversation

@juenglin

Copy link
Copy Markdown
Contributor

Description

Refs #3195

Backports the CI hardening that main received for the eMudhra/emSign CA key compromise (CVE-2026-111943, CVE-2026-115557) to 13.4.x, so that wheels released from this branch are built with patched tooling and verified downloads. Before this change the branch's cibuildwheel builds used the pip frontend with an unpatched pip, installed an unverified releases/latest sccache, and downloaded CTK archives and GPU driver installers without checksums.

  • install-sccache action: pin rapidsai/sccache v0.18.0-rapids.7 and verify its sha256 (CI: pin and verify sccache via a shared composite action #3248), and use it in test-sdist-linux.yml instead of mozilla-actions/sccache-action.
  • GPU driver installers: require and verify DRIVER_SHA256 before running the installer, with a test-matrix check (CI: verify GPU driver installer SHA-256 before execution #3249).
  • upgrade-ca-certificates action: run in build-wheel.yml, coverage.yml, test-sdist-windows.yml and the Windows pre-commit job in ci.yml (CI: add eMudhra/emSign CA stopgap for GitHub-hosted Windows runners #3270).
  • setup-hardened-python action: replaces actions/setup-python plus harden-pip everywhere, including the release verification jobs (CI: install eMudhra-fixed Pythons with a pinned uv #3275). This branch still builds and tests Python 3.10, which has no fixed release, so for series older than 3.11 the action warns and installs the sha256-pinned pip 26.2.2 wheel instead of failing.
  • cibuildwheel uses build[uv] with UV_* constraints and the hardened host Python, as main does (build: switch to uv as build frontend #2964). This drops the dummy wheel test stage (sccache stats are dumped during repair) and the before-build = "pip install delvewheel" overrides in the pyproject.toml files.
  • fetch_ctk: verify the CTK redistrib JSON against a pinned sha256 and each component archive against the sha256 in that JSON before extracting it, failing closed on a mismatch. This adds component-sha256 and verify-metadata to ci/tools/fetch_ctk_redistrib.py, with unit tests. All CUDA versions used by this branch's CI (12.6.3, 12.9.1, 13.0.2, 13.4.2) are pinned; unpinned versions only warn. This part is not on main yet.

Not backported: the runner trust probe, the "Enable Git symlinks" steps, and the lychee and pre-commit.yml workflows, which do not exist on this branch.

Notes for reviewers

The commit is a squash of cherry-picks with adaptations for this branch; the conflicts were in the workflow files, which differ from main (cuda_bindings/ instead of cuda_bindings_13/, no lychee workflows, Python 3.10 still supported).

Validation: ci/tools/tests pass locally (79 tests), pre-commit including actionlint passes, and the pinned redistrib hashes were re-checked against the live JSONs. The workflow changes have not run on GitHub Actions yet, so CI on this PR is the real test, in particular the Windows cibuildwheel jobs with build[uv] and the Python 3.10 fallback.

Checklist

  • I have read the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

… backport)

Backport the CI hardening that main received for the eMudhra/emSign CA key
compromise (CVE-2026-111943, CVE-2026-115557), so that wheels released from
this branch are built with patched tooling and verified downloads:

- install-sccache: pin rapidsai/sccache v0.18.0-rapids.7 and verify its
  sha256 instead of piping releases/latest into sudo tar (NVIDIA#3248). Also use
  it in test-sdist-linux.yml instead of mozilla-actions/sccache-action.
- GPU driver installers: verify DRIVER_SHA256 before running them as
  root/admin, with a test-matrix check (NVIDIA#3249).
- upgrade-ca-certificates: upgrade ca-certificates on GitHub-hosted Linux
  and distrust the compromised intermediates on GitHub-hosted Windows, in
  build-wheel.yml, coverage.yml, test-sdist-windows.yml and the Windows
  pre-commit job in ci.yml (NVIDIA#3270).
- setup-hardened-python: install interpreters with a pinned uv and fail on
  unpatched ones, replacing actions/setup-python + harden-pip (NVIDIA#3275). This
  branch still builds and tests Python 3.10, which has no fixed release; for
  series older than 3.11 the action warns and installs pip from the
  sha256-pinned 26.2.2 wheel instead of failing. cibuildwheel now runs with
  the hardened host Python (`python-path`).
- cibuildwheel: use build[uv] with UV_* constraints, as main does (NVIDIA#2964),
  so build dependencies are not fetched by the pip that cibuildwheel
  provides. Drop the dummy wheel test stage (sccache stats are dumped during
  repair) and the `before-build = "pip install delvewheel"` overrides.
- fetch_ctk: verify the CTK redistrib JSON against a pinned sha256 and every
  component archive against the sha256 in that JSON before extracting it,
  failing closed on a mismatch (new `component-sha256` and `verify-metadata`
  subcommands in ci/tools/fetch_ctk_redistrib.py, with tests).

Not backported: the runner trust probe, the "Enable Git symlinks" steps, and
the lychee/pre-commit workflows, which do not exist on this branch.

Refs NVIDIA#3195
@juenglin juenglin added the CI/CD CI/CD infrastructure label Oct 30, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@juenglin juenglin self-assigned this Oct 30, 2026
@github-actions github-actions Bot added the cuda.core Everything related to the cuda.core module label Oct 30, 2026
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 3541ec5

@github-actions

Copy link
Copy Markdown
Contributor

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure cuda.core Everything related to the cuda.core module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant