Skip to content

CI: harden build-wheel.yml for the eMudhra/emSign CA incident (13.4.x backport) #3305

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
juenglin wants to merge 1 commit into
base: 13.4.x
Choose a base branch
from
Draft

CI: harden build-wheel.yml for the eMudhra/emSign CA incident (13.4.x backport) #3305

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .github/actionlint.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,15 @@ paths:
".github/workflows/test-wheel-windows.yml":
ignore:
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
".github/workflows/build-wheel.yml":
ignore:
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
".github/workflows/ci.yml":
ignore:
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
".github/workflows/test-sdist-windows.yml":
ignore:
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
".github/workflows/coverage.yml":
ignore:
- 'specifying action "\$/[^"]+" in invalid format because ref is missing'
12 changes: 12 additions & 0 deletions .github/actions/fetch_ctk/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,11 @@ runs:
CTK_JSON_URL="$CTK_BASE_URL/redistrib_${{ inputs.cuda-version }}.json"
CTK_JSON_FILE="$CACHE_TMP_DIR/redistrib.json"
curl -LSs "$CTK_JSON_URL" -o "$CTK_JSON_FILE"
# Pin the metadata JSON: the per-archive sha256 checks below are only
# trustworthy if the JSON that carries those hashes is itself trusted.
python "$CTK_REDIST_TOOL" verify-metadata \
--cuda-version "${{ inputs.cuda-version }}" \
--metadata-path "$CTK_JSON_FILE"
if [[ "${{ inputs.host-platform }}" == linux* ]]; then
function extract() {
tar -xvf $1 -C $CACHE_TMP_DIR --strip-components=1
Expand Down Expand Up @@ -102,6 +107,13 @@ runs:
CTK_COMPONENT_URL="${CTK_BASE_URL}/${CTK_COMPONENT_REL_PATH}"
CTK_COMPONENT_COMPONENT_FILENAME="$(basename $CTK_COMPONENT_REL_PATH)"
download $CTK_COMPONENT_URL $CTK_COMPONENT_COMPONENT_FILENAME
# Verify the downloaded archive against the sha256 in the (pinned)
# metadata before extracting; fail closed on any mismatch.
CTK_COMPONENT_SHA256="$(python "$CTK_REDIST_TOOL" component-sha256 \
--host-platform "${{ inputs.host-platform }}" \
--component "$CTK_COMPONENT" \
--metadata-path "$CTK_JSON_FILE")"
echo "${CTK_COMPONENT_SHA256} ${CTK_COMPONENT_COMPONENT_FILENAME}" | sha256sum --check --strict -
extract $CTK_COMPONENT_COMPONENT_FILENAME
rm $CTK_COMPONENT_COMPONENT_FILENAME
}
Expand Down
28 changes: 0 additions & 28 deletions .github/actions/harden-pip/action.yml
View file
Open in desktop

This file was deleted.

75 changes: 75 additions & 0 deletions .github/actions/install-sccache/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0

name: Install pinned rapidsai/sccache
description: Download, verify, and install a pinned rapidsai/sccache binary on Linux

# Regenerate hashes when bumping SCCACHE_VERSION:
# for arch in x86_64 aarch64; do
# curl -fsSL "https://github.com/rapidsai/sccache/releases/download/<tag>/sccache-${arch}-unknown-linux-musl.tar.gz" | sha256sum
# done
# Cross-check against the `.sha256` files published on the GitHub release.
#
# Security note (eMudhra/emSign CA incident):
# - Every version bump MUST repeat the embedded-roots check on the extracted
# `sccache` binary:
# grep -ac 'ISRG Root X1' sccache # >0 means a root list is embedded
# grep -ac 'emSign ECC Root CA - G3' sccache # must be 0
# Pass criteria: either no embedded root list (both return 0; binary defers
# to the OS trust store via `rustls-platform-verifier`, as verified for
# `rapidsai/sccache` v0.18.0-rapids.7), or an embedded `webpki-roots` list
# at 1.0.10+ where `ISRG Root X1` matches and `emSign ECC Root CA - G3`
# returns 0. Match the full root name: 1.0.10 removes only that emSign root
# and still ships other emSign roots.
# - Do NOT switch to or bump `mozilla/sccache` to v0.19.0 (or later without
# repeating the roots check): `mozilla/sccache` v0.19.0 re-embeds
# `webpki-roots` 1.0.5, which still trusts `emSign ECC Root CA - G3`.

runs:
using: composite
steps:
- name: Install pinned rapidsai/sccache
shell: bash --noprofile --norc -xeuo pipefail {0}
env:
SCCACHE_VERSION: v0.18.0-rapids.7
run: |
if [[ "$(uname -s)" != "Linux" ]]; then
echo "::error::install-sccache only supports Linux (got $(uname -s))" >&2
exit 1
fi

arch="$(uname -m)"
case "${arch}" in
x86_64)
expected_sha256="44106a2e01aeff99767e66b1a5ae6be074306c518458b367f75287de4532974c"
;;
aarch64)
expected_sha256="1bad10208481786f2c2e943d7cca8006eaa7e620d122859ef2d61861a733d3d1"
;;
*)
echo "::error::Unsupported architecture for install-sccache: ${arch}" >&2
exit 1
;;
esac

tarball="sccache-${arch}-unknown-linux-musl.tar.gz"
url="https://github.com/rapidsai/sccache/releases/download/${SCCACHE_VERSION}/${tarball}"

tmp_dir="$(mktemp -d)"
trap 'rm -rf "${tmp_dir}"' EXIT

curl -fsSL -o "${tmp_dir}/${tarball}" "${url}"
(
cd "${tmp_dir}"
echo "${expected_sha256} ${tarball}" | sha256sum --check --strict -
)

if [[ "$(id -u)" -eq 0 ]]; then
tar -C /usr/local/bin -xvzf "${tmp_dir}/${tarball}" --wildcards --strip-components=1 -x '*/sccache'
else
sudo tar -C /usr/local/bin -xvzf "${tmp_dir}/${tarball}" --wildcards --strip-components=1 -x '*/sccache'
fi

echo "SCCACHE_PATH=/usr/local/bin/sccache" >> "${GITHUB_ENV}"
/usr/local/bin/sccache --version
150 changes: 150 additions & 0 deletions .github/actions/setup-hardened-python/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# SPDX-License-Identifier: Apache-2.0

name: Set up hardened Python
description: >-
Install CPython with a pinned uv, create a venv from it, and put the venv
first on PATH. setup-uv verifies the uv binary and uv verifies the
python-build-standalone archive against embedded checksums, so the
interpreter's integrity does not depend on TLS. The job then fails unless
the interpreter's ssl module rejects the compromised eMudhra/emSign CA keys
(CVE-2026-115557; first fixed releases: 3.11.18, 3.12.16, 3.13.18, 3.14.10,
3.15.2), and pip is updated through that ssl module. Unlike setup-python,
this provides patched Windows builds of maintenance series, for which
python.org publishes no installers. Series that have no fixed release
(3.10 and older, which this branch still builds and tests) are installed
with a warning instead, and their pip is replaced by a sha256-pinned
wheel, so that the pip update does not depend on TLS. See
https://github.com/NVIDIA/cuda-python/issues/3195 and
https://discuss.python.org/t/109678.

inputs:
python-version:
description: >-
uv Python request, e.g. "3.12" or "3.14t" (free-threaded). The newest
patch known to the pinned uv is installed. Series with a fixed release
listed in the interpreter check (3.11-3.15) must carry it. Series
without a fixed release (3.10 and older) are accepted with a warning,
and pip is installed from a sha256-pinned wheel.
required: true
venv:
description: Directory for the venv. Defaults to a new directory under RUNNER_TEMP.
default: ""

outputs:
python-path:
description: Absolute path of the venv's interpreter.
value: ${{ steps.venv.outputs.python-path }}
python-version:
description: Full version of the interpreter, e.g. "3.14.10".
value: ${{ steps.venv.outputs.python-version }}

runs:
using: composite
steps:
- name: Install uv
uses: astral-sh/setup-uv@68559e99cc807c1eb7e9238fd43a15bba460eec2 # v10.3.3
with:
# The newest uv whose checksum ships inside the pinned setup-uv, so
# the binary is verified against the SHA-pinned action itself. The uv
# release determines the newest patch of each CPython series, so
# bumping the setup-uv SHA also moves the CPython patch versions.
version: "latest-known"
enable-cache: false
# Some callers (e.g. release verification) have no checkout.
ignore-empty-workdir: true

- name: Install Python and create venv
id: venv
shell: bash --noprofile --norc -euo pipefail {0}
env:
PY_REQUEST: ${{ inputs.python-version }}
VENV_DIR: ${{ inputs.venv }}
# A per-job install directory ensures the newest patch known to the
# pinned uv is used, never an older interpreter left on a persistent
# self-hosted runner.
UV_PYTHON_INSTALL_DIR: ${{ runner.temp }}/setup-hardened-python/interpreters
UV_NO_CONFIG: "1"
run: |
uv python install --no-bin --no-registry "${PY_REQUEST}"
base="$(uv python find --managed-python --no-project "${PY_REQUEST}")"
venv="${VENV_DIR}"
if [[ -z "${venv}" ]]; then
# A fresh directory per call, so a second call in the same job
# (e.g. build-wheel.yml) leaves the first venv intact.
venv="$("${base}" -c 'import sys, tempfile; print(tempfile.mkdtemp(prefix="venv-", dir=sys.argv[1]))' "${RUNNER_TEMP}")"
fi
# The stdlib venv seeds pip from the interpreter's bundled wheel, so
# nothing is downloaded before the ssl check below.
"${base}" -m venv --clear "${venv}"
if [[ "${RUNNER_OS}" == Windows ]]; then
py="${venv}/Scripts/python.exe"
else
py="${venv}/bin/python"
fi
# Let the interpreter report native absolute paths (Windows included).
python_path="$("${py}" -c 'import sys; print(sys.executable)')"
"${py}" -c 'import os, sys; print(os.path.dirname(sys.executable))' >> "${GITHUB_PATH}"
{
echo "python-path=${python_path}"
echo "python-version=$("${py}" -c 'import platform; print(platform.python_version())')"
} >> "${GITHUB_OUTPUT}"

- name: Check Python interpreter is patched (CVE-2026-115557)
id: check
# Runs the venv's python through the PATH entry added above.
shell: python {0}
env:
PYTHON: ${{ steps.venv.outputs.python-path }}
run: |
import os
import platform
import sys

if os.path.normcase(sys.executable) != os.path.normcase(os.environ["PYTHON"]):
print(f"::error::PATH resolves python to {sys.executable}, not the venv interpreter {os.environ['PYTHON']}.")
sys.exit(1)

FIXED = {(3, 11): 18, (3, 12): 16, (3, 13): 18, (3, 14): 10, (3, 15): 2}
# Series that will never get a fixed release (end of life before the
# fix). Their pip is installed from a pinned wheel instead.
UNPATCHABLE_BELOW = (3, 11)
ssl_patched = "true"
v = sys.version_info
series = (v.major, v.minor)
name = f"{v.major}.{v.minor}"
full = platform.python_version()
print("sys.version:", sys.version)
if series in FIXED:
floor = f"{name}.{FIXED[series]}"
if v.micro < FIXED[series]:
print(f"::error::Python {full} is older than {floor}, the first {name} release whose ssl module rejects the compromised eMudhra CA keys (CVE-2026-115557).")
sys.exit(1)
print(f"Python {full}: ssl module carries the fix (first fixed: {floor}).")
elif series < UNPATCHABLE_BELOW:
ssl_patched = "false"
print(f"::warning::Python {full}: no {name} release rejects the compromised eMudhra CA keys (CVE-2026-115557); pip will be installed from a sha256-pinned wheel instead.")
else:
# Fail closed: a newer series is only accepted once its fixed release is listed.
print(f"::error::Python {full}: no {name} release is known to reject the compromised eMudhra CA keys (CVE-2026-115557).")
sys.exit(1)

with open(os.environ["GITHUB_OUTPUT"], "a", encoding="utf-8") as f:
f.write(f"ssl-patched={ssl_patched}\n")

- name: Update pip
shell: bash --noprofile --norc -xeuo pipefail {0}
env:
PYTHON: ${{ steps.venv.outputs.python-path }}
SSL_PATCHED: ${{ steps.check.outputs.ssl-patched }}
run: |
if [[ "${SSL_PATCHED}" == "true" ]]; then
"${PYTHON}" -m pip install -U pip
else
# The interpreter's ssl module still trusts the compromised CA keys,
# so do not fetch pip through it. The sha256 fragment guarantees the
# wheel's integrity regardless of the transport.
"${PYTHON}" -m pip install \
"pip @ https://files.pythonhosted.org/packages/3f/06/3e4420302363fbb71be3c7f35ce042f54bfca330e3e66b3a2e3c4cca579f/pip-26.2.2-py3-none-any.whl#sha256=fd7f4dfa8190e014013b9f08f3e973f4e4c26ffd179a204895533a7254813a70"
fi
Loading
Loading