Skip to content

ci(native-arch): a lane that cannot check out the commit must fail, not report OK - #778

Merged
tannevaled merged 1 commit into
mainfrom
wave62-ci-lane-must-test-the-commit
Oct 4, 2026
Merged

tannevaled merged 1 commit into
mainfrom
wave62-ci-lane-must-test-the-commit

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

A scheduled lane reported success without testing the commit it was given. Found while auditing the build and CI surface for the security review in #768.

What was there

Both remote scripts in native-arch.yml — the cfarm job and the s390x job, with the same two lines — ended their checkout with:

git fetch --depth 50 origin "$COMMIT" || true
git checkout -q "$COMMIT" || echo "== commit $COMMIT not reachable; testing default branch HEAD"

Each remote block opens with set -euo pipefail (:179, :255), so those two suffixes were defeating it deliberately.

The consequence is not the message. It is what happens next: the job went on to run go vet ./... and go test ./... against whatever the default branch pointed at, then printed

== native $GOARCH validation OK

and exited 0. Nothing downstream can distinguish that from a real pass. This is the lane that stands in for riscv64, loong64, ppc64le and s390x, which have no other native coverage — ci.yml's per-PR arch gate is amd64 + arm64 by design, and the QEMU nightly is emulation. So the one place those four are exercised on real hardware was free to report on the wrong tree.

Three ways $COMMIT goes unreachable, none of them exotic:

  • the clone is --depth 50, so a commit further back is simply absent;
  • a force-push can orphan it;
  • a workflow_dispatch off a branch passes a SHA the default branch never had.

What it is now

git fetch --depth 1 origin "$COMMIT"
git checkout -q --detach "$COMMIT"
head="$(git rev-parse HEAD)"
if [ "$head" != "$COMMIT" ]; then
  echo "== HEAD is $head but \$COMMIT is $COMMIT -- refusing to report on the wrong tree"
  exit 1
fi
echo "== testing $head"

Fetch and checkout are now fatal (the set -e above was always willing to do this), and the assertion is there because the fetch succeeding is not the same as HEAD being right — a checkout can land elsewhere and still exit 0. It names both values on failure, so the log says which tree it refused.

Applied to both sites identically. A fix to one would have left the other reporting on the wrong tree, which is the kind of half-fix this class of defect invites: the two blocks are near-duplicates 75 lines apart.

Not fixed here, deliberately

Two further findings in this same file are left for their own change, because they are different decisions and one of them needs a judgement call about the shared academic hosts:

  • the Go toolchain is fetched with curl -fsSL https://go.dev/dl/$tarball and extracted with no SHA256 check before go from it is executed (:196, :272);
  • the file documents the cfarm account, node names and the LinuxONE host address in a public repo (:36, :46-49, :137-140, :227-228) — not a credential, but it names the targets.

Verified

native-arch.yml still parses (jobs: guard, native-cfarm, native-s390x; top-level permissions: contents: read unchanged). The lane itself is schedule + workflow_dispatch only, so this change cannot be exercised by this PR's checks — it is reviewed by reading, which is why the diff is deliberately small and the two sites are identical.

🤖 Generated with Claude Code

…ot report OK

Both remote scripts ended their checkout with

    git fetch --depth 50 origin "$COMMIT" || true
    git checkout -q "$COMMIT" || echo "== commit $COMMIT not reachable; testing default branch HEAD"

Each block opens with `set -euo pipefail`, so those two suffixes were defeating
it deliberately. The consequence is not the message: it is that the job then ran
`go vet` and `go test` against whatever the default branch pointed at, and
finished by printing `== native $GOARCH validation OK` and exiting 0. A lane that
reports success without testing its subject is worse than one that fails,
because nothing downstream can tell the two apart -- and this is the lane that
stands in for four architectures with no other native coverage.

Three ways $COMMIT goes unreachable, none exotic: the clone is `--depth 50`, so
a commit further back is simply absent; a force-push can orphan it; and a
`workflow_dispatch` off a branch passes a SHA the default branch never had.

Now it fetches the SHA, checks it out detached, and asserts `git rev-parse HEAD`
equals $COMMIT before testing, naming both values when it does not. Applied to
both sites identically -- the cfarm job and the s390x job carried the same two
lines, so a fix to one would have left the other reporting on the wrong tree.

Found while auditing the build and CI surface for the security review in #768.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 43f383d into main Oct 4, 2026
9 checks passed
@tannevaled
tannevaled deleted the wave62-ci-lane-must-test-the-commit branch October 4, 2026 21:02
tannevaled added a commit that referenced this pull request Oct 4, 2026
…t of it (#781)

Both remote scripts did

    ( cd "$workdir" && curl -fsSL "https://go.dev/dl/${tarball}" -o "$tarball" && tar -xzf "$tarball" )

and then put that directory on PATH and executed `go` from it. HTTPS and nothing
else: no integrity check between downloading a 64 MB archive and running a
compiler out of it, on the GCC Compile Farm and a LinuxONE host -- machines
shared under an acceptable-use policy, where we are a guest.

Now it downloads, verifies SHA256 against a sum pinned in this file, and only
then extracts.

The sums are pinned HERE rather than fetched alongside the tarball on purpose: a
checksum served from the same origin as the file it describes cannot attest to
that file. Fetching go.dev's JSON index at run time would protect against a
truncated download and nothing else, which is the weaker thing wearing the
stronger thing's clothes.

The cost is that bumping GO_VERSION means updating three lines. The `*)` arm is
what makes that cost visible instead of silent: an unpinned version or
architecture exits 1 naming what to add, rather than skipping the check. A guard
that quietly does nothing when it does not recognise its input is the failure
mode worth designing against here.

Sums taken from https://go.dev/dl/?mode=json for go1.26.4, and cross-checked
programmatically against that index after being written into the file, so a
transcription slip cannot survive:

    ppc64le  53f49b8c...bc22e
    riscv64  2b9ba137...40555
    s390x    1e12a231...b3ca1

The compare logic was exercised locally in all four states before pushing --
matching sum passes; a mismatch exits 1 printing both values; an unpinned
GOVER and an unpinned GOARCH each exit 1 with the "refusing to run an
unverified toolchain" message. The local run used `shasum -a 256` since this is
a mac; the workflow uses `sha256sum`, which is what the Linux hosts have.

Applied at both sites. The cfarm job covers ppc64le and riscv64, the s390x job
covers s390x, and they are near-duplicate blocks 100 lines apart -- fixing one
would have left the other running an unverified toolchain.

Deferred from #778, which fixed the other defect in these same two blocks.
Found in the build/CI audit for #768.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant