Skip to content

Run pnpm install-proof as one job per Node runtime - #897

Open
Mikola Lysenko (mikolalysenko) wants to merge 2 commits into
mainfrom
ci-janitor/pnpm-compat-shards
Open

Mikola Lysenko (mikolalysenko) wants to merge 2 commits into
mainfrom
ci-janitor/pnpm-compat-shards

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

pnpm hosted compatibility is the reddest workflow in the repo right now. In the last 400 runs (2026-10-05 19:07–22:40 UTC), 20 of 28 completed pnpm runs failed (npm hosted/vendored, the next worst, was 16/28). I checked the failed install-proof legs, and none of them had a test failure. Each one was an ubuntu-latest job that was cancelled 15–18 min after it was queued, never got a runner, and has no log (the job-logs API returns 404):

  • run 37364433915 (attempt 2): 10 of 25 legs were cancelled with no runner (2.25.7, 3.0.0, 3.8.1, 5.18.11, 7.33.7, 8.0.0, 10.33.0, 11.0.0, 12.0.0, 12.4.2). The other 15 passed, and their real work took ~20 s each.
  • run 37373110769: 2 legs (6.35.1, 7.0.0) were cancelled with no runner. Every other leg passed.

The workflow spawns 25 single-version jobs (one per pnpm/Node pair). Most of them do about 20 s of real work (the two 12.x legs take ~70 s); the rest is runner setup, and every job is billed as at least one full minute. A run fails if any one of its 25 jobs fails to get a runner, so the matrix is the workflow most exposed to runner shortage during busy agent push bursts. In the run above, legs waited up to 11 min just to start.

Root cause

The matrix runs one pnpm version per job, so 25 jobs compete for runners to do about 8 minutes of total work.

Fix

The install-proof job now runs one job per Node runtime (10.24.1, 16.20.2, 24.11.1), and each job loops over its pnpm versions:

  • It installs every pnpm version of the group under Node 24, as before, then switches to the group's Node.
  • For each version, it runs pnpm-e2e pnpm_pinned_matrix and then pnpm-vendor-e2e pnpm_pinned_matrix. The vendored suite still only runs if the hosted one passed, as the two separate steps did before.
  • Each version gets its own TMPDIR. That keeps the suites' shared cache sandbox (cache_env::cache_root() = $TMPDIR/socket-patch-test-caches-$USER: HOME, PNPM_HOME, the npm cache, XDG dirs) and every fixture tempdir as empty as they were on a fresh runner. No pnpm version sees another's cache or store.
  • A failing version doesn't stop the loop. It gets an ::error title=pnpm <v>:: annotation and its own log group, and the step fails at the end listing every failed version.

The build job is unchanged.

Also included: 58e901c is a -x cherry-pick of #878 (Gradle digests through utils::digest). The CI workflow's coverage, test and test-release jobs are red on main itself, on utils::digest::tests::production_digests_go_through_the_helpers (a semantic conflict between #646 and #865). Without that fix this PR couldn't go green. The commit becomes a no-op once #878 merges.

Proof

  • The coverage is the same. I compared the sorted (pnpm, Node) pairs from origin/main's matrix with the expanded new matrix and got the same 25 pairs. actionlint is clean.
  • Local run. I ran the step's script, extracted from the workflow YAML, against locally built pnpm-e2e and pnpm-vendor-e2e binaries with PNPM_TEST_VERSIONS="9.15.9 10.33.0 0.0.0-missing". 9.15.9 and 10.33.0 passed both suites. The bogus version failed, got its ::error annotation, and the step exited 1 with Failed pnpm versions: 0.0.0-missing. Each version's cache sandbox was created under its own $RUNNER_TEMP/tmp-<v>/socket-patch-test-caches, and nothing was written to the shared /tmp.
  • This PR's own CI (run 37384301349) is green. Every version's log group shows both suites passing:
job versions job duration
install-proof (node 10.24.1) 6 1m50s
install-proof (node 16.20.2) 10 2m43s
install-proof (node 24.11.1) 9 4m14s
  • Cost. Install-proof is billed ≈10 job-minutes, down from ≥27: 25 jobs at ≥1 min each, and the 12.x legs ~2 min. Runner slots per run drop from 26 to 4.
  • Wall-clock trade-off. When runners are idle, the run takes about 2 min longer (6m42s vs 4m42s on main run 37355008271). The node-24 job now runs the two slow 12.x hosted tests (~70 s each, the same on main) one after the other. Under contention, which is when the failures happen, needing 3 runner slots instead of 25 is the bigger win.
  • The cherry-picked fix. Locally, the utils::digest, gradle_cache, jvm_jar and sidecars lib tests pass (67/67). rustfmt --check is clean, and so is CI's cargo clippy --workspace --all-features -- -D warnings.

Where each test still runs

Nothing moved and nothing was removed. All 25 pnpm/Node pairs still run on every PR that touches the workflow's paths and on every main push, in the same workflow, now under 3 jobs.

Note for the owner

The per-version check names install-proof (<pnpm>, <node>) are replaced by install-proof (node 10.24.1), install-proof (node 16.20.2) and install-proof (node 24.11.1). If any of the old names is a required status check in branch protection, it needs updating.

Related: #892 adds PR-run concurrency to this same workflow. The two changes touch different parts of the file and are complementary: #892 drops superseded runs, and this PR shrinks each run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Uej6tnJfjRU8NCUz2jdDG4


Note

Low Risk
Changes are CI orchestration and digest helper consolidation with the same test matrix coverage; branch protection may need updated required check names for the three Node jobs.

Overview
pnpm hosted compatibility no longer fans out 25 install-proof matrix jobs (one per pinned pnpm). It runs three jobs—one per Node runtime (10.24.1, 16.20.2, 24.11.1)—that install each group’s pnpm versions under a shared setup, then loop with per-version TMPDIR, log groups, and ::error annotations. Hosted and vendored e2e (pnpm-e2e then pnpm-vendor-e2e) run in one step per version; job timeout rises to 30 minutes.

Gradle/JVM code routes SHA-1/SHA-256 hashing through crate::utils::digest (sha1_hex_of, sha256_hex_of) in gradle_cache, jvm_jar, and Maven sidecars instead of inline sha1/sha2 + hex::encode (cherry-pick to unblock CI on production_digests_go_through_the_helpers).

Reviewed by Cursor Bugbot for commit 58e901c. Configure here.


Generated by Claude Code

The pnpm install-proof matrix spawned 25 single-version jobs (one
per pnpm/Node pair) whose real work is ~20 s each. Most of each job was
runner setup, and any one leg that never got a runner left the run
red: on 2026-10-05 20/28 pnpm runs failed, every failed leg checked
being an ubuntu-latest job cancelled with no runner and no log.

Group the legs by Node runtime (10, 16, 24): each job installs its
pnpm versions, then runs both pinned suites per version in turn with
a per-version TMPDIR so the shared cache sandbox starts empty, as it
did on a fresh runner. Every pnpm/Node pair still runs on every PR
and main push; a failure is reported per version via ::error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uej6tnJfjRU8NCUz2jdDG4
@mikolalysenko Mikola Lysenko (mikolalysenko) added the ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) label Oct 5, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

main has failed socket-patch-core's lib tests since Gradle support
(#646) and the digest helpers (#865) both landed. The guard test
production_digests_go_through_the_helpers flags three files #646 added
that still hash inline: crawlers/gradle_cache.rs, patch/jvm_jar.rs and
patch/sidecars/maven.rs. That breaks test, test-release and coverage on
every open PR.

Each inline sha1/sha256 call now goes through sha1_hex_of or
sha256_hex_of, which compute the same lowercase hex. Behaviour is
unchanged.

Assisted-by: Claude Code:claude-opus-5-5
(cherry picked from commit 659ac2c)
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

coverage failed on 6d45d5d with utils::digest::tests::production_digests_go_through_the_helpers. The same failure is red on main itself (run 37355008322). It comes from a semantic conflict between #646 and #865, not from this workflow-only change.

I cherry-picked #878's fix (659ac2c, -x) as 58e901c so this PR can go green. It becomes a no-op once #878 merges.

Locally I ran the digest, gradle_cache, jvm_jar and sidecars lib tests: 67/67 pass. rustfmt --check is clean on the 3 files, and so is CI's own cargo clippy --workspace --all-features -- -D warnings.


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 58e901c. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 6, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] Ready for review at 58e901c. CI: every check on this head is green, 0 failing or pending. Bugbot reviewed 58e901c with no findings, and there are no open review threads. Reviewers should know two things. First, 58e901c is a cherry-pick of #878's digest fix, which becomes a no-op once #878 merges. Second, this PR renames the per-version pnpm install-proof jobs to per-runtime shards, so if any old job name is a required status check, the branch rules need updating.


Generated by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants