Repository navigation
fix(ci): bound the gcp-cloud-run image's bun install so a hang cannot stall it - #4206
Conversation
… stall it Same shape as fix/regression-docker-build-timeout: bun install can print a tarball error and hang instead of exiting (run 35468046055). This Dockerfile ships a real prod deploy path, so a reviewer should look twice.
jrusso1020
left a comment
There was a problem hiding this comment.
Approving at 9910a324. Ten lines, both in the right direction, and every arithmetic claim in the description checks out: the job is timeout-minutes: 15 (ci.yml:119) and the new step cap is 10, so the step fails inside the budget rather than eating it. Three non-blocking notes below; the first is the one I'd act on, and it applies to #4204 verbatim since that PR carries the identical two lines.
The one I'd fix: timeout without -k is conditional on the thing that already misbehaved
RUN timeout 300 bun install --frozen-lockfile \
|| timeout 300 bun install --frozen-lockfiletimeout sends SIGTERM and then waits. If the hung process doesn't act on SIGTERM, timeout waits forever and the bound doesn't bind. The failure you're fixing is, in your own words, a process that "can print an error and then hang instead of exiting" — a process already not following its own error path is exactly the population whose signal handling I'd least want to depend on. I have not reproduced the hang, so I can't tell you whether SIGTERM clears it; the point is that nothing here makes it not matter.
In CI that's survivable, because the new timeout-minutes: 10 is a real backstop sitting outside the container. On the production path it isn't, and that's the path in the title. docs/deploy/gcp-cloud-run.mdx:30-32 and :40-41: hyperframes cloudrun deploy "builds and pushes the renderer when needed" from this Dockerfile, on the user's machine or through Cloud Build. There is no step timeout there — timeout 300 is the only bound, and it's the conditional one.
timeout -k 30 300 … escalates to SIGKILL, which cannot be ignored or blocked. Two characters, and the bound stops being conditional in the one context that has nothing behind it.
The retry re-draws from the state that poisoned it
Both attempts run in one RUN, so they share a filesystem. When attempt 1 is killed mid-extraction, whatever it wrote — a partial node_modules, a half-written tarball in ~/.bun/install/cache — is what attempt 2 starts from. The specific fault you named, a corrupt tarball extraction, is the kind that persists rather than the kind that passes.
Whether that actually defeats the retry depends on whether bun revalidates a cached tarball against the lockfile's integrity hash before reusing it. It may well; I haven't tested it, and --frozen-lockfile gives it the hashes to do so. But nothing in the diff forces the question, and clearing the slate costs one line:
RUN timeout -k 30 300 bun install --frozen-lockfile \
|| { rm -rf node_modules ~/.bun/install/cache; \
timeout -k 30 300 bun install --frozen-lockfile; }Then the retry is independent by construction instead of by hoping the cache layer is honest.
The retry can't finish in the case it was added for
300 + 300 = 600s, which is exactly the step's 10-minute cap — before the LFS checkout, the base pull, the chrome-headless-shell download and fifteen COPY layers have taken anything. So if both attempts hang to the full bound, the step cap fires during attempt 2 and the retry never completes.
That's not harmful — the step cap is doing its job — but it means the two halves cover different faults than the description implies. The retry buys you a fast transient failure (a flaky registry 5xx that exits quickly); the hang is covered by the caps alone. Worth saying because "bounded to 5 minutes with one retry" reads as though the retry is part of the hang protection, and the arithmetic says it can't be.
Verified, and what the audit turned up
- Coverage is complete for the root cause. Four Dockerfiles in the repo.
Dockerfile.testis #4204, this one is here.packages/cli/src/docker/Dockerfile.render:45installs withnpm install -g, not bun, so this fault doesn't reach it. The only otherbun install --frozen-lockfileisexamples/k8s-jobs/Dockerfile.example:108, which nothing in CI builds — genuinely last on the list, but it is the one site left. ci.ymlhas exactly onebuild-push-action, the one you capped, so there's no second uncapped image build hiding in that file.--frozen-lockfileis preserved on the retry. Easy to drop when you duplicate a line, and dropping it would let the retry silently resolve a different dependency set into a production image. It's correct here.- Citing the run id (
35468046055) in both the Dockerfile comment and the workflow comment is the right call — the next person hitting this can read the actual failure instead of trusting "flaky". - Action SHAs stay pinned.
One claim I could only partly check
"the longest observed build at under 5" — I pulled the last 25 ci.yml runs and this job executed once, at 211s wall clock for the whole job including checkout, buildx and the chrome download; every other run skipped it on the path filter. So your number is consistent with what I can see, but the evidence base is a single observation, because gcp_beginframe almost never triggers. That cuts both ways: a cold-cache install has rarely been measured here, and if one ever does legitimately exceed 300s, the retry turns it into a 10-minute hard failure reported as a step timeout rather than as a slow install. The headroom looks adequate to me — just don't read "observed" as "well characterised".
Scope
Audited: both changed files whole, the gcp-beginframe-contract job and its budget, every Dockerfile in the repo and every build-push-action in ci.yml, #4204's diff for the shared root cause, deploySite.ts and build.mjs (neither builds the image — the CLI path in the docs does), and the last 25 CI runs for this job's real durations. Not run: the image build itself, and I did not reproduce the bun hang.
CI at this head as I write: 37 passing, 7 skipped, 1 neutral, 6 running, nothing failed. Reporting it, not gating on it.
— Rames
Same defect as #4204/#4206: an erroring bun install (corrupt tarball extraction) can hang instead of exiting, eating a full job timeout. - install-workspace-dependencies.sh: portable bash-only watchdog (no GNU `timeout` on macOS/Windows runners) so its own retry loop can actually fire on a hang. - k8s-jobs/Dockerfile.example: same timeout+retry as the other two fixed Dockerfiles (Debian base, GNU timeout available). - ci.yml: route the bare Windows install step through the now guarded script instead of a raw bun install. Nothing else in this class is left. Claude-Session: https://claude.ai/code/session_018Ne5CfTmFV6GStSKZDf2dy
What changes
Companion to #4204:
packages/gcp-cloud-run/Dockerfile:101had the identical unguardedRUN bun install --frozen-lockfilethat caused a shard in the regression workflow to hang for 20-36 minutes afterbun installprinted a tarball-extraction error and then produced no further output. This is a real production Cloud Run deploy image, so a reviewer should look twice.gcp-beginframe-contractjob's build step gets its owntimeout-minutes: 10, inside the job's existing 15-minute cap.Verification
actionlintclean.gcp-beginframe-contract): build step ranges from a few seconds (cache hit) up to 4m35s observed, comfortably under the new 10-minute step cap. No genuinely cold-cache run was found in the sampled history, so a first-ever cold build's true worst case isn't directly measured.Review
An independent review found no blocking or should-fix issues. It confirmed: the base image (
node:22-bookworm-slim) matches the sibling fix exactly, sotimeoutis present; the fixed line sits on the actually-builtbeginframe-contracttarget, not a dead stage; and the retry doesn't mask a genuine--frozen-lockfilemismatch.It also found three related but out-of-scope items, not fixed here:
examples/k8s-jobs/Dockerfile.examplehas the same unguarded pattern;scripts/ci/install-workspace-dependencies.sh(used by 15 workflow steps) retries on a bad exit code but would still hang forever on this exact error-then-hang failure mode; and a Windows CI step callsbun install --frozen-lockfiledirectly, bypassing that script's retry entirely.