Skip to content

fix(ci): bound the gcp-cloud-run image's bun install so a hang cannot stall it - #4206

Merged
miguel-heygen merged 1 commit into
mainfrom
fix/gcp-cloud-run-docker-build-timeout
Sep 19, 2026
Merged

miguel-heygen merged 1 commit into
mainfrom
fix/gcp-cloud-run-docker-build-timeout

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

What changes

Companion to #4204: packages/gcp-cloud-run/Dockerfile:101 had the identical unguarded RUN bun install --frozen-lockfile that caused a shard in the regression workflow to hang for 20-36 minutes after bun install printed a tarball-extraction error and then produced no further output. This is a real production Cloud Run deploy image, so a reviewer should look twice.

  • The install now has a 300 second bound and one retry, same shape as the regression fix.
  • The gcp-beginframe-contract job's build step gets its own timeout-minutes: 10, inside the job's existing 15-minute cap.

Verification

  • actionlint clean.
  • Real job history (24 successful runs of gcp-beginframe-contract): build step ranges from a few seconds (cache hit) up to 4m35s observed, comfortably under the new 10-minute step cap. No genuinely cold-cache run was found in the sampled history, so a first-ever cold build's true worst case isn't directly measured.
  • No docker daemon is available to reproduce the original hang end to end.

Review

An independent review found no blocking or should-fix issues. It confirmed: the base image (node:22-bookworm-slim) matches the sibling fix exactly, so timeout is present; the fixed line sits on the actually-built beginframe-contract target, not a dead stage; and the retry doesn't mask a genuine --frozen-lockfile mismatch.

It also found three related but out-of-scope items, not fixed here: examples/k8s-jobs/Dockerfile.example has the same unguarded pattern; scripts/ci/install-workspace-dependencies.sh (used by 15 workflow steps) retries on a bad exit code but would still hang forever on this exact error-then-hang failure mode; and a Windows CI step calls bun install --frozen-lockfile directly, bypassing that script's retry entirely.

… stall it

Same shape as fix/regression-docker-build-timeout: bun install can print a tarball error and hang instead of exiting (run 35468046055).

This Dockerfile ships a real prod deploy path, so a reviewer should look twice.

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 9910a324. Ten lines, both in the right direction, and every arithmetic claim in the description checks out: the job is timeout-minutes: 15 (ci.yml:119) and the new step cap is 10, so the step fails inside the budget rather than eating it. Three non-blocking notes below; the first is the one I'd act on, and it applies to #4204 verbatim since that PR carries the identical two lines.

The one I'd fix: timeout without -k is conditional on the thing that already misbehaved

RUN timeout 300 bun install --frozen-lockfile \
    || timeout 300 bun install --frozen-lockfile

timeout sends SIGTERM and then waits. If the hung process doesn't act on SIGTERM, timeout waits forever and the bound doesn't bind. The failure you're fixing is, in your own words, a process that "can print an error and then hang instead of exiting" — a process already not following its own error path is exactly the population whose signal handling I'd least want to depend on. I have not reproduced the hang, so I can't tell you whether SIGTERM clears it; the point is that nothing here makes it not matter.

In CI that's survivable, because the new timeout-minutes: 10 is a real backstop sitting outside the container. On the production path it isn't, and that's the path in the title. docs/deploy/gcp-cloud-run.mdx:30-32 and :40-41: hyperframes cloudrun deploy "builds and pushes the renderer when needed" from this Dockerfile, on the user's machine or through Cloud Build. There is no step timeout there — timeout 300 is the only bound, and it's the conditional one.

timeout -k 30 300 … escalates to SIGKILL, which cannot be ignored or blocked. Two characters, and the bound stops being conditional in the one context that has nothing behind it.

The retry re-draws from the state that poisoned it

Both attempts run in one RUN, so they share a filesystem. When attempt 1 is killed mid-extraction, whatever it wrote — a partial node_modules, a half-written tarball in ~/.bun/install/cache — is what attempt 2 starts from. The specific fault you named, a corrupt tarball extraction, is the kind that persists rather than the kind that passes.

Whether that actually defeats the retry depends on whether bun revalidates a cached tarball against the lockfile's integrity hash before reusing it. It may well; I haven't tested it, and --frozen-lockfile gives it the hashes to do so. But nothing in the diff forces the question, and clearing the slate costs one line:

RUN timeout -k 30 300 bun install --frozen-lockfile \
    || { rm -rf node_modules ~/.bun/install/cache; \
         timeout -k 30 300 bun install --frozen-lockfile; }

Then the retry is independent by construction instead of by hoping the cache layer is honest.

The retry can't finish in the case it was added for

300 + 300 = 600s, which is exactly the step's 10-minute cap — before the LFS checkout, the base pull, the chrome-headless-shell download and fifteen COPY layers have taken anything. So if both attempts hang to the full bound, the step cap fires during attempt 2 and the retry never completes.

That's not harmful — the step cap is doing its job — but it means the two halves cover different faults than the description implies. The retry buys you a fast transient failure (a flaky registry 5xx that exits quickly); the hang is covered by the caps alone. Worth saying because "bounded to 5 minutes with one retry" reads as though the retry is part of the hang protection, and the arithmetic says it can't be.

Verified, and what the audit turned up

  • Coverage is complete for the root cause. Four Dockerfiles in the repo. Dockerfile.test is #4204, this one is here. packages/cli/src/docker/Dockerfile.render:45 installs with npm install -g, not bun, so this fault doesn't reach it. The only other bun install --frozen-lockfile is examples/k8s-jobs/Dockerfile.example:108, which nothing in CI builds — genuinely last on the list, but it is the one site left.
  • ci.yml has exactly one build-push-action, the one you capped, so there's no second uncapped image build hiding in that file.
  • --frozen-lockfile is preserved on the retry. Easy to drop when you duplicate a line, and dropping it would let the retry silently resolve a different dependency set into a production image. It's correct here.
  • Citing the run id (35468046055) in both the Dockerfile comment and the workflow comment is the right call — the next person hitting this can read the actual failure instead of trusting "flaky".
  • Action SHAs stay pinned.

One claim I could only partly check

"the longest observed build at under 5" — I pulled the last 25 ci.yml runs and this job executed once, at 211s wall clock for the whole job including checkout, buildx and the chrome download; every other run skipped it on the path filter. So your number is consistent with what I can see, but the evidence base is a single observation, because gcp_beginframe almost never triggers. That cuts both ways: a cold-cache install has rarely been measured here, and if one ever does legitimately exceed 300s, the retry turns it into a 10-minute hard failure reported as a step timeout rather than as a slow install. The headroom looks adequate to me — just don't read "observed" as "well characterised".

Scope

Audited: both changed files whole, the gcp-beginframe-contract job and its budget, every Dockerfile in the repo and every build-push-action in ci.yml, #4204's diff for the shared root cause, deploySite.ts and build.mjs (neither builds the image — the CLI path in the docs does), and the last 25 CI runs for this job's real durations. Not run: the image build itself, and I did not reproduce the bun hang.

CI at this head as I write: 37 passing, 7 skipped, 1 neutral, 6 running, nothing failed. Reporting it, not gating on it.

— Rames

@miguel-heygen
miguel-heygen merged commit 2060e13 into main Sep 19, 2026
52 checks passed
@miguel-heygen
miguel-heygen deleted the fix/gcp-cloud-run-docker-build-timeout branch September 19, 2026 22:33
miguel-heygen added a commit that referenced this pull request Sep 19, 2026
Same defect as #4204/#4206: an erroring bun install (corrupt
tarball extraction) can hang instead of exiting, eating a full
job timeout.

- install-workspace-dependencies.sh: portable bash-only watchdog
  (no GNU `timeout` on macOS/Windows runners) so its own retry
  loop can actually fire on a hang.
- k8s-jobs/Dockerfile.example: same timeout+retry as the other
  two fixed Dockerfiles (Debian base, GNU timeout available).
- ci.yml: route the bare Windows install step through the now
  guarded script instead of a raw bun install.

Nothing else in this class is left.

Claude-Session: https://claude.ai/code/session_018Ne5CfTmFV6GStSKZDf2dy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants