Skip to content

fix(cli): wait for the cloud's file check as long as the file's size allows - #7357

Merged
miguel-heygen merged 3 commits into
mainfrom
fix/cloud-complete-waits-for-check
Nov 7, 2026
Merged

miguel-heygen merged 3 commits into
mainfrom
fix/cloud-complete-waits-for-check

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

What

After a big upload the CLI asks the cloud to finish it, and the cloud answers 409 "still being checked" until its file check ends. The CLI waited about 5 seconds in total for that and then failed the whole export as refused, even though the file was uploaded.

  • completeWithRetry now keeps asking until a budget is spent: 30 s plus 0.5 s per MiB of the file, up to 10 minutes, polling every 0.5 s and slowing to every 5 s.
  • When the budget is spent the error says the file was uploaded but the cloud had not finished checking it after N s, instead of reading as a refusal.
  • Any answer other than 409 still stops at once.
  • The cloud sends no Retry-After on this 409, and the generated client does not expose response headers, so the wait is the size-scaled budget alone.

Why

A 449 s upload of a large file ended as "cloud refused" after the cloud had been checking for a few seconds: the check takes longer for a bigger file, and the client gave up first.

Size

One function and its constants in upload.ts plus a new test file.

Test plan

  • On the rebased head: bun run format:check exit 0 ("All matched files use the correct format."); fallow audit --base origin/main --fail-on-issues exit 0 ("No issues in 55 changed files"); oxlint exit 0; tsc reports no error in src/cloud; vitest run src/cloud/upload: 24 passed, 3 runs in a row.
  • fails without the fix: with the budget cut to 3 s (about what main waited), the three timing tests in uploadComplete.test.ts fail (budget scaling, polling through two minutes of 409s on a big file, the give-up message for a small file).
  • fake timers, no real waits: two minutes of 409s on a 600 MiB file still returns the asset; a 1 MiB file stops after 31 s with the message; a 400 stops at once.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes at 951dbb63. There's one blocker, and it's a quick fix.

Blocker: CI is red because the new test file isn't formatted. The preview-regression run fails at Preflight on bun run format:check: oxfmt --check flags packages/cli/src/cloud/uploadComplete.test.ts. Three lines go past the line width: the stillChecking constructor, the clientAnswering return, and the clientAnswering(...Array.from(...), "ok") call. I get the same result locally, so the body's "oxfmt clean" claim doesn't hold for this file. To fix it, run npx oxfmt packages/cli/src/cloud/uploadComplete.test.ts. That's +21/−5 lines and only changes formatting.

What I checked at this head:

  • npx vitest run src/cloud/upload: 24/24 passed.
  • The "fails without the fix" claim holds. I capped completeBudgetMs at 3 s, about what main waited. The three timing tests in uploadComplete.test.ts then fail: the budget scaling test, the big-file polling test and the give-up message test. The 400 test still passes. The fix line names those tests as required.
  • tsc shows no errors in src/cloud, and oxlint is clean on both files.
  • The logic looks right. Only a 409 is retried. Each sleep is capped at the time the budget has left. The give-up error keeps the 409 as cause. It's now a plain Error, so nameRefusedFile passes it through unchanged and it no longer reads as a refusal, which is the point of the change.

Not blocking:

  • completeWithRetry doesn't get opts.signal, so its sleep ignores abort. In practice this is still bounded. Callers like fetchStems put the timeout signal on the client's fetch, so the next complete call aborts with an error that isn't a 409 and the loop exits. An abort can still add up to one 5 s poll. Passing the signal into sleep(Math.min(poll, left), signal) would remove that delay.
  • The test named "two minutes of 409s" really covers about 88 s: 20 polls at 0.5, 1, 2, 4 and then 5 s each. The assertion is still valid, but the name overstates it.

I'll approve once the formatter is run and CI is green on the new head.

Base automatically changed from fix/cloud-upload-flaky-tests to main November 7, 2026 09:39
@miguel-heygen
miguel-heygen force-pushed the fix/cloud-complete-waits-for-check branch from 951dbb6 to 498dfa2 Compare November 7, 2026 09:50

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at 498dfa29.

What changed since my last review: the PR is retargeted to main and rebased onto the merge of #7352. Compared with 951dbb63, the only changes are:

  • oxfmt formatting in uploadComplete.test.ts, which fixes the blocker from my last review;
  • completeWait is no longer exported.

The diff against main is still only upload.ts and its test.

Checks at this head:

  • oxfmt --check and oxlint are clean on src/cloud, and tsc shows no errors in src/cloud.
  • vitest run src/cloud/upload: 24/24 passed.
  • The "fails without the fix" claim still holds. With completeBudgetMs capped at 3 s, the three named timing tests fail and the 400 test still passes.
  • CI is green: CI (including Flake gate), CodeQL, preview-regression (Preflight now passes), regression, PR captures, Player perf and Comments. Windows render verification is still pending. It covers rendering, not this CLI upload path.

Two non-blocking notes from my last review still apply: the wait between retries doesn't take opts.signal, and the "two minutes" test name overstates its ~88 s.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 7, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 7, 2026
@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit f46a593 Nov 7, 2026
122 of 176 checks passed
@miguel-heygen
miguel-heygen deleted the fix/cloud-complete-waits-for-check branch November 7, 2026 11:02
SanHsien added a commit to SanHsien/hyperframes that referenced this pull request Nov 7, 2026
…, heygen-com#7360, heygen-com#7347); review to PR heygen-com#7362

Adopt heygen-com#7357 and heygen-com#7361 (upload completion waits and retries), heygen-com#7360 (Chrome
start failure names its cause in check) and heygen-com#7347 (Safari perf tooling);
release v0.9.20 not applicable. Pin open heygen-com#7362 (needs its golden LFS pointer
when adopted); record issue heygen-com#7363. Monitored 126 -> 127.

Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant