Skip to content

fix(producer): a stalled capture resumes at the first unwritten frame, not frame 0 - #7104

Merged
miguel-heygen merged 2 commits into
mainfrom
fix/stall-resume
Nov 3, 2026
Merged

miguel-heygen merged 2 commits into
mainfrom
fix/stall-resume

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

What

When a capture stall is recovered during a streaming render, the render now continues from the first frame not yet written to the encoder. Before, every streaming path except one restarted the whole film at frame 0 on a fresh encoder, so a stall near the end of a long film cost the whole film again, and a second stall ended the export.

Why

The streaming stage already resumed after a stall in one place: the parallel BeginFrame path, once. Everywhere else the stall error left the stage, the orchestrator forced a fresh screenshot session, and the stage ran again from frame 0.

A frame's pixels depend on its index and time, not on the frames before it, and the encoder already holds every frame it was given in order. So the stage keeps its encoder, opens a fresh capture session in the same mode, and continues at the first unwritten frame. The parallel path's "where to resume" (the count written to the encoder) is the single rule; the sequential paths now use the same one.

How

  • Parallel streaming: the resume gate no longer requires BeginFrame and no longer stops after one resume. A resume needs new frames written since the last one.
  • Sequential streaming (plain loop and the drawElement worker-encode loop): the session lifecycle runs in a loop with a resume cursor. A stall with frames written since the last resume closes the wedged session, opens a fresh one, and both loops start at the cursor. The worker-encode loop resumes at the first frame not yet drained, not the one captured but unwritten.
  • A stall before any frame was written, or a resume that lands no new frame, throws as before, so the orchestrator's existing fallback (and the end of the render) is unchanged. No endless loop: each resume needs progress.

Not changed: the disk capture path already retries only missing frames; the HDR and segmented stages are separate.

Test plan

  • Unit tests added/updated (captureStreamingStage.test.ts, 38 tests, bun test)
  • Manual testing performed: end to end on a Linux box, below
  • Documentation updated (not applicable)
  • Comments follow CONTRIBUTING.md "Comments"

fails without the fix: a sequential capture stall after frames were written > resumes a screenshot capture at the first unwritten frame, not frame 0 (and the BeginFrame, drawElement worker-encode and parallel screenshot / drawElement siblings). At the merge-base 8 tests fail and 30 pass; at this head 38 pass, 3 runs in a row. Each new test also fails when its own code is broken: sequential resume at 0, the worker-encode loop restarting at 0, resuming without new frames (sequential and parallel), parallel resume for BeginFrame only, parallel resume once only.

End to end: a 60 s, 1800-frame 1080p30 film (GSAP text plus a video layer), rendered uninterrupted and again with the whole render process tree frozen for 100 s at about frame 714, so the 60 s watchdog fires.

path uninterrupted stalled at resumed at stalled output
sequential, BeginFrame d5c169d7…7125140 713 716 same file hash
sequential, screenshot same 714 716 same file hash
parallel streaming (2 workers), screenshot same 715 716 same file hash

All six renders have one SHA-256, d5c169d758d073d686d4aab7dc40bfccc1d633d3cc086f0210441b9db7125140, so the stalled renders are byte-identical to the uninterrupted ones.

What I did NOT exercise

  • drawElement end to end (the box has no GPU path for it); it is covered by the unit tests only.
  • A stall in a render with a video layer's late frames or the drawElement risk-frame screenshot session after a resume.
  • Real sleep or a real GPU hang; the freeze is a stand-in that trips the same watchdog.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 267 (base branch 267), smooth 82 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (8)

Unstable (2)

  • seqplainkeys-none-pct-r0-nested-z100: tracking 0.05, pressJump 0, drop 0, reload 0, render 0.02, renderKey -, undo false, teleport true / tracking 0.05, pressJump 0, drop 0, reload 0, render 0.02, renderKey -, undo true, teleport true / tracking 0.05, pressJump 0, drop 0, reload 0, render 0.02, renderKey -, undo true, teleport true
  • seqab-none-pct-r0-root-z100: tracking 0.05, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.05, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.05, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE · 0 blocker · 1 important · 0 nits · head 067dbd1e

The fails without the fix line holds. With main's captureStreamingStage.ts and this head's tests, 8 tests fail and 30 pass; at head all 38 pass. The sequential path continues from the last frame written to the encoder, which is safe because the stall race wraps only the capture call, never writeFrame. The worker-encode loop resumes at the last frame it drained, and a stall's orphaned capture never reaches the encoder. One important item is inline.

Scope

Read the stage diff end-to-end, plus raceAgainstStall, stallBudgetMs and the orchestrator's stall fallback (renderOrchestrator.ts:5134). Windows render verification failed on an unrelated engine test (browserManager.test.ts resolveHeadlessShellPath, a 5 s timeout); it passed on rerun. preview-regression was cancelled at checkout and passed on rerun. CI's File size check job was cancelled even though all of its steps passed. Producer unit and integration tests, regression, Comments and CodeQL are green. No prior reviews.

return (
error instanceof SequentialCaptureStallError &&
abortSignal?.aborted !== true &&
resume.written > resume.from

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

important: resumes now have no upper limit. Each resume only needs one new frame, so a page that wedges every few frames keeps resuming in the same mode. Each resume costs a full stall budget plus a new session: 1800 frames with a stall every 10 frames is about 180 × 60 s. Before, the first stall went to the orchestrator's fresh-screenshot fallback, which may not wedge at all. Consider a cap (for example, at most 2 or 3 resumes per stage) after which the stall is thrown, in both this check and the parallel one at :935.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0338499: resumes are capped at 3 per stage (STALL_RESUMES_BEFORE_FALLBACK), in both the sequential check and the parallel one. A fourth stall throws the typed stall even when new frames landed, so the orchestrator's fresh-session fallback runs as before. New tests drive four stalls in a row in each path and expect three resumes, then the typed stall; both fail with the cap removed (38 pass, 2 fail) and pass at head (40 of 40, 3 runs). oxlint, oxfmt and tsc exit 0.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fixed in 0338499

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE · 1 resolved, 0 open, 0 new · 067dbd1e → 03384995

The resume cap is in place in both paths. With the cap removed, the 2 new tests fail; at head all 40 pass. Every check is green on this head.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 3, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 3, 2026
@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit 3dc82f1 Nov 3, 2026
91 checks passed
@miguel-heygen
miguel-heygen deleted the fix/stall-resume branch November 3, 2026 22:43
This was referenced Nov 4, 2026
SanHsien added a commit to SanHsien/hyperframes that referenced this pull request Nov 5, 2026
Ports: heygen-com#7165 (font preflight and system TTC names), heygen-com#7166 (font
shorthand var(), weight-suffixed names, linked font sheets), heygen-com#7168
(render lock and --wait, effect-scan/session watchdog without the heygen-com#7104
retry, font_face_local_only lint), heygen-com#6972, heygen-com#7183 (adapted), heygen-com#7184; docs
heygen-com#7132, heygen-com#7196; prerequisites heygen-com#6665, heygen-com#6697, heygen-com#6716 (cache-dir helper only).
Deferred heygen-com#7182, heygen-com#7176. Package size budgets follow upstream's heygen-com#7168
increments. Watermarks: commit 15b1de4, PR 7189; monitored 92.

Co-authored-by: Miguel Ángel <miguel.sierra@heygen.com>
Co-authored-by: WaterrrForever <miao.yang@heygen.com>
Co-authored-by: Ular Kimsanov <ularkimsanov7@gmail.com>
Co-authored-by: James Russo <james.russo@heygen.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant