Skip to content

feat(preview): an unplayable video shows its own frame while its copy is made - #6589

Merged
miguel-heygen merged 5 commits into
mainfrom
feat/preview-still-frame
Nov 3, 2026
Merged

miguel-heygen merged 5 commits into
mainfrom
feat/preview-still-frame

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Builds on #6586, which is merged.

What

While the copy of a video the browser cannot play is being made (HEVC without a decoder, ProRes), the video shows its own frame at the playhead instead of black, and the existing copy-progress report now runs at Full quality for those videos, as it already did for lighter copies below Full.

  • The server cuts one JPEG of the source at the asked time (?hf-proxy=…&hf-proxy-still=<seconds>), to a tenth of a second, sized like the copy's box, with one ffmpeg -frames:v 1. It sits behind the same eligibility gate as the copy and is cached next to it, so a second ask costs nothing.
  • The runtime sets that URL as the video's poster on each ask while the video is at the playhead and has no poster of its own, and removes it when the copy has loaded. A failed copy leaves the frame up.
  • previewCopies counts the copy a file cannot play without separately from the lighter ones: it is reported at any level, and copiesReady (which the host reads for lighter copies) is unchanged.

Why

On a machine with no HEVC decoder a project of many HEVC clips showed only its captions over black until the copies were made, with nothing to say why. The copy-progress report only ran below Full quality, so at Full there was no sign of progress either.

Measured

24 separate 1080x1920 HEVC clips of 8 s, back to back, captions over them, in Studio served by the built CLI on a Linux box whose Chrome has no HEVC decoder (the same as a Windows machine without the HEVC extension). Seconds from the first clip appearing to its first picture, each run with a cold copy cache:

Run Before (the copy plays) After (the still arrives)
night 5.9 s 0.13 s
paper 8.8 s 1.8 s
paper, repeated n/a 5.3 s

The wait for the copy is the same in both, and the box was shared, so the spread follows its load: the still is a separate ffmpeg that competes for the same CPU as the copies. In every run the still came before the copy.

Before

Two seconds after the first clip appears (night, then paper): the clip is black, only its caption shows.

before, night
before, paper

After

The same moment: the clip's own frame stands in until the copy lands.

after, night
after, paper

Tests

  • mediaProxy.test.ts: the poster is set for a video at the playhead and not for an idle one, carries the source time, and is removed when the copy loads.
  • previewCopies.test.ts: the unplayable file's copy is counted at Full, a lighter one is not, and copiesReady ignores the former.
  • preview.test.ts: the route serves the JPEG behind the same gate and refuses a bad time or a request without the proxy param.
  • proxyTranscoder.test.ts: a real ffmpeg cuts a JPEG, and the same tenth of a second finds the cached one.
  • The poster test fails with the poster disabled.
  • proxyCache.test.ts: a cached still frame counts toward the cache's byte budget and is swept with the copies. It fails with the still extension left out of the cleanup's managed types; before, stills were never swept and piled up without bound.
  • mediaProxy.test.ts clears the copy-report state after each case (resetCopyReporting). Copies the file cannot play without are now reported at Full quality, and a report timer left by one case made the next case's "no diagnostic" checks depend on test order; the checks that waited for no message at all now look for a diagnostic only.

heygengenesis[bot]
heygengenesis Bot previously approved these changes Oct 27, 2026

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE · 0 blocker · 0 important · 2 nits · head 87493be9

The still route sits behind the same eligibility gate as the copy, and the poster is only set for non-light copies and cleared on load or when the copy stops being live. Two nits are inline.

Scope and checks
  • Read all 12 files. Also checked other poster readers in the runtime (sceneUrls and the asset-failure diagnostic in init.ts): both are unaffected. The retry path now also picks up failed unplayable copies at Full, through swapToProxy.
  • Ran locally on the head: core mediaProxy/previewCopies tests (66 pass); full core suite (5465 pass; the only failures are 3 studio-api files that fail on import because studio-server isn't built here, unrelated); full studio-server suite (1555 pass); resolveStill with real ffmpeg (ran, not skipped). oxlint, oxfmt and tsc are clean for core and studio-server.
  • Because the PR is stacked on #6586, the main CI workflow hasn't run. The Comments, regression, preview-regression and Player perf runs passed on this head. Required CI (including the core size budget in Build) runs after the retarget to main.
  • I couldn't check how the desktop host shows a preview-copies report at Full, because hyperframes-internal is outside my access.

const source = canonicalizeProxySource(projectDir, absoluteSourcePath);
const at = Math.max(0, Math.round(atSeconds * 10) / 10);
const key = fastCopyKey(source.sourcePath, PROXY_PARAMS_VERSION, `still@${at}`, box);
const cachePath = join(source.projectDir, CACHE_DIR_NAME, `${key}.jpg`);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: proxyCache.ts only budgets and idles out names whose extension is in PROXY_EXTENSIONS (the copy variants), so finished <key>.jpg stills are never evicted. Every cold copy of an unplayable clip leaves one behind in .transcode-cache until hyperframes clean runs. Each one is small, but a still for a source that later changes is never reclaimed. To fix, add .jpg to the swept set, or delete the still once its copy lands.

box: PreviewProxyBox | null,
): void {
const at = sourceTimeAtPlayhead(el);
if (at === undefined || stills.has(el) || el.hasAttribute("poster")) return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: stills.has(el) (and the poster you just set) pins the frame from the first ask. If someone scrubs inside the same clip while the copy is still being made, the poster keeps showing the old time instead of the frame at the playhead. That's still better than black, but it doesn't match the body's "on each ask". If you want it to follow the playhead, re-set the poster when the rounded at changes and stills.has(el) is true.

Base automatically changed from fix/proxy-playhead-first to main October 27, 2026 20:32
@miguel-heygen
miguel-heygen dismissed heygengenesis[bot]’s stale review October 27, 2026 20:32

The base branch was changed.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 267 (base branch 267), smooth 115 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (8)

@miguel-heygen
miguel-heygen force-pushed the feat/preview-still-frame branch from 87493be to e12fd3c Compare November 2, 2026 21:45

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at e12fd3c9. No blockers.

Checked:

  • Server gate: the hf-proxy-still branch runs after the existing hf-proxy checks (variant, video type, auto-proxy enabled, codec eligibility). It also 404s without a proxyVariant, and it 404s on an empty, non-finite or negative time. A request that isn't eligible never starts ffmpeg.
  • Cache key: fastCopyKey includes the source path, mtime and size plus still@<t> and the box. An edited source gets a new still, and requests for the same still share one in-flight ffmpeg run through stillsInFlight.
  • Runtime: showStill returns early when the clip isn't at the playhead, when it already set a still, or when the author set a poster. It is only wired for copies the browser can't play without (light ? undefined : …). The still is removed on loadeddata or error once the copy loads, and when the job stops being live. A failed copy keeps the still up, as the PR says.
  • previewCopies: these copies are now counted at Full quality, and the lighter ones are still dropped when the level changes. copiesReady ignores them, so the host's lighter-copy gate behaves as before.

Non-blocking notes:

  1. cleanupProxyCache only evicts names whose extension is in PROXY_EXTENSIONS (the video copy extensions). So the <key>.jpg stills are never swept or counted toward the size cap. They are small, but over time they pile up without bound. Worth a follow-up to add .jpg there.
  2. A playhead time that rounds past the end of the clip (-ss beyond the duration) makes ffmpeg write no frame. renameSync then fails and the route returns 502, which shows as a black poster, the same as before this PR. That's harmless, but every new ask runs ffmpeg again for it.
  3. This PR is typed feat, so I haven't applied the "fails without the fix:" line rule. The body does name a test that fails with the poster disabled.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at e77a5afd. No blockers.

Delta since e12fd3c9 (my earlier approval): one commit that makes the cache cleanup cover the still frames, closing my non-blocking note 1.

  • PROXY_STILL_EXTENSION = ".jpg" is now in PROXY_EXTENSIONS. The stills use the transcoder's <sha256>.jpg naming, so they pass the TRANSCODER_PROXY_NAME check too. They now count toward the size cap and are removed by the idle and oldest-first sweeps, the same as the copies. Their .tmp-…jpg leftovers already matched the temp-file pattern.
  • resolveStill builds its file name from the shared constant, so the writer and the sweep can't drift apart.
  • The new proxyCache.test.ts case fails without the change: a 6-byte still and a 6-byte copy total 12 bytes against a cap of 8. Without the change the still isn't counted, the total is 6, nothing is removed, and removed doesn't equal [oldStill].
  • If the sweep removes a still between existsSync and readFileSync in the route, the read error is caught and the request returns 502. That only means a missing poster.

The PR is still typed feat, so the "fails without the fix:" line rule doesn't apply. The test above covers the new commit anyway.

@miguel-heygen
miguel-heygen force-pushed the feat/preview-still-frame branch from e77a5af to b32b28e Compare November 3, 2026 00:07
@miguel-heygen
miguel-heygen added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit eb35629 Nov 3, 2026
98 checks passed
@miguel-heygen
miguel-heygen deleted the feat/preview-still-frame branch November 3, 2026 00:41
@jrusso1020 jrusso1020 mentioned this pull request Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant