Skip to content

fix(producer): compile probes each unique media source once however many elements use it - #6441

Merged
miguel-heygen merged 2 commits into
mainfrom
fix/compile-probe-once
Oct 26, 2026
Merged

miguel-heygen merged 2 commits into
mainfrom
fix/compile-probe-once

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

What was wrong

Compiling a composition probed a media source once per element, not once per file. A source with no content identity (anything the compiler reads in place from a URL) shares only its in-flight probe, and the shared four-slot limiter queues the other probes, so none of them is in flight together and each one starts a new ffprobe. The compile then runs those probes in rounds of four, each over the network, and a long composition with many video elements of the same few files spends its time re-reading headers, so a slow link multiplies the wait by the element count.

Measured on a project of 40 sub-compositions, each with one muted <video> over 3 sources served over HTTP (the new test): 92 ffprobe runs on main, 9 here (two per remote file: one format read, one keyframe scan, plus one for each absent local path the advisory pass tried).

Fix

compileForRender creates one SourceProbes map and hands it to every pass that resolves a media duration (the page, each sub-composition, the audio/video clamp). A unique (tag, resolved source) is probed once behind the existing four-slot limiter; every element reads its own duration from that result, and the per-element checks (media type of the element, the error naming its src) still run per element. The advisory keyframe pass skips a source it already queued. assetMediaType.ts already groups by unique path (byPath), which is the pattern this follows.

Design calls:

  • Scope the result to the compile instead of caching it process-wide: a URL's content can change between renders, which is why the engine keeps no result for it.
  • The deadline stays per probe: each probe owns its 30 s, started when it gets a slot, so waiting behind others never counts, and a failure carries the element's [src=...].
  • No change to the limit of four: a bigger pool would not change how many probes a file needs.

Test

htmlCompiler.probeCount.test.ts builds the 40-element project above and counts the ffprobe processes per source through a logging shim: at most 2 per file. On main it received 24 for one source (expected at most 2); with the fix 3 runs in a row pass. The existing limiter test now uses a distinct source per sub-composition, since it relied on one file being probed eight times. htmlCompiler suites (mediaType, hiddenProbe, scriptOrder, naturalDuration, animatedGif, parity, fontEmbed, moduleScripts, main): all pass.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES · 1 blocker · 0 important · 0 nits · head 9d791823

The compiler change is correct: I reran the new test against main's htmlCompiler.ts (24 probes for one source, fails) and at this head (passes), and the mediaType, hiddenProbe and naturalDuration suites pass locally. The one blocker is that the new test runs in the wrong CI lane, so the producer unit tests fail. ↓ inline

Scope

Read all of htmlCompiler.ts diff, its 3 call sites and the advisory pass. The probe map is keyed on the absolute resolved path and scoped to one compile. The probe never rejects, so nothing goes unhandled. withSrcContext still wraps each element separately. No prior reviews. Other CI jobs were still running when I posted this.

— Claude

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 156 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (1)

  • seqplainkeys-none-pct-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 1 resolved, 0 open, 0 new · 9d79182 → fbac394

@miguel-heygen
miguel-heygen force-pushed the fix/compile-probe-once branch from fbac394 to 8979eca Compare October 26, 2026 17:26

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 0 resolved, 0 open, 0 new · fbac394 → 8979eca (a rebase: the PR patch is the same apart from the re-pinned classification sha256, which I checked against the file; CI is green)

@miguel-heygen
miguel-heygen force-pushed the fix/compile-probe-once branch from 8979eca to 2deec47 Compare October 26, 2026 18:04

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 0 resolved, 0 open, 0 new · 8979eca → 2deec47 (a rebase: same patch apart from the re-pinned classification sha256, which matches the file; main's HEIC prep in htmlCompiler.ts runs on images after the duration probes, so the two changes do not interact; CI is green)

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 26, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 26, 2026
@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 26, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 26, 2026
@miguel-heygen
miguel-heygen force-pushed the fix/compile-probe-once branch from 2deec47 to 532737b Compare October 26, 2026 20:31

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 0 resolved, 0 open, 0 new · 2deec47 → 532737b (a rebase: the source patch is the same, and the probe-count test is now listed in integration-tests.json. Main has not touched htmlCompiler.ts since the last base. All CI is green.)

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 26, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 26, 2026
@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 26, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 26, 2026
@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 26, 2026
Merged via the queue into main with commit 165744e Oct 26, 2026
78 of 89 checks passed
@miguel-heygen
miguel-heygen deleted the fix/compile-probe-once branch October 26, 2026 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants