Skip to content

feat(cli): check catches audio the render would lose or play as silence - #6439

Merged
miguel-heygen merged 4 commits into
mainfrom
feat/check-audio
Oct 26, 2026
Merged

miguel-heygen merged 4 commits into
mainfrom
feat/check-audio

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

What

hyperframes check had no audio section, so a film whose sound the render drops, refuses or plays as silence passed with 0 errors (#2775). It now reads every clip the render mixes, from the same media list the render builds, and reports:

Code Severity When What render does (checked on a real render)
audio_source_rejected error the clip's file has no audio track stops at compile: "composition asset(s) do not match their authored media element type (expected: audio)"
audio_source_unreadable error the file does not decode as audio the audio mix fails and the render stops: "Audio mix failed; output would be video-only"
audio_starts_past_source_end warning data-media-start is at or past the end of the file the clip plays nothing
audio_clip_silent warning the whole part of the file the clip plays is silent the clip plays silence
audio_none info no clip in the film plays audio (an unmuted video with an audio track counts, as in render) a film with no sound

Findings carry the clip's #id, the composition file it was written in, and its time on the film. A 3-scene film with one defect per scene, on main and on this branch:

main:   ◇  Check passed   (exit 0)

branch:
Audio
  ⚠ audio_clip_silent: audio/vo-0.wav is silent for the whole part this clip plays (0-3.5 s of the file).
    compositions/s0.html #s0-vo t=0.5s
  ✗ audio_source_rejected: audio/vo-1.mp4 has no audio track, so render stops before it starts.
    compositions/s1.html #s1-vo t=5.5s
  ⚠ audio_starts_past_source_end: audio/vo-2.wav is 3.5 s long but the clip starts 9 s into it, so it plays nothing.
    compositions/s2.html #s2-vo t=10.5s
  1 error(s), 2 warning(s), 0 info(s)
◇  Check failed   (exit 1)

Design

  • Options: (a) metadata checks only by default, the silence decode behind a flag; (b) decode every clip in full by default; (c) by default, sample a long clip at three 1 s spots and stop at the first one with sound, reading the whole played part only when every spot is quiet, while the browser pass runs.
  • Picked (c): a silent clip is the defect Audio pipeline: reused narration silently dropped + voice-clip seam collisions — both pass every check #2775 describes and a flag nobody passes would not catch it; a full decode of a 6-minute bed cost about 0.4 s on a CPU-only box, and sampling brings a clip with sound down to one short decode. A clip is only called silent after its whole played part is read.
  • Not added: sample-rate or channel checks (render resamples every source), loudness targets (taste, and hyperframes audio reports them), overlap across tracks (music under a voice is normal; same-track overlap is already lint's duplicate_audio_track).

One owner per decision: the bundle and render media list are built once and shared by the HDR inspection and the audio pass (the HDR inspection now takes that list instead of bundling its own); the media-type rule is render's own mediaProfileMatchesElementType, now exported from the producer; the probe is the engine's cached extractAudioMetadata that render uses; the decode and silence measure are the ones hyperframes audio uses.

Timing

check --json wall time, built CLI, CPU-only Linux VM (shared, so each main/branch pair runs back to back after the load drops), 3-composition 15 s film and a 6-scene 6-minute film, each with a music bed and one voice per scene.

Fixture main median branch median delta median paired delta budget (max of +10%, +150 ms)
3 compositions, 15 s 1948 ms 1983 ms +35 ms (+1.8%) +38 ms 195 ms
6 scenes, 6 min 3755 ms 3808 ms +53 ms (+1.4%) +26 ms 376 ms

9 pairs per fixture, every run exit 0, branch reports carry an audio section with no findings (the fixtures' clips all sound). A first version that decoded every clip in full measured +695 ms (+11.5%) on the 6-minute film, over budget; that is what the spot sampling fixes.

Tests

  • checkAudio.test.ts (10): real WAV and PNG files through the real probe, render's media-type rule, decoder and silence measure: a sounding clip passes; rejected source; unreadable source; start past the end; silent played part (only that clip, only the part it plays, playback rate respected); a long clip is only called silent after a full read; composition file named; video sound counted.
  • check.test.ts (+3): an audio error fails the check and lands in its own section; audio is still checked when lint errors skip the browser; an audio pass that throws becomes an info finding, not a failed check.
  • Red on main: the 3 new pipeline tests fail and checkAudio.test.ts cannot import. One deliberate break per check turns exactly its test red. 3 green runs in a row.

Fixes #2775

check gains an Audio section, read from the media list the render mixes:
- error: a clip's source has no audio the render can read (its mix fails,
  the film ships without sound)
- warning: a clip starts past the end of its file, or the part it plays
  is silent
- info: the film has no sound at all

The bundle and media list are now built once and shared with the HDR
inspection; the audio pass runs while the browser pass does.

Fixes #2775
…dles it

A source with no audio track is refused by render's own media-type rule, now exported from the producer and called by check; a source that does not decode as audio fails the mix and stops the render.
…l of it for silence

Sound at any spot ends the decode; a clip is only called silent after its whole played part is read.
@mintlify

mintlify Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
hyperframes 🟢 Ready View Preview Oct 26, 2026, 4:14 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 166 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (1)

  • seqplainkeys-none-px-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE · 0 blocker · 0 important · 2 nits · head faf489da

  • nit — Comments (not required) fails: the comment share in checkFindings.ts rose from 2/42 to 3/61 because findingAtRoot moved there with its doc line.
  • nit — checkPipeline.ts inspectProjectMedia: if the browser pass throws, the outer catch returns without awaiting audio. The probes and decodes keep running after the failure report until they finish. That's harmless (the promise has a .catch), but it can hold the process open for a moment.
Scope

Read all 13 files and checked the claims against render. compileStage preflights compiled.audios, video -audio tracks included, with mediaProfileMatchesElementType("audio"), so audio_source_rejected matches render. An audio_processing_failed warning fails the render (renderOrchestrator.ts:838), which matches audio_source_unreadable. Locally: cli check, checkAudio, hdrPromotion and checkPipeline tests pass 200/200. I ran a built-CLI check on a fixture: a silent clip and a start past the file's end were flagged; a clip with sound only near its end, and a sounding clip, were not. CI is green apart from Comments.

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE — 0 resolved, 0 open, 0 new · faf489d → 28a0e54. This head only drops one comment line, and that fixes Comments. All CI is green on this head.

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 26, 2026
SanHsien added a commit to SanHsien/hyperframes that referenced this pull request Oct 26, 2026
Merge 2 more upstream commits (7d6c02f..acd0bc3: heygen-com#5366, heygen-com#3943).
Regenerate skills-manifest.json for the media-use hash conflict. Record
the 10 open PRs heygen-com#6430-heygen-com#6439 in docs/DECISIONS.md (none adopted; heygen-com#6436
cannot be copied because the fork wraps authored scripts in Function())
and advance the commit watermark to acd0bc3 and the PR watermark to
heygen-com#6439.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Merged via the queue into main with commit d83bf99 Oct 26, 2026
80 checks passed
@miguel-heygen
miguel-heygen deleted the feat/check-audio branch October 26, 2026 17:31
SanHsien added a commit to SanHsien/hyperframes that referenced this pull request Oct 26, 2026
Merge 20 more upstream commits (acd0bc3..ad4d3fa, v0.8.205: heygen-com#6403,
heygen-com#6363, heygen-com#6209, heygen-com#6395, heygen-com#6424, heygen-com#6426, heygen-com#6442, heygen-com#6430, heygen-com#6402, heygen-com#6435, heygen-com#6439,
heygen-com#6449, heygen-com#5436, heygen-com#6455 and docs). No conflicts. Record the 11 open PRs
heygen-com#6440-heygen-com#6458 and issue heygen-com#6443 in docs/DECISIONS.md (none adopted), re-check
the earlier open PRs, and advance the commit watermark to ad4d3fa, the
PR watermark to heygen-com#6458 and the issue watermark to heygen-com#6443.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Audio pipeline: reused narration silently dropped + voice-clip seam collisions — both pass every check

1 participant