feat(whisper+captions): language detection, audio-reactive captions, multilingual defaults by vanceingalls · Pull Request #175 · heygen-com/hyperframes

vanceingalls · 2026-04-01T04:20:47Z

Summary

Whisper improvements:

Auto-detect language and switch from .en to multilingual model when needed
Detect speech onset in WAV to strip hallucinated words before speech begins
Merge whisper-cpp token fragments: contractions (didn + 't → didn't), split capitals (C + aught → Caught), dropped-g (shin + in' → shinin')
Interpolate zero-duration word clusters for reliable karaoke timing

Captions skill updates (folded from #176):

Rewrite script-to-style mapping as an energy detection table (high → low) with mandatory animation requirements: karaoke baseline, 2+ highlight techniques, kinetic exits
Replace tl.call() per-frame audio-reactive pattern with group-level GSAP tweens — read peak bass/treble for each group's time range and modulate entrance intensity at build time
Add transcript quality check with automatic retry rules (>20% music tokens = retry with larger model)
Add caption word structure lint rule (.caption-group + <span>) for studio editor compatibility

Multilingual defaults (folded from #186):

Default whisper model changed from small.en to small to prevent silent translation of non-English audio
Added non-negotiable language rule to captions skill

Test plan

pnpm test passes (contraction merging, fragment merging, zero-duration interpolation, speech onset)
Transcribe non-English audio — verify it transcribes in original language, not translates
Skill files render correctly, cross-references resolve
dynamic-techniques.md audio-reactive section uses tl.to()/tl.set() only, no tl.call() loops

🤖 Generated with Claude Code

vanceingalls · 2026-04-01T04:21:03Z

feat(whisper+captions): language detection, audio-reactive captions, multilingual defaults #175 : 2 dependent PRs (#176 , #180 ) 👈 (View in Graphite)
feat(skills): add dynamic caption techniques and split captions skill into references #173
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

- Auto-detect language and switch from .en to multilingual model when needed - Detect speech onset in WAV to strip hallucinated words before speech begins - Merge whisper-cpp token fragments: contractions (didn + 't → didn't), split capitals (C + aught → Caught), dropped-g (shin + in' → shinin') - Interpolate zero-duration word clusters for reliable karaoke timing - Replace non-null assertions with optional chaining in tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…y checks - Rewrite script-to-style mapping as energy detection table - Add mandatory animation design requirements (karaoke baseline, technique cycling) - Simplify dynamic-techniques.md to technique selection reference - Add transcript quality check with automatic retry rules - Add multilingual model guidance and decision tree Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Never overflow: hidden on caption containers — clips glow/scale effects - Music requires audio-reactive captions (any genre, any energy level) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The player needs window.__timelines registered immediately to start playback. GSAP timelines are mutable so tl.call() entries added later in the fetch callback still fire when seeked. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Documents the inline TRANSCRIPT requirement, JSON-quoted keys, and CSS class conventions needed for the studio caption editor to detect and edit captions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Adds two lint rules for caption compositions: - caption_transcript_not_inline: warns when transcript is loaded via fetch() instead of an inline var TRANSCRIPT array - caption_transcript_parse_error: warns when the inline TRANSCRIPT array is not valid JSON (common with unquoted keys + apostrophes) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…-level GSAP tweens

…pattern

…slation .en models (small.en, medium.en) translate non-English audio into English instead of transcribing it. This silently destroys the original language. - Add non-negotiable language rule to captions skill - Update model tables to default to models without .en suffix Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…multilingual defaults (heygen-com#175) ## Summary **Whisper improvements:** - Auto-detect language and switch from `.en` to multilingual model when needed - Detect speech onset in WAV to strip hallucinated words before speech begins - Merge whisper-cpp token fragments: contractions (`didn` + `'t` → `didn't`), split capitals (`C` + `aught` → `Caught`), dropped-g (`shin` + `in'` → `shinin'`) - Interpolate zero-duration word clusters for reliable karaoke timing **Captions skill updates (folded from heygen-com#176):** - Rewrite script-to-style mapping as an energy detection table (high → low) with mandatory animation requirements: karaoke baseline, 2+ highlight techniques, kinetic exits - Replace `tl.call()` per-frame audio-reactive pattern with group-level GSAP tweens — read peak bass/treble for each group's time range and modulate entrance intensity at build time - Add transcript quality check with automatic retry rules (>20% music tokens = retry with larger model) - Add caption word structure lint rule (`.caption-group` + `<span>`) for studio editor compatibility **Multilingual defaults (folded from heygen-com#186):** - Default whisper model changed from `small.en` to `small` to prevent silent translation of non-English audio - Added non-negotiable language rule to captions skill ## Test plan - [ ] `pnpm test` passes (contraction merging, fragment merging, zero-duration interpolation, speech onset) - [ ] Transcribe non-English audio — verify it transcribes in original language, not translates - [ ] Skill files render correctly, cross-references resolve - [ ] `dynamic-techniques.md` audio-reactive section uses `tl.to()`/`tl.set()` only, no `tl.call()` loops 🤖 Generated with [Claude Code](https://claude.com/claude-code)

vanceingalls marked this pull request as ready for review April 1, 2026 04:22

This was referenced Apr 1, 2026

fix(studio): serve runtime locally instead of from CDN #178

Closed

feat(cli): add hyperframes validate command #179

Merged

vanceingalls force-pushed the vance/whisper-improvements branch from 5360ae6 to c47472a Compare April 1, 2026 05:38

vanceingalls mentioned this pull request Apr 1, 2026

feat(studio): caption designer #180

Closed

8 tasks

miguel-heygen approved these changes Apr 1, 2026

View reviewed changes

vanceingalls force-pushed the vance/whisper-improvements branch from c47472a to 28e707c Compare April 1, 2026 22:05

vanceingalls mentioned this pull request Apr 1, 2026

fix(skills): default to non-.en whisper models to prevent silent translation #186

Closed

2 tasks

vanceingalls force-pushed the vance/dynamic-captions branch from 504b446 to afa5021 Compare April 2, 2026 05:30

vanceingalls force-pushed the vance/whisper-improvements branch from 28e707c to 4b0ce61 Compare April 2, 2026 05:30

vanceingalls mentioned this pull request Apr 2, 2026

refactor(cli): remove custom skill install, delegate to vercel-labs/skills #189

Closed

4 tasks

vanceingalls changed the base branch from vance/dynamic-captions to graphite-base/175 April 2, 2026 05:47

vanceingalls force-pushed the vance/whisper-improvements branch from 4b0ce61 to 436cd55 Compare April 2, 2026 05:57

vanceingalls changed the base branch from graphite-base/175 to vance/dynamic-captions April 2, 2026 05:57

vanceingalls changed the base branch from vance/dynamic-captions to graphite-base/175 April 2, 2026 06:24

vanceingalls force-pushed the graphite-base/175 branch from 15dd060 to 159a2e7 Compare April 2, 2026 06:24

vanceingalls force-pushed the vance/whisper-improvements branch from 436cd55 to 7085bdc Compare April 2, 2026 06:24

graphite-app Bot changed the base branch from graphite-base/175 to main April 2, 2026 06:25

vanceingalls force-pushed the vance/whisper-improvements branch from 7085bdc to 4e51632 Compare April 2, 2026 06:25

vanceingalls and others added 6 commits April 2, 2026 06:25

feat(captions): add overflow and audio-reactive constraints

747dadf

- Never overflow: hidden on caption containers — clips glow/scale effects - Music requires audio-reactive captions (any genre, any energy level) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

refactor(captions): replace tl.call audio-reactive pattern with group…

2cca04d

…-level GSAP tweens

vanceingalls and others added 2 commits April 2, 2026 06:25

docs(captions): update audio-reactive references to match simplified …

0686408

…pattern

vanceingalls changed the title ~~feat(whisper): language detection, speech onset, contraction merging~~ feat(whisper+captions): language detection, audio-reactive captions, multilingual defaults Apr 2, 2026

vanceingalls merged commit 37404f2 into main Apr 2, 2026
21 of 22 checks passed

Navigation Menu

Repository navigation

feat(whisper+captions): language detection, audio-reactive captions, multilingual defaults - #175

feat(whisper+captions): language detection, audio-reactive captions, multilingual defaults#175
vanceingalls merged 9 commits into
mainfrom
vance/whisper-improvements

vanceingalls commented Apr 1, 2026 •

edited

Loading

Uh oh!

vanceingalls commented Apr 1, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

vanceingalls commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

vanceingalls commented Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vanceingalls commented Apr 1, 2026 •

edited

Loading

vanceingalls commented Apr 1, 2026 •

edited

Loading