Skip to content

fix(cli): remove the steady tones from generated speech - #6327

Closed
miguel-heygen wants to merge 1 commit into
mainfrom
fix/tts-remove-model-tone-lines
Closed

miguel-heygen wants to merge 1 commit into
mainfrom
fix/tts-remove-model-tone-lines

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

Problem

Every clip from hyperframes tts carries faint steady tones at 4800, 7200 and 9600 Hz, audible as a whistle through speech and pauses.

Cause

Kokoro's final inverse STFT leaves lines at multiples of sample_rate / 10 in the raw model output (2400, 4800, 7200, 9600 Hz at 24 kHz). They are already there before any resampling, normalising or encoding, so the CLI's own steps do not add them. They follow the speech envelope: only 14 to 28% of the excess energy sits within 2 Hz of a line, 55 to 70% within 30 Hz.

Fix

After synthesis, a zero-phase triangular notch of +-150 Hz at 4800, 7200 and 9600 Hz. Narrower notches ring into pauses; wider ones start to cut speech. 2400 Hz is left alone because it sits inside the speech band and its line is too faint to matter. The script file is renamed to synth-v4.py so cached copies are replaced.

Measured (two voices, same text)

line height over the floor, before after
4800 Hz about +19 dB about -20 dB
7200 Hz about +13 dB about -21 dB
9600 Hz about +15 dB about -25 dB
  • Loudness change: -0.016 dB. Duration identical, sample for sample.
  • Pause level at the lines is back at the original floor (about -65 to -71 dB absolute against -69 to -78 before).
  • Speech band 100 to 3000 Hz energy change is under 0.01 dB.

Test

synthesize.tones.test.ts runs the notch from the shipped script on a synthetic clip (speech-like bursts and pauses, with envelope-following lines): line height drops from above 10 dB to below 0 dB, length is unchanged, speech band and loudness are unchanged, pauses stay under -60 dB. With the notch disabled it fails (line height 8.8 dB). It needs numpy: it skips without it locally and fails on Linux CI, which now installs python3-numpy in the runtime test job.

@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

Listen

Same fixture sentence (a voice with pauses). In order: before (raw model output), after (with this fix), whistle only (before minus after, boosted +30 dB).

before.mp4
after.mp4
whistle-only-plus30dB.mp4

@github-actions

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 168 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (3)

  • seqplainkeys-none-center-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true
  • seqplainkeys-none-px-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true
  • seqheldtwice-tlhold-px-r0-nested-z100: tracking 0.41, pressJump 60.41, drop 0.39, reload 0.39, render 0.02, renderKey -, undo true, teleport true / tracking 0.09, pressJump 0.11, drop 0.39, reload 0.39, render 0.02, renderKey -, undo true, teleport true / tracking 0.09, pressJump 0.11, drop 0.39, reload 0.39, render 0.02, renderKey -, undo true, teleport true

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: APPROVE · 0 blocker · 0 important · 0 nits · head d6ee1cfc

CI is red on this head, but the PR is not the cause. Test (runtime) hit its 10-minute limit after every suite had passed (cli 403 files, core 237, engine 129), and synthesize.tones.test.ts ran with 2 passing, with numpy installed. #6326 is the fix for that timeout. Merging still needs a green Test.

Scope

Read the full diff and the synthesize_text return paths; np.asarray covers the list that the split-retry path returns. numpy is already a dependency of kokoro-onnx/onnxruntime, so the new import adds no install requirement. I ran the test harness against the shipped filter with numpy 2.x. Line height went from 15/17/19 dB to −23/−23/−20 dB, pauses sit 67–68 dB below speech, the speech-band change was 4e-6 dB, the loudness change −0.075 dB, and the length was unchanged.

Cross-PR: #6326 moves the cli tests into a new Test (cli) job that has no numpy. I requested the install line there so that main stays green whichever PR lands first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant