Repository navigation
fix(cli): remove the steady tones from generated speech - #6327
miguel-heygen wants to merge 1 commit into
Conversation
ListenSame fixture sentence (a voice with pauses). In order: before (raw model output), after (with this fix), whistle only (before minus after, boosted +30 dB). before.mp4after.mp4whistle-only-plus30dB.mp4 |
Edit accuracy: accurate 271 (base branch 271), smooth 168 of thoseThe gate passes. Quarantined, measured but not gated (0) Unstable (3)
|
There was a problem hiding this comment.
Verdict: APPROVE · 0 blocker · 0 important · 0 nits · head d6ee1cfc
CI is red on this head, but the PR is not the cause. Test (runtime) hit its 10-minute limit after every suite had passed (cli 403 files, core 237, engine 129), and synthesize.tones.test.ts ran with 2 passing, with numpy installed. #6326 is the fix for that timeout. Merging still needs a green Test.
Scope
Read the full diff and the synthesize_text return paths; np.asarray covers the list that the split-retry path returns. numpy is already a dependency of kokoro-onnx/onnxruntime, so the new import adds no install requirement. I ran the test harness against the shipped filter with numpy 2.x. Line height went from 15/17/19 dB to −23/−23/−20 dB, pauses sit 67–68 dB below speech, the speech-band change was 4e-6 dB, the loudness change −0.075 dB, and the length was unchanged.
Cross-PR: #6326 moves the cli tests into a new Test (cli) job that has no numpy. I requested the install line there so that main stays green whichever PR lands first.
Problem
Every clip from
hyperframes ttscarries faint steady tones at 4800, 7200 and 9600 Hz, audible as a whistle through speech and pauses.Cause
Kokoro's final inverse STFT leaves lines at multiples of sample_rate / 10 in the raw model output (2400, 4800, 7200, 9600 Hz at 24 kHz). They are already there before any resampling, normalising or encoding, so the CLI's own steps do not add them. They follow the speech envelope: only 14 to 28% of the excess energy sits within 2 Hz of a line, 55 to 70% within 30 Hz.
Fix
After synthesis, a zero-phase triangular notch of +-150 Hz at 4800, 7200 and 9600 Hz. Narrower notches ring into pauses; wider ones start to cut speech. 2400 Hz is left alone because it sits inside the speech band and its line is too faint to matter. The script file is renamed to
synth-v4.pyso cached copies are replaced.Measured (two voices, same text)
Test
synthesize.tones.test.tsruns the notch from the shipped script on a synthetic clip (speech-like bursts and pauses, with envelope-following lines): line height drops from above 10 dB to below 0 dB, length is unchanged, speech band and loudness are unchanged, pauses stay under -60 dB. With the notch disabled it fails (line height 8.8 dB). It needs numpy: it skips without it locally and fails on Linux CI, which now installspython3-numpyin the runtime test job.