Skip to content

[bug] Silence is transcribed as the agent's own name — proper nouns in the Whisper bias list come back verbatim #332

Closed

Description

@serge-ivo

Companion to #331. That ticket explains why the microphone was still open; this one explains what
the silence turned into.

What happened

Coder Lead, 2026-08-06. Two user messages, both carrying an audioKey, both content "Coder
Lead"
— 23:51:51 and 23:52:10. The user did not say it either time.

Immediately before, "Stop-stop." had failed to stop anything (#331), so the user had disengaged
while the mic was still live. Hands-free recorded silence and auto-submitted it.

Cause: the agent's own name is in the transcription bias list

store/console/src/pages/InstanceDetail.tsx:218:

transcribePrompt: buildTranscribePrompt(surfaces, instance?.name ? [instance.name] : []),

buildTranscribePrompt appends extra to the vocabulary list handed to Whisper. So for this
instance the prompt literally contains "Coder Lead" — and on silence, the decoder returns a
term from the list it was given. It is not imagining the user; it is echoing our own word back.

This is the unfinished half of an earlier fix

365475c fixed the fluent version of this: the prompt used to open "The speaker is talking to an
AI assistant about their work. Expect terms like: …"
, and silence came back as whole sentences
built from the term list — "I just need to refactor this function before I commit the changes to
the repo."
Making it a bare comma-separated list removed the grammar that produced prose.

It did not remove the words. A bare list still supplies candidates, and a proper noun is the
most likely thing to surface
— it is the only distinctive token in a list otherwise full of
generic vocabulary, and Whisper prefers it precisely because the prompt marked it as expected.

So the failure got quieter, not fixed: "I just need to refactor this function" became "Coder Lead".

Why not simply drop the bias

Vocabulary bias earns its place — it is what stops a developer's "bugs" being transcribed as
"bars", which is the reason buildTranscribePrompt exists. Removing it trades one wrong
transcription for another.

Better options, roughly in order of preference:

  1. Gate the bias on the speech gate. hadSpeech(peakLevel) already exists and already refuses
    to upload a recording with no speech. If that gate is doing its job, silence never reaches
    Whisper and the bias is harmless. Worth checking why these two clips got through — a whole-room
    silence should have been discarded before upload, and if it was not, that is the more valuable
    bug.
  2. Keep proper nouns out of the list. Agent names, repo names and usernames are exactly the
    tokens that come back verbatim from noise, and they are the least useful for disambiguating
    ordinary speech. Generic domain vocabulary ("refactor", "commit") is where the value is.
  3. Reject a transcript that is exactly a bias term and nothing else. Cheap, targeted, and
    isNoiseTranscript is already the place for it — it currently matches a fixed list of classic
    hallucinations ("you", "thank you") and would extend naturally.

(1) is the real fix; (2) and (3) are defence in depth for when it leaks.

Note

#319 (persist the dictation) would have made this self-evident: the bubble would have shown nothing
captured, then "Coder Lead" appearing from the final pass. Right now the two are indistinguishable
from the outside, which is why it reads as the agent imagining things.

Verification

  • A silent recording in hands-free produces no message at all.
  • A transcript identical to a single bias term is discarded.
  • "bugs" is still transcribed as "bugs", not "bars" — the bias still works for real speech.

Activity

  1. serge-ivo commented on Aug 7, 2026

    @serge-ivo
    ContributorAuthor

    Fixed in 3d7e28b — both (1) and (3); (2) deliberately not taken.

    Why the existing defences didn't catch it. Working backwards through the layers:

    • isTooShortToTranscribe — a 250ms / 512-byte floor. A couple of seconds of room tone clears it easily; it only ever caught header-only captures.
    • hadSpeech(peakLevel) and the VAD's seen flag both key off VOICE_FLOOR = 0.1. vadStep only returns "end" once seen is true, so the fact a turn ended AT ALL proves the amplitude layers heard something above the floor — a cough, typing, a door. Both correctly said "there was sound". Neither claims it was speech; that is the dictation gate's job.
    • The dictation gate is the layer written for exactly this case, and it is the one that failed. endOfTurnAction discards when a proven-alive gate heard no words — but gate.heardSpeech() latched on ANY recognised words, with no reference to the echo/paused/muted guards the same gate's interim path applies one function away. So the agent's own TTS coming back through the speakers marked the turn legitimate. The gate vouched for a turn the user never took, the clip was uploaded, and the decoder returned a term from the vocabulary prompt we handed it.
    • isNoiseTranscript then passed "Coder Lead": nine letters, not in the fixed hallucination set.

    (1) The real fix. createSpeechGate takes an acceptSpeech predicate, and the hook passes the predicate it already computes for the live transcript (!muted && !shouldIgnoreResult(guard, now)). The veto and the display now agree about whose voice they are hearing.

    Honest limit: the guard is evaluated when the recognizer delivers a result, and the echo tail is 800ms. Words the agent spoke that arrive later than that still slip through. It closes the window rather than sealing it; a text-level comparison against the last TTS utterance would seal it and is worth a follow-up if this recurs.

    (3) Defence in depth. isNoiseTranscript(text, biasPrompt) now rejects a transcript that is exactly one multi-word bias term. Single words are left alone on purpose: "commit" / "deploy" / "refactor" said alone are real speech, and this filter's whole contract is that short commands survive — dropping one would trade a phantom turn for a swallowed one, the same bug wearing the other hat. A multi-word term echoed back alone carries no instruction, so asking for it again costs nothing.

    (2) Not taken. Dropping proper nouns from the list would neuter the console's buildTranscribePrompt(surfaces, [instance.name]) call silently from inside the SDK, and (3) catches the same class — "Coder Lead" included — without giving up repo/agent-name bias for real speech. The console lane was another agent's this run and needs no change.

    Tests use the literal incident transcript ("Coder Lead"), plus the inverse: "ask Coder Lead to look at this" and "Coder Lead, what's the status?" still send.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions