Repository navigation
Voice transcription lost when agent message arrives mid-dictation #175
Description
Bug
Speak a turn in hands-free; if an agent message arrives before your transcript lands, everything you said is silently discarded.
Root cause — found, and it is not a re-render
The original "fix ideas" pointed at React state being wiped by a re-render. That is not it: the in-progress transcript lives in pendingTextRef (a ref) and, in Whisper mode, in the MediaRecorder itself. A re-render cannot touch either.
The actual path:
- An agent message arrives →
maybeSpeakResponse(use-voice.ts:626) →speakAndResume(:600). speakAndResumesetspausedForThinkingRef.current = trueand callssttRef.current.stop()(:608-609).stop()— unlikestopDiscard()— does finalise: the clip uploads and transcribes, so the user's words survive the network round-trip.- The transcript comes back and hits
if (shouldIgnoreResult(readGuard(), Date.now())) return;(:772).shouldIgnoreResultisisEchoing(...) || s.paused(machine.ts:51), andpausedis still true because the agent is mid-sentence.
So the turn is transcribed successfully and then dropped one line before it would have been sent. Nothing logs it; the user sees their words vanish.
Why the guard exists (do not simply remove it)
That paused check is load-bearing and its comment says so: a late Whisper transcript must not "fall through and send a turn the user already abandoned" — e.g. after the mode was switched off. And the echo half stops the agent transcribing its own TTS. A naive if (!paused) removal reintroduces both.
The distinction the guard cannot currently make: audio captured BEFORE the pause began is the user's real turn; audio captured DURING the pause is echo or an abandoned turn. Same flag, two meanings.
Suggested direction
Stamp the capture, not the result. Record when the current recording STARTED (lastListenStartRef already exists, :1029) and when the pause began; a result whose capture started before the pause is the user's speech and should be delivered even if it lands late. Only results whose capture overlaps the pause are echo/abandoned.
Deliver it rather than send it blind: appending the recovered transcript to the composer, so the user can see it survived and press send, is safer than auto-sending a turn into a conversation that has since moved on.
Acceptance criteria
- Dictating in hands-free while an agent reply arrives mid-turn preserves the spoken text.
- The agent still never transcribes its own TTS (the echo half of the guard is unaffected).
- A transcript from a capture that started AFTER the pause (mode switched off, abandoned turn) is still dropped.
- Unit tests on the pure guard: capture-before-pause survives, capture-during-pause does not.
Reproduce
Hands-free, sttMode: openai. Start a long sentence, and have the agent emit any reply (e.g. a second browser tab sending a message to the same instance) before you stop speaking.
Activity
fix(voice,console): stop losing a spoken turn (#175) and stop claimin…
serge-ivo commentedon Aug 6, 2026
Fixed in 534db6d.
Root cause was exactly as the (rewritten) ticket described:
speakAndResumesetspausedand stops the recogniser,stop()finalises, the transcript comes back and dies atshouldIgnoreResultbecause the agent is still mid-sentence.The guard keeps both of its jobs. What it gained is the distinction it never had — WHEN the audio was captured:
pausedAtis stamped on the false→true EDGE of the pause, through a singlesetPausedsetter, so the flag and its timestamp cannot drift (all ~20 assignment sites now go through it).classifyResult(guard + {captureStartedAt, pausedAt})(pure,voice/machine.ts) returns accept / recover / ignore. Capture started BEFORE the pause → the user's own turn; capture started during it → echo or abandoned, dropped as before. Unknown timings (0) are never recoverable, so under-stamping degrades to the old behaviour rather than to a false positive.onRecoveredTextputs it in the composer (console Assistant + Coder Co-pilot), where it is visible, editable and sendable. Auto-sending into a conversation that has moved on would be a different bug. It still passesisNoiseTranscript, so an echo-tail fragment can't land there either.Acceptance criteria: all four covered, the last by 9 new unit tests in
machine.test.ts— 17 → 26 (capture-before-pause recovers, capture-during-pause does not, TTS echo still ignored, no-timing behaves like the old guard).fix(voice): the spoken turn lives in the thread, and pressing hands-f…
feat(voice): switch agents without touching the screen — "next" + a c…
feat(console): the composer goes below the thread, and only text mode…
feat(supervision): an agent can hand you to another agent when you as…
fix(voice): mute silences the agent without deleting what the user sa…
fix(chat): the platform performs the retry it asks for, so a failed t…
feat(voice): split command-vs-content classification into fire/park/n…