Skip to content

The client telemetry channel has no level, so "the noise gate worked" is logged at level: "error" — 19 of the 40 newest rows are voice, and half of those are designed behaviour #571

Closed

Description

@serge-ivo

The client telemetry channel has no level, so "the noise gate worked" is logged at level: "error" — 19 of the 40 newest rows are voice, and half of those are designed behaviour

Observed

list_errors(limit: 40), production, 2026-08-15. 19 of the 40 most recent rows are client:voice*, every one at level: "error". Eight of those 19 (25 occurrences after repeat-collapse) report a discard that is the feature working:

client:voice            "voice turn rejected as noise — nothing was heard, discarded"   error  ×5
                        context: { transcript: "Thank you for watching.", … }
client:voice            "voice turn rejected as noise — the live capture was kept"      error  ×1
                        context: { transcript: "Thanks for watching!", … }
client:voice            "voice turn discarded at end-of-turn — the live gate heard no words"  error  ×11, ×3
client:voice            "voice turn dropped before sending — repetition"                error  ×2, ×1, ×1
                        context: { transcript: "Yes, yes, yes, yes, yes, … " }
client:voice            "result ignored (echo tail or a turn already abandoned)"        error  ×1
                        context: { transcript: "Thank you for watching." }

"Thank you for watching." is Whisper's well-known hallucination on silence. The gate catching it is the correct outcome. It is recorded at the same severity as Anthropic (400): Your credit balance is too low and coding run failed.

The read side documents error as the way to filter for bugs — workers/api/src/routes/errors.ts:65:

?level=error is how you ask for bugs only.

On this account that filter returns, mostly, discarded microphone frames.

Mechanism — one derivation, one premise, and the premise is false for a whole family

Verified. packages/sdk/src/client.ts:69 — the browser reporter takes no level:

export function reportClientError(source: string, message: string, context?: Record<string, unknown>, status?: number): void

Verified. The server derives it from status alone — workers/api/src/routes/errors.ts:44, with its reasoning stated at :40-43:

// Severity is derived here rather than trusted from the browser: a reported 4xx is a wall
// the user hit (diagnostic, not a bug), while a network failure or a thrown component
// carries no status and is a real error. Letting the client name its own level would make
// the field meaningless the first time a caller passed the wrong one.
level: status !== undefined && status >= 400 && status < 500 ? "warn" : "error",

Both halves of that comment are right about what they describe. The gap is the third case nobody had yet when it was written: telemetry that carries no status and is not a failure at all. The voice stack reports exclusively through this path with no status (packages/sdk/src/voice/use-voice.ts:586, voice/turn.ts:138,208,209), so every observation it makes — discard, retry, gate decision, working-as-designed rejection — lands on the error branch.

The distrust argument still holds and should be kept: a caller must not be able to downgrade its own real failure. The fix is not "trust the client".

What to do — cheapest first

1. Add a fourth level, info, and let a source opt into it. Keep the derivation server-side, keep the client unable to name a level for itself, and add an allowlist of source values whose statusless reports are observations rather than failures — voice discards, gate decisions. error_log.level already has three values (error-log.ts:143 coerces to warn | error); adding one is a column-value change, not a schema one.

2. Or split the source. client:voice-discard for designed rejections vs client:voice for real failures (TTS load failure, transcription timeout, mic denial), and derive warn for the former. Cheaper, no new level, and list_errors(source:…) already filters. Less general.

3. Whichever: keep the rows. They are load-bearing evidence for #511, #535 and #538, all of which reason from exactly this telemetry. Nothing here should reduce what is recorded — only what severity it claims.

Alternatives considered and rejected

Acceptance criteria

  • list_errors(level: "error") on an account with heavy voice use returns provider failures, crashes and run failures — not gate discards.
  • A designed voice discard is still recorded, still readable, and still carries its full context.
  • Test: a statusless report from an observation source and one from a failure source land on different levels.

Regression risk

collapseRepeat (error-log.ts:72) keys identity on source, level, status, user, message — changing the level of an existing message splits its collapse group, so a row currently at repeat_count: 11 will start a second row rather than continue. Harmless, but it will look like a spike in row count the day it deploys; #538 is already open on that same collapse key and should be read alongside.

Related: #511, #535, #538 (all reason from this telemetry — do not reduce it), #425 (the mic-refusal rows, which are genuine failures and should stay error).

Activity

  1. added
    bugSomething isn't working
    backendBackend / Worker / API work
    voiceVoice / STT / TTS
    on Aug 14, 2026
  2. serge-ivo commented on Aug 15, 2026

    @serge-ivo
    ContributorAuthor

    Landed as 4c0826b. CI green; Deploy API Worker and Deploy Host Worker both succeeded (the console bundles @proagentstore/sdk, so the client half needed the host deploy too).

    The design, and why option 1 was rejected

    Option 1 as written in this issue is unsound, and reading the call sites is what shows it: client:voice is not homogeneous. It carries designed decisions and genuine failures on the same string, so allowlisting the source name would have hidden the second list — the mirror of the bug being fixed.

    left at error (source stays client:voice) what it records
    use-voice.ts:491 AudioContext threw; the VAD is gone and the turn survives only on the max-dictation cap
    use-voice.ts:1452 String(err) from the STT layer — Whisper 400/401, timeouts
    use-voice.ts:1491 planRestartBail
    use-voice.ts:1807 transcription watchdog (#421)

    :1491 is the trap, and it is the one this issue does not mention. It reads like a designed outcome — the existing comment says bailing is correct — but convo.ts:48-86 shows what it actually records: "the microphone stopped responding", and every cause the notice names (permission revoked mid-session, another tab or app took the mic, the OS suspended it) is a real fault. I re-traced it independently before accepting the classification. It now carries a comment saying which it is.

    So option 2: the eight designed-decision sites (planTurnClose, planNoiseRejection on four paths, planSend, planClipGate, classifyResult's echo-tail ignore) report under a new source client:voice-decision; the four failures keep client:voice unchanged, so every existing query for real voice failures still returns level: "error".

    warn, not a fourth level. warn's own docstring already defines it as "a failure can be RECORDED without being counted as a bug" (#424), which is a designed discard exactly. info would have to be threaded through routes/errors.ts:65, listErrors and the MCP list_errors level enum before a row filed at it could be named by the tool the log is read with.

    The derivation stays server-side, in a pure deriveClientLevel(source, status) (lib/error-log.ts). routes/errors.ts:40-43's reasoning is preserved, not deleted — the client still supplies only a source, and the server decides what that name means exactly as it already decides what a status means. Status wins in both directions, so an observation source carrying a 5xx is still an error.

    Nothing was reduced

    The rows are still written, still readable, still carry their full context. #511, #535 and #538 all reason from this telemetry. Only the severity changes — and the source name, so an investigation of the gates now reads list_errors(source: "client:voice-decision") rather than "client:voice".

    The set is asserted, not exampled (ADR 0002)

    packages/sdk/src/voice/report-sources.test.ts sweeps the whole voice tree with a balanced-argument walker and asserts the denominator — "25 reportClientError call sites across 20 files" — then the count per source, the eight decision expressions, the four failure expressions, that the bare "voice" literal is never hand-written again, and that the SDK's name still matches the Worker's OBSERVATION_SOURCES. Watched red (G4): emptying OBSERVATION_SOURCES gives expected 'error' to be 'warn'; putting one gate discard back on the literal "voice" reds three of six tests with a voice report went back to the literal "voice": use-voice.ts:471.

    Deploy-day note (the regression risk in this issue, confirmed)

    collapseRepeat keys identity on (source, level, status, user, message, build), so every reclassified row starts a new bucket. A signature at repeat_count: 11 will appear to reset and the row count will look like it spiked. It has not — the collapse group split, once, on the deploy. Read alongside #538.

    Not verified

    The live level of a new row. POST /v1/errors/client is unauthenticated but returns only {ok:true}, and GET /v1/errors requires a session, so I could not read a row back to confirm the level from here — and I deliberately did not write a synthetic row into the production log to try, since that pollutes the log this issue is about cleaning up and still would not be readable by me. The derivation is unit-tested and the deploy is green; confirming the acceptance criterion needs one authenticated list_errors(level:"error") on a heavy-voice account. Worth doing before this is considered closed — which is why I am leaving it open.

  3. serge-ivo commented on Aug 15, 2026

    @serge-ivo
    ContributorAuthor

    What closes this — stated, so nobody re-derives it

    Verdict: ready to close on one measured voice turn. The fix is deployed on both sides and
    nothing in it is still unknown; only the live acceptance check is outstanding. The step below is
    the check, with the two ways it can read as a false negative.

    Verified today (2026-08-15)

    • Both halves are deployed. Deploy API Worker and Deploy Host Worker both succeeded on
      4c0826b (2026-08-15T00:21:53Z); the API has since redeployed at fcfbce0 and the console at
      4258494, both later than the fix.
    • The client half is live in the served bundle. /console/ inlines its JS into the HTML, and
      that HTML contains the minified VOICE_DECISION constant exactly once — Pc=\voice-decision`. Deployed console build id: **4258494** (BUILD, store/console/vite.config.ts:19, reported on every row via setClientBuild(BUILD), store/console/src/main.tsx:11`).
    • The server half is read, not exercised. deriveClientLevel (workers/api/src/lib/error-log.ts:77-80)
      falls through to OBSERVATION_SOURCES (:62, exactly {"client:voice-decision"}) only for a
      statusless report, and is wired at workers/api/src/routes/errors.ts:52.
    • Baseline re-measured, GET /v1/errors?limit=200 on the owner account: 0 rows at
      client:voice-decision. 62 client:voice* rows, all level:"error", newest 2026-08-12 at
      build cfe8ff7926ac. Nothing has used voice since the deploy — the fix is live and unexercised,
      as stated.

    The verification step

    1. Make sure the reporting bundle is the fixed one. Hard-reload the console — the JS is inlined
      into the /console/ document, so a cached document keeps the old JS and a soft reload is not
      enough. Confirm what production is serving with
      curl -s https://proagentstore.online/console/ | grep -c voice-decision → must print 1.
    2. Take ONE voice turn the gate is designed to discard. The deterministic input is to say
      "thank you": isNoiseTranscript matches the normalised entry thank you
      (packages/sdk/src/voice/audio.ts:86-88, tested at :137) → planSend returns
      {action:"drop", reason:"noise"} (voice/turn.ts:295) → voice/use-voice.ts:587 reports
      VOICE_DECISION, message voice turn dropped before sending — noise. Nothing is sent to the
      agent, so this costs one STT call and no model spend.
      Do not use "sit in silence" as the trigger. planTurnClose returns report: null for the
      ordinary quiet-mic idle (voice/turn.ts:186, and the comment above it says why), so a silent
      turn can legitimately write no row at all and would read as a failed verification.
    3. Read it back. list_errors(source: "client:voice-decision"), or
      GET /v1/errors?source=client:voice-decision.
      PASS = ≥1 row with level: "warn", build equal to the current console build (i.e. not
      cfe8ff7926ac), and context intact (transcript, path:"send", plus captureContext()).
      Two false negatives to tell apart: a row landing under client:voice at error with a
      discard message means the tab was still running a pre-fix bundle — read build to confirm;
      no row at all means either the utterance passed the gate (retry) or the reporter's 30 s
      per-source|message dedupe swallowed a repeat (packages/sdk/src/client.ts:77-80).
    4. The level:"error" half of AC 1 is not checkable through MCP as written. list_errors
      exposes only token / scope / source / limit
      (workers/mcp/src/instance-tools/observability.ts:105-110) — it has no level parameter,
      unlike agent_trace. Either read ?level=error on the REST route (routes/errors.ts:72) or
      filter the level field on the returned rows. Also expect the pre-fix rows to stay: nothing
      rewrites history, so the 2026-08-12 discard rows remain at error until the 30-day sweep. The
      claim to check is "no NEW discard row lands at error", keyed on created_at / build.

    On the other two acceptance criteria

    AC 2 (the row is still recorded with its full context) and AC 3 (a test separating an observation
    source from a failure source) are delivered by 4c0826b: workers/api/src/lib/error-log.test.ts:259-298
    pins the derivation both ways including the status-outranks-source case, and
    packages/sdk/src/voice/report-sources.test.ts pins the classification of all 25 reports across the
    voice tree so a new call site cannot land unclassified.

    Verified vs inferred

    • Verified: every file:line above; the two deploy runs; the voice-decision string in the
      served console HTML and the build id it reports; the 0-row / 62-row baseline; the absence of a
      level parameter on the MCP tool.
    • Inferred: that saying "thank you" reaches planSend rather than being discarded earlier by
      the clip gate. Either outcome writes a VOICE_DECISION row (use-voice.ts:1408 reports the clip
      gate under the same constant), so the check passes either way — but the message may read
      differently from the one quoted above.
    • Not done here: the live turn itself, and deliberately — writing a synthetic row into the log
      this issue is about cleaning up would corrupt the very evidence.

    Open question for the owner, not filed: list_errors having no level filter while its REST route
    and agent_trace both do. It is why AC 1 cannot be executed with the tool the issue names.

  4. added
    P2: correctnessReal defect, no live harm today — inert fields, miscounts, missing guards
    observabilityA displayed value the code cannot produce, or that means something other than its label
    on Aug 18, 2026
  5. serge-ivo commented on Aug 23, 2026

    @serge-ivo
    ContributorAuthor

    DELIVERED — closing. Commit 4c0826b3, verified against main and against the live error log.

    Each acceptance criterion

    AC Evidence at HEAD
    1. list_errors(level:"error") returns provider failures and crashes, not gate discards The derivation is now deriveClientLevel(source, status) — workers/api/src/lib/error-log.ts:78-81: status wins where present, and a statusless report from a declared observation source is warn. OBSERVATION_SOURCES is new Set(["client:voice-decision"]) (:63), and the SDK routes every designed decision there — packages/sdk/src/voice/report-source.ts:53 VOICE_DECISION = "voice-decision", :59 VOICE_FAILURE = "voice"
    2. a designed discard is still recorded, readable, and carries its full context error-log.ts:59-61 — "Nothing is reduced either way: the rows are still written, still readable, still carry their full context — #511, #535 and #538 all reason from this telemetry and none of it goes away. Only the severity it claims changes." Nothing in 4c0826b3 touches logError's write path
    3. a test asserts a statusless observation source and a failure source land on different levels error-log.test.ts (+69 lines in the commit) and packages/sdk/src/voice/report-sources.test.ts (197 lines), which walks the whole voice tree so a new call site cannot land unclassified — report-source.ts:38-39 states that contract

    The design took option 2 from the ticket, not option 1, and said why

    The ticket offered "add a fourth level info" or "split the source". The code took the split, with the reasoning recorded at error-log.ts:53-61: warn has meant "a failure can be RECORDED without being counted as a bug" since #424, and an info level would have to be threaded through routes/errors.ts, listErrors and the MCP list_errors enum — "a row nobody can name is a row nobody can filter for". The ticket's own rejected-alternatives list is honoured: a blanket allowlist on client:voice was explicitly declined at report-source.ts:13-28 because that source is not homogeneous.

    The ticket's related-issue note is also honoured: client:voice-gate and client:voice-control mic refusals stay error (they are #425's genuine failures), and isMicPermissionDenied (convo.ts:1048) is deliberately narrow so aborted/audio-capture contention cannot be mistaken for a denial.

    One residue, stated rather than hidden

    Historical rows keep the level they were written at — there is no backfill, and none was asked for. Read live today, GET /v1/errors?level=error&limit=60 still returns 8 client:voice gate discards; every one of them is dated 2026-08-08 → 2026-08-12, i.e. before 4c0826b3 deployed. GET /v1/errors?source=client:voice-decision returns 0 rows — voice has not been used in production since 2026-08-12, so no post-fix row exists to observe.

    So the mechanism is verified in code and the fix is unobservable in the log until voice is next used. That is a measurement gap, not an unmet criterion: the rows this ticket complained about cannot be rewritten, and they age out.

    Verified: every file:line; the two live GET /v1/errors responses quoted. Inferred: that the next voice session will file its discards as warn — that follows from deriveClientLevel and the source split, but has not been observed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2: correctnessReal defect, no live harm today — inert fields, miscounts, missing guardsbackendBackend / Worker / API workbugSomething isn't workingobservabilityA displayed value the code cannot produce, or that means something other than its labelvoiceVoice / STT / TTS

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions