Skip to content

Production streaming TTS + Dictation API, called as documented - #279

Merged
alexkroman merged 3 commits into
mainfrom
claude/festive-lovelace-h3ci77
Oct 22, 2026
Merged

alexkroman merged 3 commits into
mainfrom
claude/festive-lovelace-h3ci77

Conversation

@alexkroman

@alexkroman alexkroman commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Brings the voice APIs the CLI calls in line with the AssemblyAI docs and makes them work in production: streaming TTS (speak, dub, live) now runs without --sandbox, and dictate moves from the Sync STT API to the Dictation API.

Streaming TTS (docs)

  • Production host: wss://tts.us.assemblyai.com/v1/ws. speak/dub/live drop the [sandbox] help tag and sandbox-only copy, so they show in everyone's --help. session.require_available stays as a backstop for an environment with no TTS host.
  • Voices: the catalog is the 11 documented built-in voices, and each one speaks all six languages. The old 26-voice, one-language-per-voice catalog had 10 voices the service doesn't have, including juergen, the German default, which gives 3012 voice_not_found. It also had wrong languages for lola, rafael and marius. A non-built-in --voice (a UUID or a cloned-voice slug) now passes through without an aai- prefix.
  • Connect params: voice_id, language as the lowercase English name (de → german), sample_rate, and a pinned encoding=pcm_s16le.
  • Protocol: long text goes out as bounded Generate+Flush requests (≤3 in flight) on one socket, instead of a socket per chunk, which hits the new-session limit (3009). Sessions end Terminate → Termination. Unknown frames such as WordBoundaries are ignored, and the undocumented is_final is gone.
  • --list-voices: shows the built-ins plus the account's own voices from GET /v1/voices. Records are keyed by slug and paginated via next_cursor. A shared eager option on all three commands means dub --list-voices needs no --lang.
  • agent-cascade template fix: its loop waited for is_final, which the service never sends, so each sentence hung until the 2-minute idle timeout. It now ends on FlushDone.

Dictation (docs)

  • dictate now calls POST https://dictation.assemblyai.com/v1/transcribe/live (core/sync_stt.py → core/dictation.py). The config part goes first, since audio-first is a 400, and fields use the documented names: language_codes, stt_prompt, keyterms_prompt.
  • Default output is the send-ready cleanup rewrite. If the rewrite fails, it falls back to the verbatim transcript with a warning.
  • New flags: --verbatim, --instruction (llm_instruction) and --rewrite-mode fast|smart. --keyterm replaces --word-boost, which stays as a hidden deprecated alias that warns.
  • --json adds llm_response and llm_error next to the verbatim text.

Models

  • live now defaults to claude-haiku-4-5-20251001. kimi-k2.5 isn't served by the production gateway; Haiku 4.5 tied for the fastest time to first token with tools bound in my measurements.
  • Retired ids (kimi-k2.5, the gemini-*-previews) drop out of the known-models list. Every model id the CLI references is on the production gateway's model list.

Dependencies (separate commit)

Resolves all 75 open Dependabot alerts (2 critical, 40 high, 30 moderate, 3 low). All of them are in uv.lock, across 16 packages, and each is bumped to at least its first patched release: aiohttp, anyio, cryptography, httpcore2/httpx2, langgraph-sdk, langsmith, mcp, pyasn1, pydantic-settings, pyjwt, pypdf, starlette, urllib3, virtualenv, yt-dlp. The alerts are raised against main, so they close when this merges.

  • mcp stays on the patched 1.x line (1.30.0) rather than 2.x, because langchain-mcp-adapters (behind live --mcp-config) doesn't set an upper bound on it.
  • The new yt-dlp returns an int from parse_duration for whole seconds. _section_timestamp now coerces it to float so --download-sections ranges stay floats as declared.
  • AGENTS.md asks for dependency bumps in their own PR. This one is a self-contained commit (6329ebb), so it can be cherry-picked into a dedicated PR if preferred.

Verification

  • ./scripts/check.sh: All checks passed, including 100% patch coverage and the mutation gate.
  • Ran against production:
    • transcribe (file and URL), transcripts list/get, stream, llm, agent, clip, caption, eval, doctor.
    • speak: default voice, --language es --voice anna, a ~3,350-character text on one socket, a two-speaker dialogue, and a cloned-voice slug.
    • dub -l de, --list-voices, live --show-code.
    • The dictate path: cleanup, verbatim, --instruction, and JSON output.
  • Not covered:
    • The sandbox, because the available sandbox key is disabled.
    • Commands that use the browser-login session (sessions, balance, usage, limits, keys, audit) and login/onboard.
    • Microphone capture (dictate, control).

Follow-ups (not in this PR)

  • live, its --show-code script and the agent-cascade template still open a TTS socket per spoken clause. The documented pattern is one long-lived session that uses Cancel for barge-in.
  • live with a file input cuts off the agent's reply when the clip ends. This bug predates this PR and reproduces on the base commit.
  • Uploading dictation audio while still recording, and selecting the EU endpoints.

https://claude.ai/code/session_01NoQ5jA9AJQ8wPWZbqbLuNn

🤖 Generated with Claude Code

claude and others added 2 commits October 22, 2026 00:36
…oices

The TTS gateway is retiring the bare `voice=<name>` query parameter in favour
of `voice_id` (preset jane is `aai-jane`, library voices `aai-<name>`). Once
that release reaches sandbox000, `assembly speak` / `assembly dub` — and the
`live` cascade, the `--show-code` cascade script, and the `agent-cascade`
init template, all of which build the same socket URL — would fail with a
3006 invalid_parameter.

- `tts/session.py`: `SpeakConfig.query_params` now encodes the voice as
  `voice_id=aai-<name>` via the new `tts.voices.voice_id` (idempotent, so a
  copied `aai-jane` works too); the one place the param is built, so speak,
  dub, and live move together. `voice_name` strips the prefix so `live
  --voice aai-jane` still validates against the catalog.
- `code_gen/agent_cascade.py` and the `agent_cascade` template send
  `voice_id` as well (the template's knob is renamed `VOICE_ID = "aai-jane"`).
- New `tts/voice_list.py` fetches `GET /v1/voices` from the TTS host (raw-key
  auth like the socket; auth/HTTP/shape failures normalized to CLIErrors) and
  renders one line per voice. `--list-voices` on `speak`, `dub`, and `live`
  prints it (`--json` dumps the raw records). The flag is one shared eager
  option (`options.list_voices_option`) that exits in its callback, so
  `dub --list-voices` needs no `--lang`; `live --list-voices` therefore now
  needs `--sandbox` + a key instead of printing the static catalog.
- `--tts-config voice_id=…` is reserved like `voice=` (it would be silently
  overridden by `--voice`).
- The static `tts/voices.py` catalog keeps driving language defaults,
  rotation, validation, and completion.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NoQ5jA9AJQ8wPWZbqbLuNn
Streaming TTS is now in production, and both voice APIs are called exactly
as documented.

Streaming TTS (https://www.assemblyai.com/docs/tts):
- Production gets a TTS host, wss://tts.us.assemblyai.com/v1/ws, so speak,
  dub, and live run without --sandbox (the [sandbox] help tag and the
  sandbox-only copy are gone).
- The catalog is the 11 documented built-in voices, each of which speaks all
  six languages; the old 26-voice single-language catalog listed 10 voices the
  service doesn't have (incl. juergen, the German default) and mislabeled
  others. `language` is sent as the lowercase English name (`de` -> `german`)
  and `encoding=pcm_s16le` is pinned. A non-built-in --voice (UUID, cloned
  voice slug) passes through unprefixed.
- Long text goes out as bounded Generate+Flush requests (<=3 in flight) on one
  socket instead of a socket per chunk (which hits 3009), and sessions end
  Terminate -> Termination. Unknown frames are ignored; is_final is gone.
- --list-voices shows the built-ins plus the account's own voices from
  GET /v1/voices (records keyed by slug, paginated via next_cursor).
- The agent-cascade template stopped waiting on is_final (the service never
  sends it), which hung each sentence until the idle timeout; it now ends on
  FlushDone.

Dictation (https://www.assemblyai.com/docs/dictation):
- `assembly dictate` moves from the Sync STT API to the Dictation API:
  POST dictation.assemblyai.com/v1/transcribe/live with the config part first
  (audio-first is a 400), language_codes/stt_prompt/keyterms_prompt.
- It prints the send-ready cleanup rewrite by default (falling back to the
  verbatim transcript, with a warning, when the rewrite fails); --verbatim,
  --instruction, and --rewrite-mode are new; --keyterm replaces --word-boost
  (kept as a hidden deprecated alias). --json adds llm_response/llm_error.

Models: live defaults to claude-haiku-4-5-20251001 (kimi-k2.5 is not served by
the production gateway); retired ids drop out of the known-models list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread aai_cli/tts/session.py
pending = list(texts)
in_flight = 0
while pending and in_flight < _MAX_IN_FLIGHT:
_send_request(ws, pending.pop(0))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing each request with pending.pop(0) shifts all remaining chunks, causing quadratic work for long inputs. Use an index or deque to consume chunks without repeated shifting.

Details

✨ AI Reasoning
​Long inputs are split into a number of chunks that grows with input size. Each request removes the first pending item, which shifts all remaining items; repeated removals therefore add avoidable work as the document grows.

🔧 How do I fix it?
Move constant work outside loops. Use StringBuilder instead of string concatenation in loops. Cache compiled regex patterns. Use hash-based lookups instead of nested loops. Batch database operations instead of N+1 queries.

Reply on this comment to give feedback or ignore the issue.
More info

@alexkroman alexkroman Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed — the fix (collections.deque + popleft(), O(1) per chunk) is ready, but this PR entered the merge queue before it could be pushed, so it will land as a small follow-up rather than in this PR.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Ignored this finding.

@alexkroman alexkroman changed the title Add live voice listing for streaming-TTS commands Production streaming TTS + Dictation API, called as documented Oct 22, 2026
Upgrades only the 16 locked packages with open alerts, each to (at least)
its first patched release: aiohttp 3.14.5, anyio 4.14.2, cryptography 50.0.2,
httpcore2/httpx2 2.13.1, langgraph-sdk 0.4.7, langsmith 0.14.9, mcp 1.30.0,
pyasn1 0.6.4, pydantic-settings 2.16.0, pyjwt 2.15.1, pypdf 6.20.1,
starlette 1.8.0, urllib3 2.8.0, virtualenv 21.14.5, yt-dlp 2026.8.19.

mcp is held on the patched 1.x line (1.30.0) rather than jumping to 2.x:
langchain-mcp-adapters, which backs `live --mcp-config`, doesn't bound it.

yt-dlp's parse_duration now returns an int for whole seconds, so
_section_timestamp coerces to float to keep its declared contract (and the
`--show-code` download_range_func rendering) stable across versions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexkroman
alexkroman added this pull request to the merge queue Oct 22, 2026
Merged via the queue into main with commit ed26807 Oct 22, 2026
20 checks passed
@alexkroman
alexkroman deleted the claude/festive-lovelace-h3ci77 branch October 22, 2026 20:24
github-merge-queue Bot pushed a commit that referenced this pull request Oct 22, 2026
Follow-up to #279, addressing [Aikido's review
comment](#279 (comment)).
#279 entered the merge queue before this could be pushed.

The multi-request TTS loop in `tts/session.py` took chunks with
`pending.pop(0)`, which shifts every remaining chunk, so a long
`--url`/PDF text did quadratic work queueing its requests. `pending` is
now a `collections.deque` consumed with `popleft()`, which is O(1) per
chunk. Behavior is unchanged: still at most 3 requests in flight, sent
in order.

`./scripts/check.sh`: **All checks passed**. The existing multi-request
tests in `tests/test_tts_session_multi.py` cover the ordering and
in-flight bound.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Alex Kroman <alex@assemblyai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants