Production streaming TTS + Dictation API, called as documented

README.md

@@ -46,13 +46,13 @@ That's it. Run `assembly onboard` for a guided tour, or see [Installation](#-ins
 | :--- | :--- |
 | `assembly transcribe` | Transcribe files, URLs, YouTube/podcast pages, podcast RSS feeds, directories, globs, or bucket storage (`s3://`, `gs://`, `az://`) — with speaker labels, PII redaction, summarization, SRT/VTT captions, and resumable batch runs |
 | `assembly stream` | Real-time transcription from your microphone, a file, or a URL — on macOS it can capture system audio too |
-| `assembly dictate` | Signal-driven dictation: records immediately, send SIGTERM for instant text — scriptable from hotkey tools like Hammerspoon (Sync STT API, up to 120 s per utterance) |
+| `assembly dictate` | Signal-driven dictation: records immediately, send SIGTERM for instant text — scriptable from hotkey tools like Hammerspoon; prints send-ready cleaned-up text (Dictation API, up to 120 s per utterance) |
 | `assembly agent` | Full-duplex spoken conversation with a voice agent, right in your terminal |
-| `assembly live` | Talk live to a tool-using voice agent, wired client-side from Streaming STT + a deepagents brain on the LLM Gateway + streaming TTS — it can web-search, fetch URLs, and read the docs mid-conversation, like the `agent-cascade` starter (sandbox-only) |
-| `assembly speak` | Synthesize text to speech over the streaming-TTS WebSocket (sandbox-only) |
+| `assembly live` | Talk live to a tool-using voice agent, wired client-side from Streaming STT + a deepagents brain on the LLM Gateway + streaming TTS — it can web-search, fetch URLs, and read the docs mid-conversation, like the `agent-cascade` starter |
+| `assembly speak` | Synthesize text to speech over the streaming-TTS WebSocket |
 | `assembly llm` | Prompt the LLM Gateway over a transcript, files, stdin, or a live stream |
 | `assembly clip` | Cut audio/video with ffmpeg by diarized speaker, text match, LLM pick, or time range (`--video` keeps the picture for URL sources) — clip boundaries snap into nearby silence |
-| `assembly dub` | Re-voice an audio/video file or URL in another language: transcription, LLM translation, per-speaker TTS, ffmpeg track-swap (sandbox-only) |
+| `assembly dub` | Re-voice an audio/video file or URL in another language: transcription, LLM translation, per-speaker TTS, ffmpeg track-swap |
 | `assembly caption` | Burn always-visible captions into a video: transcribe (or reuse a transcript), fetch SRT, ffmpeg burns it in — audio untouched |
 | `assembly eval` | Benchmark WER against Hugging Face datasets (built-in aliases: `librispeech`, `tedlium`, …) or local manifests |
 | `assembly webhooks listen` | Open a public dev URL that prints webhook deliveries and can forward them to your local app |
@@ -68,22 +68,20 @@ Add `--show-code` to `transcribe` / `stream` / `agent` / `live` to print the equ
 
 A few one-liners that show what `assembly` can do. The everyday basics live under [Getting started](#-getting-started) below.
 
-> [!NOTE]
-> `speak` and `dub` are sandbox-only today — that's why the examples below pass `--sandbox`.
-
 **Recreate a scene with synthetic voices** — transcribe and diarize a YouTube clip, then pipe it straight into TTS with a different voice per speaker:
 
 ```sh
 assembly transcribe "https://www.youtube.com/watch?v=awmCtXzFsJo" --speaker-labels \
-  | assembly --sandbox speak --voice A=jane --voice B=mary --out scene.wav
+  | assembly speak --voice A=jane --voice B=mary --out scene.wav
 ```
 
 `speak` auto-detects `Speaker A:` labels, merges each speaker's turns, and rotates voices.
+`assembly speak --list-voices` prints the voices the TTS service offers.
 
 **Dub a video into another language** — the whole platform in one command: transcription with utterance timestamps, per-utterance LLM translation, TTS for each line (one voice per speaker), and ffmpeg laying the new track over the original video. A great demo is the first YouTube video ever, "Me at the zoo" — it's 19 seconds long, a single clear English speaker, and instantly recognizable, so the dub finishes fast and the before/after is obvious:
 
 ```sh
-assembly --sandbox dub "https://www.youtube.com/watch?v=jNQXAC9IVRw" -l de --video
+assembly dub "https://www.youtube.com/watch?v=jNQXAC9IVRw" -l de --video
 ```
 
 The video stream is copied untouched; each dubbed line lands at its original start time.
@@ -92,7 +90,7 @@ The video stream is copied untouched; each dubbed line lands at its original sta
 
 ```sh
 assembly transcribe "https://podcasts.apple.com/us/podcast/id1516093381" --speaker-labels \
-  | assembly --sandbox speak --out episode.wav
+  | assembly speak --out episode.wav
 ```
 
 **Cut the highlight reel from a speech** — `clip` downloads the video (`--video`; omit it for audio-only clips), transcribes it, has an LLM pick the windows, and cuts each one into its own file with ffmpeg (here: Steve Jobs' Stanford commencement address):

aai_cli/agent_cascade/__init__.py

@@ -4,8 +4,7 @@
 `assembly agent`, but where `agent` talks to AssemblyAI's single Voice Agent
 endpoint, this slice wires the three primitives together itself — Streaming STT
 -> the LLM Gateway -> streaming TTS — exactly like the ``agent-cascade``
-``assembly init`` template does server-side. Because it uses streaming TTS it is
-sandbox-only.
+``assembly init`` template does server-side.
 
 `engine.run_cascade` is the orchestrator; it takes injected dependencies
 (`CascadeDeps`) so tests drive the whole cascade against fakes, the same seam

aai_cli/agent_cascade/config.py

@@ -15,8 +15,10 @@
 
 # `assembly live` defaults to a fast, low-latency gateway model (override with --model) —
 # a literal rather than llm.DEFAULT_MODEL so the live agent's default is independent of the
-# one-shot `assembly llm` default. Latency matters most for a spoken back-and-forth.
-DEFAULT_MODEL = "kimi-k2.5"
+# one-shot `assembly llm` default (they coincide today). Latency matters most for a spoken
+# back-and-forth: Haiku 4.5 is among the fastest production models to first token with
+# tools bound, and it must be one the production gateway serves.
+DEFAULT_MODEL = "claude-haiku-4-5-20251001"
 DEFAULT_MAX_TOKENS = llm.DEFAULT_MAX_TOKENS
 # The realtime model the cascade transcribes with (same as the agent-cascade template).
 DEFAULT_SPEECH_MODEL = "universal-3-5-pro"

aai_cli/agent_cascade/voices.py

@@ -3,39 +3,21 @@
 The cascade's audio comes from streaming TTS, so its voices are the TTS catalog
 (`aai_cli.tts.voices`) — not the Voice Agent voices `assembly agent` uses. This
 module is the thin presentation layer over that catalog: the membership list
-that catches a typo'd ``--voice``, the completion callback, and the grouped
-``--list-voices`` rendering.
+that catches a typo'd ``--voice`` and the completion callback. (``--list-voices``
+itself is the service's live listing — see ``app.tts_voices``.)
 """
 
 from __future__ import annotations
 
 from aai_cli.core import choices
 from aai_cli.tts import voices as tts_voices
 
-DEFAULT_VOICE = "jane"
+DEFAULT_VOICE = tts_voices.DEFAULT_VOICE
 
-# The selectable voice ids, sorted for a stable --list-voices / completion order.
-VOICE_NAMES: list[str] = sorted(tts_voices.VOICE_LANGUAGES)
-
-# ISO 639-1 code -> the heading --list-voices groups that language's voices under.
-_LANGUAGE_LABELS: dict[str, str] = {
-    "en": "English",
-    "fr": "French",
-    "it": "Italian",
-    "de": "German",
-    "es": "Spanish",
-    "pt": "Portuguese",
-}
+# The selectable voice names, sorted for a stable completion order.
+VOICE_NAMES: list[str] = sorted(tts_voices.VOICES)
 
 
 def complete_voice(incomplete: str) -> list[str]:
-    """Shell-completion callback for ``--voice``: catalog ids matching the prefix."""
+    """Shell-completion callback for ``--voice``: catalog names matching the prefix."""
     return choices.complete_prefix(VOICE_NAMES, incomplete)
-
-
-def format_voice_list() -> str:
-    """Human-readable voice ids for ``--list-voices``, grouped by language."""
-    return choices.render_grouped(
-        (label, [name for name in VOICE_NAMES if tts_voices.VOICE_LANGUAGES[name] == code])
-        for code, label in _LANGUAGE_LABELS.items()
-    )

aai_cli/app/init_exec.py

@@ -106,9 +106,7 @@ def _active_env_vars() -> dict[str, str]:
         # The environment's authoritative Voice Agent host (not derived from the
         # streaming host, which only coincides by naming convention today).
         "ASSEMBLYAI_AGENTS_HOST": env.agents_host,
-        # Streaming-TTS host for the cascade (agent-cascade) template. Empty in
-        # production, where streaming TTS has no host; that template then refuses to
-        # run and points at --sandbox.
+        # Streaming-TTS host for the cascade (agent-cascade) template.
         "ASSEMBLYAI_TTS_HOST": env.streaming_tts_host,
     }
 

aai_cli/app/tts_voices.py

@@ -0,0 +1,19 @@
+"""The ``--list-voices`` body shared by the streaming-TTS commands (speak, dub, live).
+
+The listing is the built-in voices plus the account's own from the service's
+``GET /v1/voices`` (``tts.voice_list``), so it needs an API key like the synthesis
+it informs. The Typer flag that invokes it is ``options.list_voices_option``.
+"""
+
+from __future__ import annotations
+
+from aai_cli.app.context import AppState
+from aai_cli.tts import session, voice_list
+from aai_cli.ui import output
+
+
+def emit_voice_list(command: str, state: AppState, *, json_mode: bool) -> None:
+    """Print the voices: the raw JSON array, or one line per voice."""
+    session.require_available(command)
+    listing = voice_list.list_voices(state.resolve_api_key())
+    output.emit(listing, voice_list.format_voice_list, json_mode=json_mode)

aai_cli/code_gen/agent_cascade.py

@@ -6,6 +6,7 @@
 
 from aai_cli.code_gen import agent_cascade_body
 from aai_cli.core import environments
+from aai_cli.tts import session
 
 if TYPE_CHECKING:
     from aai_cli.agent_cascade.config import CascadeConfig
@@ -71,11 +72,10 @@ def _stt_url(speech_model: str, *, format_turns: bool) -> str:
 
 
 def _tts_url(voice: str, language: str | None) -> str:
-    """The streaming-TTS socket URL for the configured voice (sandbox-only host)."""
-    params: dict[str, str] = {"voice": voice, "sample_rate": "24000"}
-    if language is not None:
-        params["language"] = language
-    return f"wss://{environments.active().streaming_tts_host}/v1/ws/?{urlencode(params)}"
+    """The streaming-TTS socket URL for the configured voice — the CLI's own URL
+    builder, so the generated script can't drift from what `speak` sends."""
+    config = session.SpeakConfig(text="", voice=voice, language=language, sample_rate=24000)
+    return session.ws_url(config.query_params())
 
 
 def _cue(*, format_turns: bool) -> tuple[str, str]:

aai_cli/code_gen/agent_cascade_body.py

@@ -82,8 +82,6 @@ def synthesize(text):
             kind = frame.get("type")
             if kind == "Audio":
                 pcm += base64.b64decode(frame.get("audio", ""))
-                if frame.get("is_final"):
-                    break
             elif kind in ("FlushDone", "Error"):
                 break
         ws.send(json.dumps({"type": "Terminate"}))

aai_cli/commands/agent_cascade/__init__.py

@@ -14,11 +14,10 @@
     DEFAULT_SYSTEM_PROMPT,
 )
 from aai_cli.agent_cascade.voices import DEFAULT_VOICE
-from aai_cli.app.context import AppState, run_command, run_with_options
+from aai_cli.app.context import run_with_options
 from aai_cli.commands.agent_cascade import _exec as agent_cascade_exec
 from aai_cli.core import choices, llm
 from aai_cli.streaming.turn_presets import TurnDetectionPreset
-from aai_cli.ui import output
 from aai_cli.ui.help_text import examples_epilog
 
 # Option panels that group the per-leg knobs in `--help` instead of one flat wall.
@@ -36,39 +35,38 @@
 )
 
 
-def _emit_voice_list(_state: AppState, json_mode: bool) -> None:
-    """--list-voices body, routed through run_command so --json yields a machine-readable
-    array instead of the human list; needs no auth."""
-    payload = [{"name": name} for name in voices.VOICE_NAMES]
-    output.emit(payload, lambda _voices: voices.format_voice_list(), json_mode=json_mode)
+def _as_tuple[T](values: list[T] | None) -> tuple[T, ...]:
+    """A repeatable option's values as the frozen tuple the options dataclass carries
+    (Typer hands back None, not an empty list, when the flag is never passed)."""
+    return tuple(values or ())
 
 
 @app.command(
     name="live",
     rich_help_panel=help_panels.TRANSCRIPTION,
     epilog=examples_epilog(
         [
-            ("Start a live voice conversation", "assembly --sandbox live"),
+            ("Start a live voice conversation", "assembly live"),
             (
                 "Pick a voice and opening line",
-                'assembly --sandbox live --voice michael --greeting "Hi there"',
+                'assembly live --voice michael --greeting "Hi there"',
             ),
             (
                 "Give the agent a persona",
-                'assembly --sandbox live --system-prompt "You are a terse pirate."',
+                'assembly live --system-prompt "You are a terse pirate."',
             ),
             (
                 "Add your own MCP servers (none load by default)",
-                "assembly --sandbox live --mcp-config ~/.config/mcp/servers.json",
+                "assembly live --mcp-config ~/.config/mcp/servers.json",
             ),
             (
                 "Run a conversation without filesystem access",
-                "assembly --sandbox live --no-files",
+                "assembly live --no-files",
             ),
-            ("See available voices", "assembly --sandbox live --list-voices"),
+            ("See available voices", "assembly live --list-voices"),
             (
                 "Print equivalent Python instead of running",
-                "assembly --sandbox live --show-code",
+                "assembly live --show-code",
             ),
         ]
     ),
@@ -184,7 +182,8 @@ def live(
         rich_help_panel=_PANEL_TOOLS,
     ),
     device: int | None = typer.Option(None, "--device", help="Microphone device index"),
-    list_voices: bool = typer.Option(False, "--list-voices", help="Print known voices and exit"),
+    # Handled eagerly in the option's own callback (it exits), so the body never reads it.
+    _list_voices: bool = options.list_voices_option("live"),
     json_out: bool = options.json_option("Emit newline-delimited JSON events"),
     output_field: choices.TextOrJson | None = typer.Option(
         None,
@@ -198,15 +197,13 @@ def live(
         help="Print the equivalent Python SDK code and exit (does not start a session)",
     ),
 ) -> None:
-    """\\[sandbox] Talk live to a tool-using voice agent
+    """Talk live to a tool-using voice agent
 
     A real-time spoken conversation, wired client-side from three primitives —
     Streaming STT, a deepagents brain on the LLM Gateway, and streaming TTS. Unlike
     'assembly agent' (the Voice Agent API), the brain here is an agent that can use
     tools mid-conversation — web search, URL fetch, and the AssemblyAI docs — so it
-    answers like a live multimodal assistant. Because it uses streaming TTS it only
-    runs in the sandbox: run it as 'assembly --sandbox live' (--sandbox goes before
-    the subcommand).
+    answers like a live multimodal assistant.
 
     Use headphones: the mic stays open while the agent speaks, so on speakers it
     would hear itself and loop. Pass an audio file/URL (or --sample) to speak a
@@ -224,10 +221,6 @@ def live(
     the session.
     """
 
-    if list_voices:
-        run_command(ctx, _emit_voice_list, json=json_out)
-        return
-
     opts = agent_cascade_exec.AgentCascadeOptions(
         source=source,
         sample=sample,
@@ -241,15 +234,15 @@ def live(
         speech_model=speech_model,
         format_turns=format_turns,
         turn_detection=turn_detection,
-        stt_config=tuple(stt_config or ()),
+        stt_config=_as_tuple(stt_config),
         stt_config_file=stt_config_file,
         max_tokens=max_tokens,
-        llm_config=tuple(llm_config or ()),
+        llm_config=_as_tuple(llm_config),
         language=language,
-        tts_config=tuple(tts_config or ()),
-        mcp_config=tuple(mcp_config or ()),
+        tts_config=_as_tuple(tts_config),
+        mcp_config=_as_tuple(mcp_config),
         files=files,
-        auto_write=tuple(auto_write or ()),
+        auto_write=_as_tuple(auto_write),
         show_code=show_code,
     )
     run_with_options(ctx, agent_cascade_exec.run_agent_cascade, opts, json=json_out)

aai_cli/commands/agent_cascade/_exec.py

@@ -30,6 +30,7 @@
 from aai_cli.streaming.sources import FileSource
 from aai_cli.streaming.validate import resolve_output_modes
 from aai_cli.tts import session as tts_session
+from aai_cli.tts import voices as tts_voices
 from aai_cli.ui import output
 
 if TYPE_CHECKING:
@@ -39,18 +40,20 @@
 # message steering the user to the right place instead of silently fighting the cascade.
 _RESERVED_TTS_KEYS: dict[str, str] = {
     "voice": "Set the voice with --voice, not --tts-config.",
+    "voice_id": "Set the voice with --voice, not --tts-config.",
     "language": "Set the language with --language, not --tts-config.",
     "sample_rate": "TTS sample rate is fixed to match the live speaker and can't be overridden.",
+    "encoding": "TTS encoding is fixed to the 16-bit PCM the live speaker plays.",
 }
 
 
 @dataclass(frozen=True)
 class AgentCascadeOptions:
     """Every `assembly agent-cascade` conversation flag as plain data.
 
-    ``--list-voices`` is excluded: it dispatches to its own auth-free body in the
-    command module. ``--json`` is excluded: run_command resolves it into the
-    ``json_mode`` argument.
+    ``--list-voices`` is excluded: it is an eager flag that lists the service's
+    voices and exits in its own callback (``app.tts_voices``). ``--json`` is
+    excluded: run_command resolves it into the ``json_mode`` argument.
     """
 
     source: str | None
@@ -334,7 +337,8 @@ def _launch_tui(api_key: str, opts: AgentCascadeOptions, config: CascadeConfig)
 def run_agent_cascade(opts: AgentCascadeOptions, state: AppState, *, json_mode: bool) -> None:
     """Execute one `assembly agent-cascade` cascade from already-parsed flags."""
     text_mode, json_mode = resolve_output_modes(opts.output_field, json_mode=json_mode)
-    validate_voice(opts.voice, voices.VOICE_NAMES, command="live")
+    # A copied voice_id (aai-jane, as --list-voices prints them) names the same catalog voice.
+    validate_voice(tts_voices.voice_name(opts.voice), voices.VOICE_NAMES, command="live")
     # Streaming TTS has no production host, so the whole cascade is sandbox-only.
     tts_session.require_available("live")
     system_prompt_text = _resolve_system_prompt(opts.system_prompt, opts.system_prompt_file)