feat: governed realtime media membrane #2166

Problem Statement

Related: NVIDIA/NemoClaw#6207.
VoiceClaw referencces

• Prioritized VoiceClaw platform-issues spreadsheet
• Detailed VoiceClaw platform-issues document

P0: OpenShell needs a governed realtime media path for audio streams.
OpenShell 0.0.72 can relay WebSocket binary bytes, but only the WebSocket upgrade and text messages have policy semantics today. WebRTC/TURN-style media is also outside the current TCP CONNECT proxy model.

VoiceClaw needs realtime audio without turning the sandbox boundary into an opaque raw socket escape. Media must remain deny-by-default, auditable, rate-limited, and bound to declared destinations and sessions.

Who will use this

• VoiceClaw realtime voice assistant.
• OpenClaw voice-call and Talk sessions.
• NemoClaw sandboxes running voice-enabled OpenClaw.
• NVIDIA speech integrations such as Parakeet ASR, Magpie TTS, and PersonaPlex once its serving wrapper is confirmed.
• Operators and security teams who need media egress governed by OpenShell policy instead of hidden behind host bridges.

Concrete protocol evidence

• Parakeet is served through NVIDIA ASR NIM / Riva Speech Recognition:
  • HTTP REST for offline transcription.
  • gRPC over HTTP/2 for RivaSpeechRecognition/Recognize and bidirectional StreamingRecognize.
  • Realtime WebSocket for /v1/realtime?intent=transcription, using JSON events with base64 audio.
• Magpie is served through NVIDIA TTS NIM / Riva Speech Synthesis:
  • HTTP REST for /v1/audio/synthesize and /v1/audio/synthesize_online.
  • gRPC over HTTP/2 for RivaSpeechSynthesis/Synthesize and server-streaming SynthesizeOnline.
  • Realtime WebSocket for /v1/realtime?intent=synthesize, using JSON events.
• PersonaPlex is publicly described as a full-duplex speech-to-speech model/checkpoint, not a documented Speech NIM API surface:
  • Treat its network protocol as unconfirmed until the serving wrapper is known.
  • If wrapped as Riva/NIM, expect gRPC or realtime WebSocket.
  • If exposed directly for browser voice, expect WebSocket or WebRTC.

Proposed Design

Add a governed media protocol surface to OpenShell.

This could be a new protocol: media, or staged support through:

1. WEBSOCKET_BINARY for realtime audio over WebSocket.
2. Realtime WebSocket JSON media policy.
3. protocol: grpc for streaming gRPC.
4. Optional TURN/WebRTC support later, only when there is a concrete sandbox use case.

The media policy should include:

• transport allowlist
• codec allowlist
• destination and session binding
• direction controls
• max frame/message size
• bitrate limits
• session duration limits
• concurrency limits
• OCSF media session logging

Alternatives Considered

• Keep using raw L4 passthrough. This moves bytes, but it cannot express media policy, session caps, gRPC method policy, or useful audit semantics.
• Keep VoiceClaw audio on the host forever. This avoids sandbox risk but forces every voice/media integration to rebuild a host-side shadow plane.
• Add raw UDP/RTP/WebRTC first. The current VoiceClaw POC does not require that inside the sandbox. Start with WebSocket binary and gRPC streaming, then add TURN/WebRTC only with a concrete requirement.

Agent Investigation

• Reviewed NemoClaw #6207, which frames this as the P0 OpenShell ask: governed realtime media without raw socket escape.
• Inspected local ../openclaw Voice Call code. Carrier and provider paths use WebSocket media streams; some providers use JSON events with base64 audio and some use binary audio frames.
• Checked OpenShell docs/code. protocol: websocket governs upgrade and client text frames; binary WebSocket frames are relayed but not policy-governed. protocol: grpc is not currently listed as an L7 protocol.
• Checked NVIDIA Speech NIM docs for Parakeet and Magpie protocols.

Sources:

• NVIDIA Speech NIM overview: https://docs.nvidia.com/nim/speech/latest/about/index.html
• ASR support matrix / Parakeet: https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/asr.html
• ASR gRPC API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/asr/protos.html
• ASR realtime WebSocket API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/asr/realtime-asr.html
• TTS support matrix / Magpie: https://docs.nvidia.com/nim/speech/latest/reference/support-matrix/tts.html
• TTS HTTP API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/http-tts.html
• TTS gRPC API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/protos.html
• TTS realtime WebSocket API: https://docs.nvidia.com/nim/speech/latest/reference/api-references/tts/realtime-tts.html
• PersonaPlex paper: https://arxiv.org/abs/2602.06053

Acceptance Criteria

• Realtime audio can be allowed without granting arbitrary raw socket egress.
• WebSocket binary media is denied unless explicitly allowed.
• Realtime WebSocket JSON media can be governed by event type and size limits.
• gRPC streaming can be restricted by host, service, method, metadata, size, rate, duration, and concurrency.
• Media sessions have frame/message size, bitrate, duration, and concurrency caps.
• OCSF logs record allow, deny, and limit decisions without logging audio payloads, base64 media, or secrets.

Non-Goals

• Unrestricted raw socket escape.
• Generic UDP/RTP/WebRTC support in the first implementation.
• Audio transcription or semantic inspection inside OpenShell.
• OpenClaw provider implementation.
• NemoClaw plugin packaging work.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request