Record Custom runner sessions into OTLP traces and enforce accounting_gap and web-search Void checks #2191

This was generated by AI during a grill-with-docs session.

Parent

#1468

What to build

Record Custom runner sessions into OTLP Session traces and enforce accounting_gap and web-search Void Trial checks (#1468 Q1.6, Q2.4, Q2.6; ADR 0032).

• In-sandbox live record_command: expose the recorder command passed in request.json inside the Trial sandbox so a Custom runner can wrap supported Harness CLI calls with --role <role>, streaming live into the Trial's session folder so mid-Trial 85% quota monitoring pauses the Experiment on usage_limit and finished streams cannot be deleted by the runner.
• Post-run output_dir/sessions.json conversion: accept output_dir/sessions.json listing top-level raw Harness streams ({ role, harness, stream_file, started_at, ended_at }) and/or direct-SDK session records ({ role, transcript_file, started_at, ended_at, models: { <model>: { uncached_input_tokens, cache_read_tokens, cache_write_tokens, output_tokens } } }). Convert both into standard OTLP .logs.jsonl and .spans.jsonl in the Trial's session folder (converting uncached_input_tokens into cache-inclusive input tokens for pricing from the Experiment's frozen prices.json, and scanning converted Claude/Codex streams for quota rejections), feeding per-role accounting, fix-leak scanning, Waterfall, Trial Peek, and OTLP export through one trace representation.
• Web-search and accounting_gap Void checks: void any Trial whose normalized session transcripts contain a web-search tool call. Count outbound provider connections through HTTPS_PROXY and non-probe Harness invocations through the sandbox's PATH wrappers; if provider traffic occurred (or a Custom runner with declared harnesses/providers ended delivered or stopped_to_ask) with zero recorded sessions—or fewer Harness session entries than non-probe Harness runs that crossed the proxy—mark the Trial as a Void Trial (accounting_gap).

Acceptance criteria

• Wrapping a Harness CLI call with record_command --role <role> streams .logs.jsonl and .spans.jsonl live into the Trial's session folder and triggers live 85% usage_limit pauses.
• Both raw Harness stream entries and direct-SDK uncached_input_tokens entries in output_dir/sessions.json convert into .logs.jsonl and .spans.jsonl, price tokens accurately from frozen prices.json (never trusting runner-reported USD), scan transcripts for fix leaks, and render in Trial Peek and Waterfall.
• A Trial whose session transcript records a web-search tool call is voided.
• A Custom runner that sends provider traffic over HTTPS_PROXY, runs a Harness across the proxy without recording a matching session, or ends delivered / stopped_to_ask with declared harnesses/providers and zero recorded sessions becomes a Void Trial (accounting_gap).

Blocked by

• Enclose Custom runner Trials in a multi-provider sandbox with Trial provider keys, web-search wrappers, and panel family rules #2189