feat(simulation): add scripted user turns (simulation.turns)

.claude/notes/orchestration.md

@@ -50,9 +50,10 @@
   path. Three things are refused rather than degraded: `--junit-xml` (a report of
   verdicts, and there are none — though `reports/junit.py` still emits `<skipped>` for
   an ungraded row it encounters), `--allow-host-grading` (it decides how an ungraded row
-  is GRADED, and `execute` grades nothing), and simulation tasks (their
-  turn-continuation logic reads criteria results, so an ungraded dialog would silently
-  change its own stopping behavior). `stop_early:` blocks are inert under `execute` for
+  is GRADED, and `execute` grades nothing), and simulation tasks with an LLM-simulated
+  user (their turn-continuation logic reads criteria results, so an ungraded dialog would
+  silently change its own stopping behavior). A scripted dialog reads none, so `execute`
+  accepts it. `stop_early:` blocks are inert under `execute` for
   the same reason the kill switch exists: the full trajectory is the deliverable.
   Motivating consumer: an external harness (Harbor / Terminal-Bench 2.0) that builds its
   own container, calls coder-eval as the agent, and grades with its own tests.
@@ -95,11 +96,13 @@ the loop over the real execution facts.
 2. **The single-shot loop** stops after capturing the trajectory. Returning False keeps
    `FinalStatus` off SUCCESS and the status chain turns it into NOT_GRADED. The
    reference-integrity check is skipped too — it protects a grade that is not happening.
-3. **The dialog loop** RAISES, and is unreachable today because `execute` rejects
-   simulation tasks at the CLI: the dialog reads criteria results to decide whether to
-   keep talking, so an ungraded dialog would silently change its own stopping behavior.
-   An empty-list "defensive no-op" here would be worse than a refusal — both callers go
-   straight on to the gate, which treats an empty criteria list as a vacuous pass.
+3. **The dialog loop** checks criteria only on a graded run. Its end-of-dialog and
+   budget-gate checks are skipped when ungraded, so a scripted dialog finalizes
+   NOT_GRADED. The per-turn check still RAISES, and is unreachable today because
+   `execute` rejects LLM-simulated tasks at the CLI: that dialog reads criteria results to
+   decide whether to keep talking, so an ungraded run would silently change its own
+   stopping behavior. An empty-list "defensive no-op" there would be worse than a
+   refusal — the gate treats an empty criteria list as a vacuous pass.
 4. **The diagnostics path** records nothing: a `not_evaluated` vector would imply criteria
    we were supposed to run and could not.
 
@@ -632,10 +635,37 @@ unchanged.
 § Gate selection is fired-only — so `result.early_stop` is never assigned here and an armed
 simulation task gates strict-AND on a possibly truncated trajectory.
 
-`execute` rejects simulation tasks at the CLI, because the loop reads criteria results to
+`execute` rejects LLM-simulated tasks at the CLI, because the loop reads criteria results to
 decide whether to keep talking: an ungraded dialog would silently change its own stopping
 behaviour rather than merely skipping the score.
 
+### Scripted dialogs
+
+`simulation.turns` swaps the LLM user for `ScriptedUser`, which answers the loop's own
+`next_user_message` call. `_evaluation_loop` composes it and hands it to the loop as
+`user=`; the loop never branches on the kind afterwards, so the stop, telemetry and logging
+paths are shared.
+
+- **The cursor is the loop's history.** `ScriptedUser` returns `turns[len(dialog_pairs)]`
+  and keeps no counter, so it cannot drift from the exchanges the loop actually completed.
+  The dialog ends in `evaluate_stop` (`script_exhausted`) before the user is asked past the
+  last turn.
+- **`turns[0]` is the pinned opener.** `_evaluation_loop` hands it to the loop as
+  `initial_prompt`, so turn 1 is recorded like any pinned opener (no simulator timing, an
+  accurate `conversation.log` label) and the task carries one source of truth for its
+  first prompt: `initial_prompt` beside `turns` is rejected.
+- **The fields a script ignores are compared by value, not by `model_fields_set`.**
+  `evaluate` rebuilds the task from a full `model_dump()`, which marks every field as set,
+  so a "was it set" test would reject every recorded scripted task.
+- **Script length owns the dialog length.** Derive `max_turns` after simulation config
+  validation, including merged variant scripts, so a replacement cannot inherit a stale cap.
+  Prompt overrides are rejected before mutation because a script owns every user turn.
+- **No criteria steer a script.** `check_criteria` and `stop_on_criteria_pass` stay at their
+  defaults, so the one criteria check is the end-of-dialog verdict. That is also why
+  `execute` can accept it: nothing the dialog does depends on a verdict it lacks.
+- **No simulator, no simulator egress.** `isolation/egress.py` adds the simulator's model
+  route only for a simulated user.
+
 ## Embedded commands
 
 A recorded config is UNTRUSTED INPUT. `evaluate <run_dir>` rebuilds the task from the

CLAUDE.md

@@ -107,8 +107,8 @@ Each entry is a pointer. Full rationale: `.claude/notes/` (index: `.claude/notes
   [Sandbox Configuration](docs/TASK_DEFINITION_GUIDE.md#sandbox-configuration).
 - **Container isolation**: `driver: docker` runs one container per task — see
   [Docker Isolation](docs/DOCKER_ISOLATION.md).
-- **Dialog mode**: `simulation/` drives a multi-turn LLM user — see
-  [Dialog Mode](docs/DIALOG_MODE.md).
+- **Dialog mode**: `simulation/` drives a multi-turn dialog. The user is an LLM or an
+  authored script (`simulation.turns`) — see [Dialog Mode](docs/DIALOG_MODE.md).
 
 ## Success Criteria
 
@@ -275,7 +275,7 @@ Tasks are YAML. Full reference: [Task Definition Guide](docs/TASK_DEFINITION_GUI
 | Task YAML, criteria, run limits, sandbox | [Task Definition Guide](docs/TASK_DEFINITION_GUIDE.md) |
 | Variants across the same tasks (A/B) | [A/B Experiments](docs/AB_EXPERIMENTS.md) |
 | Dataset fan-out and suite thresholds | [Bring Your Own Dataset](docs/DATASETS.md) |
-| Multi-turn user simulation | [Dialog Mode](docs/DIALOG_MODE.md) |
+| Multi-turn user simulation, scripted turns | [Dialog Mode](docs/DIALOG_MODE.md) |
 | `driver: docker`, custom images | [Docker Isolation](docs/DOCKER_ISOLATION.md) |
 | Run/report JSON consumed downstream | [Report Schema](docs/REPORT_SCHEMA.md) |
 

docs/DATASETS.md

@@ -71,10 +71,11 @@ Each line is one JSON object — the same shape as an inline row:
 
 ## Substituting row values
 
-`${row.<field>}` placeholders are replaced with that row's value in exactly two places:
+`${row.<field>}` placeholders are replaced with that row's value in:
 
-1. the task's `initial_prompt`, and
-2. every **string leaf** of every `success_criteria` entry (at any nesting depth).
+1. the task's `initial_prompt`,
+2. each scripted `simulation.turns` entry, and
+3. every **string leaf** of every `success_criteria` entry (at any nesting depth).
 
 Nothing else is substituted — not the task-level `description`, not `pre_run`, not `sandbox`. Keep
 row-varying data in the prompt and the criteria. (A *criterion's* own `description` is a string leaf

docs/DIALOG_MODE.md

@@ -13,7 +13,8 @@ back requirements until asked.
 
 Dialog mode replaces the single-shot loop with a conversation between the coding agent and a
 **simulated user** — a second LLM driven by a persona and a goal. It is the only way to measure the
-behaviors that only exist in conversation.
+behaviors that only exist in conversation. To replay a fixed list of authored questions instead of
+simulating a user, see [Scripted dialogs](#scripted-dialogs).
 
 ## When to use it
 
@@ -71,7 +72,8 @@ The mechanics:
 **Agent-kind constraint.** The *subject* agent can be any registered kind — the dialog driver only
 calls the agent's `communicate()`, so Codex and plugin agents work. The *simulator*, however, is
 always a Claude Code agent. A dialog run therefore needs the `claude` CLI on `PATH` and working
-Anthropic or Bedrock credentials **even when the agent under test is not Claude**.
+Anthropic or Bedrock credentials **even when the agent under test is not Claude**. A
+[scripted dialog](#scripted-dialogs) has no simulator and needs neither.
 
 **Early stop is not available here.** [Criterion-level `stop_early:` arming](TASK_DEFINITION_GUIDE.md#stop_early-opt-in-early-stop)
 is single-shot-only; an armed criterion alongside `simulation.enabled` is a hard error at resolution
@@ -82,6 +84,62 @@ Every field's default and constraint lives in one place — the
 [Task Definition Guide's simulation section](TASK_DEFINITION_GUIDE.md#simulation-multi-turn-user-dialog).
 This page deliberately does not restate them.
 
+## Scripted dialogs
+
+Sometimes the conversation itself is the benchmark: a fixed list of authored questions, asked in
+order within one session, whose answers you read afterwards. A simulated user would paraphrase them
+and add an LLM call per turn. Set `turns` instead of `persona` and `goal`, and the dialog replays the
+list verbatim:
+
+```yaml
+task_id: onboarding-questions
+description: Ask the same three questions in one session and keep each answer.
+agent:
+  type: codex                          # Any agent: a script needs no `claude` CLI or credentials of its own.
+working_directory_prefix: false        # Send each turn exactly as written.
+simulation:
+  enabled: true
+  turns:
+    - "What does the knowledge base say about onboarding?"
+    - "And about reimbursements?"
+    - "Summarize both answers in two sentences."
+  max_turns: 3                         # Must be at least len(turns).
+success_criteria:
+  - type: llm_judge
+    description: "Each answer is grounded in the knowledge base"
+    prompt: "Read the DIALOG block and grade whether every answer is grounded in the knowledge base."
+    include_dialog: true
+```
+
+What changes against a simulated dialog:
+
+- **One session, no simulator.** Every turn goes to the same agent instance, so the agent keeps its
+  context from turn to turn. No LLM plays the user: no `claude` CLI, no credentials, no simulator
+  route, and `simulation.simulator_model` is `null` with zero simulator tokens.
+- **`turns[0]` is the first prompt.** The task sets no `initial_prompt` or `initial_prompt_file`, so
+  there is one source of truth. Turns are sent byte for byte when `working_directory_prefix: false`;
+  by default each still carries the `Your working directory is: …` line in front.
+- **The script ends the dialog.** After the last turn the dialog stops with
+  `stop_reason: script_exhausted`, whatever the criteria say. Nothing in a script reads a criteria
+  result, so `stop_on_criteria_pass` and `check_criteria: every_turn` / `both` make no sense for it,
+  and neither do `persona`, `goal`, `constraints`, `model` or `stop_token`: each must keep its
+  default, or the task is rejected at load time. `max_turns` is derived from the script length
+  (1–100 turns), so an authored script needs no second dialog-length setting. Variant prompt
+  overrides and mutations are rejected; replace `simulation.turns` to change the script.
+- **A failed turn ends the dialog.** A turn that still fails after its retries, exhausts
+  `run_limits.max_turns` or breaches a `run_limits` budget stops the script. Later turns are not sent,
+  and the turns that ran stay on the record.
+- **It runs without grading.** `coder-eval execute` accepts a scripted dialog and finalizes it
+  `NOT_GRADED`; `coder-eval evaluate <run_dir>` grades it later. Criteria are checked once, after the
+  last turn.
+
+Each exchange is one entry in `iterations` of `task.json` (see [TurnRecord](REPORT_SCHEMA.md#turnrecord)):
+that turn's `user_input` (the prompt as the agent received it), `agent_output`, `commands`,
+`token_usage` and timings, with the authored turn as `messages[0]`. `agent_output` is the agent's own
+rendering of its reply: Claude Code's is a tagged transcript, and its SDK result text is also in
+`result_summary.result`. `n_trials: N` repeats the whole script N times, each in its own session, to
+measure variance.
+
 ## Designing the persona and goal
 
 `persona` is *who is talking*; `goal` is *what they want*. Keep them separate — a persona that
@@ -121,15 +179,17 @@ After each exchange the driver evaluates the stop conditions **in this order**,
 2. **`stop_on_criteria_pass`** (`criteria_passed`) — every success criterion passes. Requires
    per-turn checking (`check_criteria: every_turn` or `both`); pairing it with the default
    `end_of_dialog` is rejected at load time, since there would be nothing to check against.
-3. **`max_turns`** (`max_turns`) — the hard cap on exchanges.
-4. **`max_total_tokens`** (`budget`) — the dialog-wide budget across simulator **and** agent. The
+3. **Script exhausted** (`script_exhausted`) — a [scripted dialog](#scripted-dialogs) has had its
+   last turn answered. It takes the place of `max_turns` there.
+4. **`max_turns`** (`max_turns`) — the hard cap on exchanges.
+5. **`max_total_tokens`** (`budget`) — the dialog-wide budget across simulator **and** agent. The
    dialog ends and the task is **still scored** — unlike
    [`run_limits.max_total_tokens`](TASK_DEFINITION_GUIDE.md#run-limits), which covers the subject
    agent only and aborts.
-5. **`run_limits.max_turns`** (`agent_max_turns`): the agent used up its own model-call cap inside
+6. **`run_limits.max_turns`** (`agent_max_turns`): the agent used up its own model-call cap inside
    one exchange. That cap restarts on every exchange, so this reason means one exchange ran long, not
    that the dialog ran out of exchanges.
-6. **`stop_token`** (`stop_token`) — only if none of the above fired is the simulator asked for
+7. **`stop_token`** (`stop_token`) — only if none of the above fired is the simulator asked for
    another message; the sentinel token in *that fresh utterance* ends the dialog. This is the
    workhorse in practice — the simulator decides, in character, that it got what it wanted — but it
    is evaluated **last**, so a turn that trips `max_turns` or the budget never gets the chance to
@@ -192,8 +252,9 @@ with them. That guard is what keeps a chatty simulator from poisoning the grade.
 Budget roughly `max_turns × n_trials` agent turns per (task, variant) — the worst case, since the
 dialog usually stops on the stop token first. On top of that:
 
-- **Simulator tokens**, one generation per turn. Small next to the agent's, but not free, and they
-  are reported separately as `simulation.simulator_input_tokens` / `simulator_output_tokens`.
+- **Simulator tokens**, one generation per turn (none in a scripted dialog). Small next to the
+  agent's, but not free, and they are reported separately as `simulation.simulator_input_tokens` /
+  `simulator_output_tokens`.
 - **Criterion cost**, multiplied by turn count under `check_criteria: every_turn` or `both`.
 
 Cap it with [`run_limits`](TASK_DEFINITION_GUIDE.md#run-limits) — but note the accounting boundary:
@@ -217,9 +278,9 @@ variants:
       persona: "A terse, impatient user who gives one-line answers."
 ```
 
-Productive axes: persona (terse vs. chatty, technical vs. not), how much the goal withholds, and
-`n_trials` as a cost/confidence trade. See [A/B Experiments](AB_EXPERIMENTS.md) for the full
-experiment layer.
+A variant's `turns` replaces the task's script as a whole. Productive axes: persona (terse vs.
+chatty, technical vs. not), how much the goal withholds, and `n_trials` as a cost/confidence trade.
+See [A/B Experiments](AB_EXPERIMENTS.md) for the full experiment layer.
 
 ## Security posture
 

docs/DOCKER_ISOLATION.md

@@ -108,7 +108,7 @@ call a model, and adds nothing for the parts that do not:
 
 | Source | Hosts added |
 |---|---|
-| API backend (`API_BACKEND`), only when a `claude-code` agent, an enabled `simulation:`, or an `llm_judge` / `agent_judge` criterion uses it | `direct`: `api.anthropic.com:443`, `platform.claude.com:443` (Claude Code OAuth refresh). `bedrock`: `bedrock-runtime.<AWS_REGION>.amazonaws.com:443`, `bedrock.<AWS_REGION>.amazonaws.com:443`. `litellm`: the host of the forwarded `LITELLM_BASE_URL` (a `localhost` value becomes `host.docker.internal`) |
+| API backend (`API_BACKEND`), only when a `claude-code` agent, an enabled `simulation:` with a simulated user (not scripted `turns`), or an `llm_judge` / `agent_judge` criterion uses it | `direct`: `api.anthropic.com:443`, `platform.claude.com:443` (Claude Code OAuth refresh). `bedrock`: `bedrock-runtime.<AWS_REGION>.amazonaws.com:443`, `bedrock.<AWS_REGION>.amazonaws.com:443`. `litellm`: the host of the forwarded `LITELLM_BASE_URL` (a `localhost` value becomes `host.docker.internal`) |
 | Agent | codex: the host of the forwarded `CODEX_BASE_URL`, else `api.openai.com:443`. antigravity: `generativelanguage.googleapis.com:443`. pi: the host of the `provider/` prefix of `agent.model` (`openrouter`, `anthropic`, `openai`, `google`; anything else or no model gives `openrouter.ai:443`). claude-code, opencode, delegate, none: nothing beyond the backend row |
 | `system_one_judge` criteria | the host of each `base_url` (default `api.typesafe.ai:443`) |
 | `egress_allowlist` | your entries |