Keep Passes clear of live Experiments

GLOSSARY.md

@@ -49,6 +49,7 @@ _Avoid_: Claim limit (former name), WIP limit, concurrency (it counts issues wai
 
 **Command limit**:
 The number of live **Commands** on one repository on one machine at which a **Pass** there is skipped, set in the **User config**, one by default. Every live Command counts, a Ticket run started by hand on an **Issue URL** included, together with the Spec run or Ticket run each Pass dispatched, but only a Pass is ever skipped for it: a Command a person types always runs. Passes of different kinds may run side by side up to the limit, but never two **Architect runs** or two **Security runs** on one repository at once. It counts no other repository; load across a machine is the **User config**'s memory and load waits.
+An Experiment independently holds back every Pass of its OS account across the machine's repositories, including while it waits for live Runs to end.
 _Avoid_: Pass lock (former, one-wide form), concurrency limit, parallel
 
 **Sweep**:
@@ -147,7 +148,8 @@ One **Command** that investigates a **Report** or an existing open issue by a **
 _Avoid_: intake
 
 **Pass**:
-An **Architect run**, a **Pickup run**, a **Security run**, a **Review run** started with `--author`, a **Fix run** started with no pull request, a **Triage run** started with neither a **Report** nor an **Issue URL**, or a **Custom process** started with no issue and neither an Issue-branch nor a Spec-branch worktree: a **Command** started with no **Issue URL**, which decides before any work whether it is skipped. Such a Custom process is skipped when the repository is at its **Command limit** on the machine, and it does not take an Architect run's Ready-issue or Architect-idea gates; a file declaring either branch worktree needs an issue, and a Custom process started with an issue is not a Pass. Every Pass is skipped while an **Experiment** is live on its machine, including one still waiting for live Runs to end.
+An **Architect run**, a **Pickup run**, a **Security run**, a **Review run** started with `--author`, a **Fix run** started with no pull request, a **Triage run** started with neither a **Report** nor an **Issue URL**, or a **Custom process** started with no issue and neither an Issue-branch nor a Spec-branch worktree: a **Command** started with no **Issue URL**, which decides before any work whether it is skipped. Such a Custom process is skipped when the repository is at its **Command limit** on the machine, and it does not take an Architect run's Ready-issue or Architect-idea gates; a file declaring either branch worktree needs an issue, and a Custom process started with an issue is not a Pass.
+Every Pass is skipped while an **Experiment** is live on its machine for the same OS account, including one still waiting for live Runs to end. A Command started on an Issue URL still runs, with a warning that it competes with the Experiment's Trials.
 _Avoid_: tick, cycle
 
 **Process**:
@@ -432,7 +434,7 @@ What a **Grader** decides about one **Trial** against its **Eval task**'s succes
 _Avoid_: grade, rating, feedback
 
 **Experiment**:
-The **Command** that compares a **Baseline arm** with a **Candidate arm** on the **Eval suites** a change touches, against a **Bar** fixed before it starts, and ends in a **Verdict**. It shows whether the change makes the agents better or worse before it reaches production.
+The **Command** that compares a **Baseline arm** with a **Candidate arm** on the **Eval suites** a change touches, against a **Bar** fixed before it starts, and ends in a **Verdict**. It waits for the same OS account's other live Commands on its machine before starting Trials, including on resume; the Day shift can bypass the wait, with the number of competing Commands recorded beside the Verdict.
 _Avoid_: replay, eval run, benchmark run, A/B test
 
 **Bar**:

console/tests/experiments.test.cjs

@@ -2,12 +2,68 @@ const { test, before, after } = require('node:test');
 const assert = require('node:assert/strict');
 const { spawn, execFileSync } = require('node:child_process');
 const { once } = require('node:events');
-const { mkdtemp, mkdir, writeFile, rm } = require('node:fs/promises');
-const { tmpdir } = require('node:os');
+const { mkdtemp, mkdir, readFile, writeFile, rm } = require('node:fs/promises');
+const { tmpdir, hostname } = require('node:os');
 const { join, resolve } = require('node:path');
 const { createInterface } = require('node:readline');
 const { chromium } = require('./playwright.cjs');
 const fixture = require('./fixtures/experiments.cjs');
+const { replaceRecord } = require('./records.cjs');
+
+test('the Inbox names the Commands blocking a waiting Experiment', async t => {
+  const folder = join(home, '.thirdshift/logs/acme/widgets/commands/experiment');
+  await mkdir(folder, { recursive: true });
+  const stat = await readFile(`/proc/${process.pid}/stat`, 'utf8');
+  const start = `${(await readFile('/proc/sys/kernel/random/boot_id', 'utf8')).trim()}:${stat.slice(stat.lastIndexOf(') ') + 2).trim().split(/\s+/)[19]}`;
+  const now = new Date().toISOString();
+  const path = join(folder, 'host-wait.command.json');
+  await replaceRecord(path, JSON.stringify({
+    version: 1, kind: 'Experiment', repository: 'acme/widgets', name: 'host-wait', issue: null,
+    hostname: hostname(), process_id: process.pid, process_start: start, harness: 'claude',
+    model: null, effort: null, started_at: now, ended_at: null, outcome: null, cause: null,
+    session_log: null, issue_branch: null, pull_request: null,
+    stages: [{ name: 'Waiting for Commands', began_at: now, session_log: null }],
+    waiting_for: ['Ticket run unique/blocker#7 <script>']
+  }));
+  t.after(() => rm(folder, { recursive: true, force: true }));
+  for (const viewport of [{ width: 1440, height: 900 }, { width: 390, height: 844 }]) {
+    const context = await browser.newContext({ viewport });
+    try {
+      const page = await context.newPage();
+      await page.goto(origin);
+      const row = page.locator('.command').filter({ hasText: 'host-wait' }).first();
+      assert.match(await row.locator('.now').textContent(), /Waiting for Commands.*Ticket run unique\/blocker#7 <script>/);
+      assert.equal(await row.locator('script').count(), 0);
+      assert.equal(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth), true);
+    } finally { await context.close(); }
+  }
+});
+
+test('host waits and bypassed Command counts appear on Experiment pages and lists', async t => {
+  t.after(() => fixture.save(home, fixture.record()));
+  for (const viewport of [{ width: 1440, height: 900 }, { width: 390, height: 844 }]) {
+    const context = await browser.newContext({ viewport });
+    try {
+      const page = await context.newPage();
+      const waiting = fixture.record();
+      waiting.waiting_for = ['Ticket run acme/widgets#7 <script>'];
+      await fixture.save(home, waiting);
+      await page.goto(`${origin}/experiments/${fixture.id}`);
+      assert.match(await page.locator('.experiment-lead').textContent(), /Waiting for Commands: Ticket run acme\/widgets#7 <script>/);
+      assert.equal(await page.locator('.experiment-lead script').count(), 0);
+      await page.goto(`${origin}/experiments`);
+      assert.match(await page.locator('.experiment-row .now').textContent(), /Waiting for Commands: Ticket run acme\/widgets#7/);
+      const completed = fixture.finish(fixture.record());
+      Object.assign(completed, { no_wait: true, commands_at_start: 2 });
+      await fixture.save(home, completed);
+      for (const route of [`/experiments/${fixture.id}`, '/experiments']) {
+        await page.goto(`${origin}${route}`);
+        assert.equal(await page.locator('.experiment-contention').textContent(), 'Started with --no-wait: 2 other Commands live at start');
+        assert.equal(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth), true);
+      }
+    } finally { await context.close(); }
+  }
+});
 
 test('passing cheaper Candidates share the Verdict line with JSON and text on desktop and phone', async t => {
   const value = fixture.finish(fixture.record());

docs/experiment-results.md

@@ -1,4 +1,4 @@
-# Experiment results (format 10)
+# Experiment results (format 11)
 
 `thirdshift experiments <id> --json` emits one UTF-8 JSON object matching
 `Experiment` below. `thirdshift experiments --json` emits an array of those
@@ -9,8 +9,8 @@ These results complement the [Eval task input format](eval-tasks.md) and
 
 ## Stability and versioning
 
-`format: 10` identifies this result contract. All fields and types below, their
-presence/null rules, enum values and meanings are stable within format 10.
+`format: 11` identifies this result contract. All fields and types below, their
+presence/null rules, enum values and meanings are stable within format 11.
 Adding, removing, renaming or retyping a field, changing its meaning or presence
 rules, or changing an enum requires a new format number and an updated document
 and drift test in the same change. Readers should reject unsupported formats.
@@ -20,6 +20,9 @@ Model names and Void causes) vary with the Experiment. Unknown measurements are
 `null`, never zero. Timestamps are RFC 3339 strings; paths are JSON strings local
 to the machine retaining the Experiment.
 
+Format 11 adds `waiting_for`, `no_wait` and `commands_at_start`: the live host
+wait and the number of other live Commands at a start using `--no-wait`.
+
 The private `experiment.json` already has `format: 1`. The CLI view projects its
 metadata and retained evidence; it is not a copy of that internal storage file.
 Older CLI builds without a `format` field predate this documented contract.
@@ -213,6 +216,13 @@ change in any state without changing the Experiment's identity, Bar or results.
 true on later resumes. These fields do not participate in Experiment identity
 or the Bar.
 
+`waiting_for` lists the other live Commands this invocation is waiting for on
+the same machine and OS account; it is empty outside that wait. `no_wait` says
+whether the latest invocation bypassed the wait. `commands_at_start` counts its
+other live Commands at startup with `--no-wait`, zero otherwise. These facts
+are saved on each start or resume and displayed beside the Verdict; older
+Experiments default to an empty list, false and zero.
+
 `running_trials` gives active Trial IDs and start times. `sessions` inventories
 retained Trial Sessions: opaque Session ID, Trial ID, zero-based attempt index
 and whether the readable file is an OTLP trace. Read one with
@@ -433,7 +443,7 @@ Trial files, rejecting undocumented, missing and retyped fields recursively.
     }
   },
   "Experiment": {
-    "format": 10,
+    "format": 11,
     "name": "string",
     "name_set": "boolean",
     "id": "string",
@@ -453,6 +463,9 @@ Trial files, rejecting undocumented, missing and retyped fields recursively.
     "resume_command": "string?",
     "command_log": "string?",
     "release": "boolean",
+    "waiting_for": ["string"],
+    "no_wait": "boolean",
+    "commands_at_start": "integer",
     "created_at": "string",
     "tasks": [{"suite": "string", "name": "string"}],
     "running_trials": [{"id": "string", "started_at": "string"}],

docs/guide.md

@@ -22,6 +22,15 @@ A [Ticket run](#what-a-ticket-run-does), one issue in and one pull request out,
 
 Architect, Pickup and Security runs, `review --author <login>`, and `fix` with no PR target are **Passes**, as is a Custom process started with no issue and no Issue-branch worktree: they take no Issue URL, so your operating system's scheduler can start them. A Custom process started with an issue is not a Pass. A file that declares an Issue-branch worktree is refused when no issue is given, before any session, rather than skipped. See [A Pickup run on a schedule](#a-pickup-run-on-a-schedule), [Architect runs every few minutes](#architect-runs-every-few-minutes) for Architect runs, and [Security runs on a schedule, with fixing allowed](#security-runs-on-a-schedule-with-fixing-allowed) for Security runs.
 
+Every Pass, including Triage with no Report or Issue URL and a Custom process
+started with no issue, skips before work while an Experiment of the same OS
+account is live on this machine, across all repositories. A waiting Experiment
+counts too. The Activity log names the Experiment, once per change of reason,
+and the Pass exits successfully. This includes the nightly Security run.
+Commands started on an Issue or pull request URL still run, with a warning on
+stderr that they will compete with the Experiment's Trials. Existing Runs keep
+running; Passes on other machines carry on.
+
 Every session runs on the **Harness** you choose: Claude Code by default, or Codex, Antigravity CLI, Grok Build, Muse Code, OpenCode or Kimi Code. See [Harness, Model and Effort](#harness-model-and-effort).
 
 ## What a Ticket run does
@@ -1216,6 +1225,21 @@ Harness, Model or Effort comparison with the same build in both Arms.
 Before building either Arm, an Experiment names every selected approved Eval
 task missing a `[trial]` table. This check also precedes the release Baseline probe.
 
+At startup and on resume, an Experiment becomes live, so every Pass of the same
+OS account on this machine skips. Before starting Trials, it waits for that
+account's other live Commands across repositories to end. Its Command log,
+`status` and the Console name the Commands it waits on and refresh the list as
+they end. Stop or interrupt during the wait ends the Experiment cleanly as
+interrupted; resuming waits again. Multiple waiting Experiments take turns once
+the other Commands have ended.
+
+Use `thirdshift experiment <candidate> --no-wait` to start immediately. Results
+record `no_wait: true` and `commands_at_start`, the number of other live Commands
+at this invocation's start, displayed beside the Verdict in text and the Console.
+A normal invocation records `no_wait: false` and zero. `waiting_for` names the
+current blockers and is empty after the wait. These fields do not change the Bar
+or Experiment identity. The structured Console API option is `no_wait: true`.
+
 For an **A/A Experiment**, use one commit and the same flags in both Arms.
 This measures how often the Arms disagree when nothing differs, so you can
 assess ordinary Trial variation before weighing a change. For example:
@@ -1747,7 +1771,7 @@ That includes a changed discriminating list or any changed Difficulty reading
 for its retained Eval tasks, even when the new reading stays inside the band.
 `thirdshift experiments --json` lists them and `thirdshift experiments <id>
 --json` shows one. Both use the versioned
-[Experiment results format](experiment-results.md), identified by `format: 10`.
+[Experiment results format](experiment-results.md), identified by `format: 11`.
 The starting command's `--json` ends with one JSON line
 containing the Verdict and suite intervals alongside the Trial results and Arm
 minutes. Both views also show each Arm's tokens by observed Model and its cost
@@ -1805,7 +1829,7 @@ versions or end times where those facts were not retained.
 Each invocation that does work keeps a Command record, Command log and Activity
 log lines under `<logs.dir>/JacobStephens2/thirdshift/commands/experiment/`
 (the Activity log is at the repository level). The record names both Arms and
-the Bar and moves through Arm builds, Eval tasks and Trials. It appears in the
+the Bar and moves through Waiting for Commands when needed, Arm builds, Eval tasks and Trials. A waiting Command record includes `waiting_for`, an array of Command names; it is omitted when empty. It appears in the
 Inbox: a pass goes in Done, and a fail, inconclusive Verdict or waiting rechecks
 go in Needs you. Rechecking a Score updates the latest invocation's Command
 outcome and Inbox group alongside the saved Verdict, keeping its completion time.