[JacobStephens2]

Summary

An Experiment can now be given a name, and one without a name gets a readable default instead of a bare commit such as "71c7a497".

 thirdshift experiment <candidate> … [--json]
+  --name <text>                        # trimmed, one line, ≤ 100 chars; refused before any build
 thirdshift experiments [<id>] [--json]
+thirdshift experiments <id> --name <text> [--json]   # any state, even running; '' clears it

The name is kept beside the Bar, never inside it. It is not part of the Experiment's identity, so it never changes Trials, Scores or the Verdict and never refuses a resume.

name = saved name
    ?? names::default(description, details)        # the one default, shared by the CLI and Console
         same commit and no Arm setup change   → "A/A at <Candidate ref | short commit>"
         --release (saved, or in an older record's command) → "Release <Candidate ref>"
         otherwise                              → what differs, as before ("Model: x → y", "v1 → branch")

Shown in: the experiments list line and experiments <id> (<id> · <name> · <state> · …); --json as name plus name_set; and the Console's list, detail heading and browser-tab title. The live refresh carries the title over, so a rename from the CLI reaches an open page.

 experiment.json  (Stored)
+  name?          # absent in older records → default name
   details
+    release      # sticky once --release ran
 every experiment.json write
+  takes metadata.lock (brief), keeps the name on disk unless it is the rename itself

A running Experiment holds its run lock for the whole run, so a rename can't take that lock. Instead, every experiment.json write (the run's own saves, rechecks and renames) now takes a short metadata.lock. The run's saves keep whatever name is on disk, so a rename made mid-run survives the run's later writes. The resume command shown for an Experiment leaves --name out, so following it keeps a later rename.

names::setup_changes is now the single list of Arm differences (Harness, Model, Effort, Harness version). The default name and the Console's "What it tests" both use it, so a page can't be titled "A/A" while it lists a change.

Tested at two pre-agreed seams:

1. the CLI (thirdshift experiment … --name, thirdshift experiments <id> [--name] [--json]), driven through the fake Harnesses in tests/experiments/names.rs and tests/experiments/release.rs;
2. the Console's Experiment list and detail pages, in tests/console/experiments.rs and console/tests/experiments.test.cjs.

tests/experiments.rs's Fixture now copies the Arm binaries with support::copy_executable. The in-process fs::copy hit ExecutableFileBusy locally once the new tests ran in parallel; that helper exists for exactly this race.

Evidence

• Before: the new tests fail on main's behaviour. For example:

  names::a_named_experiment_shows_its_name_and_renames_never_refuse_its_resume  FAILED
  release::release_uses_the_last_tag_…  left: Null  right: "Release candidate"
  Console "A CLI rename reaches an open Experiment page and its tab title"
     → page.waitForFunction: Timeout (heading updated, tab title stayed)
  

  After:

  cargo test --test experiments -- names:: release::      10 passed
  cargo test --test console -- experiments::              8 passed
  node --test console/tests/experiments.test.cjs          5 passed
  npm test --prefix site                                  51 passed
  

• Behaviours pinned by tests: a paused --name Experiment renamed and then resumed (no Bar refusal; the name is kept); a rename while it runs survives the run's later writes; --name '' brings back main (merge base) → candidate; an A/A run reads A/A at v0.1.0; a release reads Release candidate; , a two-line name and 101 characters are refused before any Experiment exists, and 100 characters are accepted; a record without name gets its default in the CLI and on both Console pages.

• Full Rust suite (cargo test --no-fail-fast, head 7688a01, before merging current main; after the merge, the experiments, console and unit tests above and the Console browser tests passed again): 3250 passed, 44 failed. Every failure is a sandboxed test on a host without socat (sandbox failed to start: socat is not on PATH, or a Trial voided as sandbox_unavailable), plus one bug_origins settlement test that needs Python's tomllib, which this host's Python lacks. None is in code this PR changes; CI has both tools. scripts/affected-tests main selected the full suite and stopped at the same sandbox unit tests.

Merge Danger

Door: two-way

The only new field written to experiment.json is an optional name (plus the release default). Older binaries ignore unknown keys there, and older records load with defaults.

Blast Radius: Experiments

The plain-text experiments line gains a name column after the ID, so a script that splits that line by position would shift; --json only adds keys. Stored::save now takes &mut self and a brief file lock on every metadata write.

Review

Reports: Standards 9 findings (judgement calls, no hard violations), Spec 3. Fixed: the shown resume command undid renames (Spec 1 / Standards 8); older release records lacked the release default (Spec 2); the A/A check ignored a Harness version change (Standards 5); the store's module doc missed the new lock (Standards 3). Commits 59adfc7 and 7688a01 make those fixes after the review pass; they touch only files the review read. Every changed file was read by both axes, and site/docs/index.html is generated output, confirmed with npm run docs:check and also inspected.

Unaddressed findings

Standards

• 1. Duplicated lock/set-name/write shape in store.rs. Declined: store::rename must read the record under metadata.lock. A nested flock on a second file description from the same process would block on itself, so it can't call Stored::rename. Each body is three to five lines (src/evals/experiment/store.rs, fn rename and Stored::save).
• 2. save silently replaces self.name with the name on disk. Declined: that invariant is what keeps a mid-run rename. It is stated in Stored::save's doc comment and pinned by names::renaming_a_running_experiment_survives_its_later_writes, and the only writer of the name is rename.
• 4. short duplicated in names.rs and render.rs. Declined: render::short serves several commit displays in the Console. Sharing a one-line helper would expose it crate-wide for no behaviour change.
• 6. The name is a plain String (Primitive Obsession). Declined: the rules are enforced at both entry points through names::parse, and name_set must stay its own JSON field, as the issue asks.
• 7. names::default reads like the Default trait. Declined: it is always called by its module path, names::default(…).
• 9. Unrelated Fixture copy change (Divergent Change). Kept and explained under Summary: the new tests triggered the existing ExecutableFileBusy race.

Spec

• 3. The Fixture copy change is outside add a way to name Experiments, right now the "71c7a497" title is pretty machine #1378's scope. Kept for the same reason as Standards 9, and disclosed above.

Closes #1378

Built with claude 2.1.312 · claude-opus-5-5 · high

Security review outcome

No unaddressed introduced findings.