[JacobStephens2]

Summary

0.16.0 adds Experiments to compare a Candidate with a fresh Baseline before release, Scoring for weekly production samples, and Run metrics that keep each signal separate (#1122, #1138, #1141).

• thirdshift experiment and experiments [id] --json add resumable paired Trials, mechanical checks, blind model Graders and Bayesian Verdicts with accuracy/wrong-finding intervals and pass³ (Add resumable Experiment Command and Trial workers #1128, Add paired Bayesian Verdicts for Experiments #1123, Grade Experiment Trials and publish Verdicts #1131, Grade written Eval criteria with blind model Graders #1145). thirdshift recheck <experiment> <trial> pass|fail --reason <text> records human decisions; unreviewed borderline or decisive Scores keep the Verdict at waiting_on_rechecks (Keep Experiment Verdicts waiting for human Score rechecks #1157).

• New thirdshift evals actions: list, touched, prepare, verify, prepare-arms, import-replays and approve <suite> <task>. These add private manifests, repositories exposing only base history, cached Arm builds with fixed Harness paths/versions, replay import and CLI/Console proposal approval (Add private Eval task format and time-safe repositories #1120, Cache Arm builds and record fixed Harness paths #1124, Import #429 replay material as private Eval suites #1127, Approve proposed Eval tasks from the CLI and Console #1273).

• Hidden thirdshift --trial <request.json> runs Spec-review and Code review sessions, retains reports and allows one Resume when background work was killed (Add hidden Trial flag for Spec review sessions #1121, Add Code review Trials and suite execution #1135).

• thirdshift score samples merged factory PRs, excluding Security fixes, builds blind merge-time packets and collects five independent model Scores; scores [--json] reads retained samples (Add Scoring runs and weekly factory PR samples #1137, Build blind packets for sampled pull requests #1139, Score production samples with independent model Graders #1277). scores <PR> --mine takes all five human Scores before revealing model Scores, with each criterion calibrated at 18 matches among its latest 20 pairs (Add your Scores and criterion calibration #1282, Spec: the Scoring run, which scores a weekly sample of production work #1138).

• thirdshift bug-origins adds confirm/correct/reject actions for origins proposed during Scoring. Confirmations attribute the bug to a Run and create proposed code-review Eval tasks requiring separate approval (Propose Bug origins for closed bugs during Scoring #1276, Settle Bug origins and propose code-review Eval tasks #1287).

• thirdshift metrics <command> shows a Run and its child Runs; metrics shows independent weekly signals, with week and repository filters and text/JSON output (Show Run metrics and weekly trends #1150).

• Releases check touched Eval suites against the last release tag, wait for human rechecks and ask before continuing after fail/inconclusive Verdicts. Interrupted releases retain their Candidate and refuse to publish content absent from it (Wait for release Experiments before publishing #1235, Spec: Experiments, which measure a change before it reaches production #1122).

• Experiment Trials and Graders use bwrap and memory-capped scopes; session networking uses a provider allowlist, while Code Graders run offline. Experiments require bwrap, socat and a reachable systemd user manager (Sandbox Trial sessions in bwrap with per-Harness logins #1199, Trial sandboxes get their own network behind an allowlisting proxy #1214, Give every Trial and Grader a memory-capped scope #1225, Sandbox Experiment Trials and Graders #1231). Setup can save Claude Trial tokens; OpenCode Trials require opencode.api_key (Save Claude setup tokens in Credentials #1125, Sandbox Trial sessions in bwrap with per-Harness logins #1199). agy’s provider-side web search remains enabled (Trial sandboxes get their own network behind an allowlisting proxy #1214).

• Agent failures and timeouts fail Trials; external causes get up to two fresh retries, then drop the task from both Arms. More than 10% dropped makes the Verdict inconclusive; recovery obeys saved retry rules (Bound Void Trial retries and report causes per Arm #1133, Deepen Trial execution and fixed-Bar recovery #1310). Claude/Codex limits pause Experiments, including older Arms (Pause Experiments at Harness plan-limit warnings #1132, Spec: Experiments, which measure a change before it reaches production #1122); quota detection follows the launch HOME, and memory-cap evidence survives scope teardown (Make Recording own Interpretation construction #1296, Retain scoped memory evidence through shared execution #1294). Harnesses exiting 126/127 before any turn produce Void Trials on both platforms (Fix macOS test failures on main #1281).

• Completed Runs retain outcome, Repairs, CI rounds, stage times, Usual-range comparisons, review counts and separate Unaddressed Standards/Spec/Security counts (Keep Run metrics in completed Command records #1140, Record review finding counts in Run metrics #1144, Count Unaddressed findings by kind in Run metrics #1146). Session signals include tokens/cache by observed Model, reported cost, errors, denied/killed/repeated work, Harness-version trends and advisory destructive-command flags (Record Session-trace signals and Harness-version trends #1158, Flag destructive commands in Run metrics #1163). Confirmed Bug origins join their Run and week when metrics are read, using shared attribution and checkpoint readers (Count confirmed Bug origins against their Runs #1304, Unify retained pull-request attribution for Scoring and metrics #1312, Deepen Scoring's confirmed Bug-origin reads #1314).

• Console adds live Experiment pages and Scores/Bug origins pages (Add live Experiment pages to the Console #1270, Add Scores and Bug origins pages to the Console #1291). Command details include historical and late-arriving metrics, preserve zeroes and mark unavailable signals “Missing” (Spec: Run metrics, the signals each Run leaves, kept apart #1141, Show Run metrics and missing signals in Console details #1151). Dismiss/Undo shows immediate pending feedback and permits retry after failure (Show immediate feedback while dismissing Console rows #1259); Stages fills the window, including with Command details open (Let Stages fill the window's full width #1271).

• Experiment Traces join Trial, Resume and Grader evidence and report Arm tokens, reported cost and minutes (Join Experiment Trial traces and report Arm usage #1143). Selected Session evidence remains readable despite unrelated corrupt Trial metadata or usage spans (Read retained Trial Session evidence independently of accounting #1303).

• Replay imports fill missing Spec-review Trial inputs while preserving edits; new Experiments reject missing inputs before building Arms (Complete Spec-review Trial inputs before Experiment builds #1272). Complete Trial repository preparation uses isolated Git policy (Complete Trial repository preparation under isolated Git policy #1300). Scoring updates stay attached to the exact loaded sample file, and failed settlements preserve existing Eval proposals (Keep Scoring updates attached to retained sample files #1306).

• Undo leaves hidden History dismissals alone (Deepen Inbox evaluation for rows, refresh work and deadlines #1254). Same-second Claim hand-backs count in either label order (Count same-second Claim hand-backs in either label order #1263). macOS proxy tunnels keep accepted sockets blocking across idle periods (Keep macOS proxy tunnels blocking after accept #1320).

• Internal log, Experiment reader/storage, Trial execution/discovery, Code Grader and assessment refactors preserve existing formats (Share one Log inventory capture across Command observations #1248, Deepen Command reading with shared captured log activity #1260, Capture retained Experiment views in one observation #1284, Own code Grader execution in one lifecycle #1286, Deepen Trial results behind one Experiment storage interface #1289, Deepen Trial execution through completed mechanical Scores #1292, Deepen Trial Session discovery across retained views and Traces #1299, Deepen Experiment assessment behind a read-only interface #1308).

Also: site, docs and CI updates (#1256, #1269, #1301, #1316).

Version diff

diff --git a/Cargo.lock b/Cargo.lock
index 9ca3a6ad..8062a4ab 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -2184,3 +2184,3 @@ dependencies = [
 name = "thirdshift"
-version = "0.15.0"
+version = "0.16.0"
 dependencies = [
diff --git a/Cargo.toml b/Cargo.toml
index 3275f932..8a85d4c3 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -2,3 +2,3 @@
 name = "thirdshift"
-version = "0.15.0"
+version = "0.16.0"
 edition = "2024"