[JacobStephens2]

Summary

Command records now say which thirdshift version ran them and, when they fail, where and how they stopped. Metrics can be sliced by version, and every failure cluster links through to its Commands.

 CommandRecord::new
+  thirdshift_version = CARGO_PKG_VERSION
 CommandRecord::finish(account)
   outcome, cause
+  if ended Err and not paused
+    stopped_at       = Recovery stage in record stage words, else last stage
+    failure_category = held | waiting | crashed (SessionFailure) | stage_failed
 Snapshot reading (metrics, details)
+  record without thirdshift_version
+    -> first Session trace's service.version, via the retained Session-signals cache
+       (in memory only; an unchanged trace is not parsed again)
 metrics trends / /api/v1/metrics / /metrics
+  --version / version= filter
+  versions: {rows by version (recorded / recovered), transitions per bucket}
+    Harness / Model transitions only once a filter narrows the view
+  where_runs_failed: rows by (stopped_at, category); a Command with no record of its own
+    falls back to its last stage and died / failed (observed / missing)
+  outcomes[].failed_commands: Failed selectors per bucket
 /metrics page
+  ◆ transition rows on the Outcomes timeline, By thirdshift version table
+  Where runs failed table: row -> History, name -> /peek Command details
+  Failed count per bucket -> History
 /history
+  stopped_at & category | failed_on & failed_kind & failed_account (+ the view's filters)
+    -> re-reads that metrics view and shows just those Failed Commands

The drilldown link names the cluster and the page's own filters, not the Commands in it. So it stays short however many Commands failed: a 120-Command cluster used to need a 13.8 kB URL.

New modules: src/logs/metrics/failures.rs (Where runs failed) and src/logs/metrics/versions.rs (version breakdown and transitions). failed_run::category classifies the typed error; recovery::Stage::recorded maps a Recovery stage to Command record stage words.

Closes #2067

Evidence

Execution evidence

• Before: the three new tests/metrics.rs tests, run against the base branch's code (origin/issue-2036 worktree with this PR's tests/metrics.rs). Command: cargo test --test metrics -- historical_versions a_narrowed where_runs_failed

  test a_narrowed_view_marks_harness_and_model_changes_between_buckets ... FAILED
  test where_runs_failed_groups_recorded_endings_and_falls_back_for_older_and_died_commands ... FAILED
  test historical_versions_come_from_session_traces_once_and_slice_metrics_by_release ... FAILED
    left: (Null, Null, Null)
   right: (Number(4), Number(2), Number(2))
  test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 45 filtered out
  

  After: the same command on this branch:

  test a_narrowed_view_marks_harness_and_model_changes_between_buckets ... ok
  test historical_versions_come_from_session_traces_once_and_slice_metrics_by_release ... ok
  test where_runs_failed_groups_recorded_endings_and_falls_back_for_older_and_died_commands ... ok
  test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 45 filtered out
  

• Before: the review's P1 test, as written, on ee55846. Command: cargo test --test console failure_drilldown_links_stay_within_a_proxy_request_line

  panicked at tests/console/metrics.rs:450:13:
  13816 bytes
  test result: FAILED. 0 passed; 1 failed
  

  After: the same command, plus where_runs_failed_and_failed_buckets_open_the_matching_commands_in_history:

  test metrics::failure_drilldown_links_stay_within_a_proxy_request_line ... ok
  test metrics::where_runs_failed_and_failed_buckets_open_the_matching_commands_in_history ... ok
  test result: ok. 2 passed; 0 failed
  

• Unit tests: cargo test --bin thirdshift covers a_command_records_its_version_at_start_and_where_a_failure_stopped_at_its_end (Recovery Self-merge + MergeHold → held; no Recovery → last stage + stage_failed; plan limit → nothing) and the_failure_category_comes_from_the_typed_error_not_its_words. Both pass.
• Full suite: cargo test --no-fail-fast on this branch: every target passes except tests/process/findings.rs::merged_fix_is_rechecked_after_a_later_fix_fails. That test fails the same way on the base branch: I ran cargo test --test process merged_fix_is_rechecked in a detached origin/issue-2036 worktree and got test result: FAILED. 0 passed; 1 failed. npm test --prefix console: 201 pass, 0 fail. npm test --prefix site: pass. npm run docs:check --prefix site: clean.

Visual evidence

/metrics?window=7d with a fixture of 21 Ticket runs across two thirdshift versions, some recording failure attribution and some falling back.

Desktop 1440×900:

Mobile 390×844 (the new tables scroll inside their own regions; the page does not overflow, as checked by console/tests/metrics.test.cjs in every Theme):

Review

The thirdshift-code-review reviewers worked against refs/remotes/origin/issue-2036 at ee55846: Standards raised 8 findings, Spec raised 4. Both axes read every changed file; the Spec axis needed one follow-up for two files. site/docs/index.html was excluded as generated output: npm run docs:check --prefix site is clean, and both reviewers reviewed docs/guide.md instead. The fixes in 69ab184 came after the review and touch only files the reviewers had already read.

Addressed:

• S1 (bucket half): outcomes[].failed_commands is now in the JSON, so the CLI and API can name what the page links.
• S2: renamed to failed_commands. Stopped.recorded is now Ending::recorded().
• S3: Ending { Recorded(FailureCategory), Died, Failed } replaces the tuple and strings.
• S4: added CommandRecord::last_stage. query_values is gone. A bar macro is shared by both tables.
• S7: the template compares against Class::Failed.
• S8: renamed to Dimension::label.
• P1: drilldown links are bounded. History re-reads the view from its filters and the cluster's key.
• P2: a version link keeps the view's window and filters. On ee55846 the template linked href="/metrics?version={{ version|urlencode }}", which drops them. P2's assertion now passes, with & in place of & because that is how Askama escapes &.

Unaddressed findings

Standards

• S1 (History half): the History drilldown filter has no CLI twin (ADR 0016/0026). Declined here. History's existing outcome= and repository= filters are already page-only (src/console.rs:1409, read with query_value; thirdshift status --local --all --json takes neither). The drilldown also has a CLI path: metrics --json now lists every cluster's and bucket's selectors (where_runs_failed.rows[].commands[].selector, outcomes[].failed_commands) for thirdshift show <selector>. Whether History's filters need a CLI twin is a call for the Day shift, so I filed it as History filters (outcome, repository, Metrics failure drilldown) have no CLI twin #2084.
• S3 (part): CommandRecord.stopped_at is a String, not StageName. Declined. Persisted stage names are deliberately strings so they stay tolerant of new stages (src/logs/command_record.rs:246, "persisted names stay tolerant of new stages"). stages[].name is stored the same way, and the fallback compares against those strings.
• S5: the filter set is a data clump / shotgun surgery. Declined. This predates the change (the reviewer says so), and this diff only follows it. The Console's list now lives in one FILTERS constant (src/console/metrics.rs).
• S6: a reader-only version_recovered field sits on the persisted record. Declined. It follows the existing precedent in the same struct, #[serde(skip)] pub end_time_estimated: bool ("Reader-only provenance", src/logs/command_record.rs:112).

Spec

• P3: Commands stopped by a person also record attribution. Declined. An interrupted ending that nobody asked to stop counts as Failed (src/logs/metrics/outcomes.rs:84-87), so it needs stopped_at and failure_category to be attributed. CommandRecord::finish cannot tell the two apart: a Stop request is a separate file, read by logs::stop::requested. A stopped Command's attribution never reaches Where runs failed, which reads Failed Commands only. docs/guide.md documents that interrupted endings record it.
• P4: the History drilldown filter is Console-only. This is the same gap as S1's History half. Declined for the same reason and filed as History filters (outcome, repository, Metrics failure drilldown) have no CLI twin #2084.

Note from the Spec review: stopped_at uses the Recovery stage in Command record stage words, which is coarser than the recorded stages (Implementation and Resume both become implement; Push becomes pull request). A new record and an older fallback record can therefore name the same failure differently.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

The new record fields are optional and additive (skip_serializing_if = "Option::is_none"). Older binaries ignore them, and older records stay unchanged on disk.

Blast Radius: contained

• The extra service.version field invalidates nothing: the retained Session-signals cache is already keyed by executable identity.
• Version recovery parses the first Session trace of any older record that has no thirdshift_version, including kinds without Run metrics. That happens once per trace per binary, and reads after that use the cache.
• The /metrics JSON gains where_runs_failed, versions and outcomes[].failed_commands.
• History gains query keys that it otherwise ignores.

Built with claude 2.1.314 · opus · default effort