Skip to content

Spec: Run metrics, the signals each Run leaves, kept apart - #1141

Merged
JacobStephens2 merged 36 commits into
mainfrom
issue-981
Oct 25, 2026
Merged

JacobStephens2 merged 36 commits into
mainfrom
issue-981

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Summary

Run metrics keep each Run's friction and Session-trace signals separate, in its Command record and local read-only views.

Run completes
  Command record v2: outcome, Repairs, CI rounds, findings, stage times
  Session traces: tokens/cache by Model, reported cost, errors, denied/killed/repeated work
    advisory destructive-command checks (never fail the Run)

Local records + Session traces + confirmed Bug origins
  metrics <command> [--json]: Run and child Runs
  metrics [--json]: weekly signals by repository and session setup
  show --json / Console details: the same available Run metrics

Completion accounting keeps Unaddressed findings by Standards, Spec and Security from Delivery's final PR body, and review counts from machine-readable summaries, counted once across Resume. Stage durations are compared with repository Usual ranges. Every Ticket Run and Base fix keeps its own metrics.

Weekly trends keep each signal independent, split by repository, Harness, Harness version, requested Model, Effort, Run kind and session kind, with week/repository filters. Tokens retain the observed Model and cache split; cost is used only when reported. Historical reads reconstruct available signals, preserve missing values and never rewrite old records. Confirmed Bug origins are joined from local Scoring data when read.

The whole-Spec review fixed two integration gaps: historical metrics and late Bug-origin confirmations now appear in show --json and Console details through the same projection as metrics. Regression tests cover both views and a confirmation changing to a rejection between Console requests. Two misleading test names were corrected. No price table, combined score or remote metrics fetch is added.

Closes #981

Evidence

  • Before: targeted regression commands demonstrate that historical metrics and late Bug-origin confirmations disappear from show --json despite appearing in metrics.
    After: both exact commands pass; historical minutes are 30.0 and the late Bug-origin count is 1.
    cargo test --test issue_981_spec_review show_includes_metrics_reconstructed_for_a_past_run -- --exact --nocapture
    cargo test --test issue_981_spec_review show_includes_confirmed_bug_origins_from_read_time_scoring_data -- --exact --nocapture
  • Test seams: whole binary against fixture records/fake Harnesses, one normalized Session trace, and Console HTTP/browser details.
  • A further regression preserves legacy Usual ranges without inventing unrecorded historical minutes; it failed before the reader ordering fix and now passes alongside the Run-health and Run-metrics tests.
  • npm test --prefix console: 60 passed, including historical/unfinished Runs and phone-width details.
  • cargo fmt --check and cargo clippy --all-targets -- -D warnings: passed.
  • Required scripts/affected-tests main: the site tests pass; the Rust run encounters an unrelated host warning from the installed snapd.service in the strict deployment-unit check. SYSTEMD_UNIT_PATH=/usr/lib/systemd/system cargo test --test site_deploy passes all 7 deployment tests. All 7 targets after deployment (spec_push_race, spec_run, status, stop, toolchain, update, user_config) pass in a separate continuation with that same unit-path isolation.
  • The final selector run took 1,120.82 seconds (18m41s), including compilation and site checks, before the host's deployment warning stopped it. The continuation passed 156 tests; all selected Rust targets were verified across these runs.
  • Full Rust validation uses GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=tag.gpgSign GIT_CONFIG_VALUE_0=false to isolate tag fixtures from this host's global signing setting. User configuration is unchanged.

Unaddressed findings

Standards

None.

Spec

None.

Review coverage

Standards: 1 P3 naming finding, addressed. Spec: 2 P2 details-view integration findings, reproduced and addressed. Each reviewer read all 57 changed files, including generated output, plus supporting standards, domain definitions and ADRs. Unread changed files: none. Generated-output exclusions: none. Final reports and combined files-read lists are retained as standards.md and spec.md with the Spec-review Session evidence.

Merge Danger

Door: two-way

Readers accept Command records with or without metrics; historical reads do not rewrite them. Older binaries may omit version-2 records from their views, while the records remain on disk. A release carrying the changed Factory code-review skill requires an Experiment once #977 is available, as specified by #981.

Blast Radius: metrics

Completion recording, Session-trace accounting, CLI metrics, Console details and their documentation. Destructive-command flags remain advisory.

Built with codex · gpt-6.1-sol · high

Tickets

Keep Run metrics in completed Command records
Record review finding counts in Run metrics
Count Unaddressed findings by kind in Run metrics
Show Run metrics and missing signals in Console details
Show Run metrics and weekly trends
Record Session-trace signals and Harness-version trends
Flag destructive commands in Run metrics
@JacobStephens2
JacobStephens2 marked this pull request as ready for review October 25, 2026 07:36
Preserve Run metrics with the captured log inventory and shared Session call lifecycle. Keep details on the common read-time metrics projection, regenerate prompts, and retain the current Ticket run terminology.

Keep help within its line budget and bound test scope launchers during version probes. Rust unit and integration targets, site and Console tests, release-generator checks, clippy, and formatting passed across the affected-test run and continuation.
Preserve the shared read-time Run metrics projection while adopting retained Scoring sample handles. Keep settlement decisions attached to their exact source files and withdraw superseded Eval proposals only after a successful save.

Validation: cargo check --tests; cargo fmt --check; cargo clippy --all-targets -- -D warnings; scripts/affected-tests main (passed in 22m03s with the installed Python 3.11 and test-only Git signing/systemd isolation). The default Python 3.9 lacked tomllib; the affected Bug-origin regression passed with Python 3.11 before rerunning the full selector.
@JacobStephens2
JacobStephens2 merged commit da52a5c into main Oct 25, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-981 branch October 25, 2026 09:26
@JacobStephens2 JacobStephens2 mentioned this pull request Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Spec: Run metrics, the signals each Run leaves, kept apart

1 participant