[JacobStephens2]

Summary

Experiments and Process evals keep suspended Trials' outcomes and costs while leaving their time out of comparisons, summaries and future timing estimates. Results use format 11 and reports disclose nonzero exclusion counts.

Command records Suspended time during a Trial
  → retain suspension in the Trial's attempt history
  → keep outcome, Scores, usage and cost
  → exclude the Trial from timing
  → report the number excluded; no rerun

Closes #2035

Evidence

Execution evidence

• Repair validation at 7335ab3d: scripts/affected-tests issue-2026 passed: 68 site tests, cargo build --locked, 198 Console browser tests, and 4,249 Rust tests (97 existing skips). The selector measured cargo nextest run at 3,381.94s wall time including compilation; Nextest reported Summary [3178.451s] 4249 tests run: 4249 passed (23 slow), 97 skipped. The exact failing Pickup test also passed in isolation (1 passed, 0 failed). No tracked files changed, no commit or push was needed, and CI was not re-run.

• Before: cargo test --test experiments suspension::suspended_trials_keep_outcomes_and_cost_but_leave_time_out_of_reports -- --exact
  failed: left: Number(6.0), right: 4.0.
  After: the same command passed: 1 passed; 0 failed.

• cargo test --test experiments suspension:: -- --test-threads=2 passed all 3 regressions, including live Experiment and Process eval suspension and comparisons using the remaining timing samples.

• The review's schema reproduction failed before the fix with AssertionError: {'const': 10, 'type': 'integer'} and passed afterward (exit 0):

  target/debug/thirdshift console openapi | python3 -c 'import json,sys; d=json.load(sys.stdin); s=d["paths"]["/api/v1/experiments/{id}"]["get"]["responses"]["200"]["content"]["application/json"]["schema"]; assert s["properties"]["format"]["const"] == 11, s["properties"]["format"]'

• cargo test --test experiments result_format::documented_format_matches_real_experiment_results -- --exact passed, retaining agreement checks for both Experiment endpoint schemas.

• cargo test --test experiments accounting::estimates_use_each_arms_own_history_and_include_frozen_costs -- --exact failed with a suspended 9-minute estimate before the change and passed afterward, preserving its $9 cost.

• node --test --test-name-pattern='suspended Trials disclose' console/tests/experiments.test.cjs passed on desktop and mobile.

• The implementation session ran scripts/affected-tests refs/remotes/origin/issue-2026: site (68 tests), locked build and Console (198 tests) passed. Its two stale accounting expectations were corrected for time_pairs, and the exact failing test passed.

• Earlier validation passed in CI at e6d96c39: 4,243 Rust tests passed, 97 skipped, with cargo nextest run reporting Summary [1490.357s] 4243 tests run: 4243 passed (3 slow), 97 skipped. Test compilation took 1m 41s. Formatting, clippy, generator, site and Console checks passed; both Linux artifact builds also passed. The continuation reuses successful CI for this exact head under docs/testing.md, without repeating the full suite.

The implementation's local /usr/bin/time -p cargo nextest run reached 1,126 tests before the existing OTLP timing check failed (baseline 3.269s; stalled 5.420s; local wall time 1,453.43s, compilation included). Its exact isolated check, cargo test --test otlp_endpoint a_stalled_receiver_cannot_delay_a_session_by_its_span_count -- --exact, then passed (1 passed, 6.52s); the full CI run above also passed.

Visual evidence

Before captures use base commit 408359cf; each After uses the same fixture, viewport and expanded state. Captures contain synthetic data.

Console Experiment - 1440×900

Console Experiment - 390×844

Architecture accounting explanation - 1440×900

Architecture accounting explanation - 390×844

Test seams

• Experiment and Process eval CLI execution, retained results, JSON/text reports and resume.
• Command recording at its clock seam: a child-only wall-clock shim with real monotonic timers.
• Console Experiment report in Chromium at desktop and mobile widths.
• Console OpenAPI response schemas compared with real Experiment results.

Review coverage

• Fresh independent reviews of the whole branch used git diff refs/remotes/origin/issue-2026...HEAD, with fixed point 0056ac95 and reviewed head e6d96c39.
• Standards: 0 findings. Spec: 0 findings. The earlier review's OpenAPI mismatch was already fixed, with its regression retained.
• Each reviewer inspected all 23 changed files, including the final accounting expectations and generated site/docs/index.html. Unread changed files: none. Generated-output exclusions: none.
• Reports and each axis's final files-read list are saved in .thirdshift-review-0SZeZh/standards.md and spec.md; summary.json records {"standards":0,"spec":0}. No further source changes or commits were needed.

Unaddressed findings

Standards

None.

Spec

None.

Security

None.

Security review outcome

No unaddressed introduced findings.

CI notes

• CI run 43157136759 failed in pickup_run::at_a_command_limit_of_2_two_pickup_runs_run_at_once_and_a_third_is_skipped_naming_the_count: unlocked Scenario::gh_state can read an empty gh-state.json while fake gh::save truncates and rewrites it; the test, helper and writer are byte-identical to issue-2026 (0056ac95), whose exported fake reproduced a zero-byte read after four saves, and the exact failing test passed locally. No code changed for this unrelated fixture race.

Merge Danger

Door: two-way

The code change can be reverted. Older retained Trials without suspension fields read as having no Suspended time. API consumers must accept Experiment result format 11.

Blast Radius: Experiments

Changes Experiment and Process eval timing reports and estimates; the published Console schema also advances to format 11.

Built with codex codex-cli 0.167.2 · gpt-6.1-sol · xhigh