Skip to content

Grade Experiment Trials and publish Verdicts - #1131

Merged
JacobStephens2 merged 2 commits into
issue-977from
issue-1004
Oct 23, 2026
Merged

JacobStephens2 merged 2 commits into
issue-977from
issue-1004

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Summary

Trial repository changed → void + fresh retry
Trial reports + mechanical checks → Score with reasons
paired suite Scores → Verdict + intervals + Arm minutes
Command outcome → pass: Done; fail/inconclusive: Needs you

Closes #1004

Test seams

  • The whole binary against fake Harnesses: Experiment results, Scores, Verdict, Command outcome, Inbox grouping, private Grader evidence and Arm minutes through the CLI.

Evidence

  • Before: the new integration test found no Verdict. Repository mutations remained succeeded, and private check output entered factory logs.
    After: the Experiment integration tests pass with Scores, intervals, correct Inbox grouping and private results.
  • Before: both initial Spec reproductions failed: ignored files remained succeeded; resumed minutes omitted limit-paused sessions. The follow-up reproduction showed generated instruction fallback links incorrectly making clean Trials void.
    After: cargo test --test experiments spec_review_ -- --nocapture passes all 3 retained regressions, covering agy and OpenCode too.
  • Final validation: formatting and cargo clippy --all-targets -- -D warnings pass locally and in CI. The full Rust CI job, site and Console jobs pass on ea4dae0f9bcbc0e95e457cba0171b8fc84125666.
  • NEXTEST_TEST_THREADS=2 NEXTEST_STATUS_LEVEL=pass /usr/bin/time -p scripts/affected-tests b8b9ec9 passed all 43 site tests. Its redundant local Rust run was stopped after the exact-head full Rust CI job passed, following docs/testing.md's CI-reuse rule. The selector took 782.73 seconds including compilation and the interrupted Rust run; this is not a completed full-suite benchmark.
  • An earlier local Rust run stopped at a Console password-code HTTP timeout. That exact test passed separately and the final full Rust CI job is green.

Merge Danger

Door: two-way

Blast Radius: Experiments

Private results gain mechanical Scores, suite Verdicts and minutes. Model grading and Trial isolation remain separate tickets.

Unaddressed findings

Standards

None. The P3 Arm-minutes representation finding was addressed with a typed record; the reviewer confirmed the correction.

Spec

None. All 3 P2 findings were reproduced with the reviewers' tests, fixed, and retained as regressions. No decision was deferred and no triage issue was needed.

Review coverage

Both axes read all 16 changed files, including the generated manual's changed hunks. Unread changed files: none. Generated-output exclusions: none.

Fixed point: issue-977, pinned to its fetched remote tip b8b9ec9c7c42a7d0e47b50ddca85509782cb85d6. Reports, final files-read lists and dispositions are saved locally in .thirdshift-review-b240Hz/standards.md and .thirdshift-review-b240Hz/spec.md.

Built with codex · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 merged commit 105f581 into issue-977 Oct 23, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1004 branch October 23, 2026 05:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant