Skip to content

Deepen Experiment assessment behind a read-only interface - #1308

Merged
JacobStephens2 merged 1 commit into
mainfrom
issue-1307
Oct 25, 2026
Merged

JacobStephens2 merged 1 commit into
mainfrom
issue-1307

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Summary

Experiment storage obtains the provisional Verdict, suite assessments and pending rechecks through one read-only grading assessment.

 Stored::refresh_verdict
-  rechecks::assess(&tasks, &mut results, &bar)
-    grading::judge + mutate Score + grading::decision + restore Score
+  grading::assess(&tasks, &results, &bar)
+    effective paired Scores + one projected scalar flip + whole-Experiment decision

Pending record serialization, human history publication, legacy upgrades and CLI/Console behavior stay compatible. Navigation now identifies grading as the assessment owner.

Closes #1307

Evidence

  • Before: recheck selection required mutable Trial results and coordinated separate Experiment judge/decision calls.
    After: storage and completed-Experiment upgrades use one shared-input assessment; five focused tests cover complete observations, whole-Experiment decisiveness across suites, separate flips, human overrides, Void pairs, validation errors, repeatability and unchanged inputs.
  • Passed: cargo check; cargo test --bin thirdshift evals::experiment::grading::tests (5); cargo test --test experiments rechecks:: (7); cargo test --bin thirdshift verdict::tests (10).
  • Final local validation: /usr/bin/time -p scripts/affected-tests main passed, selecting cargo nextest run: 2,935 passed, 62 skipped, no failures. Wall time 1,499.92 seconds, including compilation (55.97 seconds); nextest execution was 1,442.051 seconds. Timing depends on this host and build cache. Full output is saved beside the Session trace in .thirdshift-review-jzhTLU/validation.log.
  • CI passed for the reviewed head 674e6e5ba27e1d3fc83d9c1ed64b94e457189314, including Linux tests, site, Console, the macOS gate and release artifact checks.

Test seams

  • grading::assess(&[Task], &[TrialResult], &Bar): complete Verdict and pending records, repeatability, unchanged inputs, paired validation, effective human Scores, Void exclusion and whole-Experiment decisive flips.
  • Existing Experiment recheck CLI and generic Verdict tests remain at their current seams.

Review coverage

Independent Standards and Spec reviews used git diff main...HEAD, with main pinned at d414b0f108fe3702e47eaaa8fc469000396ca5a1. Both axes reported zero findings and read every changed file:

  • docs/code-navigation.md
  • src/evals/experiment.rs
  • src/evals/experiment/grading.rs
  • src/evals/experiment/grading/tests.rs
  • src/evals/experiment/rechecks.rs
  • src/evals/experiment/store.rs

Unread changed files: none on either axis. Generated-output exclusions: none. Full reports and each axis's final supporting files-read list are saved beside the Session trace in .thirdshift-review-jzhTLU/standards.md and .thirdshift-review-jzhTLU/spec.md.

Unaddressed findings

  • Standards: none.
  • Spec: none.
  • Security: none. Guidance-mode review of git diff d414b0f108fe3702e47eaaa8fc469000396ca5a1...HEAD plus supporting code found no introduced finding: the change is a pure in-process refactor (mutable flip-and-restore counterfactuals become projected single-scalar flips over read-only inputs) with no new trust boundary, principal, sink, file path, network call, or serialized-format change; aggregation, Void-exclusion, human-overlay and recheck-gating rules are preserved. No repository SECURITY.md or threat model exists. Validated with cargo test --bin thirdshift evals::experiment::grading::tests (5 passed), cargo test --test experiments rechecks:: (7 passed) and cargo test --bin thirdshift verdict::tests (10 passed). No proof-of-concept failed, so no fix was made; pre-existing findings, if any, are recorded only in the private security-review record.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Reversible implementation ownership change with no saved-format migration.

Blast Radius: Experiments

Verdict assessment and pending recheck selection; recording and views retain their existing interfaces.

Built with codex · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 merged commit 9df88d6 into main Oct 25, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1307 branch October 25, 2026 09:01
@JacobStephens2 JacobStephens2 mentioned this pull request Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deepen Experiment Verdict assessment behind one read-only interface

1 participant