Repository navigation
Add paired Bayesian Verdicts for Experiments - #1123
Merged
Merged
Conversation
This was referenced Oct 23, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the standalone Verdict module. The default Bar uses a signed ten-point tolerance (
-0.10), a 95% credible level, one uniform pseudo Eval task, and 20,000 seeded draws.Closes #999
Test seams
verdict::judgeaccepts paired Eval task Scores and a Bar, and returns the serializable Verdict. This is parent Spec: Experiments, which measure a change before it reaches production #977's pre-agreed seam 2.Evidence
After:
cargo test --bin thirdshift verdict::testspasses all 10 focused tests, including a sure loss inside the margin and the first forecast count reaching 0.8 assurance. Independent Python simulations corroborated the one-Trial interval and the small-gap forecast range.cargo fmt --all --checkandcargo clippy --all-targets -- -D warningspass./usr/bin/time -p scripts/affected-tests issue-977selectedcargo nextest run: 2,386 passed, 56 skipped. Wall time 2,513.50s, including compilation (3m 20s); nextest execution 2,309.858s on Linux.Merge Danger
Door: two-way
Blast Radius: Experiments
The statistical module has no session or storage dependencies. Forecasts assume new non-void Eval tasks and record their finite simulation budget; exhaustion returns an explicit unfinished estimate.
Review coverage
Reviewed against
issue-977(d05aa971). Standards: one naming finding, addressed by using “Eval task” in the three test names. Spec: no confirmed findings. Both reviewers inspected all seven changed files, includingCargo.lock.Unread changed files: none on either axis. Generated-output exclusions: none. Final reports are saved locally as
.thirdshift-review-bzWxTy/standards.mdand.thirdshift-review-bzWxTy/spec.md.Unaddressed findings
Standards
None.
Spec
None.
Built with codex · gpt-6.1-sol · xhigh