Skip to content

Add paired Bayesian Verdicts for Experiments - #1123

Merged
JacobStephens2 merged 4 commits into
issue-977from
issue-999
Oct 23, 2026
Merged

JacobStephens2 merged 4 commits into
issue-977from
issue-999

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Summary

paired Trial Scores + fixed Bar
  → paired Dirichlet intervals for accuracy and wrong findings
  → pass / fail / inconclusive, with the paired dropout guard
  → posterior-predictive task estimates when inconclusive
  → JSON with the Bar, sampler provenance, counts and pass³

Adds the standalone Verdict module. The default Bar uses a signed ten-point tolerance (-0.10), a 95% credible level, one uniform pseudo Eval task, and 20,000 seeded draws.

Closes #999

Test seams

Evidence

  • Before: the focused regressions failed for the unimplemented interval, missing decision rules, retained void tasks, unreversed wrong findings, absent pass³, missing forecasts and missing JSON provenance.
    After: cargo test --bin thirdshift verdict::tests passes all 10 focused tests, including a sure loss inside the margin and the first forecast count reaching 0.8 assurance. Independent Python simulations corroborated the one-Trial interval and the small-gap forecast range.
  • cargo fmt --all --check and cargo clippy --all-targets -- -D warnings pass.
  • /usr/bin/time -p scripts/affected-tests issue-977 selected cargo nextest run: 2,386 passed, 56 skipped. Wall time 2,513.50s, including compilation (3m 20s); nextest execution 2,309.858s on Linux.

Merge Danger

Door: two-way

Blast Radius: Experiments

The statistical module has no session or storage dependencies. Forecasts assume new non-void Eval tasks and record their finite simulation budget; exhaustion returns an explicit unfinished estimate.

Review coverage

Reviewed against issue-977 (d05aa971). Standards: one naming finding, addressed by using “Eval task” in the three test names. Spec: no confirmed findings. Both reviewers inspected all seven changed files, including Cargo.lock.

Unread changed files: none on either axis. Generated-output exclusions: none. Final reports are saved locally as .thirdshift-review-bzWxTy/standards.md and .thirdshift-review-bzWxTy/spec.md.

Unaddressed findings

Standards

None.

Spec

None.

Built with codex · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 merged commit 7f88dd6 into issue-977 Oct 23, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-999 branch October 23, 2026 03:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant