Skip to content

Add your Scores and criterion calibration - #1282

Merged
JacobStephens2 merged 3 commits into
issue-978from
issue-1027
Oct 24, 2026
Merged

JacobStephens2 merged 3 commits into
issue-978from
issue-1027

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Summary

score → historical 20 + weekly sample → independent Grader
scores <PR> --mine → five human Scores → reveal model Scores
paired Scores → last 20 per criterion → calibrated at 18 matches
Scoring traces → local only; launcher environment values → redacted

Closes #1027

Test seams

  • The whole binary against fake GitHub and Harness executables, including the terminal interaction and private Scoring records.

Evidence

  • Before: cargo test --test scoring mine_requires_all_five_scores_before_revealing_the_model failed on the unsupported CLI; the historical-selection test found zero marked PRs.
    After: terminal entry, the historical 20, weekly selection, independent rolling calibration and a history exceeding 1,000 PRs pass through the binary against fakes.
  • Before: both Spec regression commands reproduced credential retention and Scoring OTLP export.
    After: cargo test --test scoring grading::scoring_grader_ -- --nocapture passes both retained regressions. The token regression also checks that the sandbox still receives its real dummy credential.
  • cargo check, cargo fmt --check and cargo clippy --all-targets -- -D warnings pass. All 32 Scoring tests pass. The initial affected-test run found the existing compact-help limit; its unchanged regression passes after combining the Scores forms onto one line.
  • Final scripts/affected-tests issue-978 passes: 2,424 Rust tests, 57 skipped fixture/platform tests, and 43 browser checks. Used nextest with eight test threads: 1,416.714s Rust execution, 26.57s compilation, 1,539.66s for the complete affected-check command including browser checks, measured with /usr/bin/time -p.
  • GitHub CI and release/smoke checks pass at 3c15c2a. An earlier attempt failed while Git cloned a temporary release-test repository; cargo test --test release main_moving_during_a_completed_ci_read_is_followed -- --exact passed unchanged, and the failed CI job passed on rerun.

Merge Danger

Door: two-way

Blast Radius: Scoring

Adds private human Scores and selection metadata to existing checkpoints, with defaults for older records. Includes the #1026 Grader prerequisite in the fixed-point review. No destructive migration.

Unaddressed findings

Standards

  • Initial Spec: thirdshift, turn a GitHub issue into a ready-for-review PR unattended #2, dynamic JSON view (withdrawn): src/scoring.rs:295 enriches the existing serialized Week/PullRequest schema only at the presentation layer; Score and Criterion remain typed in src/scoring/grader.rs:18. A parallel typed view would duplicate the persisted schema. calibration_requires_twenty_pairs_tracks_eighteen_matches_and_rolls_independently exercises each criterion's JSON. The reviewer accepted this implementation decision and withdrew the suggestion.
  • Initial Walking skeleton: a fresh run on the happy path #3, separate Harness switches (withdrawn): src/proxy.rs:57 selects provider hosts, src/sandbox.rs:646 provisions credentials, and src/sandbox.rs:937 chooses mounts according to the binary's filesystem layout. These implement different decisions; combining them would couple distinct operations. The reviewer accepted this implementation decision and withdrew the suggestion.

Spec

None. Both privacy findings were reproduced, fixed and retained as regression tests.

Security

None.

Security review outcome

No unaddressed introduced findings.

Review coverage

Independent Standards and Spec reviewers used issue-978 (8905eb4) as the fixed point and reviewed the implementation at 5e32972. Each read all 42 changed files, including generated output; no exclusions and no unread changed files at that reviewed head. Standards: three suggestions fixed, two withdrawn. Spec: two findings fixed. Reports, including each axis's final combined files-read list, are saved in .thirdshift-review-Z0yy4R/standards.md and .thirdshift-review-Z0yy4R/spec.md beside the Session evidence.

Post-review unread change: src/main.rs:85 was subsequently compacted onto one help line at 3c15c2a; this final hunk was not reread by either reviewer. The rest of that file was read. The existing help_is_a_compact_reference_with_a_link_to_the_details regression passes unchanged. No other final changed file was left unread.

Built with codex · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 merged commit 92a47e7 into issue-978 Oct 24, 2026
24 of 25 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-1027 branch October 24, 2026 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant