Your Scores and calibration #1027

Parent

#978

What to build

• thirdshift scores <pull request> --mine asks for your five Scores, each pass or fail with a reason. It shows the model's Scores only once you've entered all five.
• The first 20: the first Scoring run picks 20 past pull requests for you to score, spread across Harnesses and Models, and the Grader scores them too.
• Two a week: after that, two pull requests from each week's sample are marked for you.
• Calibration: a criterion's model Scores count as calibrated while they match yours on at least 18 of the last 20 pull requests you both scored. Otherwise they show as uncalibrated. A criterion that never calibrates stays marked, and doesn't hold up the others.
• thirdshift scores [<pull request>] [--json] shows each criterion's calibration beside its Scores.

Acceptance criteria

• --mine hides the model's Scores until all five of yours are entered.
• The first Scoring run picks 20 past pull requests spread across Harnesses and Models, and later runs mark two a week.
• Calibration follows the 18-of-20 rule for each criterion, and an uncalibrated criterion doesn't change the others.
• thirdshift scores --json shows each criterion's calibration.