thirdshift score: the Scoring run and its weekly sample #1024

Parent

#978

What to build

thirdshift score starts a Scoring run, a new kind of Command, as #978 describes.

• The Command: it writes a Command record, a Command log and Activity log lines in thirdshift's own repository's logs folder, under a new scoring folder. It shows in the Inbox, and can be Stopped. It's skipped, doing nothing, while another Scoring run is running on the machine. A failed Scoring run lands in Needs you.

• Its window runs from the end of the previous Scoring run's window to its own start. The first one looks back 7 days.

• The [scoring] section of the User config names:

  • the owners whose repositories are searched;
  • the sample size, about 10 by default;
  • the Grader's Harness and Model, and a second Harness and Model for when the first matches a Run's Model.

  Setup and the documented defaults describe it.

• The sample:

  • It searches GitHub for pull requests merged in the window whose body holds the Built-with marker, across those owners.
  • It leaves out every pull request whose issue carries the security-fix label.
  • It draws a random sample spread across the Harness and Model pairs that the Built-with lines name, and records the seed.
  • It saves the sample under ~/.thirdshift/evals/scoring/.

• thirdshift scores [--json] lists the sample by week.

Acceptance criteria

• An integration test with a fake gh that serves a window's merged factory pull requests checks that the sample is spread across Harnesses and Models, leaves out security-fix work, and records its seed.
• The Command record, Command log and Activity log lines are written, and a second Scoring run while one is running is skipped.
• A second window scores nothing twice, and the first looks back 7 days.
• thirdshift scores --json lists the sample by week.
• The [scoring] section is documented with the User config's defaults.