The Eval task format, suites and time-safe repositories #1000

Parent

#977

What to build

• The format: the documented Eval task format. Each Eval task is a folder with a manifest that names:

  • its repository and commits: the base, and the change under review;
  • its session kind;
  • its inputs, such as a Spec's body as first published;
  • its success criteria: written ones for model Graders, and optional mechanical checks for code Graders.

• Suites: they live privately under ~/.thirdshift/evals/suites/<suite>/. Each one declares what it depends on (Factory skills, Session prompts, and Harness, Model and Effort defaults) and its Trials per Arm. Those declarations decide which suites a change touches.

• Preparing an Eval task gives a throwaway repository with:

  • history only up to the base commit;
  • no origin, tags or other branches;
  • its reflog expired and pruned.

  Preparation fails if git log --all shows any later commit. A Spec-review Eval task also gets a local bare repository as its origin.

• Proposals: a proposed Eval task, from Spec: the Scoring run, which scores a weekly sample of production work #978's Bug origins, has a state that keeps it out of its suite until it's approved. Ticket Approve a proposed Eval task #1021 adds the approval.

The format is documented in the repository. The Eval tasks are not.

Acceptance criteria

• The format is documented, with an example built from made-up data.
• Preparing fixture Eval tasks gives a repository at the base commit with no later commit, origin, tags or other branches, and a planted later commit fails preparation.
• A Spec-review Eval task gets a bare origin, and a push to it works.
• A suite's dependency declarations decide which suites a set of changed paths touches.
• Proposed Eval tasks are listed but never run.