[JacobStephens2]

These renames and new terms come from the #698 grilling. They match the words that eval tools and Anthropic's eval writing use.

Renamed

• Replay → Experiment. LangSmith defines an experiment as "Results of evaluating a specific application version on a dataset", and Braintrust, Langfuse and Phoenix use the same word. In those tools each version's pass is an experiment of its own, so one Experiment here holds two of theirs.
• Grade → Score. In Anthropic's "Demystifying evals for AI agents", graders produce scores: "combined grader scores must hit a threshold". Inspect and Langfuse also call the result a score, and LangSmith calls its feedback "the scores from evaluation". Grader stays, since it's Anthropic's word.

Widened

• Eval now covers both kinds the tools distinguish. Offline, it runs on an Eval suite in an Experiment; online, it runs on sampled merged work. These are LangSmith's "offline evaluation" and "online evaluation".
• Grader and Score now cover a Run in the production sample as well as a Trial.
• Command now lists the Scoring run.

Added

• Scoring run: the weekly Command that does thirdshift's online evaluation. It scores a sample of merged factory pull requests from every machine, and proposes a Bug origin for each bug fixed that week.
• Run metrics: what a Run's Command record and Session logs show about how it went, each kept apart rather than combined into one score. The tools call tokens, latency and cost "metrics".
• Bug origin: the pull request that introduced a bug, and the Run that made it, as a session proposed it and the Day shift confirmed it.

Also

• The Avoid lists gain the tools' synonyms:
  • Eval suite: dataset;
  • Eval task: example;
  • Experiment: replay and A/B test;
  • Score: grade and feedback.
• The Security finding entry said its triage decision "is also its grade". Since "grade" is now a word to avoid, it now says the decision "settles whether it was real".

Trace and Session trace still join the glossary when Session traces are built (ADR 0019). Span and log record stay OpenTelemetry's own terms.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BnFUcJ51dAFoj97E8FSXQF