Repository navigation
Add the #698 evals grilling, split by round, and its research notes - #971
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record round 3's answers, check them against main's v0.13.0 Console, and ask round 4: the Console as one UI for Replays, the Replay command, the Baseline build, the bar's statistics, Run scores, Trace structure, old Session logs and the Spec split. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BnFUcJ51dAFoj97E8FSXQF
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BnFUcJ51dAFoj97E8FSXQF
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BnFUcJ51dAFoj97E8FSXQF
The grilling's single file becomes .scratch/grill-with-docs/698-evals/: one file per round, with the Settled list, the research results and the pending research in files of their own, and a README that indexes them. Questions are renumbered <round>.<question>. Rounds 5 to 7 are new, and every question is answered. docs/research gains the four notes behind Rounds 5 and 6: eval viewers, eval runners, the OTLP file writer and eval sandboxes. The landscape note's references to Q2 now read Q1.2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BnFUcJ51dAFoj97E8FSXQF
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The grilling, split by round.
.scratch/grill-with-docs/698-evals.mdbecomes the folder.scratch/grill-with-docs/698-evals/. It holds:round-1.mdtoround-7.md;settled.mdfor what follows from the answers;research.mdfor findings, andwaiting.mdfor research still pending;README.md, which indexes the rounds and says where each part goes.Questions are renumbered
<round>.<question>, so Q55 is now Q6.2.mainhas only the first round. Rounds 2 to 7 are new here, and every question is answered.Four research notes behind Rounds 5 and 6:
docs/research/eval-viewers.md,eval-runners.md,otlp-file-writer.mdandeval-sandboxes.md.docs/research/eval-tools-landscape.md: its three references to the grilling's Q2 now read Q1.2.Why
The four Specs that come out of #698, for Session traces, Experiments, the Scoring run and Run metrics, will link to these decisions and notes, so they need to be on
main.Notes
eval-sandboxes.mddescribes thethirdshifthost's setup, including its loopback listeners and memory, as the grilling's earlier research results do.🤖 Generated with Claude Code
https://claude.ai/code/session_01BnFUcJ51dAFoj97E8FSXQF