Skip to content

Grill #1484: process Eval suite decisions, ADRs 0027–0028 and research - #1890

Merged
JacobStephens2 merged 14 commits into
mainfrom
eval-suite
Nov 9, 2026
Merged

JacobStephens2 merged 14 commits into
mainfrom
eval-suite

Conversation

@JacobStephens2

Copy link
Copy Markdown
Owner

Summary

Documentation only: the #1484 grill, the decisions it reached, and the research behind them. Specs #1484 and #1872 cite these files.

 GLOSSARY.md                        # Arm names a Process; Trial may be many sessions;
                                    # Bar fixes who grades; new term: Grader panel
 docs/adr/
+├── 0027-a-trial-carries-out-a-process-but-is-not-a-command.md
+└── 0028-who-decides-a-verdict-is-fixed-in-the-bar.md
 docs/grill-with-docs/done/
+└── 1484-implement-suite.md        # 10 rounds, Q1.1–Q10.2
 docs/research/
+├── eval-process-arms-and-human-review.md   # Langfuse et al., external runners, blind review
+└── eval-grader-panel.md                    # same-family Graders, panel size, Kimi/DeepSeek Harnesses

What the grill decided, and where it was published:

Evidence

Execution evidence

No code changes. scripts/affected-tests origin/main selects the whole cargo test suite (the glossary changed). thirdshift is Linux-only, so it runs in the Lima VM. The result is in a comment below, and CI runs it too.

Merge Danger

Door: two-way

Revert the commit. The Specs would then cite missing files.

Blast Radius: documentation

The glossary wording for Arm, Trial and Bar changes for every agent that reads it. The new wording describes behaviour #1484 and #1872 will build, so today's code doesn't yet match the Arm's Process clause, a multi-session Trial or the Bar's grading settings.

@JacobStephens2

Copy link
Copy Markdown
Owner Author

Execution evidence: scripts/affected-tests origin/main at fd89f44, run on Ubuntu 24.04 aarch64 in the Lima VM. Every changed file triggered a conservative full-suite selection, which ran cargo test → PASS in 999 s (3929 passed, 0 failed, 91 ignored).

@JacobStephens2
JacobStephens2 merged commit 0f27843 into main Nov 9, 2026
4 checks passed
@JacobStephens2
JacobStephens2 deleted the eval-suite branch November 9, 2026 21:43
@JacobStephens2 JacobStephens2 mentioned this pull request Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant