You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Evals layer: test data and calibration metrics for rules #116
Evals layer: test data and calibration metrics for rules
Summary
A rule is only trustworthy if it's measured. Proposal: an evals layer where each rule — especially judge-backed semantic rules — carries or references a small labelled test dataset, and rule effectiveness is measured before and while the rule is in use.
What it looks like
Test fixtures: small labelled node sets per rule, with a deliberate mix of violation and non-violation cases — including tricky ones (e.g. a child that merely restates its parent, a well-framed node that looks like a violation).
Calibration metrics: recall/specificity of a rule against its labels, surfaced per rule and per schema.
Same invocation everywhere: a rule runs against its fixture set or a live space with the same command, so iteration against real data is cheap and the results carry straight into shipped behaviour. Experimentation-friendly by design — not an internal-only harness.
Refreshable calibration sets: judges calibrated on curated fixtures can still fail on realistic content; the layer should make it easy to add fixtures drawn from real-world cases and re-measure.
Why
Prevents shipping rules that look right and misfire. Published experience with LLM-as-judge evaluation shows judges that look calibrated on hand-made sets can saturate (near-100% error) when run against production data — upstream quality issues in the data being judged are a common cause.
Measurable methodology: turns published discovery-quality guidance (e.g. what makes a well-framed opportunity) from prose into something with testable, versioned effectiveness.
Terminology note (from the skill-eval exploration running in parallel):
To keep "eval" unambiguous across the emerging eval layers (skills / rules / content), a qualified vocabulary is now in use — eval terms are object × purpose, so they compose and relate cleanly. Relevant to this issue:
Rule fixture — a labelled case set used to calibrate a judge rule (deliberate violation/non-violation mix, including tricky cases). Distinct from an eval fixture (seeded, deterministic environment data a skill case runs against).
Rule calibration eval — the purpose-qualified name for what this issue builds: a judge rule (or skill judge) run against its rule fixtures → recall/specificity, surfaced per rule and per schema.
Content judge vs skill judge — an LLM judge grading space content (a semantic rule) vs one grading a candidate skill's run output/trajectory.
Bare "eval" isn't used unqualified in artefacts.
This also sharpens the relationship with #38: that issue is about skill regression evals (the object is an agent skill), this one about rule calibration evals (the object is a judge rule). Different objects, same machinery class — rubric + invocation + labelled fixtures + calibration metrics — so ideas should transfer even where implementations stay separate.
Decision: calibration gates blocking. An uncalibrated judge rule reports but can't block downstream evaluation. It becomes eligible to block (per #115) only once it has a labelled calibration set and measured recall/specificity above a threshold. Thresholds and how calibration status is recorded are still open here.
Idea for the evals layer: build on the live judge tests from #151.
judge-test/ holds a fixture space (schema, config, nodes) and runs sctx validate --json --judge <name> for each provider whose API key is set. It asserts verdicts and gating on clear-cut cases. It is opt-in with bun run test:judge (SCTX_LIVE_JUDGE_TESTS=1).
To grow that into evals:
Fixtures as datasets. Use one fixture space per schema, each with labelled expected verdicts. Hard cases go in alongside the clear-cut ones.
A provider × schema matrix. Run every configured provider on every fixture space.
Report scores instead of failing on a mismatch. Measure agreement with the expected labels, calibration (confidence compared with correctness), cost and latency.
A calibration case for this layer, from #152: a rule with `semantic.scope` or `semantic.includeContent` is offered the `missing-context` verdict, and the provider is told to use it only where a trim marker shows content was cut. Measure how often a provider answers `missing-context` when nothing was trimmed. A non-trivial rate means the option itself skews verdicts. The batching drift test in `tests/judge/live.test.ts` is a starting point.
Activity
mindsocket commentedon Sep 16, 2026
Terminology note (from the skill-eval exploration running in parallel):
To keep "eval" unambiguous across the emerging eval layers (skills / rules / content), a qualified vocabulary is now in use — eval terms are object × purpose, so they compose and relate cleanly. Relevant to this issue:
This also sharpens the relationship with #38: that issue is about skill regression evals (the object is an agent skill), this one about rule calibration evals (the object is a judge rule). Different objects, same machinery class — rubric + invocation + labelled fixtures + calibration metrics — so ideas should transfer even where implementations stay separate.
mindsocket commentedon Sep 25, 2026
Decision: calibration gates blocking. An uncalibrated judge rule reports but can't block downstream evaluation. It becomes eligible to block (per #115) only once it has a labelled calibration set and measured recall/specificity above a threshold. Thresholds and how calibration status is recorded are still open here.
mindsocket commentedon Oct 14, 2026
Idea for the evals layer: build on the live judge tests from #151.
judge-test/holds a fixture space (schema, config, nodes) and runssctx validate --json --judge <name>for each provider whose API key is set. It asserts verdicts and gating on clear-cut cases. It is opt-in withbun run test:judge(SCTX_LIVE_JUDGE_TESTS=1).To grow that into evals:
mindsocket commentedon Oct 16, 2026
A calibration case for this layer, from #152: a rule with `semantic.scope` or `semantic.includeContent` is offered the `missing-context` verdict, and the provider is told to use it only where a trim marker shows content was cut. Measure how often a provider answers `missing-context` when nothing was trimmed. A non-trivial rate means the option itself skews verdicts. The batching drift test in `tests/judge/live.test.ts` is a starting point.