Skip to content

Evals layer: test data and calibration metrics for rules #116

Open

Description

@mindsocket

Evals layer: test data and calibration metrics for rules

Summary

A rule is only trustworthy if it's measured. Proposal: an evals layer where each rule — especially judge-backed semantic rules — carries or references a small labelled test dataset, and rule effectiveness is measured before and while the rule is in use.

What it looks like

  • Test fixtures: small labelled node sets per rule, with a deliberate mix of violation and non-violation cases — including tricky ones (e.g. a child that merely restates its parent, a well-framed node that looks like a violation).
  • Calibration metrics: recall/specificity of a rule against its labels, surfaced per rule and per schema.
  • Same invocation everywhere: a rule runs against its fixture set or a live space with the same command, so iteration against real data is cheap and the results carry straight into shipped behaviour. Experimentation-friendly by design — not an internal-only harness.
  • Refreshable calibration sets: judges calibrated on curated fixtures can still fail on realistic content; the layer should make it easy to add fixtures drawn from real-world cases and re-measure.

Why

  • Prevents shipping rules that look right and misfire. Published experience with LLM-as-judge evaluation shows judges that look calibrated on hand-made sets can saturate (near-100% error) when run against production data — upstream quality issues in the data being judged are a common cause.
  • Measurable methodology: turns published discovery-quality guidance (e.g. what makes a well-framed opportunity) from prose into something with testable, versioned effectiveness.

Relates

Activity

  1. mindsocket commented on Sep 16, 2026

    @mindsocket
    OwnerAuthor

    Terminology note (from the skill-eval exploration running in parallel):

    To keep "eval" unambiguous across the emerging eval layers (skills / rules / content), a qualified vocabulary is now in use — eval terms are object × purpose, so they compose and relate cleanly. Relevant to this issue:

    • Rule fixture — a labelled case set used to calibrate a judge rule (deliberate violation/non-violation mix, including tricky cases). Distinct from an eval fixture (seeded, deterministic environment data a skill case runs against).
    • Rule calibration eval — the purpose-qualified name for what this issue builds: a judge rule (or skill judge) run against its rule fixtures → recall/specificity, surfaced per rule and per schema.
    • Content judge vs skill judge — an LLM judge grading space content (a semantic rule) vs one grading a candidate skill's run output/trajectory.
    • Bare "eval" isn't used unqualified in artefacts.

    This also sharpens the relationship with #38: that issue is about skill regression evals (the object is an agent skill), this one about rule calibration evals (the object is a judge rule). Different objects, same machinery class — rubric + invocation + labelled fixtures + calibration metrics — so ideas should transfer even where implementations stay separate.

  2. mindsocket commented on Sep 25, 2026

    @mindsocket
    OwnerAuthor

    Decision: calibration gates blocking. An uncalibrated judge rule reports but can't block downstream evaluation. It becomes eligible to block (per #115) only once it has a labelled calibration set and measured recall/specificity above a threshold. Thresholds and how calibration status is recorded are still open here.

  3. mindsocket commented on Oct 14, 2026

    @mindsocket
    OwnerAuthor

    Idea for the evals layer: build on the live judge tests from #151.

    judge-test/ holds a fixture space (schema, config, nodes) and runs sctx validate --json --judge <name> for each provider whose API key is set. It asserts verdicts and gating on clear-cut cases. It is opt-in with bun run test:judge (SCTX_LIVE_JUDGE_TESTS=1).

    To grow that into evals:

    • Fixtures as datasets. Use one fixture space per schema, each with labelled expected verdicts. Hard cases go in alongside the clear-cut ones.
    • A provider × schema matrix. Run every configured provider on every fixture space.
    • Report scores instead of failing on a mismatch. Measure agreement with the expected labels, calibration (confidence compared with correctness), cost and latency.
    • Keep it out of CI, as the live tests are now.
  4. mindsocket commented on Oct 16, 2026

    @mindsocket
    OwnerAuthor

    A calibration case for this layer, from #152: a rule with `semantic.scope` or `semantic.includeContent` is offered the `missing-context` verdict, and the provider is told to use it only where a trim marker shows content was cut. Measure how often a provider answers `missing-context` when nothing was trimmed. A non-trivial rate means the option itself skews verdicts. The batching drift test in `tests/judge/live.test.ts` is a starting point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions