Repository navigation
Closed
Description
Semantic rule evaluation: make descriptive rules executable (LLM-as-judge)
Summary
Rules currently have two execution modes:
- Structural — JSON Schema (AJV) validation per node
- Executable — JSONata expressions in
$metadata.rules(seedocs/rules.md)
A third class of rule — ones that "require judgment" — currently exists only as inert descriptive guidance. This issue proposes completing the #16 design: descriptive rules become executable through semantic evaluation, rather than introducing a separate rule kind. The descriptive rule text is the natural source for the judge's rubric.
Motivation
docs/torres-alignment-research.md§3 documents opportunity-framing rules that cannot be enforced structurally ("they require judgment") — e.g. framed in the customer's voice, problem-space not solution-space, not a business goal in disguise. These exist as descriptive rules today but are never evaluated.- The emerging pattern for AI-generated structured content pairs deterministic shape checks with semantic judges — deterministic checks act as auditors/guardrails; judges handle the quality dimension. A published example: Teresa Torres' Vistaly work uses deterministic tree-shape checks alongside LLM judges for grouping quality, and found upstream structural errors actively contaminate downstream judge accuracy.
- Structured quality guidance complements published discovery methodologies rather than just describing them.
Proposed shape (open to iteration)
- Evaluation mode: a rule declares that it needs semantic evaluation. The descriptive rule text (plus optional examples) becomes the judge rubric.
- Judge interface: provider-agnostic. The rule declares what it needs; configuration supplies how it's fulfilled.
- Opt-in and lazy: default
validatestays deterministic and free. Semantic rules evaluate only when explicitly requested (flag or config). Safe defaults automatically apply cascading — semantic rules should not run on nodes that failed structural or executable checks upstream (see cascading issue). - Sequencing and result synthesis live inside structured-context; the consuming agent loop stays external, working through the CLI.
Open questions
- Built-in provider vs external agent — does the CLI call an LLM API itself (provider config, keys, cost), or expose an interface that an external agent fulfils? Scope note: this project stops at markdown-centric modelling, CLI tools, and basic agent skills — either design must fit that boundary. This issue does not pre-decide it.
- Atomic vs batched evaluation — per-node/per-rule calls are precise but expensive and prevent cross-rule context in a single evaluation; batching is cheaper but risks missed details. Likely both, selectable per run.
- Caching — judge output for a node could be remembered when its inputs haven't changed, provided the cache key accounts for upstream/downstream dependencies. Needs a dependency-aware invalidation story.
- Experimentation-friendly by design — the same invocation should run a rule against a fixture dataset or a live space, so judge rules can be iterated against real data and shipped, not just tinkered with internally. (See companion evals-layer issue.)
- Category design — the
validation | coherence | workflow | best-practicecategories indocs/rules.mdare a useful working model but not locked in; grading of semantic results ("could be better" vs "is wrong") interacts with Rule categories and severity #60.
Relates
- Support descriptive and executable rules as schema metadata #16 (descriptive/executable rules — the design this completes)
- Rule categories and severity #60 (rule categories and severity)
- Layered rule evaluation: cascade gating and coverage annotation #115 (layered evaluation — cascading gates semantic rules)
- Create strict_ost schema aligned with Teresa Torres' Opportunity Solution Tree model #15 (strict_ost — proving ground for the first semantic rules)
docs/torres-alignment-research.md§3
Metadata
Metadata
Assignees
Labels
No labels
Activity
mindsocket commentedon Sep 25, 2026
Decision on the provider question: external fulfilment. sctx defines judge rules (the descriptive rule text is the rubric), sequences them after deterministic checks, emits judge requests and aggregates responses — it doesn't call an LLM itself. That keeps default
validatefree and deterministic and testable, and lets different runners fulfil requests (an agent using the structured-context skill is one; CI or an eval harness are others). Any schema, bundled or custom, can declare judge rules. The request/response format is still to be designed here.mindsocket commentedon Oct 7, 2026
Two additions from thinking about this more.
Jev as a fulfilment option. Jev (TypeSafe's decision model, via OpenRouter) is a narrow decision model that returns typed, calibrated judgments, rather than a general-purpose LLM. It looks like a good fit for judge rules: each judgment is cheap and fast, and the output is structured rather than prose that has to be parsed. External fulfilment keeps sctx out of provider choice, so this is a runner concern. Still, the request format should make it easy to fulfil with a decision model as well as a chat LLM: a closed set of verdicts and a confidence score, not free-form text. Worth trying early, with a strict_ost opportunity-framing rule as the first case (#15).
Rules declare the context they need. A judge rule needs its rubric, plus a declaration of how much of the space must be in the request for a human or model to make the judgment:
The declared context scope then drives:
feat(rules): semantic rules judged by model or decision model (#114)
feat(rules): semantic rules judged by model or decision model (#114)