Skip to content

Semantic rule evaluation: make descriptive rules executable (LLM-as-judge) #114

Closed

Description

@mindsocket

Semantic rule evaluation: make descriptive rules executable (LLM-as-judge)

Summary

Rules currently have two execution modes:

  1. Structural — JSON Schema (AJV) validation per node
  2. Executable — JSONata expressions in $metadata.rules (see docs/rules.md)

A third class of rule — ones that "require judgment" — currently exists only as inert descriptive guidance. This issue proposes completing the #16 design: descriptive rules become executable through semantic evaluation, rather than introducing a separate rule kind. The descriptive rule text is the natural source for the judge's rubric.

Motivation

  • docs/torres-alignment-research.md §3 documents opportunity-framing rules that cannot be enforced structurally ("they require judgment") — e.g. framed in the customer's voice, problem-space not solution-space, not a business goal in disguise. These exist as descriptive rules today but are never evaluated.
  • The emerging pattern for AI-generated structured content pairs deterministic shape checks with semantic judges — deterministic checks act as auditors/guardrails; judges handle the quality dimension. A published example: Teresa Torres' Vistaly work uses deterministic tree-shape checks alongside LLM judges for grouping quality, and found upstream structural errors actively contaminate downstream judge accuracy.
  • Structured quality guidance complements published discovery methodologies rather than just describing them.

Proposed shape (open to iteration)

  • Evaluation mode: a rule declares that it needs semantic evaluation. The descriptive rule text (plus optional examples) becomes the judge rubric.
  • Judge interface: provider-agnostic. The rule declares what it needs; configuration supplies how it's fulfilled.
  • Opt-in and lazy: default validate stays deterministic and free. Semantic rules evaluate only when explicitly requested (flag or config). Safe defaults automatically apply cascading — semantic rules should not run on nodes that failed structural or executable checks upstream (see cascading issue).
  • Sequencing and result synthesis live inside structured-context; the consuming agent loop stays external, working through the CLI.

Open questions

  1. Built-in provider vs external agent — does the CLI call an LLM API itself (provider config, keys, cost), or expose an interface that an external agent fulfils? Scope note: this project stops at markdown-centric modelling, CLI tools, and basic agent skills — either design must fit that boundary. This issue does not pre-decide it.
  2. Atomic vs batched evaluation — per-node/per-rule calls are precise but expensive and prevent cross-rule context in a single evaluation; batching is cheaper but risks missed details. Likely both, selectable per run.
  3. Caching — judge output for a node could be remembered when its inputs haven't changed, provided the cache key accounts for upstream/downstream dependencies. Needs a dependency-aware invalidation story.
  4. Experimentation-friendly by design — the same invocation should run a rule against a fixture dataset or a live space, so judge rules can be iterated against real data and shipped, not just tinkered with internally. (See companion evals-layer issue.)
  5. Category design — the validation | coherence | workflow | best-practice categories in docs/rules.md are a useful working model but not locked in; grading of semantic results ("could be better" vs "is wrong") interacts with Rule categories and severity #60.

Relates

Activity

  1. mindsocket commented on Sep 25, 2026

    @mindsocket
    OwnerAuthor

    Decision on the provider question: external fulfilment. sctx defines judge rules (the descriptive rule text is the rubric), sequences them after deterministic checks, emits judge requests and aggregates responses — it doesn't call an LLM itself. That keeps default validate free and deterministic and testable, and lets different runners fulfil requests (an agent using the structured-context skill is one; CI or an eval harness are others). Any schema, bundled or custom, can declare judge rules. The request/response format is still to be designed here.

  2. mindsocket commented on Oct 7, 2026

    @mindsocket
    OwnerAuthor

    Two additions from thinking about this more.

    Jev as a fulfilment option. Jev (TypeSafe's decision model, via OpenRouter) is a narrow decision model that returns typed, calibrated judgments, rather than a general-purpose LLM. It looks like a good fit for judge rules: each judgment is cheap and fast, and the output is structured rather than prose that has to be parsed. External fulfilment keeps sctx out of provider choice, so this is a runner concern. Still, the request format should make it easy to fulfil with a decision model as well as a chat LLM: a closed set of verdicts and a confidence score, not free-form text. Worth trying early, with a strict_ost opportunity-framing rule as the first case (#15).

    Rules declare the context they need. A judge rule needs its rubric, plus a declaration of how much of the space must be in the request for a human or model to make the judgment:

    • Node-local: the rule judges the node's own content or attributes, e.g. "framed in the customer's voice" or "problem-space, not solution-space". The request carries the node and the rubric.
    • Relational: the rule judges the node against its neighbours up or down the hierarchy, e.g. "this opportunity is aligned with the goal/strategy above it", or "this opportunity has a reasonably broad range of solutions or experiments explored beneath it". The request must include those adjacent nodes: ancestors, descendants, or siblings, to some depth.

    The declared context scope then drives:

    • request size and cost (relational rules are much more expensive)
    • atomic vs batched evaluation (open question 2): node-local rules batch easily, relational ones less so
    • cache invalidation (open question 3): the declared scope is the dependency set for the cache key
    • gating (Layered rule evaluation: cascade gating and coverage annotation #115): a relational rule should be skipped if any node in its context scope failed upstream, not just the target node
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions