Skip to content

Semantic rules: rule scope and judge input (what the model sees) #152

Closed

Description

@mindsocket

Summary

Semantic rules (#114, PR #151) judge a node in isolation, plus its parent titles. Many of the rules worth judging need wider context: "this solution addresses its parent opportunity", "this opportunity is distinct from its siblings", "this outcome is consistent with the goal above it". This issue is to decide how a rule asks for that context and what the judge is sent.

Problem

Today the judge receives the node's title, type, parent titles and schema data. A rule cannot ask for more, and there is no rule-level control over how much is sent. Widening it ad hoc would change cost, privacy exposure and caching for every semantic rule at once.

Questions

  1. Scope vocabulary. A per-rule scope (node-local stays the default). Candidates: ancestors, parent, siblings, children, descendants, branch. Which are worth supporting, and are they enumerated or parameterised (e.g. depth)? Note scope: "global" already means "once for the whole space" for expression rules, and semantic rules reject it, so the semantic vocabulary needs to read distinctly.
  2. Judge input format. Decision models take a structured state object (Jev: 32k token limit for state plus questions); chat models take prose. How is wider context laid out so it is the same information in both? What is included for each related node (title and type only, full schema data, body text)?
  3. Bounding. Widening scope can blow past model context limits. Truncation, summarisation or per-scope caps, and what happens when a node's scope is too large.
  4. Gating interaction. Related nodes that are themselves gated (known-bad) are poor context. Should a rule's scope exclude gated nodes, or flag them?
  5. Cost and batching. The judge is called once per node with all of its semantic rules. Rules with different scopes need different inputs, so this either splits calls by scope or sends the union. Which?
  6. validate-file. It judges only the file and its gating ancestors. Sibling or descendant scopes would need those nodes' verdicts too; decide whether scopes read other nodes' data only (cheap) or other nodes' verdicts (costly).
  7. Caching. The cache key described in Semantic rule evaluation: make descriptive rules executable (LLM-as-judge) #114 must include everything in scope, so scope has to be settled before caching.

Relates

Activity

  1. mindsocket commented on Oct 14, 2026

    @mindsocket
    OwnerAuthor

    Update from PR #151: two things that bear on scope.

    • when (JSONata precondition, any per-node rule, not combined with scope: global) now exists and runs before the judge, so a skipped node costs nothing. It also applies to the ancestor rules validate-file checks. Scope decides what the judge is sent; when decides whether to judge at all. They should stay separate.
    • semantic rules still reject scope: global, as raised above. Verdict guidance ({description, guidance}) is user-facing only and never sent to the judge, so wider scope doesn't change what guidance can say.
    • A rule can also name a provider (judge) and a run can override it (--judge), so per-scope input formats may need to vary by provider type (decision model state vs chat prose), which sharpens question 2.
  2. mindsocket commented on Oct 15, 2026

    @mindsocket
    OwnerAuthor

    This should depend on the change to include body content, and include a commentary option for whether to include content body (opt in).

  3. mindsocket commented on Oct 15, 2026

    @mindsocket
    OwnerAuthor

    There's a case to be made that scope global is perfectly valid for a semantic rule on a whole space, within practical limits

  4. mindsocket commented on Oct 15, 2026

    @mindsocket
    OwnerAuthor

    For cost and batching, consider how we might batch across scopes (assumption is that a node still sets boundary for a call). Eg one call provides all ancestors and descendents as input elements. Each rule references the relevant ancestor and or descendent nodes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions