Repository navigation
Closed
Description
Summary
Semantic rules (#114, PR #151) judge a node in isolation, plus its parent titles. Many of the rules worth judging need wider context: "this solution addresses its parent opportunity", "this opportunity is distinct from its siblings", "this outcome is consistent with the goal above it". This issue is to decide how a rule asks for that context and what the judge is sent.
Problem
Today the judge receives the node's title, type, parent titles and schema data. A rule cannot ask for more, and there is no rule-level control over how much is sent. Widening it ad hoc would change cost, privacy exposure and caching for every semantic rule at once.
Questions
- Scope vocabulary. A per-rule
scope(node-local stays the default). Candidates:ancestors,parent,siblings,children,descendants,branch. Which are worth supporting, and are they enumerated or parameterised (e.g. depth)? Notescope: "global"already means "once for the whole space" for expression rules, and semantic rules reject it, so the semantic vocabulary needs to read distinctly. - Judge input format. Decision models take a structured
stateobject (Jev: 32k token limit for state plus questions); chat models take prose. How is wider context laid out so it is the same information in both? What is included for each related node (title and type only, full schema data, body text)? - Bounding. Widening scope can blow past model context limits. Truncation, summarisation or per-scope caps, and what happens when a node's scope is too large.
- Gating interaction. Related nodes that are themselves gated (known-bad) are poor context. Should a rule's scope exclude gated nodes, or flag them?
- Cost and batching. The judge is called once per node with all of its semantic rules. Rules with different scopes need different inputs, so this either splits calls by scope or sends the union. Which?
validate-file. It judges only the file and its gating ancestors. Sibling or descendant scopes would need those nodes' verdicts too; decide whether scopes read other nodes' data only (cheap) or other nodes' verdicts (costly).- Caching. The cache key described in Semantic rule evaluation: make descriptive rules executable (LLM-as-judge) #114 must include everything in scope, so scope has to be settled before caching.
Relates
- Semantic rule evaluation: make descriptive rules executable (LLM-as-judge) #114 (semantic rule evaluation), Evals layer: test data and calibration metrics for rules #116 (evals layer: calibration data needs a defined input)
Metadata
Metadata
Assignees
Labels
No labels
Activity
mindsocket commentedon Oct 14, 2026
Update from PR #151: two things that bear on scope.
when(JSONata precondition, any per-node rule, not combined withscope: global) now exists and runs before the judge, so a skipped node costs nothing. It also applies to the ancestor rulesvalidate-filechecks. Scope decides what the judge is sent;whendecides whether to judge at all. They should stay separate.semanticrules still rejectscope: global, as raised above. Verdict guidance ({description, guidance}) is user-facing only and never sent to the judge, so wider scope doesn't change what guidance can say.judge) and a run can override it (--judge), so per-scope input formats may need to vary by provider type (decision model state vs chat prose), which sharpens question 2.mindsocket commentedon Oct 15, 2026
This should depend on the change to include body content, and include a commentary option for whether to include content body (opt in).
mindsocket commentedon Oct 15, 2026
There's a case to be made that scope global is perfectly valid for a semantic rule on a whole space, within practical limits
mindsocket commentedon Oct 15, 2026
For cost and batching, consider how we might batch across scopes (assumption is that a node still sets boundary for a call). Eg one call provides all ancestors and descendents as input elements. Each rule references the relevant ancestor and or descendent nodes
feat(judge): batch Jev semantic questions across nodes that share a r…