Repository navigation
Description
Layered rule evaluation: cascade gating and coverage annotation
Summary
As the rules engine grows beyond structural checks (executable rules, and semantic/evaluated rules to come), evaluation order and error propagation become first-class concerns. Proposal:
- Safe defaults: deterministic layers (schema validation, executable rules) always run, and automatically gate downstream evaluation where possible.
- Gating semantics: a failed or errored check on a node suppresses dependent downstream checks for that node and its subtree. Downstream checks shouldn't run on inputs known to be bad — and where the downstream check is an LLM judge, upstream errors contaminate judge accuracy as well as adding cost.
- Opt-in evaluative layers: semantic rules run only when explicitly requested; cascading applies automatically by default.
- Blocking is per-rule, not global: a mechanism to declare whether a given rule blocks downstream checks. Cheap, bottom-up checks (orphans, content quality) should still be surfacable even when a top-down blocker has fired — if asked for.
CLI surface idea
Options to evaluate at a chosen layer/depth of the schema hierarchy: start at the top and evaluate breadth-first. An agent can then walk the tree, fixing issues at the top before requesting evaluation of lower layers — a natural fit for an agentic fix loop (fix upstream, then re-evaluate downstream).
Output must annotate coverage
For both human and agent consumption, results should clearly state what scope was covered and what was skipped and why:
- which rules were gated, by which upstream failure, on which node
- where the current validation "edge" is — e.g. "this blocked goal means all opportunities beneath it were not evaluated" — visible at a glance
This has implications for the report format (#93): coverage annotation is part of the contract an agent needs to respond to the report correctly.
Motivation
- Judge contamination: upstream errors cause downstream judges to produce false positives, so gating is both noise reduction and cost reduction (avoids spending calls on inputs the judge would misread).
- Deterministic-first economics: catch structure problems before spending LLM calls.
Relates
- Rule categories and severity #60 (rule categories and severity)
- Output validation report in a format that can be responded to #93 (response-ready report format)
- Semantic rule evaluation: make descriptive rules executable (LLM-as-judge) #114 (semantic rule evaluation)
Activity
mindsocket commentedon Sep 25, 2026
Design note on what "dependent downstream checks" means: gating shouldn't be limited to the hierarchy subtree. Dependencies often run along non-hierarchy relationships (e.g. one node establishes something that others require). Proposal:
--jsonoutput, so downstream tools can consume it without re-deriving the graph walk.feat(rules): cascade gating for rule evaluation (#115)
feat(rules): semantic rules judged by model or decision model (#114)
feat(rules): semantic rules judged by model or decision model (#114)