Skip to content

Layered rule evaluation: cascade gating and coverage annotation #115

Closed

Description

@mindsocket

Layered rule evaluation: cascade gating and coverage annotation

Summary

As the rules engine grows beyond structural checks (executable rules, and semantic/evaluated rules to come), evaluation order and error propagation become first-class concerns. Proposal:

  • Safe defaults: deterministic layers (schema validation, executable rules) always run, and automatically gate downstream evaluation where possible.
  • Gating semantics: a failed or errored check on a node suppresses dependent downstream checks for that node and its subtree. Downstream checks shouldn't run on inputs known to be bad — and where the downstream check is an LLM judge, upstream errors contaminate judge accuracy as well as adding cost.
  • Opt-in evaluative layers: semantic rules run only when explicitly requested; cascading applies automatically by default.
  • Blocking is per-rule, not global: a mechanism to declare whether a given rule blocks downstream checks. Cheap, bottom-up checks (orphans, content quality) should still be surfacable even when a top-down blocker has fired — if asked for.

CLI surface idea

Options to evaluate at a chosen layer/depth of the schema hierarchy: start at the top and evaluate breadth-first. An agent can then walk the tree, fixing issues at the top before requesting evaluation of lower layers — a natural fit for an agentic fix loop (fix upstream, then re-evaluate downstream).

Output must annotate coverage

For both human and agent consumption, results should clearly state what scope was covered and what was skipped and why:

  • which rules were gated, by which upstream failure, on which node
  • where the current validation "edge" is — e.g. "this blocked goal means all opportunities beneath it were not evaluated" — visible at a glance

This has implications for the report format (#93): coverage annotation is part of the contract an agent needs to respond to the report correctly.

Motivation

  • Judge contamination: upstream errors cause downstream judges to produce false positives, so gating is both noise reduction and cost reduction (avoids spending calls on inputs the judge would misread).
  • Deterministic-first economics: catch structure problems before spending LLM calls.

Relates

Activity

  1. mindsocket commented on Sep 25, 2026

    @mindsocket
    OwnerAuthor

    Design note on what "dependent downstream checks" means: gating shouldn't be limited to the hierarchy subtree. Dependencies often run along non-hierarchy relationships (e.g. one node establishes something that others require). Proposal:

    • The schema declares which relationships propagate gating (e.g. a flag on the relationship definition). sctx follows the declared edges generically, without knowing what the node types mean. Hierarchy parent→child could be a default propagating edge.
    • A node's gating scope = itself plus everything reachable along propagating edges. An error-severity violation (Rule categories and severity #60) on a node gates semantic/judge rules (Semantic rule evaluation: make descriptive rules executable (LLM-as-judge) #114) on nodes in that scope; warnings and info don't gate.
    • Coverage annotation reports gated rule, gated node, and the originating violation (rule id + node), in both text and --json output, so downstream tools can consume it without re-deriving the graph walk.
    • Cycles along propagating edges need handling (Detect cycles in non-hierarchy relationships #118).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions