Skip to content

Rule categories and severity #60

Closed

Description

@mindsocket

Currently, the validate script treats all rule violations the same. The immediate impact of that is that the smoke tests fail when run on a project that's work in progress. Introducing severity levels would allow rules to be evaluated, but return with a more nuanced assessment of validation.

These things can be defined at several levels. One would be in the schema itself alongside the rules. Another could be a default configuration based on category that could be something like a workflow is a warning. Where is something else is a critical. It can be overridden on the command line and it could be overridden in a config file.

For the sake of simplicity, it's probably best that we design for categories of rules to be the default grouping of severity. It would be allowable to override a specific rule, but the preference would be not to do it that way by default; prefer higher levels of categorisation

Activity

  1. mindsocket commented on Sep 16, 2026

    @mindsocket
    OwnerAuthor

    Adding context from current rules-engine thinking: the existing categories (validation | coherence | workflow | best-practice) already encode the intent spectrum — from hard correctness constraints through to advisory quality checks. What's missing for #60 to close the loop is the reporting dimension on top of that:

    • Guard vs guide grading: coherence/best-practice violations should be able to report as advisory ("could be better") rather than as failures ("is wrong"), so work-in-progress spaces don't fail smoke tests on advisory categories (the original motivation in this issue's body).
    • Interaction with future semantic evaluation (Semantic rule evaluation: make descriptive rules executable (LLM-as-judge) #114): judged results likely need a graded scale rather than pass/fail, and severity defaults would want to be set per category, overridable per rule — as proposed here.
    • Interaction with cascading (Layered rule evaluation: cascade gating and coverage annotation #115): severity/category is also the natural place to declare whether a rule blocks downstream evaluation — cheap bottom-up checks (orphans, content quality) plausibly shouldn't block, while failed structural checks should.

    So: category = default grouping for severity (per this issue's design note), with rule-level override as the exception. That still holds; the additions above are the new requirements it needs to accommodate.

  2. mindsocket commented on Sep 25, 2026

    @mindsocket
    OwnerAuthor

    Concrete need from downstream use: rules need three levels — error / warning / info — with the category supplying the default severity and a per-rule override. Example: a coverage-gap rule ("this node has no linked implementation") is informational, while an advisory consistency check is a warning, and neither should fail validation. Currently category is a reporting label only (docs/rules.md), so every rule violation counts as an error. Also needed: gating uses severity, so errors block downstream semantic rules on the affected subtree and warnings don't (see #115).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions