diff --git a/AGENTS.md b/AGENTS.md index 1a6097a0..b4a25890 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -58,9 +58,9 @@ There are several places that need reviewing and updating with any new feature o - `schemas/` — Bundled default schema files (JSON5) using the structured-context schema dialect and top-level `$metadata`. Files starting with `_` are "partials" (fragments for `$ref`). - `src/schema/metadata-contract.ts` — Single source of truth for the `$metadata` contract - `schemas/generated/latest/_structured_context_schema_meta.json` — Generated metaschema for the newest dialect (generated on build or with `bun run generate:schema-meta`). `schemas/generated//` folders are frozen copies, created by the `version` script (`generate:schema-meta --release`) only when a release changes the dialect; never edit them. `src/schema/meta-schemas.ts` loads them at runtime. -- `src/schema/gating.ts` — Cascade gating: computes hierarchy depth and evaluates rules with subtree gating (schema/hierarchy/blocking-rule failures suppress downstream rule checks). See [docs/rules.md](docs/rules.md#cascade-gating). -- `src/judge/` — Semantic rule judges: provider interface and types (`types.ts`), verdict resolution (`verdicts.ts`), per-node and grouped dispatch (`judge-node.ts`), and providers (`provider/anthropic.ts`, `provider/openai.ts`, `provider/jev.ts`, `provider/fake.ts`), provider hydration and the bundled catalogue (`factory.ts`), and `sctx providers` (`src/commands/providers.ts`). See [docs/semantic-rules.md](docs/semantic-rules.md). -- `src/schema/rule-filter.ts` — `--rule-filter` parsing and rule filtering by mode, category, severity, type or id. +- `src/schema/gating.ts` — Cascade gating: computes hierarchy depth and evaluates rules with subtree gating (schema/hierarchy/blocking-rule failures suppress downstream rule evaluation). See [docs/rules.md](docs/rules.md#cascade-gating). +- `src/judge/` — Semantic rule evaluation (the "judge" is its internal name): provider interface and types (`types.ts`), verdict resolution (`verdicts.ts`), per-node and grouped dispatch (`judge-node.ts`), and providers (`provider/anthropic.ts`, `provider/openai.ts`, `provider/jev.ts`, `provider/fake.ts`), provider hydration and the bundled catalogue (`factory.ts`), and `sctx providers` (`src/commands/providers.ts`). See [docs/semantic-rules.md](docs/semantic-rules.md). +- `src/schema/rule-filter.ts` — `--rule-filter` parsing and rule filtering by category, severity, type or id. ## Testing For most development only the main unit tests need re-running regularly. diff --git a/README.md b/README.md index 1834bb40..dd9f667a 100644 --- a/README.md +++ b/README.md @@ -115,8 +115,8 @@ Schemas are JSON Schema 2020-12 plus a top-level `$metadata` block for hierarchy "id": "active-outcome-count", "category": "workflow", "description": "Only one outcome should be active at a time", - "scope": "global", - "check": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" + "global": true, + "expression": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" } ] } @@ -131,17 +131,17 @@ See [docs/schemas.md](docs/schemas.md) (`sctx docs schema`) for the dialect, com ### Validate nodes ```bash -sctx validate [--watch] [--depth ] [--include-gated] [--semantic] [--judge ] [--rule-filter ] +sctx validate [--watch] [--depth ] [--include-gated] [--semantic [provider]] [--rule-filter ] ``` Validates markdown files against the JSON schema: - Extracts YAML frontmatter from each `.md` file - Skips files without frontmatter or without a `type` field - Reports validation results with counts and per-file errors -- Runs executable rules (`$metadata.rules`) with **cascade gating**: a schema, hierarchy, or blocking-rule failure on a node suppresses rule evaluation for that node's hierarchy subtree. Gated checks are reported in a Coverage section so you know what wasn't evaluated and why. See [docs/rules.md](docs/rules.md#cascade-gating) for details. +- Runs executable rules (`$metadata.rules`) with **cascade gating**: a schema, hierarchy, or blocking-rule failure on a node suppresses rule evaluation for that node's hierarchy subtree. Gated rules are reported in a Coverage section so you know what wasn't evaluated and why. See [docs/rules.md](docs/rules.md#cascade-gating) for details. - `--depth ` limits evaluation to hierarchy nodes within `n` breadth-first levels from the top — useful for staged agentic fix loops (fix the top level, then request the next). - `--include-gated` evaluates rules even on gated nodes (still reported as gated) — for surfacing cheap bottom-up checks on request. -- `--semantic` also evaluates `mode: semantic` rules with the configured judge (Jev, Anthropic or OpenAI-compatible); `--judge ` uses that configured model provider instead and implies `--semantic`; `--rule-filter` narrows which rules run. See [docs/semantic-rules.md](docs/semantic-rules.md) (`sctx docs semantic-rules`). +- `--semantic` also evaluates the `semantic` part of rules with the configured model provider (Jev, Anthropic or OpenAI-compatible); `--semantic ` uses that configured model provider instead; `--rule-filter` narrows which rules run. See [docs/semantic-rules.md](docs/semantic-rules.md) (`sctx docs semantic-rules`). ### Render space diff --git a/config.example.json b/config.example.json index 10b0d94b..9292f980 100644 --- a/config.example.json +++ b/config.example.json @@ -18,5 +18,5 @@ } ], "schema": "/path/to/custom/schemas/my-general-schema.json", - "judge": { "provider": "openrouter-jev" } + "semantic": { "provider": "openrouter-jev" } } diff --git a/docs/concepts.md b/docs/concepts.md index 738aff07..2f02fa23 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -163,17 +163,19 @@ Rules may be: Rules are distinct from schema validation: the schema checks structure; rules check meaning and quality. +A rule's `when` is its **applicability condition**: a test that decides whether the rule applies to a node. Its `expression` is the test the node must pass. + Each executable rule has a **category** (`validation`, `coherence`, `workflow`, `best-practice`) that supplies its default **severity** (`error`, `warning`, `info`). A rule may override its category's severity, but categories are the preferred grouping. Only `error` severity violations fail validation. See [docs/rules.md](rules.md) for the rules reference, including JSONata expression syntax and the full `$metadata` field reference. ### Semantic rules -A **semantic rule** (`mode: semantic`) asks a model to judge a node, for example whether an opportunity is a customer need. It has a `rubric` and may have a `check`, an expression precondition that runs before the judge. +A **semantic rule** is a rule with a `semantic` part. It asks a model to evaluate a node, for example whether an opportunity is a customer need. The `semantic` object can carry a `rubric`, `verdicts` and a verdict `severity` map. A rule can also have an `expression`, a JSONata test that runs before the model. -A **judge** is the model that judges semantic rules: a decision model (Jev) or a chat model (Claude, OpenAI-compatible). For each rule, the judge gives one **verdict** from the rule's `verdicts`, for example `customer-need`, `vague` or `solution`. The rule's `severity` says which verdicts report a violation. Semantic rules cost money, so they are judged only when a run asks for it (`--semantic` or `--judge`); a rule's `check` always runs. The judge is a configured **model provider**: a bundled provider or an entry in `modelProviders`. See [docs/semantic-rules.md](semantic-rules.md). +A **model provider** evaluates semantic rules: a decision model (Jev) or a chat model (Claude, OpenAI-compatible). It is a bundled provider or an entry in `modelProviders`. For each rule, the model gives one **verdict** from the rule's `semantic.verdicts`, for example `customer-need`, `vague` or `solution`. The rule's `semantic.severity` says which verdicts report a violation. Semantic rules cost money, so they are evaluated only when a run asks for it (`--semantic`); a rule's `expression` always runs. See [docs/semantic-rules.md](semantic-rules.md). -A **node ruleset group** is the set of nodes that share the same semantic rules. It forms within one evaluation step of [cascade gating](#cascade-gating): one hierarchy depth, or the set of non-hierarchy nodes. Nodes are keyed on the rules left after `when` and `check` filtering. A judge on the `openrouter-decisions` or `typesafe-systemone` API judges one group in as few decision requests as it can. +A **node ruleset group** is the set of nodes that share the same semantic rules. It forms within one evaluation step of [cascade gating](#cascade-gating): one hierarchy depth, or the set of non-hierarchy nodes. Nodes are keyed on the rules left after `when` and `expression` filtering. A model provider on the `openrouter-decisions` or `typesafe-systemone` API evaluates one group in as few decision requests as it can. A **decision request** is one call to a decisions API. It carries the state of one or more nodes and a list of decision questions. @@ -183,7 +185,7 @@ A **decision question** is one choice question about one node under one semantic **Cascade gating** is the mechanism by which a schema failure, hierarchy structure violation, or blocking rule violation on a node suppresses rule evaluation for that node's `hierarchy` subtree. It exists so downstream checks — especially costly semantic (model-judged) rules — aren't run against inputs already known to be invalid. -A **coverage gate** records one instance of this: the gating node, what kind of failure triggered it (`schema`, `hierarchy`, or a specific rule `id`), and the set of nodes whose rule checks were suppressed as a result. `sctx validate` reports coverage gates so both humans and agents can see the validation "edge" — where evaluation stopped and why. See [docs/rules.md](rules.md#cascade-gating) for the full model, including the per-rule `gatesDescendants` field and the `--include-gated` / `--depth` CLI options. +A **coverage gate** records one instance of this: the gating node, what kind of failure triggered it (`schema`, `hierarchy`, or a specific rule `id`), and the set of nodes whose rules were suppressed as a result. `sctx validate` reports coverage gates so both humans and agents can see the validation "edge" — where evaluation stopped and why. See [docs/rules.md](rules.md#cascade-gating) for the full model, including the per-rule `gatesDescendants` field and the `--include-gated` / `--depth` CLI options. --- diff --git a/docs/config.md b/docs/config.md index dea778e9..1bfd52a2 100644 --- a/docs/config.md +++ b/docs/config.md @@ -65,9 +65,9 @@ All plugin names must start with `sctx-` (the prefix is optional in config and n Plugins can also contribute schemas and partials via a `/schemas/` directory (or `"sctx": { "schemas": "..." }` in `package.json`). If a space specifies a plugin-prefixed schema (e.g. `"sctx-wardley-mapping/wardley_map.json"`), schemas are resolved statically without executing plugin code, and the plugin's code hooks are loaded when parsing or rendering. -## Model providers and judge +## Model providers and semantic rules -structured-context judges [semantic rules](semantic-rules.md) with a **model provider**: a named definition that says which wire API to speak, which model to use and where. A provider definition combines an `api`, explicit values or environment-variable names, and the key source. +structured-context evaluates [semantic rules](semantic-rules.md) with a **model provider**: a named definition that says which wire API to speak, which model to use and where. A provider definition combines an `api`, explicit values or environment-variable names, and the key source. **Bundled providers** work without any configuration: a fixed model per vendor chosen by this project, plus `-env` variants that also read the vendor's usual model and endpoint variables (`OPENAI_MODEL`, `OPENAI_BASE_URL`, `ANTHROPIC_MODEL`), falling back to the bundle's defaults where a variable is unset. Bundles without `-env` read no environment beyond the key: they behave the same on every machine. @@ -105,7 +105,7 @@ Bundled provider names follow the pattern: | Field | Description | |---|---| -| `api` | **Required.** The wire API (see the table above and [Choosing a judge](semantic-rules.md#choosing-a-judge)), or `fake` (for tests). | +| `api` | **Required.** The wire API (see the table above and [Choosing a model provider](semantic-rules.md#choosing-a-model-provider)), or `fake` (for tests). | | `model` | A fixed model. Its default depends on the API. | | `modelEnv` | Environment variable holding a model override. An explicit `model` wins; a variable set but empty is an error. | | `endpoint` | A fixed API URL. | @@ -117,18 +117,18 @@ Resolution per field: the explicit value wins, then the named environment variab The environment holds only keys and the variables a `-env` bundle names. No environment variable selects a provider. -The top-level `judge` names the model provider that judges semantic rules: a bundled provider or an entry in `modelProviders`. A space can set its own `judge`, which replaces the top-level one for that space, for example to keep a private space on a local model: +The top-level `semantic.provider` names the model provider that evaluates semantic rules: a bundled provider or an entry in `modelProviders`. A space can set its own `semantic`, which replaces the top-level one for that space, for example to keep a private space on a local model: ```json { - "judge": { "provider": "openrouter-jev" }, + "semantic": { "provider": "openrouter-jev" }, "spaces": [ - { "name": "private-notes", "path": "/path/to/notes", "judge": { "provider": "local" } } + { "name": "private-notes", "path": "/path/to/notes", "semantic": { "provider": "local" } } ] } ``` -Config does not turn judging on. Each run asks for it with `--semantic`, or with `--judge `, which also selects a different provider for that run. +Config does not turn semantic evaluation on. Each run asks for it with `--semantic`. `--semantic ` also selects a different provider for that run. ## Markdown plugin config diff --git a/docs/rules.md b/docs/rules.md index ab0afdfe..9b1a39e4 100644 --- a/docs/rules.md +++ b/docs/rules.md @@ -4,10 +4,12 @@ title: "Executable Rules" # Executable Rules -Rules check content after JSON Schema validation, for example cross-node checks and workflow constraints. A schema declares rules in `$metadata.rules`. There are two kinds of rule: +Rules check content after JSON Schema validation, for example cross-node checks and workflow constraints. A schema declares rules in `$metadata.rules`. A rule has an `expression`, a `semantic` part, or both: -- An **expression rule** (the default) is a JSONata `check` that must return `true`. -- A **semantic rule** asks a model to judge a node. See [Semantic rules](semantic-rules.md). +- An **`expression`** is a JSONata test that must return `true`. +- A **`semantic`** part asks a model to evaluate a node. See [Semantic rules](semantic-rules.md). + +A rule with neither is a schema error. For metadata structure and composition behavior, see [docs/schemas.md](schemas.md). @@ -21,8 +23,8 @@ Rules are a flat array. "id": "active-outcome-count", "category": "workflow", "description": "Only one outcome should be active at a time", - "scope": "global", - "check": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" + "global": true, + "expression": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" } ] ``` @@ -36,7 +38,7 @@ Every rule has a category, which describes its intent and supplies a default sev | `workflow` | Process/operating-discipline checks | `warning` | | `best-practice` | Advisory quality checks | `info` | -Severity can also be set per rule. +Severity can also be set per rule. It applies to a failed `expression` and to the default `fail` verdict of a [semantic rule](semantic-rules.md#severity). | Severity | Meaning | Fails validation | |---|---|---| @@ -55,7 +57,7 @@ Prefer choosing the right category and relying on its default. Set `severity` on "severity": "error", "description": "Goals must not be left blocked", "type": "goal", - "check": "current.status != 'blocked'" + "expression": "current.status != 'blocked'" } ``` @@ -63,28 +65,32 @@ Prefer choosing the right category and relying on its default. Set `severity` on ## Selecting nodes -- A rule with `scope: "global"` runs once for the space. +- A rule with `global: true` runs once for the space. It needs an `expression`, and it cannot have `when`, `type` or `semantic`. - Other rules run once for each node. - `type` limits a rule to nodes with that `resolvedType`. -- `when` limits a rule to nodes where a test returns `true`. See [Conditional rules](#conditional-rules-when). +- `when` limits a rule to nodes where the applicability condition returns `true`. See [Conditional rules](#conditional-rules-when). ### Conditional rules (`when`) -`when` is a JSONata test that determines whether a rule runs on a node. It has the same context as `check` and runs before it. Use `when` when `type` cannot select the nodes a rule applies to: +`when` is the rule's applicability condition: a JSONata test that determines whether a rule applies to a node. It has the same context as `expression` and runs before it. Use `when` when `type` cannot select the nodes a rule applies to: ```json { "id": "ready-has-owner", "category": "workflow", "type": "task", "description": "Ready tasks have an owner", - "when": "current.status = 'ready'", "check": "$exists(current.owner)" } + "when": "current.status = 'ready'", "expression": "$exists(current.owner)" } ``` -`when` costs almost nothing, so use it to stop an expensive rule from running where it does not apply. For a semantic rule, this saves a model call on each node that `when` excludes. A semantic rule can also carry a `check`; see [Preconditions](semantic-rules.md#preconditions-check) for the order. +`when` costs almost nothing, so use it to stop an expensive rule from running where it does not apply. For a semantic rule, this saves a model call on each node that `when` excludes. + +### Order within a rule + +A rule runs its parts in this order: `when`, then `expression`, then the `semantic` part. If the `expression` fails, the `semantic` part does not run on that node. -An error in a `when` or `check` is reported as a non-gating `error` violation. +An error in a `when` or `expression` is reported as a non-gating `error` violation. ## Writing expressions -An expression rule's `check` (and `when`) is JSONata. This section covers what an expression can see and how to write checks. +An `expression` (and `when`) is JSONata. This section covers what an expression can see and how to write expressions. ### Expression context @@ -154,8 +160,8 @@ Example override: "override": true, "category": "workflow", "description": "Require exactly one active outcome", - "scope": "global", - "check": "$count(nodes[resolvedType='outcome' and status='active']) = 1" + "global": true, + "expression": "$count(nodes[resolvedType='outcome' and status='active']) = 1" } ``` @@ -163,22 +169,22 @@ Example override: When a schema uses `$metadata.hierarchy`, rule evaluation cascades top-down through the hierarchy tree (breadth-first from roots and orphans). A **gate** occurs when a node fails -in a way that makes its descendants' checks untrustworthy or wasteful to run: +in a way that makes its descendants' rules untrustworthy or wasteful to run: -- **Schema validation failure** on a node — always gates. The node's own rule checks are +- **Schema validation failure** on a node — always gates. The node's own rules are suppressed too, since its data is structurally invalid. - **Hierarchy structure violation** on a node (e.g. an invalid parent) — always gates the node's subtree, though the node's own rules still run. -- **A rule violation with `gatesDescendants: true`** — gates the node's subtree, though the - violating node's own other rules still run. +- **A rule failure with `gatesDescendants: true`** — gates the node's subtree, though the + failing node's own other rules still run. -Gating exists so that expensive or semantic downstream checks (see [Semantic rules](semantic-rules.md)) aren't wasted +Gating exists so that expensive or semantic downstream rules (see [Semantic rules](semantic-rules.md)) aren't wasted evaluating nodes whose ancestors are already known-bad, and so noise doesn't drown out the actionable failure. It is per-subtree: once a node is gated, its descendants inherit the same gate (the original blocker), rather than each generating a new one. Cascade gating only applies to nodes within the hierarchy DAG. -The default is `gatesDescendants: true` for `error`-severity rules and `false` for `warning` and `info` rules. Set it explicitly when that default doesn't fit. +A rule has one `gatesDescendants` setting. It applies to whichever part failed, the `expression` or the `semantic` part. Its default comes from the severity of that failure: `true` for `error`, and `false` for `warning` and `info`. Set it explicitly when that default doesn't fit. ### Coverage output @@ -196,9 +202,9 @@ don't depend on an ancestor being valid. `sctx validate-file ` applies the same gating as `sctx validate`, so a file's rule results match its entry in a full run. -- Expression rules are evaluated for the whole space; the output is restricted to the file. -- Semantic rules are judged only for the file and its gating ancestors (see [Semantic rules](semantic-rules.md#single-file-validation)). -- If the file is gated, its rule checks are skipped and a `gated:` entry under `warnings` names the blocker, so a gated file doesn't read as clean. `--include-gated` evaluates the file's rules anyway. +- Expressions are evaluated for the whole space; the output is restricted to the file. +- Semantic parts are evaluated only for the file and its gating ancestors (see [Semantic rules](semantic-rules.md#how-semantic-evaluation-works)). +- If the file is gated, its rules are skipped and a `gated:` entry under `warnings` names the blocker, so a gated file doesn't read as clean. `--include-gated` evaluates the file's rules anyway. ## Controlling validation runs @@ -208,9 +214,9 @@ These options restrict which rules run, without changing rule definitions. `--rule-filter =[,]` runs only the rules that match. It works on `sctx validate` and `sctx validate-file`. -- Dimensions: `mode`, `category`, `severity`, `type` and `id`. +- Dimensions: `category`, `severity`, `type` and `id`. - Values in one dimension combine with OR. Repeat the option to add a dimension; dimensions combine with AND. -- A semantic rule with a severity map matches each severity that it can report. +- A rule matches each severity that it can report, including the severities in a semantic verdict map. - Gating comes only from the rules that ran. For example, `--rule-filter category=workflow,coherence --rule-filter severity=error` runs only workflow and coherence rules that can report an error. @@ -225,7 +231,7 @@ the next). Excluded nodes are reported under `skippedByDepth`, separate from gat ## Output - **`sctx validate`** — the summary counts rule violations per severity, and details are grouped by severity with each line showing the rule's category and id. -- **`sctx validate --json`** — `errors` includes only `error` severity rule violations (keyed `rule:`, with a `severity` field). A `ruleViolations` array lists every violation with `file`, `ruleId`, `category`, `severity` and `description` (`file` is empty for global rules). +- **`sctx validate --json`** — `errors` includes only `error` severity rule violations (keyed `rule:`, with a `severity` field). A `ruleViolations` array lists every violation with `file`, `ruleId`, `category`, `severity` and `description` (`file` is empty for global rules). A violation from a failed `expression` has the evidence `expression failed: ` when the rule also has a `semantic` part, and no verdict. - **`sctx validate-file`** — `error` severity violations appear under `errors`; `warning` and `info` violations appear under `warnings`. Rule entries carry a `severity` field. Gated files: see [Single-file validation](#single-file-validation). diff --git a/docs/schemas.md b/docs/schemas.md index 5035dc4b..a1c19b65 100644 --- a/docs/schemas.md +++ b/docs/schemas.md @@ -139,8 +139,8 @@ Top-level metadata shape: "id": "active-outcome-count", "category": "workflow", "description": "Only one outcome should be active at a time", - "scope": "global", - "check": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" + "global": true, + "expression": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" } ] } @@ -159,7 +159,7 @@ Top-level metadata shape: | `hierarchy.allowSkipLevels` | `boolean` | Optional; allows parent to be any ancestor level | | `relationships` | `Relationship[]` | Optional; defines related node links outside the primary hierarchy. A relationship may set `override: true` | | `aliases` | `Record` | Optional type alias map. Use `{ "type": "goal", "override": true }` to replace an imported alias | -| `rules` | `Rule[]` | Optional flat rule array. Rules are `expression` (JSONata `check`, the default) or `semantic` (judged against a rubric); see [Rules](rules.md) | +| `rules` | `Rule[]` | Optional flat rule array. A rule has an `expression` (a JSONata test), a `semantic` part (evaluated by a model against a rubric), or both; see [Rules](rules.md) | ### Relationships @@ -369,8 +369,8 @@ How hierarchies are used: "override": true, "category": "workflow", "description": "Require exactly one active outcome", - "scope": "global", - "check": "$count(nodes[resolvedType='outcome' and status='active']) = 1" + "global": true, + "expression": "$count(nodes[resolvedType='outcome' and status='active']) = 1" } ] } diff --git a/docs/semantic-rules.md b/docs/semantic-rules.md index a963ce97..3037c218 100644 --- a/docs/semantic-rules.md +++ b/docs/semantic-rules.md @@ -1,16 +1,16 @@ # Semantic Rules -A semantic rule asks a model to judge a node, for example "Is this opportunity a customer need, or a solution in disguise?". Use a semantic rule for a check that an expression cannot do. +A semantic rule asks a model to evaluate a node, for example "Is this opportunity a customer need, or a solution in disguise?". Use a semantic rule for a test that an expression cannot do. -This page assumes you know [Rules](rules.md). Semantic rules use the same fields, severities, [`when`](rules.md#conditional-rules-when) and [cascade gating](rules.md#cascade-gating) as expression rules. This page gives only the differences. +A rule is semantic when it has a `semantic` object. This page assumes you know [Rules](rules.md). A semantic rule uses the same severities, [`when`](rules.md#conditional-rules-when) and [cascade gating](rules.md#cascade-gating) as any other rule. This page gives only the differences. ## Quick start 1. Set `OPENROUTER_API_KEY` in your environment. -2. Choose Jev as the judge in `config.json`. Jev is a decision model and the recommended judge (see [Choosing a judge](#choosing-a-judge)): +2. Choose Jev as the model provider in `config.json`. Jev is a decision model and the recommended provider (see [Choosing a model provider](#choosing-a-model-provider)): ```json - { "judge": { "provider": "openrouter-jev" } } + { "semantic": { "provider": "openrouter-jev" } } ``` `openrouter-jev` is a bundled provider. To use Claude instead, set `"provider": "anthropic"` and `ANTHROPIC_API_KEY`. @@ -20,113 +20,114 @@ This page assumes you know [Rules](rules.md). Semantic rules use the same fields { "id": "opportunity-customer-need", "category": "best-practice", - "mode": "semantic", "type": "opportunity", - "description": "The opportunity is a customer need or pain point, not a solution or a business metric" + "description": "The opportunity is a customer need or pain point, not a solution or a business metric", + "semantic": {} } ``` 4. Run `sctx validate --semantic`. -## Run `sctx validate` or `sctx validate-file` with semantic rule checking +## Run `sctx validate` or `sctx validate-file` with semantic rules -Semantic rules cost money and send node content to the judge's provider. Thus they run only when you ask, on each run: +Semantic rules cost money and send node content to the model provider. Thus they run only when you ask, on each run: -- `--semantic` runs them with the configured judge. -- `--judge ` runs them with a specific model provider instead. +- `--semantic` runs them with the configured model provider. +- `--semantic ` runs them with a specific model provider instead. -A semantic rule's [`check`](#preconditions-check) costs no model call, so it runs on every validation. Without a judge, only that expression runs. +A rule's `expression` costs no model call, so it runs on every validation. Without `--semantic`, a rule with both an `expression` and a `semantic` part runs only the expression, and a rule with only a `semantic` part does not run. ## Rule fields +A semantic rule has the [common rule fields](rules.md) and a `semantic` object. `semantic: {}` takes every default. + | Field | Required | Description | |---|---|---| -| `mode` | yes | `"semantic"` | -| `check` | no | An expression that a node must pass before the judge sees it. Runs even without a judge. See [Preconditions](#preconditions-check) | -| `rubric` | no | Tells the judge what to assess. Default: `description` | -| `verdicts` | no | The answers the judge can give. Default: `pass` and `fail`. See [Verdicts](#verdicts) | -| `severity` | if `verdicts` is set | The severity of each verdict. See [Severity](#severity) | +| `semantic.rubric` | no | Tells the model what to assess. Default: `description` | +| `semantic.verdicts` | no | The answers the model can give. Default: `pass` and `fail`. See [Verdicts](#verdicts) | +| `semantic.severity` | if `semantic.verdicts` is set | The severity of each verdict. See [Severity](#severity) | -A semantic rule cannot have `scope: "global"`. +A semantic rule cannot be `global`. A rule can also have an `expression`; see [Combining an expression](#combining-an-expression). -## Preconditions (`check`) +## Combining an expression -A `check` costs no model call. It uses the same [expression syntax and context](rules.md#writing-expressions) as an expression rule. Use it for a requirement that an expression can verify, so the judge only sees nodes worth the cost: +A rule can have both an `expression` and a `semantic` part. An `expression` costs no model call. It uses the same [expression syntax and context](rules.md#writing-expressions) as any other rule. Use it for a requirement that an expression can verify, so the model only sees nodes worth the cost: ```json { "id": "opportunity-customer-voice", "category": "coherence", "type": "opportunity", - "mode": "semantic", "description": "Opportunity is framed in customer voice", - "check": "$exists(current.source)" } + "description": "Opportunity is framed in customer voice", + "expression": "$exists(current.source)", "semantic": {} } ``` -The order is `when`, then `check`, then the judge. Without a judge, the order is `when`, then `check`. - -- If `check` returns `true`, the judge runs. Without a judge, nothing is reported. The verdict maps to a severity as usual. -- If `check` does not return `true`, structured-context reports a violation and does not call the judge. The violation has no verdict, and its evidence reads `expression check failed: `. -- If `check` cannot be evaluated, structured-context reports a non-gating `error` violation and does not call the judge. +The order is `when`, then `expression`, then the `semantic` part. -A failing `check` reports the severity that an expression rule would report: the rule's `severity`, or the default for its category. With a [severity map](#severity), it reports the most severe severity in the map. If every entry in the map is `ignore`, it reports `info`. +- If the `expression` returns `true`, the model runs. Without `--semantic`, nothing is reported. +- If the `expression` does not return `true`, structured-context reports a violation and does not call the model. The violation has no verdict, and its evidence reads `expression failed: `. +- If the `expression` cannot be evaluated, structured-context reports a non-gating `error` violation and does not call the model. -A failing `check` [gates](rules.md#cascade-gating) the node's descendants as an expression rule does. It gates by default when it reports `error` severity. Set `gatesDescendants` to override that either way. +A failed `expression` reports the rule's `severity`, or the default for its category. The verdict severities do not apply to it. -`--rule-filter mode=semantic` selects these rules with or without a judge. +A failed `expression` [gates](rules.md#cascade-gating) the node's descendants as it does for any rule. It gates by default when it reports `error` severity. Set `gatesDescendants` to override that either way. ## Verdicts -For each node, the judge gives exactly one verdict for each rule. The verdict must come from the rule's `verdicts`. A fixed set of verdicts makes results comparable between runs. +For each node, the model gives exactly one verdict for each rule. The verdict must come from the rule's `semantic.verdicts`. A fixed set of verdicts makes results comparable between runs. Each verdict has a purpose for one audience: -- The **description** tells the judge when to choose that verdict. -- The **guidance** (optional) tells the user what to do when the judge chooses that verdict. structured-context shows it with the violation and does not send it to the judge. +- The **description** tells the model when to choose that verdict. +- The **guidance** (optional) tells the user what to do when the model chooses that verdict. structured-context shows it with the violation and does not send it to the model. A verdict is a description string, or an object with `description` and `guidance`: ```json -"verdicts": { - "customer-need": "A customer need or pain point", - "vague": { - "description": "A need, but too vague to act on", - "guidance": "Say who has the need and what they are trying to do." +"semantic": { + "verdicts": { + "customer-need": "A customer need or pain point", + "vague": { + "description": "A need, but too vague to act on", + "guidance": "Say who has the need and what they are trying to do." + }, + "solution": "A solution or business goal in disguise" }, - "solution": "A solution or business goal in disguise" -}, -"severity": { "vague": "info", "solution": "warning" } + "severity": { "vague": "info", "solution": "warning" } +} ``` ## Severity The verdict is a classification. The severity says what the verdict means for the user. -- With custom verdicts, `severity` is a map from verdict to `error`, `warning`, `info` or `ignore`. +- With custom verdicts, `semantic.severity` is a map from verdict to `error`, `warning`, `info` or `ignore`. It is required. - A verdict with `error`, `warning` or `info` reports a violation at that severity. - A verdict that is not in the map reports no violation. At least one verdict must report no violation. - `ignore` also reports no violation. Use it to show that you chose to ignore a verdict. -- With the default verdicts, `severity` is one value for `fail`, the same as for an expression rule. +- With the default verdicts, `fail` takes the rule's top-level `severity`, the same as for an `expression`. `fail` must not appear in `semantic.severity`. +- With custom verdicts and no `expression`, the top-level `severity` has no failure to describe, so it is a schema error. With an `expression`, it sets the severity of the failed expression. Gating uses the severity of the verdict. A verdict with `error` severity gates the node's subtree, as for any rule. `gatesDescendants` changes this, as for any rule. -## How judging works +## How semantic evaluation works -- **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge. -- **Grouping.** A judge on the `openrouter-decisions` or `typesafe-systemone` API judges each node ruleset group (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. This includes the bundled `openrouter-openai-decisions` provider. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request on tokens is sent whole as its own decision request. A node with more than 40 rules has its rules split across decision requests. If a node's questions are split across several requests and one fails, all of that node's rules are reported as errors. Other providers, including `openai-decisions` and chat models, judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Grouping can shift verdicts and probabilities slightly compared with judging each node alone. -- **Gating.** structured-context does not judge a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated. -- **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. If one decision request fails, only the rules on the nodes in that decision request report the error. -- **Input.** The judge gets the node's title, type, parent titles and schema data, including frontmatter. -- **Output.** A violation from a semantic rule also includes the verdict, the judge's evidence and any guidance. +- **Order.** On each node, structured-context runs each rule's [`expression`](#combining-an-expression) first. It sends the semantic parts of the rules that pass to the model provider. +- **Grouping.** A model provider on the `openrouter-decisions` or `typesafe-systemone` API evaluates each node ruleset group (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. This includes the bundled `openrouter-openai-decisions` provider. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request on tokens is sent whole as its own decision request. A node with more than 40 rules has its rules split across decision requests. If a node's questions are split across several requests and one fails, all of that node's rules are reported as errors. Other providers, including `openai-decisions` and chat models, evaluate one node per request. Within a space, the nodes at one hierarchy depth are evaluated together, because their verdicts decide whether the next depth is evaluated. Grouping can shift verdicts and probabilities slightly compared with evaluating each node alone. +- **Gating.** structured-context does not evaluate the semantic part on a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated. +- **Evaluation errors.** If the model provider fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation with the suffix `(Semantic evaluation error: ...)`. An evaluation error does not gate. If one decision request fails, only the rules on the nodes in that decision request report the error. +- **Input.** The model provider gets the node's title, type, parent titles and schema data, including frontmatter. +- **Output.** A violation from a semantic verdict also includes the verdict, the model's evidence and any guidance. -`sctx validate-file` judges only the file and the rules on its ancestors that can gate it. Thus one file costs at most one judge call for each ancestor level, plus one. +`sctx validate-file` evaluates the semantic part only for the file and the rules on its ancestors that can gate it. Thus one file costs at most one model call for each ancestor level, plus one. -## Choosing a judge +## Choosing a model provider Use a decision model when you can. Its results are repeatable and cheap, and they include a confidence. Use a chat model when the rubric needs open reasoning or a written explanation. Bundled providers give you a model with no configuration; the decision models among them speak a decisions API. Run `sctx providers` for the full list with effective models, endpoints and key variables. -structured-context uses only the provider that you configure. It does not choose a provider from the keys in your environment. To override models, endpoints or keys, add an entry of the same name to `modelProviders`, or define a new one (see [Model providers and judge](config.md#model-providers-and-judge)). +structured-context uses only the provider that you configure. It does not choose a provider from the keys in your environment. To override models, endpoints or keys, add an entry of the same name to `modelProviders`, or define a new one (see [Model providers and semantic rules](config.md#model-providers-and-semantic-rules)). -A rule does not choose its judge: that is configuration, not schema. To use a different provider for one run, name it with `--judge`. For example, to get a second opinion from a chat model on one rule and one file: +A rule does not choose its model provider: that is configuration, not schema. To use a different provider for one run, name it with `--semantic `. For example, to get a second opinion from a chat model on one rule and one file: ```bash -sctx validate-file --rule-filter id= --judge claude +sctx validate-file --rule-filter id= --semantic claude ``` diff --git a/plugin/.claude-plugin/plugin.json b/plugin/.claude-plugin/plugin.json index 21c2d184..4172ef25 100644 --- a/plugin/.claude-plugin/plugin.json +++ b/plugin/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "structured-context", - "version": "0.12.14", + "version": "0.12.15", "description": "Structured content authoring for Obsidian — automatic validation hooks, slash commands, and agent skills to keep markdown files compliant with your schema.", "homepage": "https://github.com/mindsocket/structured-context", "repository": "https://github.com/mindsocket/structured-context", diff --git a/plugin/skills/structured-context/references/commands.md b/plugin/skills/structured-context/references/commands.md index 0e160a6f..da9c67d9 100644 --- a/plugin/skills/structured-context/references/commands.md +++ b/plugin/skills/structured-context/references/commands.md @@ -11,7 +11,7 @@ Then use the `sctx` command for all operations. Always pass `--config ` wh ## validate ```bash -sctx validate [--watch] [--config ] [--depth ] [--include-gated] [--semantic] [--judge ] [--rule-filter ] +sctx validate [--watch] [--config ] [--depth ] [--include-gated] [--semantic [provider]] [--rule-filter ] ``` Validates all `.md` files in the space against the JSON schema. For each file: @@ -29,8 +29,8 @@ Validates all `.md` files in the space against the JSON schema. For each file: the top — useful for fixing a space level by level (fix the top, re-run with a deeper `--depth`, repeat). `--include-gated` evaluates rules on gated nodes anyway, still reporting them as gated — use it to surface cheap bottom-up checks even when an upstream blocker fired. -`--semantic` also runs `mode: semantic` rules through a configured model judge (costly; off by -default; `--judge ` judges with that configured model provider and implies `--semantic`), and `--rule-filter =` (repeatable; mode, category, severity, type, id) +`--semantic` also evaluates the `semantic` part of rules through a configured model provider (costly; off by +default; `--semantic ` uses that configured model provider instead), and `--rule-filter =` (repeatable; category, severity, type, id) narrows which rules run. `validate-file` accepts the same two flags. Exit codes: `0` = clean (warning and info rule violations don't fail), `1` = validation errors found. diff --git a/plugin/skills/structured-context/references/schema-authoring.md b/plugin/skills/structured-context/references/schema-authoring.md index 129ec4e7..069beb81 100644 --- a/plugin/skills/structured-context/references/schema-authoring.md +++ b/plugin/skills/structured-context/references/schema-authoring.md @@ -30,8 +30,8 @@ See `~/src/structured-context/schemas/` for examples (`strategy_general.json`, ` "id": "active-outcome-count", "category": "workflow", "description": "Only one outcome should be active at a time", - "scope": "global", - "check": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" + "global": true, + "expression": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" } ] } @@ -66,7 +66,7 @@ Relationships define how sub-entities (nodes inside other files) are parsed and Rules are a flat array. Each rule's `category` (`validation`, `coherence`, `workflow`, `best-practice`) supplies its default severity: `validation` → `error`, `coherence` and `workflow` → `warning`, `best-practice` → `info`. Only `error` violations fail validation; warnings and info are reported only. An optional per-rule `severity` (`error` | `warning` | `info`) overrides the category default — prefer picking the right category over overriding. -Rules default to `mode: "expression"` (a JSONata `check`). Set `mode: "semantic"` for rules an LLM or decision model judges against a `rubric` (falls back to `description`): they take no `scope` and may take a `check` (a JSONata precondition that runs before the judge, and even without `--semantic`; a failing `check` reports a violation without calling the judge), may declare `verdicts` (verdict → description, or `{ description, guidance }`; default `pass`/`fail`) with a `severity` map saying which verdicts are `error`/`warning`/`info`/`ignore`, and are judged only with `--semantic` or `--judge`. Run `sctx docs semantic-rules` for the full semantic rule reference. A `when` JSONata test (any rule) limits a rule to the nodes where it returns true. An optional `gatesDescendants` controls whether a violation suppresses rule checks on the node's subtree (default: true for `error` severity). +A rule has an `expression` (a JSONata test that must return `true`), a `semantic` object (a model evaluates the node against a `rubric`, which falls back to `description`), or both; a rule with neither is rejected. `global: true` runs an `expression` once for the space (no `when`, `type` or `semantic`). `semantic: {}` takes every default; it may declare `verdicts` (verdict → description, or `{ description, guidance }`; default `pass`/`fail`) with a `semantic.severity` map saying which verdicts are `error`/`warning`/`info`/`ignore` (required with custom verdicts). The top-level `severity` is that of a failed `expression` and of the default `fail` verdict. A rule's `expression` always runs and, when it fails, the `semantic` part is skipped; the `semantic` part runs only with `--semantic`. Run `sctx docs semantic-rules` for the full semantic rule reference. A `when` JSONata test (the applicability condition, any rule) limits a rule to the nodes where it returns true. An optional `gatesDescendants` controls whether a failure suppresses rules on the node's subtree (default: true for `error` severity). ## Metadata composition semantics diff --git a/plugin/skills/structured-context/references/schema-design.md b/plugin/skills/structured-context/references/schema-design.md index 4503ad81..b1328ea7 100644 --- a/plugin/skills/structured-context/references/schema-design.md +++ b/plugin/skills/structured-context/references/schema-design.md @@ -111,7 +111,7 @@ This signals the field has identity and makes it easy to add an enum later: For each wikilink or array-of-wikilink field, consider writing a rule: -| Relationship | Rule type | Example check | +| Relationship | Rule type | Example expression | |-------------|----------|--------------| | Required single link | workflow | `$exists(current.phase) = true` | | Required array (min 1) | workflow | `$count(current.performed_by) >= 1` | diff --git a/schemas/_knowledge_wiki.json b/schemas/_knowledge_wiki.json index c15424bc..2ae52cfd 100644 --- a/schemas/_knowledge_wiki.json +++ b/schemas/_knowledge_wiki.json @@ -50,14 +50,14 @@ "category": "coherence", "description": "Syntheses should reference at least one contributing source in their sources array", "type": "synthesis", - "check": "$exists(current.sources) and $count(current.sources) >= 1" + "expression": "$exists(current.sources) and $count(current.sources) >= 1" }, { "id": "concept-has-summary", "category": "best-practice", "description": "Concept pages should have a short summary for index discoverability", "type": "concept", - "check": "$exists(current.summary) and $length(current.summary) > 0" + "expression": "$exists(current.summary) and $length(current.summary) > 0" } ] }, diff --git a/schemas/_okf.json b/schemas/_okf.json index 3cd54d8a..a99b22e0 100644 --- a/schemas/_okf.json +++ b/schemas/_okf.json @@ -13,27 +13,27 @@ "category": "best-practice", "severity": "warning", "description": "Concepts should have a one-line description (used by index generators and search snippets)", - "check": "$exists(current.description) and $length(current.description) > 0" + "expression": "$exists(current.description) and $length(current.description) > 0" }, { "id": "okf-stale-after-generated", "category": "validation", "description": "stale_after must be later than generated.at", - "check": "$not($exists(current.stale_after)) or $not($exists(current.generated.at)) or $toMillis(current.stale_after) > $toMillis(current.generated.at)" + "expression": "$not($exists(current.stale_after)) or $not($exists(current.generated.at)) or $toMillis(current.stale_after) > $toMillis(current.generated.at)" }, { "id": "okf-usage-count-window", "category": "coherence", "severity": "warning", "description": "A source's usage_count is only meaningful within a usage_window (on the source or shared at the top level)", - "check": "$not($exists(current.sources)) or $count(current.sources[$exists(usage_count) and $not($exists(usage_window)) and $not($exists($$.current.usage_window))]) = 0" + "expression": "$not($exists(current.sources)) or $count(current.sources[$exists(usage_count) and $not($exists(usage_window)) and $not($exists($$.current.usage_window))]) = 0" }, { "id": "okf-computation-present", "category": "validation", "type": "Attested Computation", "description": "An Attested Computation needs its computation: a `computation` path or a body `# Computation` section. Known limitation: the section is found by matching a `# Computation` line in the body text, so such a line inside a code fence also counts", - "check": "$exists(current.computation) or $contains(current.content, /^# Computation[ \\t]*$/m)" + "expression": "$exists(current.computation) or $contains(current.content, /^# Computation[ \\t]*$/m)" } ] }, diff --git a/schemas/_ost_strict.json b/schemas/_ost_strict.json index 08fad84b..82aeb99c 100644 --- a/schemas/_ost_strict.json +++ b/schemas/_ost_strict.json @@ -13,36 +13,37 @@ "id": "active-outcome-count", "category": "workflow", "description": "Only one outcome should be active at a time", - "scope": "global", - "check": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" + "global": true, + "expression": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" }, { "id": "active-opportunity-count", "category": "workflow", "description": "Only one target opportunity should be active at a time", - "scope": "global", - "check": "$count(nodes[resolvedType='opportunity' and status='active']) <= 1" + "global": true, + "expression": "$count(nodes[resolvedType='opportunity' and status='active']) <= 1" }, { "id": "active-node-parent-active", "category": "workflow", "description": "An active node's parent should also be active", - "check": "current.status != 'active' or $exists(parent) = false or parent.status = 'active'" + "expression": "current.status != 'active' or $exists(parent) = false or parent.status = 'active'" }, { "id": "solution-quantity", "category": "best-practice", "description": "Explore multiple candidate solutions (aim for at least three) for the target opportunity", "type": "opportunity", - "check": "(current.status != 'exploring' and current.status != 'active') or $count(nodes[resolvedParentTitle=$$.current.title and resolvedType='solution']) >= 3" + "expression": "(current.status != 'exploring' and current.status != 'active') or $count(nodes[resolvedParentTitle=$$.current.title and resolvedType='solution']) >= 3" }, { "id": "opportunity-customer-voice", "category": "best-practice", - "mode": "semantic", "type": "opportunity", "description": "Opportunity must be framed in the customer's voice (something a customer would actually say) as a need or pain point, not a solution or business goal", - "rubric": "Verify that the opportunity is stated from the customer's perspective as a problem, pain point, or desire. It must not be a solution in disguise (e.g. 'build X feature') or a business metric (e.g. 'increase revenue')." + "semantic": { + "rubric": "Verify that the opportunity is stated from the customer's perspective as a problem, pain point, or desire. It must not be a solution in disguise (e.g. 'build X feature') or a business metric (e.g. 'increase revenue')." + } } ] }, diff --git a/schemas/generated/latest/_structured_context_schema_meta.json b/schemas/generated/latest/_structured_context_schema_meta.json index c1fa17d7..96b9d963 100644 --- a/schemas/generated/latest/_structured_context_schema_meta.json +++ b/schemas/generated/latest/_structured_context_schema_meta.json @@ -197,24 +197,13 @@ "enum": ["validation", "coherence", "workflow", "best-practice"] }, "severity": { - "oneOf": [ - { - "enum": ["error", "warning", "info"] - }, - { - "type": "object", - "additionalProperties": { - "enum": ["error", "warning", "info", "ignore"] - }, - "minProperties": 1 - } - ] + "enum": ["error", "warning", "info"] }, "description": { "type": "string", "minLength": 1 }, - "check": { + "expression": { "type": "string", "minLength": 1 }, @@ -222,46 +211,63 @@ "type": "string", "minLength": 1 }, - "mode": { - "enum": ["expression", "semantic"] - }, - "rubric": { - "type": "string", - "minLength": 1 - }, - "verdicts": { - "type": "object", - "additionalProperties": { - "oneOf": [ - { - "type": "string", - "minLength": 1 - }, - { - "type": "object", - "properties": { - "description": { - "type": "string", - "minLength": 1 + "semantic": { + "oneOf": [ + { + "const": true + }, + { + "type": "object", + "properties": { + "rubric": { + "type": "string", + "minLength": 1 + }, + "verdicts": { + "type": "object", + "additionalProperties": { + "oneOf": [ + { + "type": "string", + "minLength": 1 + }, + { + "type": "object", + "properties": { + "description": { + "type": "string", + "minLength": 1 + }, + "guidance": { + "type": "string", + "minLength": 1 + } + }, + "required": ["description"], + "additionalProperties": false + } + ] }, - "guidance": { - "type": "string", - "minLength": 1 - } + "minProperties": 2 }, - "required": ["description"], - "additionalProperties": false - } - ] - }, - "minProperties": 2 + "severity": { + "type": "object", + "additionalProperties": { + "enum": ["error", "warning", "info", "ignore"] + }, + "minProperties": 1 + } + }, + "additionalProperties": false + } + ] }, "type": { "type": "string", "minLength": 1 }, - "scope": { - "enum": ["global"] + "global": { + "type": "boolean" }, "override": { "type": "boolean" @@ -271,66 +277,110 @@ } }, "required": ["id", "category", "description"], - "not": { - "required": ["when", "scope"] - }, - "if": { - "properties": { - "mode": { - "const": "semantic" - } - }, - "required": ["mode"] - }, - "then": { - "not": { - "required": ["scope"] - }, - "if": { - "required": ["verdicts"] - }, - "then": { - "required": ["severity"], - "properties": { - "severity": { - "type": "object" + "additionalProperties": false, + "allOf": [ + { + "anyOf": [ + { + "required": ["expression"] + }, + { + "required": ["semantic"] } - } + ] }, - "else": { - "properties": { - "severity": { - "oneOf": [ + { + "if": { + "required": ["global"], + "properties": { + "global": { + "const": true + } + } + }, + "then": { + "not": { + "anyOf": [ + { + "required": ["when"] + }, { - "enum": ["error", "warning", "info"] + "required": ["type"] }, { - "type": "object" + "required": ["semantic"] } ] } } - } - }, - "else": { - "required": ["check"], - "not": { - "anyOf": [ - { - "required": ["verdicts"] - }, - { - "required": ["rubric"] + }, + { + "if": { + "required": ["semantic"], + "properties": { + "semantic": { + "type": "object", + "required": ["verdicts"] + } } - ] + }, + "then": { + "properties": { + "semantic": { + "type": "object", + "required": ["severity"] + } + } + } }, - "properties": { - "severity": { - "enum": ["error", "warning", "info"] + { + "if": { + "required": ["semantic"], + "properties": { + "semantic": { + "type": "object", + "not": { + "required": ["verdicts"] + } + } + } + }, + "then": { + "properties": { + "semantic": { + "type": "object", + "properties": { + "severity": { + "type": "object", + "not": { + "required": ["fail"] + } + } + } + } + } + } + }, + { + "if": { + "required": ["semantic"], + "properties": { + "semantic": { + "type": "object", + "required": ["verdicts"] + } + }, + "not": { + "required": ["expression"] + } + }, + "then": { + "not": { + "required": ["severity"] + } } } - }, - "additionalProperties": false + ] } } }, diff --git a/src/commands/schemas.ts b/src/commands/schemas.ts index d6cb99fb..16343288 100644 --- a/src/commands/schemas.ts +++ b/src/commands/schemas.ts @@ -9,6 +9,7 @@ import { resolveSpacePluginSchemas, shortenPluginName, } from '../plugins/util'; +import { ruleSemantic } from '../schema/rule-severity'; import { bundledSchemasDir, type EntityInfo, @@ -226,15 +227,16 @@ function generateMermaidErd(metadata: SchemaMetadata, entities: EntityInfo[]): s return mmd; } -/** Bracketed notes on a rule: mode, severity (or verdict severities) and `when`. Empty for a plain rule. */ +/** Bracketed notes on a rule: semantic, severity (and verdict severities) and `when`. Empty for a plain rule. */ function ruleAnnotations(rule: Rule): string { const notes: string[] = []; - if (rule.mode === 'semantic') notes.push('semantic'); - if (typeof rule.severity === 'string') { - notes.push(`severity: ${rule.severity}`); - } else if (rule.severity) { + const semantic = ruleSemantic(rule); + if (rule.global) notes.push('global'); + if (semantic) notes.push('semantic'); + if (rule.severity) notes.push(`severity: ${rule.severity}`); + if (semantic?.severity) { notes.push( - `severity: ${Object.entries(rule.severity) + `verdict severity: ${Object.entries(semantic.severity) .map(([verdict, sev]) => `${verdict}=${sev}`) .join(', ')}`, ); diff --git a/src/commands/validate-file.ts b/src/commands/validate-file.ts index f14c482b..c7f5d275 100644 --- a/src/commands/validate-file.ts +++ b/src/commands/validate-file.ts @@ -14,13 +14,13 @@ export { validateFile } from '../validate'; */ export async function validateFileCommand( filePath: string, - options: { json?: boolean; includeGated?: boolean; semantic?: boolean; judge?: string; ruleFilter?: string[] } = {}, + options: { json?: boolean; includeGated?: boolean; semantic?: boolean | string; ruleFilter?: string[] } = {}, ): Promise { const config = loadConfig(); const resolved = resolveFileSpace(filePath, config); let result: ValidateFileOutput; if (resolved) { - const judge = judgeForRun(config, resolved.space, options); + const judge = judgeForRun(config, resolved.space, options.semantic); result = await validateResolvedFile(filePath, resolved.label, createSpaceContext(resolved.space.name, config), { includeGated: options.includeGated, judge, diff --git a/src/commands/validate.ts b/src/commands/validate.ts index ce9722b0..489ed21c 100644 --- a/src/commands/validate.ts +++ b/src/commands/validate.ts @@ -14,8 +14,7 @@ export async function validate( json?: boolean; depth?: number; includeGated?: boolean; - semantic?: boolean; - judge?: string; + semantic?: boolean | string; ruleFilter?: string[]; } = {}, ): Promise { @@ -23,7 +22,7 @@ export async function validate( const result = await validateSpace(context, { depth: options.depth, includeGated: options.includeGated, - judge: judgeForRun(context.config, context.space, options), + judge: judgeForRun(context.config, context.space, options.semantic), ruleFilter: options.ruleFilter && parseRuleFilter(options.ruleFilter), }); const ruleCounts: Record = { error: 0, warning: 0, info: 0 }; diff --git a/src/config.ts b/src/config.ts index 7f90c84b..315727c8 100644 --- a/src/config.ts +++ b/src/config.ts @@ -28,7 +28,7 @@ const MODEL_PROVIDER_SCHEMA = { additionalProperties: false, } as const; -const JUDGE_CONFIG_SCHEMA = { +const SEMANTIC_CONFIG_SCHEMA = { type: 'object', properties: { provider: { type: 'string', minLength: 1 } }, additionalProperties: false, @@ -55,7 +55,7 @@ const CONFIG_SCHEMA = { additionalProperties: false, }, }, - judge: JUDGE_CONFIG_SCHEMA, + semantic: SEMANTIC_CONFIG_SCHEMA, }, required: ['name', 'path'], additionalProperties: false, @@ -64,7 +64,7 @@ const CONFIG_SCHEMA = { schema: { type: 'string' }, includeSpacesFrom: { type: 'array', items: { type: 'string' } }, modelProviders: { type: 'object', additionalProperties: MODEL_PROVIDER_SCHEMA }, - judge: JUDGE_CONFIG_SCHEMA, + semantic: SEMANTIC_CONFIG_SCHEMA, }, anyOf: [{ required: ['spaces'] }, { required: ['includeSpacesFrom'] }], additionalProperties: false, @@ -78,8 +78,8 @@ export type SpaceConfig = { plugins?: Record>; /** Named filter views for this space. Keys are view names; values contain the filter expression. */ views?: Record; - /** Judge for this space's semantic rules; overrides the top-level `judge`. */ - judge?: JudgeConfig; + /** Model provider for this space's semantic rules; overrides the top-level `semantic`. */ + semantic?: JudgeConfig; /** * Directory of the config file that defined this space, used to anchor relative plugin-config * paths. Set by loadConfig; absent for a hand-assembled Config, whose callers must pass an @@ -94,8 +94,8 @@ export type Config = { includeSpacesFrom?: string[]; /** Model providers by name: how to reach each model. */ modelProviders?: Record; - /** Judge for semantic rules (a model provider name). A space's `judge` overrides it. */ - judge?: JudgeConfig; + /** Model provider for semantic rules (a provider name). A space's `semantic` overrides it. */ + semantic?: JudgeConfig; }; let _configPathOverride: string | undefined; diff --git a/src/index.ts b/src/index.ts index 3c69d752..63a6a60b 100755 --- a/src/index.ts +++ b/src/index.ts @@ -38,7 +38,7 @@ export function buildSpaceContext(spaceName: string): SpaceContext { } } -/** Run a validate command, reporting a misconfigured judge as a plain error rather than a stack trace. */ +/** Run a validate command, reporting a misconfigured model provider as a plain error rather than a stack trace. */ async function reportJudgeConfigErrors(run: () => Promise): Promise { try { return await run(); @@ -75,18 +75,16 @@ program ) .option( '--rule-filter ', - 'Evaluate only matching rules; dimensions: mode, category, severity, type, id (repeatable, e.g. mode=semantic)', + 'Evaluate only matching rules; dimensions: category, severity, type, id (repeatable, e.g. severity=error)', (v: string, prev: string[] = []) => [...prev, v], ) - .option('--semantic', 'Evaluate semantic rules with the configured judge') - .option('--judge ', 'Judge semantic rules with this model provider (implies --semantic)') + .option('--semantic [provider]', 'Evaluate semantic rules, with the configured model provider or the one named') .action(async (filePath, options) => { process.exitCode = await reportJudgeConfigErrors(() => validateFileCommand(filePath, { json: options.json, includeGated: options.includeGated, semantic: options.semantic, - judge: options.judge, ruleFilter: options.ruleFilter, }), ); @@ -109,11 +107,10 @@ program ) .option( '--rule-filter ', - 'Evaluate only matching rules; dimensions: mode, category, severity, type, id (repeatable, e.g. mode=semantic)', + 'Evaluate only matching rules; dimensions: category, severity, type, id (repeatable, e.g. severity=error)', (v: string, prev: string[] = []) => [...prev, v], ) - .option('--semantic', 'Evaluate semantic rules with the configured judge') - .option('--judge ', 'Judge semantic rules with this model provider (implies --semantic)') + .option('--semantic [provider]', 'Evaluate semantic rules, with the configured model provider or the one named') .action(async (spaceName, options) => { const context = buildSpaceContext(spaceName); if (options.watch) { @@ -126,7 +123,6 @@ program depth: options.depth, includeGated: options.includeGated, semantic: options.semantic, - judge: options.judge, ruleFilter: options.ruleFilter, }), ); diff --git a/src/judge/factory.ts b/src/judge/factory.ts index 66f5a349..dd006281 100644 --- a/src/judge/factory.ts +++ b/src/judge/factory.ts @@ -82,7 +82,7 @@ const DEFAULT_KEY_ENV: Record = { 'typesafe-systemone': 'TYPESAFE_API_KEY', } as const; -/** The judge is misconfigured (no provider, unknown provider, missing key). Reported as a plain CLI error. */ +/** Semantic evaluation is misconfigured (no provider, unknown provider, missing key). Reported as a plain CLI error. */ export class JudgeConfigError extends Error {} /** @@ -102,7 +102,7 @@ export function hydrateProvider(name: string, def: ModelProviderConfig, env: Env const envValue = env[envName]; if (envValue !== undefined && envValue.trim() === '') { throw new JudgeConfigError( - `Model provider "${name}" has ${envName}=${envName} set but empty; set a value or unset the variable.`, + `Model provider "${name}" has ${envName} set but empty; set a value or unset the variable.`, ); } return envValue; @@ -148,33 +148,33 @@ export function createModelProvider( } /** - * The judge a CLI run asks for: none without `--semantic` or `--judge `; the named provider with - * `--judge`; else the configured one. + * The provider a CLI run asks for: none without `--semantic`; the named one with `--semantic `; + * else the configured one. */ export function judgeForRun( config: Parameters[0], space: Parameters[1], - options: { semantic?: boolean; judge?: string }, + semantic?: boolean | string, ): SemanticJudgeProvider | undefined { - if (!options.semantic && options.judge === undefined) return undefined; - return createJudge(config, space, options.judge); + if (!semantic) return undefined; + return createJudge(config, space, typeof semantic === 'string' ? semantic : undefined); } /** - * The judge for a run: the model provider named by `name` (the `--judge` override), else the space's - * `judge.provider`, else the top-level one. The name resolves against `modelProviders` first, then + * The provider for a run: the one named by `name` (the `--semantic ` override), else the space's + * `semantic.provider`, else the top-level one. The name resolves against `modelProviders` first, then * the bundled providers. Throws {@link JudgeConfigError} when it cannot be built. */ export function createJudge( - config: { modelProviders?: Record; judge?: { provider?: string } }, - space: { judge?: { provider?: string } } = {}, + config: { modelProviders?: Record; semantic?: { provider?: string } }, + space: { semantic?: { provider?: string } } = {}, name?: string, env: Env = process.env, ): SemanticJudgeProvider { - const chosen = name ?? space.judge?.provider ?? config.judge?.provider; + const chosen = name ?? space.semantic?.provider ?? config.semantic?.provider; if (!chosen) { throw new JudgeConfigError( - 'No judge configured. Set "judge.provider" in config.json to a bundled provider (e.g. "openrouter-jev") or to the name of an entry in "modelProviders".', + '--semantic needs a model provider. Set "semantic.provider" in config.json, or pass --semantic .', ); } const def = config.modelProviders?.[chosen] ?? BUNDLED_PROVIDERS[chosen]; diff --git a/src/judge/judge-node.ts b/src/judge/judge-node.ts index 844a0a75..c2d9d308 100644 --- a/src/judge/judge-node.ts +++ b/src/judge/judge-node.ts @@ -4,7 +4,7 @@ import { isJudgeError, resolveSemanticRule } from './verdicts'; export type NodeJudgement = { rule: Rule; - /** True when the verdict fails the rule (it has a severity), or the judge errored. */ + /** True when the verdict fails the rule (it has a severity), or the provider errored. */ failed: boolean; /** Severity of the failing verdict; unset for a judge error, which takes the rule's own severity. */ severity?: RuleSeverity; @@ -36,12 +36,12 @@ function interpret(rules: Rule[], resolved: ResolvedRules, results: JudgeResults return rules.map((rule, i) => { const result = results[rule.id]; if (!result || isJudgeError(result)) { - const message = result ? result.error : 'Judge returned no result for this rule'; + const message = result ? result.error : 'Model provider returned no result for this rule'; return { rule, failed: true, evidence: `Error during evaluation: ${message}`, error: message }; } const severity = resolved[i]!.severities[result.verdict]; if (severity === undefined && !(result.verdict in resolved[i]!.spec.verdicts)) { - const message = `Judge returned unknown verdict "${result.verdict}" (allowed: ${Object.keys(resolved[i]!.spec.verdicts).join(', ')})`; + const message = `Model provider returned unknown verdict "${result.verdict}" (allowed: ${Object.keys(resolved[i]!.spec.verdicts).join(', ')})`; return { rule, failed: true, evidence: `Error during evaluation: ${message}`, error: message }; } const failed = severity !== undefined; diff --git a/src/judge/prompt.ts b/src/judge/prompt.ts index 1d62876f..05712de9 100644 --- a/src/judge/prompt.ts +++ b/src/judge/prompt.ts @@ -80,7 +80,7 @@ export function parseJudgeResponse(rawText: string, allowed: string[]): Semantic if (verdict === undefined || !allowed.includes(verdict)) { return { - error: `Judge returned ${verdict === undefined ? 'no verdict' : `unknown verdict "${verdict}"`} (allowed: ${allowed.join(', ')}): ${rawText.slice(0, 200)}`, + error: `Model provider returned ${verdict === undefined ? 'no verdict' : `unknown verdict "${verdict}"`} (allowed: ${allowed.join(', ')}): ${rawText.slice(0, 200)}`, }; } return { diff --git a/src/judge/types.ts b/src/judge/types.ts index c11ad7be..645d98e9 100644 --- a/src/judge/types.ts +++ b/src/judge/types.ts @@ -81,7 +81,7 @@ export interface HydratedProvider { keyEnv?: string; } -/** Which model provider judges semantic rules: config `judge`, at the top level or on a space. */ +/** Which model provider evaluates semantic rules: config `semantic`, at the top level or on a space. */ export interface JudgeConfig { /** Name of an entry in `modelProviders`, or a bundled provider. */ provider?: string; diff --git a/src/judge/verdicts.ts b/src/judge/verdicts.ts index 1f593c8b..2ff0d4f8 100644 --- a/src/judge/verdicts.ts +++ b/src/judge/verdicts.ts @@ -1,4 +1,5 @@ -import { ruleSeverity } from '../schema/rule-severity'; +import { RESERVED_VERDICTS } from '../schema/metadata-contract'; +import { ruleSemantic, ruleSeverity } from '../schema/rule-severity'; import type { Rule, RuleSeverity } from '../types'; import type { SemanticJudgeResult, SemanticRuleSpec } from './types'; @@ -23,7 +24,9 @@ export interface ResolvedSemanticRule { * and work out which verdicts fail at what severity. Throws on an inconsistent rule. */ export function resolveSemanticRule(rule: Rule): ResolvedSemanticRule { - const declared = rule.verdicts ?? DEFAULT_VERDICTS; + const semantic = ruleSemantic(rule); + if (!semantic) throw new Error(`Rule "${rule.id}" has no semantic part.`); + const declared = semantic.verdicts ?? DEFAULT_VERDICTS; const verdicts: Record = {}; const guidance: Record = {}; for (const [name, value] of Object.entries(declared)) { @@ -36,25 +39,29 @@ export function resolveSemanticRule(rule: Rule): ResolvedSemanticRule { } let mapping: Record; - if (typeof rule.severity === 'object') { - mapping = rule.severity; - } else if (rule.verdicts) { - throw new Error( - `Semantic rule "${rule.id}" declares verdicts, so its severity must map the failing verdicts, e.g. { "${Object.keys(verdicts)[0]}": "warning" }.`, - ); + if (semantic.verdicts) { + if (!semantic.severity) { + throw new Error( + `Semantic rule "${rule.id}" declares verdicts, so semantic.severity must map the failing verdicts, e.g. { "${Object.keys(verdicts)[0]}": "warning" }.`, + ); + } + mapping = semantic.severity; } else { - mapping = { fail: ruleSeverity(rule) }; + mapping = { ...semantic.severity, fail: ruleSeverity(rule) }; } - const unknown = Object.keys(mapping).filter((v) => !(v in verdicts)); + const reserved: readonly string[] = RESERVED_VERDICTS; + const unknown = Object.keys(mapping).filter((v) => !(v in verdicts) && !reserved.includes(v)); if (unknown.length > 0) { - throw new Error(`Semantic rule "${rule.id}" has severity for verdicts not in its verdicts: ${unknown.join(', ')}.`); + throw new Error( + `Semantic rule "${rule.id}" has semantic.severity for verdicts not in its verdicts: ${unknown.join(', ')}.`, + ); } const severities: Record = {}; for (const [verdict, severity] of Object.entries(mapping)) { if (severity !== 'ignore') severities[verdict] = severity; } - if (Object.keys(severities).length >= Object.keys(verdicts).length) { + if (Object.keys(severities).filter((v) => v in verdicts).length >= Object.keys(verdicts).length) { throw new Error(`Semantic rule "${rule.id}" has no passing verdict: every verdict has a severity.`); } @@ -62,7 +69,7 @@ export function resolveSemanticRule(rule: Rule): ResolvedSemanticRule { spec: { id: rule.id, description: rule.description, - rubric: rule.rubric, + rubric: semantic.rubric, category: rule.category, verdicts, }, diff --git a/src/schema/evaluate-rule.ts b/src/schema/evaluate-rule.ts index aaf2c584..038bacf9 100644 --- a/src/schema/evaluate-rule.ts +++ b/src/schema/evaluate-rule.ts @@ -68,7 +68,7 @@ export class ExpressionError extends Error { } } -/** Result of evaluating a rule's `check` or `when`: a value, or the error that stopped it (never thrown). */ +/** Result of evaluating a rule's `expression` or `when`: a value, or the error that stopped it (never thrown). */ export type ExpressionOutcome = { value: boolean | string | number } | { error: string }; /** Like {@link evaluateExpression} but reports a failure as data, so callers can turn it into a violation. */ @@ -130,7 +130,7 @@ export function buildEvalContext( }; } -/** The violation reported when a rule's `check` or `when` cannot be evaluated: an authoring error, so always `error`. */ +/** The violation reported when a rule's `expression` or `when` cannot be evaluated: an authoring error, so always `error`. */ export function expressionErrorViolation(rule: Rule, file: string, error: string): RuleViolation { return { file, @@ -141,42 +141,40 @@ export function expressionErrorViolation(rule: Rule, file: string, error: string }; } +/** The outcome of a rule's `expression` on one node. */ +export type ExpressionRuleResult = + | { passed: true } + /** `failed` is the expression returning something other than true; otherwise it could not be evaluated. */ + | { passed: false; violation: RuleViolation; failed: boolean }; + /** - * Applies the `check` precondition of semantic rules on a node. A rule whose `check` fails or errors reports a - * violation (at its own severity, or `error` for an evaluation failure) and is left out of `judgeable`, so the - * judge never sees it. A failing check's violation names the expression in its evidence, and its rule is listed in - * `failed` so the caller can honour `gatesDescendants`; an evaluation error is never listed. + * Evaluate a rule's `expression` against a node (`file` empty and `node` a sentinel for a global rule). A failure + * reports the rule's severity; a rule with a semantic part names the expression in the evidence, so the + * violation is not mistaken for a verdict. An evaluation error reports `error` and is not a `failed` result. */ -export async function applySemanticChecks( - rules: Rule[], +export async function evaluateRuleExpression( + rule: Rule, node: SpaceNode, allNodes: SpaceNode[], nodeIndex: Map, -): Promise<{ judgeable: Rule[]; violations: RuleViolation[]; failed: Rule[] }> { - const judgeable: Rule[] = []; - const violations: RuleViolation[] = []; - const failed: Rule[] = []; - for (const rule of rules) { - if (!rule.check) { - judgeable.push(rule); - continue; - } - const outcome = await tryEvaluateExpression(rule.check, buildEvalContext(node, allNodes, nodeIndex)); - if ('error' in outcome) { - violations.push(expressionErrorViolation(rule, node.label, outcome.error)); - } else if (outcome.value !== true) { - violations.push({ - file: node.label, - ruleId: rule.id, - category: rule.category, - severity: ruleSeverity(rule), - description: rule.description, - evidence: `expression check failed: ${rule.check}`, - }); - failed.push(rule); - } else { - judgeable.push(rule); - } + file: string = node.label, +): Promise { + const expression = rule.expression as string; + const outcome = await tryEvaluateExpression(expression, buildEvalContext(node, allNodes, nodeIndex)); + if ('error' in outcome) { + return { passed: false, failed: false, violation: expressionErrorViolation(rule, file, outcome.error) }; } - return { judgeable, violations, failed }; + if (outcome.value === true) return { passed: true }; + return { + passed: false, + failed: true, + violation: { + file, + ruleId: rule.id, + category: rule.category, + severity: ruleSeverity(rule), + description: rule.description, + ...(rule.semantic !== undefined && { evidence: `expression failed: ${expression}` }), + }, + }; } diff --git a/src/schema/gating.ts b/src/schema/gating.ts index 70727871..77836f32 100644 --- a/src/schema/gating.ts +++ b/src/schema/gating.ts @@ -4,16 +4,16 @@ import type { SemanticJudgeProvider } from '../judge/types'; import type { SpaceGraph } from '../space-graph'; import type { Rule, RuleSeverity, RuleViolation, SpaceNode } from '../types'; import { - applySemanticChecks, buildEvalContext, + evaluateRuleExpression, expressionErrorViolation, tryEvaluateExpression, } from './evaluate-rule'; import { ruleSeverities, ruleSeverity } from './rule-severity'; /** - * Whether a violation of this rule gates its subtree: explicit `gatesDescendants`, else true only for `error` severity. - * `severity` is that of the violation; a semantic rule's verdicts can differ. + * Whether a failure of this rule gates its subtree: explicit `gatesDescendants`, else true only for `error` severity. + * `severity` is that of the failure: the rule's own for an expression, the verdict's for a semantic part. */ export function ruleGatesDescendants(rule: Rule, severity: RuleSeverity = ruleSeverity(rule)): boolean { return rule.gatesDescendants ?? severity === 'error'; @@ -62,12 +62,12 @@ export type GatedRuleEvaluation = { export interface GatingOptions { /** Evaluate rules on gated nodes anyway (gates are still reported). */ includeGated?: boolean; - /** Judge for `mode: semantic` rules. Without it, semantic rules are not evaluated. */ + /** Evaluates `semantic` rules. Without it, a rule's semantic part is not evaluated. */ provider?: SemanticJudgeProvider; /** * Label of the one node whose result is wanted (`validate-file`). Gating is still computed over the * whole space, but the judge is only consulted for that node and for its hierarchy ancestors' gate-capable - * semantic rules (top-down, so a gated ancestor stops the walk). Other nodes' semantic rules are not judged. + * semantic rules (top-down, so a gated ancestor stops the walk). Other nodes' semantic rules are not evaluated. */ judgeTarget?: string; } @@ -103,7 +103,7 @@ export function computeHierarchyDepths(graph: SpaceGraph): Map { * * Traversal is breadth-first from hierarchy roots and orphans. Non-hierarchy nodes * (not part of the hierarchy DAG) are always evaluated — cascade gating only applies - * within hierarchy subtrees. Global-scope rules always run, unaffected by gating. + * within hierarchy subtrees. Global rules always run, unaffected by gating. */ export async function evaluateRulesWithGating( nodes: SpaceNode[], @@ -143,27 +143,18 @@ export async function evaluateRulesWithGating( return origin; } - const globalRules = rules.filter((r) => r.scope === 'global' && r.mode !== 'semantic'); + const globalRules = rules.filter((r) => r.global && r.expression !== undefined); const sentinel = nodes[0]; if (sentinel) { for (const rule of globalRules) { - const context = buildEvalContext(sentinel, nodes, nodeIndex); - const outcome = await tryEvaluateExpression(rule.check as string, context); - if ('error' in outcome) { - violations.push(expressionErrorViolation(rule, '', outcome.error)); - } else if (outcome.value !== true) { - violations.push({ - file: '', - ruleId: rule.id, - category: rule.category, - severity: ruleSeverity(rule), - description: rule.description, - }); - } + const result = await evaluateRuleExpression(rule, sentinel, nodes, nodeIndex, ''); + if (!result.passed) violations.push(result.violation); } } - const perNodeRules = rules.filter((r) => r.scope !== 'global' && (r.mode !== 'semantic' || judge || r.check)); + const perNodeRules = rules.filter( + (r) => !r.global && (r.expression !== undefined || (r.semantic !== undefined && judge)), + ); // With a judge target, only the target and its hierarchy ancestors can need judging. const targetAncestors = new Set(); @@ -189,7 +180,12 @@ export async function evaluateRulesWithGating( const applicable: Rule[] = []; for (const r of perNodeRules) { if (r.type && r.type !== node.resolvedType) continue; - if (r.mode === 'semantic' && options.judgeTarget !== undefined && node.label !== options.judgeTarget) { + if ( + judge && + r.semantic !== undefined && + options.judgeTarget !== undefined && + node.label !== options.judgeTarget + ) { // Ancestors only matter through the gates they could raise. if (!(targetAncestors.has(node.title) && ruleCanGate(r))) continue; } @@ -206,7 +202,7 @@ export async function evaluateRulesWithGating( return applicable; } - /** A node whose deterministic rules have run and whose semantic rules await the judge. */ + /** A node whose expressions have run and whose semantic parts await the judge. */ type PreparedNode = { violations: RuleViolation[]; semantic: Rule[]; @@ -225,41 +221,39 @@ export async function evaluateRulesWithGating( rule: Rule, detail: { severity?: RuleSeverity; verdict?: string; guidance?: string; evidence?: string; error?: string } = {}, ): void { - const severity = detail.severity ?? ruleSeverity(rule); + const severity = detail.severity ?? ruleSeverities(rule)[0] ?? ruleSeverity(rule); local.push({ file: node.label, ruleId: rule.id, category: rule.category, severity, - description: detail.error ? `${rule.description} (Judge error: ${detail.error})` : rule.description, + description: detail.error + ? `${rule.description} (Semantic evaluation error: ${detail.error})` + : rule.description, ...(detail.evidence !== undefined && { evidence: detail.evidence }), ...(detail.verdict !== undefined && { verdict: detail.verdict }), ...(detail.guidance !== undefined && { guidance: detail.guidance }), }); - // A judge error says nothing about the node, so it never gates. + // An evaluation error says nothing about the node, so it never gates. if (!detail.error && ruleGatesDescendants(rule, severity) && !prepared.blockingRuleId) { prepared.blockingRuleId = rule.id; } } - // Cheap deterministic rules first; the judge is only consulted afterwards. - for (const rule of applicable.filter((r) => r.mode !== 'semantic')) { - const outcome = await tryEvaluateExpression(rule.check as string, buildEvalContext(node, nodes, nodeIndex)); - if ('error' in outcome) local.push(expressionErrorViolation(rule, node.label, outcome.error)); - else if (outcome.value !== true) report(rule); + // Cheap expressions first; a semantic part is only evaluated when its own expression passes. + for (const rule of applicable) { + if (rule.expression !== undefined) { + const result = await evaluateRuleExpression(rule, node, nodes, nodeIndex); + if (!result.passed) { + local.push(result.violation); + if (result.failed && ruleGatesDescendants(rule, result.violation.severity) && !prepared.blockingRuleId) { + prepared.blockingRuleId = rule.id; + } + continue; + } + } + if (rule.semantic !== undefined && judge) prepared.semantic.push(rule); } - - // A semantic rule's `check` fails fast before the judge. Its violation gates like an expression rule's, at the severity it reports. - const prechecked = await applySemanticChecks( - applicable.filter((r) => r.mode === 'semantic'), - node, - nodes, - nodeIndex, - ); - local.push(...prechecked.violations); - const gatingRule = prechecked.failed.find((r) => ruleGatesDescendants(r)); - if (gatingRule && !prepared.blockingRuleId) prepared.blockingRuleId = gatingRule.id; - if (judge) prepared.semantic = prechecked.judgeable; prepared.judged = (judgements) => { for (const j of judgements) { if (j.failed) { diff --git a/src/schema/metadata-contract.ts b/src/schema/metadata-contract.ts index 46586f65..eb63ad42 100644 --- a/src/schema/metadata-contract.ts +++ b/src/schema/metadata-contract.ts @@ -57,35 +57,20 @@ const HIERARCHY_LEVEL_SCHEMA = { const SEVERITY_SCHEMA = { enum: ['error', 'warning', 'info'] } as const; -const RULE_SCHEMA = { +/** The verdicts the evaluation reserves besides a rule's own; a verdict map may set a severity for them. */ +export const RESERVED_VERDICTS = ['missing-context'] as const; + +const VERDICT_SEVERITY_SCHEMA = { enum: ['error', 'warning', 'info', 'ignore'] } as const; + +/** The semantic part of a rule: a model evaluates the node against a rubric and returns a verdict. */ +const SEMANTIC_SCHEMA = { type: 'object', properties: { - id: { type: 'string', minLength: 1 }, - category: { enum: ['validation', 'coherence', 'workflow', 'best-practice'] }, - /** - * A severity, or (semantic rules only) a map of verdict to severity. In a map, a verdict mapped to - * `ignore` is judged but not reported, and a verdict left out passes. Defaults from the category. - */ - severity: { - oneOf: [ - SEVERITY_SCHEMA, - { - type: 'object', - additionalProperties: { enum: ['error', 'warning', 'info', 'ignore'] }, - minProperties: 1, - }, - ], - }, - description: { type: 'string', minLength: 1 }, - /** Expression rules: the test a node must pass. Semantic rules: a precondition evaluated after `when` and before the judge; a failing node is reported without judging. */ - check: { type: 'string', minLength: 1 }, - /** Optional JSONata applicability test, evaluated per node before the rule runs. The rule applies only when it returns true. */ - when: { type: 'string', minLength: 1 }, - mode: { enum: ['expression', 'semantic'] }, + /** What the model judges the node against. Falls back to the rule's `description`. */ rubric: { type: 'string', minLength: 1 }, /** - * Semantic rules: the closed set of verdicts a judge may return. Each is a description (what the verdict - * means, for the judge) or `{ description, guidance }`, where `guidance` is shown to the user when it fires. + * The closed set of verdicts the model may return. Each is a description (what the verdict means, for the + * model) or `{ description, guidance }`, where `guidance` is shown to the user when it fires. * Defaults to `pass`/`fail`. */ verdicts: { @@ -103,32 +88,81 @@ const RULE_SCHEMA = { }, minProperties: 2, }, + /** + * Severity per verdict in effect (declared, default or reserved). A verdict mapped to `ignore` is judged but + * not reported, and a verdict left out passes. Required with `verdicts`; with the default verdicts, `fail` + * takes the rule's own `severity` instead. + */ + severity: { + type: 'object', + additionalProperties: VERDICT_SEVERITY_SCHEMA, + minProperties: 1, + }, + }, + additionalProperties: false, +} as const; + +const RULE_SCHEMA = { + type: 'object', + properties: { + id: { type: 'string', minLength: 1 }, + category: { enum: ['validation', 'coherence', 'workflow', 'best-practice'] }, + /** + * Severity of a failed `expression` and of the default `fail` verdict. Defaults from the category. + */ + severity: SEVERITY_SCHEMA, + description: { type: 'string', minLength: 1 }, + /** JSONata test a node must pass. When the rule also has `semantic`, a failing node is reported without being evaluated by the model. */ + expression: { type: 'string', minLength: 1 }, + /** Optional JSONata applicability condition, evaluated per node before the rule runs. The rule applies only when it returns true. */ + when: { type: 'string', minLength: 1 }, + /** A model evaluates the node against a rubric. `{}` takes every default; `true` means the same. */ + semantic: { oneOf: [{ const: true }, SEMANTIC_SCHEMA] }, type: { type: 'string', minLength: 1 }, - scope: { enum: ['global'] }, + /** Run once for the whole space instead of once per node. */ + global: { type: 'boolean' }, override: { type: 'boolean' }, - /** Whether a violation of this rule gates (suppresses) downstream checks on the node's hierarchy subtree. Defaults to true for `error` severity rules and false otherwise. */ + /** Whether a failure of this rule gates (suppresses) downstream rules on the node's hierarchy subtree. Defaults to true when the failure's severity is `error`. */ gatesDescendants: { type: 'boolean' }, }, required: ['id', 'category', 'description'], - // A global rule has no node to test, so `when` needs a per-node rule. - not: { required: ['when', 'scope'] }, - // Semantic rules are judged per node (not global, `check` optional) and with custom verdicts must map each failing - // verdict to a severity. Every other rule needs a `check` and takes none of the semantic-only fields. - if: { properties: { mode: { const: 'semantic' } }, required: ['mode'] }, - // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then`/`else` keyword, not a thenable - then: { - not: { required: ['scope'] }, - if: { required: ['verdicts'] }, - // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then`/`else` keyword, not a thenable - then: { required: ['severity'], properties: { severity: { type: 'object' } } }, - else: { properties: { severity: { oneOf: [SEVERITY_SCHEMA, { type: 'object' }] } } }, - }, - else: { - required: ['check'], - not: { anyOf: [{ required: ['verdicts'] }, { required: ['rubric'] }] }, - properties: { severity: SEVERITY_SCHEMA }, - }, additionalProperties: false, + allOf: [ + // A rule has an expression, a semantic part, or both. + { anyOf: [{ required: ['expression'] }, { required: ['semantic'] }] }, + // A global rule has no node to apply to or to evaluate. + { + if: { required: ['global'], properties: { global: { const: true } } }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable + then: { not: { anyOf: [{ required: ['when'] }, { required: ['type'] }, { required: ['semantic'] }] } }, + }, + // Custom verdicts need a severity for each failing verdict. + { + if: { required: ['semantic'], properties: { semantic: { type: 'object', required: ['verdicts'] } } }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable + then: { properties: { semantic: { type: 'object', required: ['severity'] } } }, + }, + // With the default verdicts, `fail` takes the rule's severity. + { + if: { required: ['semantic'], properties: { semantic: { type: 'object', not: { required: ['verdicts'] } } } }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable + then: { + properties: { + semantic: { type: 'object', properties: { severity: { type: 'object', not: { required: ['fail'] } } } }, + }, + }, + }, + // A rule with custom verdicts and no expression has no failure for the top-level severity to describe. + { + if: { + required: ['semantic'], + properties: { semantic: { type: 'object', required: ['verdicts'] } }, + not: { required: ['expression'] }, + }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then` keyword, not a thenable + then: { not: { required: ['severity'] } }, + }, + ], } as const; const REF_SCHEMA = { @@ -238,4 +272,5 @@ export type MetadataContractRelationship = FromSchema; export type SharedEmbeddingFields = FromSchema; export type Rule = FromSchema; +export type RuleSemantic = FromSchema; export type MetadataRef = FromSchema; diff --git a/src/schema/rule-filter.ts b/src/schema/rule-filter.ts index 4393e8a3..3e8b3e7a 100644 --- a/src/schema/rule-filter.ts +++ b/src/schema/rule-filter.ts @@ -1,7 +1,7 @@ import type { Rule } from '../types'; import { ruleSeverities } from './rule-severity'; -export const RULE_FILTER_DIMENSIONS = ['mode', 'category', 'severity', 'type', 'id'] as const; +export const RULE_FILTER_DIMENSIONS = ['category', 'severity', 'type', 'id'] as const; export type RuleFilterDimension = (typeof RULE_FILTER_DIMENSIONS)[number]; /** Narrows which rules are evaluated. Dimensions combine with AND; values within a dimension with OR. */ @@ -28,8 +28,6 @@ export function parseRuleFilter(specs: string[]): RuleFilter { function dimensionValues(rule: Rule, dimension: RuleFilterDimension): string[] { switch (dimension) { - case 'mode': - return [rule.mode ?? 'expression']; case 'severity': return ruleSeverities(rule); case 'category': diff --git a/src/schema/rule-severity.ts b/src/schema/rule-severity.ts index 969fd30c..c6c862c4 100644 --- a/src/schema/rule-severity.ts +++ b/src/schema/rule-severity.ts @@ -1,4 +1,5 @@ import type { Rule, RuleCategory, RuleSeverity } from '../types'; +import type { RuleSemantic } from './metadata-contract'; /** Default severity for each rule category. A rule's own `severity` field overrides this. */ export const DEFAULT_SEVERITY_BY_CATEGORY: Record = { @@ -10,15 +11,27 @@ export const DEFAULT_SEVERITY_BY_CATEGORY: Record = const SEVERITY_ORDER: RuleSeverity[] = ['error', 'warning', 'info']; -/** Every severity this rule can report. A verdict map lists several; `ignore` entries report nothing. */ -export function ruleSeverities(rule: Rule): RuleSeverity[] { - if (rule.severity === undefined) return [DEFAULT_SEVERITY_BY_CATEGORY[rule.category]]; - if (typeof rule.severity === 'string') return [rule.severity]; - const mapped = new Set(Object.values(rule.severity).filter((s): s is RuleSeverity => s !== 'ignore')); - return SEVERITY_ORDER.filter((s) => mapped.has(s)); +/** The rule's semantic part as an object (`semantic: true` means `{}`), or undefined for a rule without one. */ +export function ruleSemantic(rule: Rule): RuleSemantic | undefined { + if (rule.semantic === undefined) return undefined; + return rule.semantic === true ? {} : rule.semantic; } -/** Effective severity of a rule: its explicit `severity`, else its category default. With a verdict map, the most severe verdict. */ +/** Severity of a failed `expression` and of the default `fail` verdict: the rule's `severity`, else its category default. */ export function ruleSeverity(rule: Rule): RuleSeverity { - return ruleSeverities(rule)[0] ?? 'info'; + return rule.severity ?? DEFAULT_SEVERITY_BY_CATEGORY[rule.category]; +} + +/** + * Every severity this rule can report: that of its failed `expression` (or default `fail` verdict), plus those + * its semantic verdict map sets. `ignore` entries report nothing. Most severe first. + */ +export function ruleSeverities(rule: Rule): RuleSeverity[] { + const semantic = ruleSemantic(rule); + const found = new Set(); + if (rule.expression !== undefined || (semantic && !semantic.verdicts)) found.add(ruleSeverity(rule)); + for (const severity of Object.values(semantic?.severity ?? {})) { + if (severity !== 'ignore') found.add(severity); + } + return SEVERITY_ORDER.filter((s) => found.has(s)); } diff --git a/src/schema/validate-rules.ts b/src/schema/validate-rules.ts index 232a1bc5..d9639a1e 100644 --- a/src/schema/validate-rules.ts +++ b/src/schema/validate-rules.ts @@ -2,17 +2,17 @@ import { judgeNodes } from '../judge/judge-node'; import type { SemanticJudgeProvider } from '../judge/types'; import type { Rule, RuleViolation, SpaceNode } from '../types'; import { - applySemanticChecks, buildEvalContext, + evaluateRuleExpression, expressionErrorViolation, tryEvaluateExpression, } from './evaluate-rule'; -import { ruleSeverity } from './rule-severity'; +import { ruleSeverities, ruleSeverity } from './rule-severity'; export { DEFAULT_SEVERITY_BY_CATEGORY, ruleSeverity } from './rule-severity'; export interface ValidateRulesOptions { - /** Judge for semantic rules. Without one, a semantic rule runs only its `check`. */ + /** Evaluates semantic rules. Without one, a rule runs only its `expression`, and a semantic-only rule is skipped. */ judge?: SemanticJudgeProvider; /** Label of the only node to judge (`validate-file`); other nodes' semantic rules are skipped. */ judgeTarget?: string; @@ -22,7 +22,7 @@ export interface ValidateRulesOptions { /** * Validate nodes against rules metadata. - * Evaluates deterministic expression rules and (when opt-in) semantic judge rules. + * Evaluates expressions and (when opt-in) semantic parts. * * @param nodes - All nodes in the space * @param rules - Rules metadata list @@ -36,8 +36,7 @@ export async function validateRules( ): Promise { const violations: RuleViolation[] = []; - // Track nodes that have failed checks (for cascading gating) - const failedNodeLabels = new Set(options.gatedNodeLabels || []); + const judge = options.judge; // Build node index for efficient lookups const nodeIndex = new Map(); @@ -48,107 +47,69 @@ export async function validateRules( } } - // Separate rules into expression rules and semantic rules - const expressionRules: Rule[] = []; - const semanticRules: Rule[] = []; + const sentinel = nodes[0]; + const runnable = rules.filter((r) => r.expression !== undefined || (r.semantic !== undefined && judge)); - for (const rule of rules) { - if (rule.mode === 'semantic') { - semanticRules.push(rule); - } else if (rule.check) { - expressionRules.push(rule); + // The nodes each per-node rule applies to: matching `type` and passing the applicability condition (`when`). + const applicable = new Map(); + for (const rule of runnable) { + if (rule.global) continue; + const matching: SpaceNode[] = []; + for (const node of nodes) { + if (rule.type && rule.type !== node.resolvedType) continue; + if (rule.when) { + const when = await tryEvaluateExpression(rule.when, buildEvalContext(node, nodes, nodeIndex)); + if ('error' in when) { + violations.push(expressionErrorViolation(rule, node.label, when.error)); + continue; + } + if (when.value !== true) continue; + } + matching.push(node); } + applicable.set(rule, matching); } - // 1. Evaluate deterministic expression rules first - for (const rule of expressionRules) { - const check = rule.check as string; - const severity = ruleSeverity(rule); - - if (rule.scope === 'global') { - // Global rules are evaluated once against the full node set. - // A sentinel node provides the evaluation context (nodes array is what matters). - const sentinel = nodes[0]; - if (sentinel) { - const context = buildEvalContext(sentinel, nodes, nodeIndex); - const outcome = await tryEvaluateExpression(check, context); - if ('error' in outcome) { - violations.push(expressionErrorViolation(rule, '', outcome.error)); - } else if (outcome.value !== true) { - violations.push({ - file: '', - ruleId: rule.id, - category: rule.category, - severity, - description: rule.description, - }); - } - } - } else { - const targetNodes = rule.type ? nodes.filter((n) => n.resolvedType === rule.type) : nodes; - for (const node of targetNodes) { - const context = buildEvalContext(node, nodes, nodeIndex); - if (rule.when) { - const when = await tryEvaluateExpression(rule.when, context); - if ('error' in when) { - violations.push(expressionErrorViolation(rule, node.label, when.error)); - continue; - } - if (when.value !== true) continue; - } - const outcome = await tryEvaluateExpression(check, context); - if ('error' in outcome) { - violations.push(expressionErrorViolation(rule, node.label, outcome.error)); - } else if (outcome.value !== true) { - violations.push({ - file: node.label, - ruleId: rule.id, - category: rule.category, - severity, - description: rule.description, - }); - failedNodeLabels.add(node.label); - } - } + // 1. Expressions, in rule order. A global rule is evaluated once against the full node set; a sentinel node + // provides the evaluation context (the nodes array is what matters). + const failedNodeLabels = new Set(options.gatedNodeLabels); // nodes that failed an expression (or upstream) + const notEvaluable = new Set(); // `${rule.id}\0${label}` whose own expression failed or errored + for (const rule of runnable) { + if (rule.expression === undefined) continue; + if (rule.global) { + if (!sentinel) continue; + const result = await evaluateRuleExpression(rule, sentinel, nodes, nodeIndex, ''); + if (!result.passed) violations.push(result.violation); + continue; + } + for (const node of applicable.get(rule) ?? []) { + const result = await evaluateRuleExpression(rule, node, nodes, nodeIndex); + if (result.passed) continue; + violations.push(result.violation); + notEvaluable.add(`${rule.id}\0${node.label}`); + if (result.failed) failedNodeLabels.add(node.label); } } - // 2. Evaluate semantic rules. Without a judge only their `check` runs. - const judge = options.judge; - const runnableSemanticRules = judge ? semanticRules : semanticRules.filter((r) => r.check); - if (runnableSemanticRules.length > 0) { + // 2. Semantic parts, for nodes that passed their own expression. A node that failed an expression or + // upstream is left out (safe cascading default). + if (judge) { const perNode: RuleViolation[][] = []; // kept in node order const pending: Array<{ node: SpaceNode; rules: Rule[]; found: RuleViolation[] }> = []; for (const node of nodes) { if (options.judgeTarget !== undefined && node.label !== options.judgeTarget) continue; const found: RuleViolation[] = []; perNode.push(found); - const applicable: Rule[] = []; - for (const r of runnableSemanticRules) { - if (r.type && r.type !== node.resolvedType) continue; - if (r.when) { - const when = await tryEvaluateExpression(r.when, buildEvalContext(node, nodes, nodeIndex)); - if ('error' in when) { - found.push(expressionErrorViolation(r, node.label, when.error)); - continue; - } - if (when.value !== true) continue; - } - applicable.push(r); - } - if (applicable.length === 0) continue; - - // Safe cascading default: gate semantic evaluation if node failed upstream checks if (failedNodeLabels.has(node.label)) continue; - - // A semantic rule's `check` fails fast before the judge. - const prechecked = await applySemanticChecks(applicable, node, nodes, nodeIndex); - found.push(...prechecked.violations); - if (judge) pending.push({ node, rules: prechecked.judgeable, found }); + const semantic = runnable.filter( + (r) => + r.semantic !== undefined && applicable.get(r)?.includes(node) && !notEvaluable.has(`${r.id}\0${node.label}`), + ); + if (semantic.length > 0) pending.push({ node, rules: semantic, found }); } - // One judge call for every node, so a judge with `evaluateNodes` can share model calls. - const judged = judge ? await judgeNodes(judge, pending) : []; + // One provider call for every node, so a provider with `evaluateNodes` can share model calls. + const judged = await judgeNodes(judge, pending); pending.forEach(({ node, found }, i) => { for (const j of judged[i] ?? []) { if (!j.failed) continue; @@ -156,8 +117,8 @@ export async function validateRules( file: node.label, ruleId: j.rule.id, category: j.rule.category, - severity: j.severity ?? ruleSeverity(j.rule), - description: j.error ? `${j.rule.description} (Judge error: ${j.error})` : j.rule.description, + severity: j.severity ?? ruleSeverities(j.rule)[0] ?? ruleSeverity(j.rule), + description: j.error ? `${j.rule.description} (Semantic evaluation error: ${j.error})` : j.rule.description, evidence: j.evidence, ...(j.verdict !== undefined && { verdict: j.verdict }), ...(j.guidance !== undefined && { guidance: j.guidance }), diff --git a/src/validate.ts b/src/validate.ts index af79e2f6..547b659f 100644 --- a/src/validate.ts +++ b/src/validate.ts @@ -169,11 +169,11 @@ export interface ValidateSpaceOptions { includeGated?: boolean; /** Evaluate only rules matching this filter (see `parseRuleFilter`). */ ruleFilter?: RuleFilter; - /** Judge for `mode: semantic` rules. Without one, a semantic rule runs only its `check`. */ + /** Evaluates `semantic` rules. Without one, a rule runs only its `expression`, and a semantic-only rule is skipped. */ judge?: SemanticJudgeProvider; } -/** One-line text for a rule violation: rule, description, and any judge evidence and verdict guidance. */ +/** One-line text for a rule violation: rule, description, and any evidence and verdict guidance. */ export function ruleViolationMessage(v: RuleViolation): string { const evidence = v.evidence ? ` (Evidence: ${v.evidence})` : ''; const guidance = v.guidance ? ` Guidance: ${v.guidance}` : ''; @@ -440,7 +440,7 @@ export interface ValidateFileOptions { includeGated?: boolean; /** Evaluate only rules matching this filter (see `parseRuleFilter`). */ ruleFilter?: RuleFilter; - /** Judge for `mode: semantic` rules. Without one, a semantic rule runs only its `check`. */ + /** Evaluates `semantic` rules. Without one, a rule runs only its `expression`, and a semantic-only rule is skipped. */ judge?: SemanticJudgeProvider; } diff --git a/tests/commands/validate-judge-flag.test.ts b/tests/commands/validate-judge-flag.test.ts index e88d6f6b..c6bac169 100644 --- a/tests/commands/validate-judge-flag.test.ts +++ b/tests/commands/validate-judge-flag.test.ts @@ -5,7 +5,7 @@ import { makeTempSpace } from '../helpers/temp-space'; const ROOT = join(import.meta.dir, '..', '..'); const space = makeTempSpace( - [{ id: 'step-fit', type: 'step', mode: 'semantic', category: 'coherence', description: 'Step fits its task' }], + [{ id: 'step-fit', type: 'step', semantic: {}, category: 'coherence', description: 'Step fits its task' }], { judging: { modelProviders: { f: { api: 'fake' } } } }, ); const configPath = join(space.dir, 'config.json'); @@ -22,20 +22,20 @@ function spawn(...args: string[]) { return { ...result, stderr: new TextDecoder().decode(result.stderr) }; } -describe('--judge on the CLI', () => { - // An unknown name fails only if the flag reaches the judge. +describe('--semantic on the CLI', () => { + // An unknown name fails only if the flag reaches the provider factory. for (const args of [ ['validate', 'ts'], ['validate-file', space.file('S.md')], ]) { - it(`${args[0]} passes --judge to the judge`, () => { - const result = spawn(...args, '--judge', 'missing'); + it(`${args[0]} passes the provider name`, () => { + const result = spawn(...args, '--semantic', 'missing'); expect(result.exitCode).toBe(1); expect(result.stderr).toContain('missing'); }); } - it('runs no judge without --semantic or --judge', () => { + it('runs no semantic evaluation without --semantic', () => { expect(spawn('validate', 'ts').exitCode).toBe(0); }); }); diff --git a/tests/config.test.ts b/tests/config.test.ts index 96f1993a..6f0f9817 100644 --- a/tests/config.test.ts +++ b/tests/config.test.ts @@ -387,7 +387,7 @@ describe('loadConfig with includeSpacesFrom', () => { expect(mainConfig.spaces[0]!.name).toBe('main-space'); }); - it('accepts model providers, a top-level judge and a space judge override', () => { + it('accepts model providers, a top-level semantic provider and a space override', () => { const configPath = join(testDir, 'judge-config.json'); writeFileSync( configPath, @@ -396,14 +396,14 @@ describe('loadConfig with includeSpacesFrom', () => { jev: { api: 'openrouter-decisions' }, claude: { api: 'anthropic-messages', model: 'claude-haiku-5-5' }, }, - judge: { provider: 'jev' }, - spaces: [{ name: 's', path: '/p', judge: { provider: 'claude' } }], + semantic: { provider: 'jev' }, + spaces: [{ name: 's', path: '/p', semantic: { provider: 'claude' } }], }), ); setConfigPath(configPath); const config = loadConfig(); - expect(config.judge?.provider).toBe('jev'); + expect(config.semantic?.provider).toBe('jev'); expect(config.modelProviders?.claude?.model).toBe('claude-haiku-5-5'); - expect(config.spaces?.[0]?.judge?.provider).toBe('claude'); + expect(config.spaces?.[0]?.semantic?.provider).toBe('claude'); }); }); diff --git a/tests/fixtures/rule-severity/schema.json b/tests/fixtures/rule-severity/schema.json index f337517a..37fa6a33 100644 --- a/tests/fixtures/rule-severity/schema.json +++ b/tests/fixtures/rule-severity/schema.json @@ -11,14 +11,14 @@ "category": "best-practice", "description": "Best-practice rule that always fails (default severity: info)", "type": "goal", - "check": "false" + "expression": "false" }, { "id": "always-warning", "category": "coherence", "description": "Coherence rule that always fails (default severity: warning)", "type": "goal", - "check": "false" + "expression": "false" }, { "id": "not-blocked", @@ -26,7 +26,7 @@ "severity": "error", "description": "Goals must not be blocked (workflow rule overridden to error)", "type": "goal", - "check": "current.status != 'blocked'" + "expression": "current.status != 'blocked'" } ] }, diff --git a/tests/fixtures/schema-composition/_conflict-pack-a.json b/tests/fixtures/schema-composition/_conflict-pack-a.json index 8da24218..aa970d3b 100644 --- a/tests/fixtures/schema-composition/_conflict-pack-a.json +++ b/tests/fixtures/schema-composition/_conflict-pack-a.json @@ -10,8 +10,8 @@ "id": "active-outcome-count", "category": "workflow", "description": "Conflict pack A", - "scope": "global", - "check": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" + "global": true, + "expression": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" } ] }, diff --git a/tests/fixtures/schema-composition/_conflict-pack-b.json b/tests/fixtures/schema-composition/_conflict-pack-b.json index 737571e1..1e8d7842 100644 --- a/tests/fixtures/schema-composition/_conflict-pack-b.json +++ b/tests/fixtures/schema-composition/_conflict-pack-b.json @@ -7,8 +7,8 @@ "id": "active-outcome-count", "category": "workflow", "description": "Conflict pack B", - "scope": "global", - "check": "$count(nodes[resolvedType='outcome' and status='active']) = 1" + "global": true, + "expression": "$count(nodes[resolvedType='outcome' and status='active']) = 1" } ] }, diff --git a/tests/fixtures/schema-composition/_merge-leaf.json b/tests/fixtures/schema-composition/_merge-leaf.json index 1330178c..7055aa56 100644 --- a/tests/fixtures/schema-composition/_merge-leaf.json +++ b/tests/fixtures/schema-composition/_merge-leaf.json @@ -13,7 +13,7 @@ "id": "leaf-rule", "category": "coherence", "description": "Leaf coherence rule", - "check": "true" + "expression": "true" } ] }, diff --git a/tests/fixtures/schema-composition/_merge-pack-a.json b/tests/fixtures/schema-composition/_merge-pack-a.json index b9dedb10..e8674c63 100644 --- a/tests/fixtures/schema-composition/_merge-pack-a.json +++ b/tests/fixtures/schema-composition/_merge-pack-a.json @@ -11,7 +11,7 @@ "id": "pack-a-rule", "category": "validation", "description": "Pack A validation rule", - "check": "true" + "expression": "true" } ] }, diff --git a/tests/fixtures/schema-composition/_override-pack-a.json b/tests/fixtures/schema-composition/_override-pack-a.json index 1ce5c1f0..b4bd3399 100644 --- a/tests/fixtures/schema-composition/_override-pack-a.json +++ b/tests/fixtures/schema-composition/_override-pack-a.json @@ -10,8 +10,8 @@ "id": "active-outcome-count", "category": "workflow", "description": "Override pack A", - "scope": "global", - "check": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" + "global": true, + "expression": "$count(nodes[resolvedType='outcome' and status='active']) <= 1" } ] }, diff --git a/tests/fixtures/schema-composition/_override-pack-b.json b/tests/fixtures/schema-composition/_override-pack-b.json index f79348c6..300140d4 100644 --- a/tests/fixtures/schema-composition/_override-pack-b.json +++ b/tests/fixtures/schema-composition/_override-pack-b.json @@ -8,8 +8,8 @@ "override": true, "category": "workflow", "description": "Override pack B", - "scope": "global", - "check": "$count(nodes[resolvedType='outcome' and status='active']) = 1" + "global": true, + "expression": "$count(nodes[resolvedType='outcome' and status='active']) = 1" } ] }, diff --git a/tests/fixtures/schema-composition/merge-root.json b/tests/fixtures/schema-composition/merge-root.json index edabebdc..332db68c 100644 --- a/tests/fixtures/schema-composition/merge-root.json +++ b/tests/fixtures/schema-composition/merge-root.json @@ -14,8 +14,8 @@ "id": "root-rule", "category": "workflow", "description": "Root-level workflow rule", - "scope": "global", - "check": "true" + "global": true, + "expression": "true" } ] }, diff --git a/tests/fixtures/schema-metadata/invalid-metadata.json b/tests/fixtures/schema-metadata/invalid-metadata.json index 9fc8a024..b9dbff8b 100644 --- a/tests/fixtures/schema-metadata/invalid-metadata.json +++ b/tests/fixtures/schema-metadata/invalid-metadata.json @@ -7,7 +7,7 @@ "id": "invalid-category", "category": "not-a-category", "description": "Invalid category should fail schema validation", - "check": "true" + "expression": "true" } ] }, diff --git a/tests/fixtures/schema-metadata/invalid-rule-severity.json b/tests/fixtures/schema-metadata/invalid-rule-severity.json index 58a59f1d..b15f3e68 100644 --- a/tests/fixtures/schema-metadata/invalid-rule-severity.json +++ b/tests/fixtures/schema-metadata/invalid-rule-severity.json @@ -8,7 +8,7 @@ "category": "workflow", "severity": "critical", "description": "Invalid severity should fail schema validation", - "check": "true" + "expression": "true" } ] }, diff --git a/tests/fixtures/schema-metadata/invalid-type-reference.json b/tests/fixtures/schema-metadata/invalid-type-reference.json index 0aaec0b7..bfc7cf80 100644 --- a/tests/fixtures/schema-metadata/invalid-type-reference.json +++ b/tests/fixtures/schema-metadata/invalid-type-reference.json @@ -21,7 +21,7 @@ "category": "validation", "description": "Test rule", "type": "nonexistent", - "check": "true" + "expression": "true" } ] }, diff --git a/tests/helpers/temp-space.ts b/tests/helpers/temp-space.ts index 4cde0ae6..6a41dcb5 100644 --- a/tests/helpers/temp-space.ts +++ b/tests/helpers/temp-space.ts @@ -26,7 +26,7 @@ export interface TempSpace { */ export function makeTempSpace( rules: object[], - options: { frontmatter?: Record; judging?: Pick } = {}, + options: { frontmatter?: Record; judging?: Pick } = {}, ): TempSpace { const dir = mkdtempSync(join(tmpdir(), 'sctx-temp-space-')); const schemaPath = join(dir, 'schema.json'); diff --git a/tests/judge/factory.test.ts b/tests/judge/factory.test.ts index a9993ea0..a4cf9087 100644 --- a/tests/judge/factory.test.ts +++ b/tests/judge/factory.test.ts @@ -54,7 +54,9 @@ describe('createModelProvider', () => { throw new Error('stop'); }) as unknown as typeof fetch; try { - const judge = createJudge({ judge: { provider: 'openrouter-jev' } }, {}, undefined, { OPENROUTER_API_KEY: 'k' }); + const judge = createJudge({ semantic: { provider: 'openrouter-jev' } }, {}, undefined, { + OPENROUTER_API_KEY: 'k', + }); await judge .evaluate({ node: { @@ -109,7 +111,7 @@ describe('createModelProvider', () => { describe('createJudge', () => { const config = { modelProviders: { mine: { api: 'openrouter-decisions' }, claude: { api: 'anthropic-messages' } } as const, - judge: { provider: 'claude' }, + semantic: { provider: 'claude' }, }; const env = { OPENROUTER_API_KEY: 'r', ANTHROPIC_API_KEY: 'a', TYPESAFE_API_KEY: 't' }; @@ -118,27 +120,27 @@ describe('createJudge', () => { }); it("uses the space's judge over the top-level one, and a run's over both", () => { - const space = { judge: { provider: 'mine' } }; + const space = { semantic: { provider: 'mine' } }; expect(createJudge(config, space, undefined, env).name).toBe('jev'); expect(createJudge(config, space, 'claude', env).name).toBe('anthropic'); }); it('accepts a bundled provider as the judge, with the type defaults', () => { - expect(createJudge({}, { judge: { provider: 'openrouter-jev' } }, undefined, env).name).toBe('jev'); - expect(createJudge({ judge: { provider: 'anthropic' } }, {}, undefined, env).name).toBe('anthropic'); + expect(createJudge({}, { semantic: { provider: 'openrouter-jev' } }, undefined, env).name).toBe('jev'); + expect(createJudge({ semantic: { provider: 'anthropic' } }, {}, undefined, env).name).toBe('anthropic'); }); it('ships the pinned and env variants of each vendor bundle', () => { - expect(createJudge({}, { judge: { provider: 'anthropic-haiku-5-5' } }, undefined, env).name).toBe('anthropic'); - expect(createJudge({}, { judge: { provider: 'anthropic-env' } }, undefined, env).name).toBe('anthropic'); - expect(createJudge({}, { judge: { provider: 'openai-env' } }, undefined, env).name).toBe('openai'); - expect(createJudge({}, { judge: { provider: 'typesafe-jev' } }, undefined, env).name).toBe('jev'); + expect(createJudge({}, { semantic: { provider: 'anthropic-haiku-5-5' } }, undefined, env).name).toBe('anthropic'); + expect(createJudge({}, { semantic: { provider: 'anthropic-env' } }, undefined, env).name).toBe('anthropic'); + expect(createJudge({}, { semantic: { provider: 'openai-env' } }, undefined, env).name).toBe('openai'); + expect(createJudge({}, { semantic: { provider: 'typesafe-jev' } }, undefined, env).name).toBe('jev'); }); it('lets a configured model provider replace a bundled provider of the same name, keeping type defaults for fields it leaves out', () => { const augmented = { modelProviders: { anthropic: { api: 'anthropic-messages', model: 'claude-sonnet-5-5' } } as const, - judge: { provider: 'anthropic' }, + semantic: { provider: 'anthropic' }, }; expect(createJudge(augmented, {}, undefined, env).name).toBe('anthropic'); }); @@ -147,7 +149,7 @@ describe('createJudge', () => { const keys = { ANTHROPIC_API_KEY: 'a', OPENAI_API_KEY: 'o', OPENROUTER_API_KEY: 'r' }; expect(() => createJudge({}, {}, undefined, keys)).toThrow(JudgeConfigError); expect(() => createJudge({ modelProviders: config.modelProviders }, {}, undefined, keys)).toThrow( - 'No judge configured', + '--semantic needs a model provider', ); }); @@ -161,25 +163,25 @@ describe('createJudge', () => { }); it('does not bundle the test-only fake provider', () => { - expect(() => createJudge({}, { judge: { provider: 'fake' } }, undefined, env)).toThrow('not configured'); + expect(() => createJudge({}, { semantic: { provider: 'fake' } }, undefined, env)).toThrow('not configured'); }); }); describe('judgeForRun (CLI)', () => { const config = { modelProviders: { f: { api: 'fake' }, g: { api: 'fake' } } as const, - judge: { provider: 'f' }, + semantic: { provider: 'f' }, }; - it('gives no judge unless the run asks with --semantic or --judge', () => { - expect(judgeForRun(config, {}, {})).toBeUndefined(); - expect(judgeForRun({}, {}, {})).toBeUndefined(); // no judge config needed when not asked - expect(judgeForRun(config, {}, { semantic: true }) instanceof FakeJudgeProvider).toBe(true); - expect(judgeForRun(config, {}, { judge: 'g' }) instanceof FakeJudgeProvider).toBe(true); + it('gives no provider unless the run asks with --semantic', () => { + expect(judgeForRun(config, {})).toBeUndefined(); + expect(judgeForRun({}, {})).toBeUndefined(); // no judge config needed when not asked + expect(judgeForRun(config, {}, true) instanceof FakeJudgeProvider).toBe(true); + expect(judgeForRun(config, {}, 'g') instanceof FakeJudgeProvider).toBe(true); }); it('fails when asked and no judge is configured, or the named provider is missing', () => { - expect(() => judgeForRun({}, {}, { semantic: true })).toThrow('No judge configured'); - expect(() => judgeForRun(config, {}, { judge: 'nope' })).toThrow('Model provider "nope" is not configured'); + expect(() => judgeForRun({}, {}, true)).toThrow('--semantic needs a model provider'); + expect(() => judgeForRun(config, {}, 'nope')).toThrow('Model provider "nope" is not configured'); }); }); diff --git a/tests/judge/judge-node.test.ts b/tests/judge/judge-node.test.ts index d8ac8615..cbc0f082 100644 --- a/tests/judge/judge-node.test.ts +++ b/tests/judge/judge-node.test.ts @@ -20,9 +20,7 @@ describe('judgeNode', () => { id: passFailRule.id, description: passFailRule.description, category: passFailRule.category, - mode: 'semantic', - severity: { fail: 'error' }, - verdicts: passFailRule.verdicts, + semantic: { verdicts: passFailRule.verdicts, severity: { fail: 'error' } }, } as unknown as Rule; const judgements: NodeJudgement[] = await judgeNode( canned({ 'customer-voice': { verdict: 'unexpected', evidence: 'hmm' } }), @@ -42,9 +40,7 @@ describe('judgeNodes', () => { id: passFailRule.id, description: passFailRule.description, category: passFailRule.category, - mode: 'semantic', - severity: { fail: 'error' }, - verdicts: passFailRule.verdicts, + semantic: { verdicts: passFailRule.verdicts, severity: { fail: 'error' } }, } as unknown as Rule; const other = { ...node, title: 'Other', label: 'other.md' } as SpaceNode; diff --git a/tests/judge/verdicts.test.ts b/tests/judge/verdicts.test.ts index 15b62e01..b43eb4e3 100644 --- a/tests/judge/verdicts.test.ts +++ b/tests/judge/verdicts.test.ts @@ -5,7 +5,7 @@ import { resolveSemanticRule } from '../../src/judge/verdicts'; import type { Rule } from '../../src/types'; import { graded, node, passFailRule } from './helpers'; -const base: Rule = { id: 'r', category: 'best-practice', description: 'd', mode: 'semantic' }; +const base: Rule = { id: 'r', category: 'best-practice', description: 'd', semantic: {} }; describe('resolveSemanticRule', () => { it('defaults to pass/fail with fail at the rule severity', () => { @@ -13,13 +13,28 @@ describe('resolveSemanticRule', () => { expect(Object.keys(resolved.spec.verdicts)).toEqual(['pass', 'fail']); expect(resolved.severities).toEqual({ fail: 'info' }); // best-practice default expect(resolveSemanticRule({ ...base, severity: 'error' }).severities).toEqual({ fail: 'error' }); + expect(resolveSemanticRule({ ...base, semantic: { severity: { 'missing-context': 'info' } } }).severities).toEqual({ + fail: 'info', + 'missing-context': 'info', + }); + }); + + it('treats `semantic: true` like `semantic: {}`', () => { + expect(resolveSemanticRule({ ...base, semantic: true }).severities).toEqual({ fail: 'info' }); + }); + + it('uses the semantic rubric, falling back to nothing so the prompt can use the description', () => { + expect(resolveSemanticRule({ ...base, semantic: { rubric: 'R' } }).spec.rubric).toBe('R'); + expect(resolveSemanticRule(base).spec.rubric).toBeUndefined(); }); it('maps declared verdicts to severities; unmapped and ignored verdicts pass', () => { const resolved = resolveSemanticRule({ ...base, - verdicts: { ok: 'O', minor: 'M', major: 'J', skip: 'S' }, - severity: { minor: 'warning', major: 'error', skip: 'ignore' }, + semantic: { + verdicts: { ok: 'O', minor: 'M', major: 'J', skip: 'S' }, + severity: { minor: 'warning', major: 'error', skip: 'ignore' }, + }, }); expect(resolved.severities).toEqual({ minor: 'warning', major: 'error' }); }); @@ -27,25 +42,30 @@ describe('resolveSemanticRule', () => { it('splits a verdict into the judge description and user guidance', () => { const resolved = resolveSemanticRule({ ...base, - verdicts: { - ok: 'Fine', - vague: { description: 'Too vague to act on', guidance: 'Name the customer and the pain.' }, + semantic: { + verdicts: { + ok: 'Fine', + vague: { description: 'Too vague to act on', guidance: 'Name the customer and the pain.' }, + }, + severity: { vague: 'warning' }, }, - severity: { vague: 'warning' }, }); expect(resolved.spec.verdicts).toEqual({ ok: 'Fine', vague: 'Too vague to act on' }); expect(resolved.guidance).toEqual({ vague: 'Name the customer and the pain.' }); }); it('rejects custom verdicts without a severity map, unknown verdicts, and sets with no passing verdict', () => { - expect(() => resolveSemanticRule({ ...base, verdicts: { a: 'A', b: 'B' } })).toThrow( + expect(() => resolveSemanticRule({ ...base, semantic: { verdicts: { a: 'A', b: 'B' } } })).toThrow( 'must map the failing verdicts', ); - expect(() => resolveSemanticRule({ ...base, verdicts: { a: 'A', b: 'B' }, severity: { z: 'error' } })).toThrow( - 'not in its verdicts', - ); expect(() => - resolveSemanticRule({ ...base, verdicts: { a: 'A', b: 'B' }, severity: { a: 'error', b: 'warning' } }), + resolveSemanticRule({ ...base, semantic: { verdicts: { a: 'A', b: 'B' }, severity: { z: 'error' } } }), + ).toThrow('not in its verdicts'); + expect(() => + resolveSemanticRule({ + ...base, + semantic: { verdicts: { a: 'A', b: 'B' }, severity: { a: 'error', b: 'warning' } }, + }), ).toThrow('no passing verdict'); }); }); diff --git a/tests/schema/gating.test.ts b/tests/schema/gating.test.ts index 794ce6aa..4d8d29f6 100644 --- a/tests/schema/gating.test.ts +++ b/tests/schema/gating.test.ts @@ -46,20 +46,20 @@ describe('evaluateRulesWithGating', () => { id: 'always-fail', category: 'validation', description: 'always fails', - check: 'false', + expression: 'false', }; const nonBlockingFailRule: Rule = { id: 'non-gating', category: 'validation', description: 'fails but does not block', - check: 'false', + expression: 'false', gatesDescendants: false, }; const passRule: Rule = { id: 'always-pass', category: 'validation', description: 'always passes', - check: 'true', + expression: 'true', }; function buildTree() { @@ -125,8 +125,8 @@ describe('evaluateRulesWithGating', () => { }); it('gatesDescendants defaults from severity: error gates, warning and info do not, explicit value overrides', async () => { - const advisory: Rule = { id: 'advisory', category: 'best-practice', description: 'info', check: 'false' }; - const warning: Rule = { id: 'warn', category: 'coherence', description: 'warning', check: 'false' }; + const advisory: Rule = { id: 'advisory', category: 'best-practice', description: 'info', expression: 'false' }; + const warning: Rule = { id: 'warn', category: 'coherence', description: 'warning', expression: 'false' }; expect(ruleGatesDescendants(alwaysFailRule)).toBe(true); expect(ruleGatesDescendants(advisory)).toBe(false); expect(ruleGatesDescendants(warning)).toBe(false); @@ -177,8 +177,8 @@ describe('evaluateRulesWithGating', () => { id: 'global-fail', category: 'workflow', description: 'fails globally', - check: 'false', - scope: 'global', + expression: 'false', + global: true, }; const seeds: GatingSeed[] = [{ file: goal.label, title: goal.title, reason: 'schema' }]; @@ -192,9 +192,7 @@ describe('evaluateRulesWithGating', () => { id: 'sem', category: 'best-practice', description: 'semantic', - mode: 'semantic', - severity: { fail: 'error' }, - verdicts: { pass: 'ok', fail: 'not ok' }, + semantic: { verdicts: { pass: 'ok', fail: 'not ok' }, severity: { fail: 'error' } }, } as unknown as Rule; it('judges each depth in one evaluateNodes call and still gates children of a failing node', async () => { diff --git a/tests/schema/rule-filter.test.ts b/tests/schema/rule-filter.test.ts index 37054f98..74459cc1 100644 --- a/tests/schema/rule-filter.test.ts +++ b/tests/schema/rule-filter.test.ts @@ -3,16 +3,16 @@ import { filterRules, parseRuleFilter } from '../../src/schema/rule-filter'; import type { Rule } from '../../src/types'; const rules: Rule[] = [ - { id: 'a', category: 'validation', description: 'a', check: 'true', type: 'goal' }, - { id: 'b', category: 'best-practice', description: 'b', mode: 'semantic', type: 'opportunity' }, - { id: 'c', category: 'coherence', description: 'c', check: 'true', severity: 'error' }, + { id: 'a', category: 'validation', description: 'a', expression: 'true', type: 'goal' }, + { id: 'b', category: 'best-practice', description: 'b', semantic: {}, type: 'opportunity' }, + { id: 'c', category: 'coherence', description: 'c', expression: 'true', severity: 'error' }, ]; const ids = (r: Rule[]) => r.map((x) => x.id); describe('rule filter', () => { it('parses repeated specs and merges values per dimension', () => { - expect(parseRuleFilter(['mode=semantic', 'category=coherence,validation', 'category=workflow'])).toEqual({ - mode: ['semantic'], + expect(parseRuleFilter(['type=goal', 'category=coherence,validation', 'category=workflow'])).toEqual({ + type: ['goal'], category: ['coherence', 'validation', 'workflow'], }); }); @@ -20,12 +20,11 @@ describe('rule filter', () => { it('rejects malformed specs and unknown dimensions', () => { expect(() => parseRuleFilter(['semantic'])).toThrow('Invalid rule filter'); expect(() => parseRuleFilter(['colour=red'])).toThrow('Invalid rule filter'); - expect(() => parseRuleFilter(['mode='])).toThrow('Invalid rule filter'); + expect(() => parseRuleFilter(['mode=semantic'])).toThrow('Invalid rule filter'); + expect(() => parseRuleFilter(['type='])).toThrow('Invalid rule filter'); }); - it('filters by mode (defaulting to expression), severity (effective), category, type and id', () => { - expect(ids(filterRules(rules, { mode: ['semantic'] }))).toEqual(['b']); - expect(ids(filterRules(rules, { mode: ['expression'] }))).toEqual(['a', 'c']); + it('filters by severity (effective), category, type and id', () => { expect(ids(filterRules(rules, { severity: ['error'] }))).toEqual(['a', 'c']); expect(ids(filterRules(rules, { category: ['coherence'] }))).toEqual(['c']); expect(ids(filterRules(rules, { type: ['goal', 'opportunity'] }))).toEqual(['a', 'b']); @@ -33,7 +32,7 @@ describe('rule filter', () => { }); it('ANDs dimensions and treats an undefined filter as everything', () => { - expect(ids(filterRules(rules, { severity: ['error'], mode: ['semantic'] }))).toEqual([]); + expect(ids(filterRules(rules, { severity: ['error'], type: ['opportunity'] }))).toEqual([]); expect(filterRules(rules, undefined)).toBe(rules); }); }); diff --git a/tests/schema/schema-composition.test.ts b/tests/schema/schema-composition.test.ts index b267b00c..dd81288b 100644 --- a/tests/schema/schema-composition.test.ts +++ b/tests/schema/schema-composition.test.ts @@ -68,7 +68,7 @@ describe('schema composition metadata', () => { expect(metadata.rules).toHaveLength(1); expect(metadata.rules?.[0]).toMatchObject({ id: 'active-outcome-count', - check: "$count(nodes[resolvedType='outcome' and status='active']) = 1", + expression: "$count(nodes[resolvedType='outcome' and status='active']) = 1", }); expect((metadata.rules?.[0] as Record | undefined)?.override).toBeUndefined(); }); diff --git a/tests/schema/semantic-rule-contract.test.ts b/tests/schema/semantic-rule-contract.test.ts index 6d923f72..526d84aa 100644 --- a/tests/schema/semantic-rule-contract.test.ts +++ b/tests/schema/semantic-rule-contract.test.ts @@ -8,7 +8,7 @@ const META = 'https://raw.githubusercontent.com/mindsocket/structured-context/main/schemas/generated/latest/_structured_context_schema_meta.json'; const base = { id: 'r', category: 'validation', description: 'd' }; -describe('rule contract: check and semantic mode', () => { +describe('rule contract: expression and semantic', () => { let dir: string; beforeAll(() => { dir = mkdtempSync(join(tmpdir(), 'sctx-rule-contract-')); @@ -29,51 +29,77 @@ describe('rule contract: check and semantic mode', () => { return loadMetadata(path); }; - it('requires check on expression rules', () => { - expect(() => load('no-check', base)).toThrow(); + it('requires an expression or a semantic part', () => { + expect(() => load('neither', base)).toThrow(); }); - it('accepts a semantic rule without check', () => { - expect(load('semantic-ok', { ...base, mode: 'semantic' }).rules).toHaveLength(1); + it('accepts `semantic: {}` and `semantic: true` as the all-defaults form', () => { + expect(load('semantic-ok', { ...base, semantic: {} }).rules).toHaveLength(1); + expect(load('semantic-true', { ...base, semantic: true }).rules).toHaveLength(1); + expect(() => load('semantic-false', { ...base, semantic: false })).toThrow(); }); - it('accepts a semantic rule that declares a check precondition', () => { - expect(load('semantic-check', { ...base, mode: 'semantic', check: 'true' }).rules).toHaveLength(1); + it('accepts a rule with both an expression and a semantic part', () => { + expect(load('both', { ...base, semantic: {}, expression: 'true' }).rules).toHaveLength(1); }); - it('rejects a global semantic rule', () => { - expect(() => load('semantic-global', { ...base, mode: 'semantic', scope: 'global' })).toThrow(); + it('rejects unknown rule fields', () => { + expect(() => load('unknown-field', { ...base, expression: 'true', scope: 'global' })).toThrow(); + }); + + it('rejects a global semantic rule, and `when` or `type` on a global rule', () => { + expect(() => load('semantic-global', { ...base, semantic: {}, global: true })).toThrow(); + expect(() => load('both-global', { ...base, semantic: {}, expression: 'true', global: true })).toThrow(); + expect(() => load('global-when', { ...base, expression: 'true', global: true, when: 'true' })).toThrow(); + expect(() => load('global-type', { ...base, expression: 'true', global: true, type: 'goal' })).toThrow(); + expect(load('global-ok', { ...base, expression: 'true', global: true }).rules).toHaveLength(1); + expect(load('global-false', { ...base, expression: 'true', global: false, type: 'x' }).rules).toHaveLength(1); }); it('accepts a verdict map with guidance, a severity map (including ignore) and a when', () => { const rule = { ...base, - mode: 'semantic', when: "current.status = 'ready'", - verdicts: { ok: 'Fine', vague: { description: 'Too vague', guidance: 'Be specific.' } }, - severity: { vague: 'warning' }, + semantic: { + rubric: 'Is it specific?', + verdicts: { ok: 'Fine', vague: { description: 'Too vague', guidance: 'Be specific.' } }, + severity: { vague: 'warning' }, + }, }; expect(load('shape-ok', rule).rules).toHaveLength(1); - expect(load('ignore-ok', { ...rule, severity: { vague: 'ignore' } }).rules).toHaveLength(1); + expect( + load('ignore-ok', { ...rule, semantic: { ...rule.semantic, severity: { vague: 'ignore' } } }).rules, + ).toHaveLength(1); }); it('requires a severity map when custom verdicts are declared', () => { const verdicts = { ok: 'Fine', bad: 'Bad' }; - expect(() => load('verdicts-no-severity', { ...base, mode: 'semantic', verdicts })).toThrow(); - expect(() => load('verdicts-scalar', { ...base, mode: 'semantic', verdicts, severity: 'error' })).toThrow(); + expect(() => load('verdicts-no-severity', { ...base, semantic: { verdicts } })).toThrow(); + expect(() => load('verdicts-scalar', { ...base, semantic: { verdicts, severity: 'error' } })).toThrow(); + }); + + it('takes a verdict map alongside default verdicts only for verdicts other than `fail`', () => { + expect(load('reserved-ok', { ...base, semantic: { severity: { 'missing-context': 'info' } } }).rules).toHaveLength( + 1, + ); + expect(() => load('fail-in-map', { ...base, semantic: { severity: { fail: 'error' } } })).toThrow(); + expect(load('semantic-true-severity', { ...base, semantic: true, severity: 'error' }).rules).toHaveLength(1); }); - it('keeps semantic-only fields off expression rules', () => { - const expr = { ...base, check: 'true' }; + it('rejects a top-level severity on a semantic-only rule with custom verdicts', () => { + const semantic = { verdicts: { ok: 'Fine', bad: 'Bad' }, severity: { bad: 'error' } }; + expect(() => load('custom-top-severity', { ...base, semantic, severity: 'error' })).toThrow(); + expect( + load('custom-with-expression', { ...base, semantic, expression: 'true', severity: 'error' }).rules, + ).toHaveLength(1); + }); + + it('keeps a verdict map off the top-level severity and rules choose no provider', () => { + const expr = { ...base, expression: 'true' }; expect(() => load('expr-map', { ...expr, severity: { fail: 'error' } })).toThrow(); - // Rules do not choose a judge: that is config, not schema. - expect(() => load('expr-judge', { ...expr, judge: 'llm' })).toThrow(); - expect(() => load('semantic-judge', { ...base, mode: 'semantic', judge: 'llm' })).toThrow(); + expect(() => load('expr-provider', { ...expr, provider: 'llm' })).toThrow(); + expect(() => load('semantic-provider', { ...base, semantic: { provider: 'llm' } })).toThrow(); expect(() => load('expr-verdicts', { ...expr, verdicts: { a: 'A', b: 'B' } })).toThrow(); expect(load('expr-when', { ...expr, when: 'true' }).rules).toHaveLength(1); }); - - it('rejects `when` on a global rule', () => { - expect(() => load('global-when', { ...base, check: 'true', scope: 'global', when: 'true' })).toThrow(); - }); }); diff --git a/tests/schema/validate-rules-semantic.test.ts b/tests/schema/validate-rules-semantic.test.ts index 22753984..1505ef94 100644 --- a/tests/schema/validate-rules-semantic.test.ts +++ b/tests/schema/validate-rules-semantic.test.ts @@ -55,15 +55,16 @@ describe('validateRules - semantic evaluation', () => { id: 'active-opportunity-count', category: 'workflow', description: 'At most 2 active opportunities', - check: '$count(nodes[resolvedType="opportunity" and status="active"]) <= 2', + expression: '$count(nodes[resolvedType="opportunity" and status="active"]) <= 2', }, { id: 'opportunity-customer-voice', category: 'best-practice', - mode: 'semantic', type: 'opportunity', description: 'Opportunity must be framed in customer voice, not a solution or business goal', - rubric: 'Must describe a customer pain point or need in customer language, not a solution to build.', + semantic: { + rubric: 'Must describe a customer pain point or need in customer language, not a solution to build.', + }, }, ]; @@ -124,13 +125,13 @@ describe('validateRules - semantic evaluation', () => { id: 'require-interviews-source', category: 'validation', type: 'opportunity', - check: '$contains(current.source, "interview") = true', + expression: '$contains(current.source, "interview") = true', description: 'Source must mention interview', }, { id: 'opportunity-customer-voice', category: 'best-practice', - mode: 'semantic', + semantic: {}, type: 'opportunity', description: 'Must be in customer voice', }, @@ -158,9 +159,9 @@ describe('validateRules - semantic evaluation', () => { { id: 'opportunity-customer-voice', category: 'best-practice', - mode: 'semantic', + semantic: {}, type: 'opportunity', - check: 'current.source = "User interview with Jane"', + expression: 'current.source = "User interview with Jane"', description: 'Must be in customer voice', }, ]; @@ -168,18 +169,18 @@ describe('validateRules - semantic evaluation', () => { const violations = await validateRules(sampleNodes, checked, { judge: fake }); - // bad-opp.md fails the check: reported at the rule severity, never sent to the judge. + // bad-opp.md fails the expression: reported at the rule severity, never sent to the judge. expect(fake.callHistory.map((c) => c.node.label)).toEqual(['valid-opp.md']); expect(violations).toEqual([ - expect.objectContaining({ file: 'valid-opp.md', evidence: 'Not customer voice' }), { file: 'bad-opp.md', ruleId: 'opportunity-customer-voice', category: 'best-practice', severity: 'info', description: 'Must be in customer voice', - evidence: 'expression check failed: current.source = "User interview with Jane"', + evidence: 'expression failed: current.source = "User interview with Jane"', }, + expect.objectContaining({ file: 'valid-opp.md', evidence: 'Not customer voice' }), ]); }); @@ -187,10 +188,10 @@ describe('validateRules - semantic evaluation', () => { const checked: Rule = { id: 'opportunity-source-interview', category: 'coherence', - mode: 'semantic', + semantic: {}, type: 'opportunity', description: 'Opportunity comes from an interview', - check: '$contains(current.source, "interview")', + expression: '$contains(current.source, "interview")', }; it('runs the check without a judge: failing reports, passing is silent', async () => { @@ -200,14 +201,16 @@ describe('validateRules - semantic evaluation', () => { file: 'bad-opp.md', ruleId: checked.id, severity: 'warning', - evidence: 'expression check failed: $contains(current.source, "interview")', + evidence: 'expression failed: $contains(current.source, "interview")', }), ]); }); it('reports an unevaluable check as an error', async () => { const fake = new FakeJudgeProvider(); - const violations = await validateRules(sampleNodes, [{ ...checked, check: '$notAFunction(' }], { judge: fake }); + const violations = await validateRules(sampleNodes, [{ ...checked, expression: '$notAFunction(' }], { + judge: fake, + }); expect(fake.callHistory).toHaveLength(0); expect(violations.map((v) => v.severity)).toEqual(['error', 'error']); expect(violations[0]?.description).toContain('Rule error'); diff --git a/tests/schema/validate-rules.test.ts b/tests/schema/validate-rules.test.ts index 9c60aba6..7813cfec 100644 --- a/tests/schema/validate-rules.test.ts +++ b/tests/schema/validate-rules.test.ts @@ -93,21 +93,22 @@ describe('validate-rules', () => { category: 'validation', description: 'Parent must be an opportunity', type: 'solution', - check: '$exists(parent) = false or $exists(nodes[title=$$.current.parent and resolvedType="opportunity"])', + expression: + '$exists(parent) = false or $exists(nodes[title=$$.current.parent and resolvedType="opportunity"])', }, { id: 'experiment-parent-type', category: 'validation', description: 'Parent must be a solution', type: 'experiment', - check: '$exists(parent) and $exists(nodes[title=$$.current.parent and resolvedType="solution"])', + expression: '$exists(parent) and $exists(nodes[title=$$.current.parent and resolvedType="solution"])', }, { id: 'outcome-no-parent', category: 'validation', description: 'Outcome nodes should not have a parent', type: 'outcome', - check: '$exists(parent) = false', + expression: '$exists(parent) = false', }, ]; @@ -173,7 +174,7 @@ describe('validate-rules', () => { category: 'best-practice', description: 'Explore multiple candidate solutions for the target opportunity', type: 'opportunity', - check: '$count(nodes[resolvedParentTitle=$$.current.title and resolvedType="solution"]) >= 3', + expression: '$count(nodes[resolvedParentTitle=$$.current.title and resolvedType="solution"]) >= 3', }, ]; @@ -259,14 +260,14 @@ describe('validate-rules', () => { id: 'active-outcome-count', category: 'workflow', description: 'Only one outcome should be active at a time', - scope: 'global', - check: '$count(nodes[resolvedType="outcome" and status="active"]) <= 1', + global: true, + expression: '$count(nodes[resolvedType="outcome" and status="active"]) <= 1', }, { id: 'active-node-parent-active', category: 'workflow', description: "An active node's parent should also be active", - check: "current.status != 'active' or $exists(parent) = false or parent.status = 'active'", + expression: "current.status != 'active' or $exists(parent) = false or parent.status = 'active'", }, ]; @@ -389,14 +390,15 @@ describe('validate-rules', () => { category: 'validation', description: 'Parent must be an opportunity', type: 'solution', - check: '$exists(parent) = false or $exists(nodes[title=$$.current.parent and resolvedType="opportunity"])', + expression: + '$exists(parent) = false or $exists(nodes[title=$$.current.parent and resolvedType="opportunity"])', }, { id: 'active-outcome-count', category: 'workflow', description: 'Only one outcome should be active at a time', - scope: 'global', - check: '$count(nodes[resolvedType="outcome" and status="active"]) <= 1', + global: true, + expression: '$count(nodes[resolvedType="outcome" and status="active"]) <= 1', }, ]; @@ -460,7 +462,7 @@ describe('validate-rules', () => { id: 'always-fails', category: 'workflow', description: 'Always fails', - check: 'false', + expression: 'false', ...overrides, }); @@ -486,7 +488,7 @@ describe('validate-rules', () => { [node], [ failingRule({ id: 'node-default', category: 'coherence' }), - failingRule({ id: 'global-override', scope: 'global', severity: 'error' }), + failingRule({ id: 'global-override', global: true, severity: 'error' }), ], ); expect(violations.map((v) => [v.ruleId, v.severity])).toEqual([ diff --git a/tests/validate/semantic-rule-shape.test.ts b/tests/validate/semantic-rule-shape.test.ts index bee6abab..111c2f03 100644 --- a/tests/validate/semantic-rule-shape.test.ts +++ b/tests/validate/semantic-rule-shape.test.ts @@ -14,16 +14,17 @@ afterAll(() => { const stepRule = { id: 'step-fit', type: 'step', - mode: 'semantic', category: 'coherence', description: 'Step fits its task', - verdicts: { - fits: 'The step fits', - minor: { description: 'Slightly off', guidance: 'Tighten the wording.' }, - major: { description: 'Unrelated', guidance: 'Move it under a better task.' }, - unsure: 'Cannot tell', + semantic: { + verdicts: { + fits: 'The step fits', + minor: { description: 'Slightly off', guidance: 'Tighten the wording.' }, + major: { description: 'Unrelated', guidance: 'Move it under a better task.' }, + unsure: 'Cannot tell', + }, + severity: { minor: 'warning', major: 'error', unsure: 'ignore' }, }, - severity: { minor: 'warning', major: 'error', unsure: 'ignore' }, }; function setup(rules: object[], options?: Parameters[1]) { @@ -107,7 +108,7 @@ describe('rule `when` precondition', () => { id: 'needs-owner', category: 'validation', description: 'owner', - check: '$exists(current.owner)', + expression: '$exists(current.owner)', when, }; const { context } = setup([rule], { frontmatter: { T: 'status: ready', S: 'status: draft' } }); @@ -147,14 +148,14 @@ describe('judge option', () => { }); describe('rule expression errors', () => { - const broken = { id: 'broken', category: 'validation', description: 'Broken check', check: '$notAFunction(' }; + const broken = { id: 'broken', category: 'validation', description: 'Broken check', expression: '$notAFunction(' }; const brokenWhen = { id: 'bad-when', category: 'coherence', description: 'Task rule', type: 'task', when: '1 +', - check: 'true', + expression: 'true', }; it('reports a failing check or when as an error violation instead of passing silently', async () => { @@ -172,10 +173,10 @@ describe('semantic rule check precondition', () => { const gated = { id: 'step-fit', type: 'step', - mode: 'semantic', + semantic: {}, category: 'coherence', description: 'Step fits its task', - check: '$exists(current.owner)', + expression: '$exists(current.owner)', gatesDescendants: true, }; const judgedLabels = (judge: FakeJudgeProvider) => judge.callHistory.map((c) => c.node.label); @@ -200,18 +201,24 @@ describe('semantic rule check precondition', () => { category: 'coherence', severity: 'warning', description: 'Step fits its task', - evidence: 'expression check failed: $exists(current.owner)', + evidence: 'expression failed: $exists(current.owner)', }, ]); }); - it('reports a failing check at the most severe mapped severity', async () => { - const { context } = setup([{ ...stepRule, check: '$exists(current.owner)' }]); - const judge = new FakeJudgeProvider({ 'step-fit': { verdict: 'minor' } }); - const result = await validateSpace(context(), { judge }); - expect(judge.callHistory).toHaveLength(0); - expect(result.ruleViolations).toEqual([expect.objectContaining({ ruleId: 'step-fit', severity: 'error' })]); - expect(result.ruleViolations[0]?.verdict).toBeUndefined(); + it('reports a failing expression at the top-level severity, not a verdict severity', async () => { + const rule = { ...stepRule, expression: '$exists(current.owner)' }; + const run = async (r: object) => { + const { context } = setup([r]); + const judge = new FakeJudgeProvider({ 'step-fit': { verdict: 'minor' } }); + const result = await validateSpace(context(), { judge }); + expect(judge.callHistory).toHaveLength(0); + return result.ruleViolations; + }; + const byCategory = await run(rule); + expect(byCategory).toEqual([expect.objectContaining({ ruleId: 'step-fit', severity: 'warning' })]); + expect(byCategory[0]?.verdict).toBeUndefined(); + expect(await run({ ...rule, severity: 'error' })).toEqual([expect.objectContaining({ severity: 'error' })]); }); it('gates descendants of a failing check like an expression rule: by severity, unless overridden', async () => { @@ -266,7 +273,7 @@ describe('semantic rule check precondition', () => { expect.objectContaining({ ruleId: 'step-fit', file: 'S.md', - evidence: 'expression check failed: $exists(current.owner)', + evidence: 'expression failed: $exists(current.owner)', }), ]); }); @@ -277,25 +284,29 @@ describe('semantic rule check precondition', () => { expect(result.ruleViolations).toEqual([]); }); - it('still skips a semantic rule that has no check', async () => { + it('still skips a semantic-only rule', async () => { const { context } = setup([stepRule]); expect((await validateSpace(context(), {})).ruleViolations).toEqual([]); }); - it('honours --rule-filter mode=semantic', async () => { + it('honours --rule-filter', async () => { const { context } = setup([gated]); - const kept = await validateSpace(context(), { ruleFilter: { mode: ['semantic'] } }); + const kept = await validateSpace(context(), { ruleFilter: { id: ['step-fit'] } }); expect(kept.ruleViolations).toHaveLength(1); - const dropped = await validateSpace(context(), { ruleFilter: { mode: ['expression'] } }); + const dropped = await validateSpace(context(), { ruleFilter: { id: ['other'] } }); expect(dropped.ruleViolations).toEqual([]); }); }); - it('reports a failing check with an all-ignore severity map at info', async () => { - const rule = { ...stepRule, severity: { minor: 'ignore', major: 'ignore' }, check: '$exists(current.owner)' }; + it('reports a failing expression the same way when every verdict is ignored', async () => { + const rule = { + ...stepRule, + semantic: { ...stepRule.semantic, severity: { minor: 'ignore', major: 'ignore' } }, + expression: '$exists(current.owner)', + }; const { context } = setup([rule]); const result = await validateSpace(context(), {}); - expect(result.ruleViolations).toEqual([expect.objectContaining({ ruleId: 'step-fit', severity: 'info' })]); + expect(result.ruleViolations).toEqual([expect.objectContaining({ ruleId: 'step-fit', severity: 'warning' })]); }); it('evaluates when before check', async () => { @@ -307,7 +318,7 @@ describe('semantic rule check precondition', () => { }); it('reports a check that cannot be evaluated as an error without judging', async () => { - const { context } = setup([{ ...gated, check: '$notAFunction(' }]); + const { context } = setup([{ ...gated, expression: '$notAFunction(' }]); const judge = new FakeJudgeProvider(); const result = await validateSpace(context(), { judge }); expect(judge.callHistory).toHaveLength(0); diff --git a/tests/validate/semantic-validation.test.ts b/tests/validate/semantic-validation.test.ts index 845a47c6..e9f14979 100644 --- a/tests/validate/semantic-validation.test.ts +++ b/tests/validate/semantic-validation.test.ts @@ -133,7 +133,7 @@ describe('validateFile judges only the target and its gate-capable ancestor rule const rule = (id: string, type: string, severity: string) => ({ id, type, - mode: 'semantic', + semantic: {}, category: 'validation', severity, description: id, @@ -209,21 +209,21 @@ describe('Semantic rules: verdicts, batching and rule filter', () => { const fake = new FakeJudgeProvider({ error: 'rate limited' }); const result = await validateSpace(context, { judge: fake }); const violation = result.ruleViolations.find((v) => v.ruleId === 'opportunity-customer-voice'); - expect(violation?.description).toContain('Judge error: rate limited'); + expect(violation?.description).toContain('Semantic evaluation error: rate limited'); expect(result.coverageGates.filter((g) => g.ruleId === 'opportunity-customer-voice')).toEqual([]); }); - it('--rule-filter narrows evaluation: semantic-only runs no expression rules, expression-only runs no judge', async () => { + it('--rule-filter narrows evaluation: filtering to the semantic rule runs no expression rules, filtering it out runs no judge', async () => { const semanticOnly = await validateSpace(context, { judge: new FakeJudgeProvider({ pass: false }), - ruleFilter: { mode: ['semantic'] }, + ruleFilter: { id: ['opportunity-customer-voice'] }, }); expect(semanticOnly.ruleViolations.every((v) => v.ruleId === 'opportunity-customer-voice')).toBe(true); const fake = new FakeJudgeProvider(); const expressionOnly = await validateSpace(context, { judge: fake, - ruleFilter: { mode: ['expression'] }, + ruleFilter: { category: ['workflow'] }, }); expect(fake.callHistory).toHaveLength(0); expect(expressionOnly.ruleViolations.some((v) => v.verdict !== undefined)).toBe(false); diff --git a/tests/validate/validate-views.test.ts b/tests/validate/validate-views.test.ts index 94fbe02b..6220038a 100644 --- a/tests/validate/validate-views.test.ts +++ b/tests/validate/validate-views.test.ts @@ -80,7 +80,7 @@ describe('judging a node that fails an expression rule', () => { { id: 'always-fails', type: 'item', - check: 'false', + expression: 'false', category: 'validation', severity: 'error', description: 'always fails', @@ -88,7 +88,7 @@ describe('judging a node that fails an expression rule', () => { { id: 'sound', type: 'item', - mode: 'semantic', + semantic: {}, category: 'validation', severity: 'warning', description: 'sound',