Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,9 +169,9 @@ See [docs/rules.md](rules.md) for the rules reference, including JSONata express

### Semantic rules

A **semantic rule** (`mode: semantic`) asks a model to judge a node, for example whether an opportunity is a customer need. It has a `rubric` instead of a `check`.
A **semantic rule** (`mode: semantic`) asks a model to judge a node, for example whether an opportunity is a customer need. It has a `rubric` and may have a `check`, an expression precondition that runs before the judge.

A **judge** is the model that judges semantic rules: a decision model (Jev) or a chat model (Claude, OpenAI-compatible). For each rule, the judge gives one **verdict** from the rule's `verdicts`, for example `customer-need`, `vague` or `solution`. The rule's `severity` says which verdicts report a violation. Semantic rules cost money, so they run only when a run asks for them (`--semantic` or `--judge`). The judge is a configured **model provider**: a bundled provider or an entry in `modelProviders`. See [docs/semantic-rules.md](semantic-rules.md).
A **judge** is the model that judges semantic rules: a decision model (Jev) or a chat model (Claude, OpenAI-compatible). For each rule, the judge gives one **verdict** from the rule's `verdicts`, for example `customer-need`, `vague` or `solution`. The rule's `severity` says which verdicts report a violation. Semantic rules cost money, so they are judged only when a run asks for it (`--semantic` or `--judge`); a rule's `check` always runs. The judge is a configured **model provider**: a bundled provider or an entry in `modelProviders`. See [docs/semantic-rules.md](semantic-rules.md).

### Cascade gating

Expand Down
2 changes: 1 addition & 1 deletion docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ The top-level `judge` names the model provider that judges semantic rules: a bun
}
```

Config does not turn semantic rules on. Each run asks for them with `--semantic`, or with `--judge <provider>`, which also selects a different provider for that run.
Config does not turn judging on. Each run asks for it with `--semantic`, or with `--judge <provider>`, which also selects a different provider for that run.

## Markdown plugin config

Expand Down
2 changes: 1 addition & 1 deletion docs/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Prefer choosing the right category and relying on its default. Set `severity` on
"when": "current.status = 'ready'", "check": "$exists(current.owner)" }
```

`when` costs almost nothing, so use it to stop an expensive rule from running where it does not apply. For a semantic rule, this saves a model call on each node that `when` excludes.
`when` costs almost nothing, so use it to stop an expensive rule from running where it does not apply. For a semantic rule, this saves a model call on each node that `when` excludes. A semantic rule can also carry a `check`; see [Preconditions](semantic-rules.md#preconditions-check) for the order.

An error in a `when` or `check` is reported as a non-gating `error` violation.

Expand Down
29 changes: 27 additions & 2 deletions docs/semantic-rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,41 @@ Semantic rules cost money and send node content to the judge's provider. Thus th
- `--semantic` runs them with the configured judge.
- `--judge <provider>` runs them with a specific model provider instead.

A semantic rule's [`check`](#preconditions-check) costs no model call, so it runs on every validation. Without a judge, only that expression runs.

## Rule fields

| Field | Required | Description |
|---|---|---|
| `mode` | yes | `"semantic"` |
| `check` | no | An expression that a node must pass before the judge sees it. Runs even without a judge. See [Preconditions](#preconditions-check) |
| `rubric` | no | Tells the judge what to assess. Default: `description` |
| `verdicts` | no | The answers the judge can give. Default: `pass` and `fail`. See [Verdicts](#verdicts) |
| `severity` | if `verdicts` is set | The severity of each verdict. See [Severity](#severity) |

A semantic rule cannot have `check` or `scope: "global"`.
A semantic rule cannot have `scope: "global"`.

## Preconditions (`check`)

A `check` costs no model call. It uses the same [expression syntax and context](rules.md#writing-expressions) as an expression rule. Use it for a requirement that an expression can verify, so the judge only sees nodes worth the cost:

```json
{ "id": "opportunity-customer-voice", "category": "coherence", "type": "opportunity",
"mode": "semantic", "description": "Opportunity is framed in customer voice",
"check": "$exists(current.source)" }
```

The order is `when`, then `check`, then the judge. Without a judge, the order is `when`, then `check`.

- If `check` returns `true`, the judge runs. Without a judge, nothing is reported. The verdict maps to a severity as usual.
- If `check` does not return `true`, structured-context reports a violation and does not call the judge. The violation has no verdict, and its evidence reads `expression check failed: <the check expression>`.
- If `check` cannot be evaluated, structured-context reports a non-gating `error` violation and does not call the judge.

A failing `check` reports the severity that an expression rule would report: the rule's `severity`, or the default for its category. With a [severity map](#severity), it reports the most severe severity in the map. If every entry in the map is `ignore`, it reports `info`.

A failing `check` [gates](rules.md#cascade-gating) the node's descendants as an expression rule does. It gates by default when it reports `error` severity. Set `gatesDescendants` to override that either way.

`--rule-filter mode=semantic` selects these rules with or without a judge.

## Verdicts

Expand Down Expand Up @@ -83,7 +108,7 @@ Gating uses the severity of the verdict. A verdict with `error` severity gates t

## How judging works

- **Order.** On each node, expression rules run first. Then structured-context sends all of the node's semantic rules to the judge in one call.
- **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge in one call.
- **Gating.** structured-context does not judge a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated.
- **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate.
- **Input.** The judge gets the node's title, type, parent titles and schema data, including frontmatter.
Expand Down
2 changes: 1 addition & 1 deletion plugin/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "structured-context",
"version": "0.12.13",
"version": "0.12.14",
"description": "Structured content authoring for Obsidian — automatic validation hooks, slash commands, and agent skills to keep markdown files compliant with your schema.",
"homepage": "https://github.com/mindsocket/structured-context",
"repository": "https://github.com/mindsocket/structured-context",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ Relationships define how sub-entities (nodes inside other files) are parsed and

Rules are a flat array. Each rule's `category` (`validation`, `coherence`, `workflow`, `best-practice`) supplies its default severity: `validation` → `error`, `coherence` and `workflow` → `warning`, `best-practice` → `info`. Only `error` violations fail validation; warnings and info are reported only. An optional per-rule `severity` (`error` | `warning` | `info`) overrides the category default — prefer picking the right category over overriding.

Rules default to `mode: "expression"` (a JSONata `check`). Set `mode: "semantic"` for rules an LLM or decision model judges against a `rubric` (falls back to `description`): they take no `check` or `scope`, may declare `verdicts` (verdict → description, or `{ description, guidance }`; default `pass`/`fail`) with a `severity` map saying which verdicts are `error`/`warning`/`info`/`ignore`, and only run with `--semantic`. Run `sctx docs semantic-rules` for the full semantic rule reference. A `when` JSONata test (any rule) limits a rule to the nodes where it returns true. An optional `gatesDescendants` controls whether a violation suppresses rule checks on the node's subtree (default: true for `error` severity).
Rules default to `mode: "expression"` (a JSONata `check`). Set `mode: "semantic"` for rules an LLM or decision model judges against a `rubric` (falls back to `description`): they take no `scope` and may take a `check` (a JSONata precondition that runs before the judge, and even without `--semantic`; a failing `check` reports a violation without calling the judge), may declare `verdicts` (verdict → description, or `{ description, guidance }`; default `pass`/`fail`) with a `severity` map saying which verdicts are `error`/`warning`/`info`/`ignore`, and are judged only with `--semantic` or `--judge`. Run `sctx docs semantic-rules` for the full semantic rule reference. A `when` JSONata test (any rule) limits a rule to the nodes where it returns true. An optional `gatesDescendants` controls whether a violation suppresses rule checks on the node's subtree (default: true for `error` severity).

## Metadata composition semantics

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -284,14 +284,7 @@
},
"then": {
"not": {
"anyOf": [
{
"required": ["check"]
},
{
"required": ["scope"]
}
]
"required": ["scope"]
},
"if": {
"required": ["verdicts"]
Expand Down
41 changes: 41 additions & 0 deletions src/schema/evaluate-rule.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import jsonata from 'jsonata';
import type { Rule, RuleViolation, SpaceNode } from '../types';
import { ruleSeverity } from './rule-severity';

const expressionCache = new Map<string, ReturnType<typeof jsonata>>();

Expand Down Expand Up @@ -139,3 +140,43 @@ export function expressionErrorViolation(rule: Rule, file: string, error: string
description: `${rule.description} (Rule error: ${error})`,
};
}

/**
* Applies the `check` precondition of semantic rules on a node. A rule whose `check` fails or errors reports a
* violation (at its own severity, or `error` for an evaluation failure) and is left out of `judgeable`, so the
* judge never sees it. A failing check's violation names the expression in its evidence, and its rule is listed in
* `failed` so the caller can honour `gatesDescendants`; an evaluation error is never listed.
*/
export async function applySemanticChecks(
rules: Rule[],
node: SpaceNode,
allNodes: SpaceNode[],
nodeIndex: Map<string, SpaceNode>,
): Promise<{ judgeable: Rule[]; violations: RuleViolation[]; failed: Rule[] }> {
const judgeable: Rule[] = [];
const violations: RuleViolation[] = [];
const failed: Rule[] = [];
for (const rule of rules) {
if (!rule.check) {
judgeable.push(rule);
continue;
}
const outcome = await tryEvaluateExpression(rule.check, buildEvalContext(node, allNodes, nodeIndex));
if ('error' in outcome) {
violations.push(expressionErrorViolation(rule, node.label, outcome.error));
} else if (outcome.value !== true) {
violations.push({
file: node.label,
ruleId: rule.id,
category: rule.category,
severity: ruleSeverity(rule),
description: rule.description,
evidence: `expression check failed: ${rule.check}`,
});
failed.push(rule);
} else {
judgeable.push(rule);
}
}
return { judgeable, violations, failed };
}
21 changes: 18 additions & 3 deletions src/schema/gating.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,12 @@ import { judgeNode } from '../judge/judge-node';
import type { SemanticJudgeProvider } from '../judge/types';
import type { SpaceGraph } from '../space-graph';
import type { Rule, RuleSeverity, RuleViolation, SpaceNode } from '../types';
import { buildEvalContext, expressionErrorViolation, tryEvaluateExpression } from './evaluate-rule';
import {
applySemanticChecks,
buildEvalContext,
expressionErrorViolation,
tryEvaluateExpression,
} from './evaluate-rule';
import { ruleSeverities, ruleSeverity } from './rule-severity';

/**
Expand Down Expand Up @@ -157,7 +162,7 @@ export async function evaluateRulesWithGating(
}
}

const perNodeRules = rules.filter((r) => r.scope !== 'global' && (r.mode !== 'semantic' || judge));
const perNodeRules = rules.filter((r) => r.scope !== 'global' && (r.mode !== 'semantic' || judge || r.check));

// With a judge target, only the target and its hierarchy ancestors can need judging.
const targetAncestors = new Set<string>();
Expand Down Expand Up @@ -232,7 +237,17 @@ export async function evaluateRulesWithGating(
else if (outcome.value !== true) report(rule);
}

const semantic = applicable.filter((r) => r.mode === 'semantic');
// A semantic rule's `check` fails fast before the judge. Its violation gates like an expression rule's, at the severity it reports.
const prechecked = await applySemanticChecks(
applicable.filter((r) => r.mode === 'semantic'),
node,
nodes,
nodeIndex,
);
violations.push(...prechecked.violations);
const gatingRule = prechecked.failed.find((r) => ruleGatesDescendants(r));
if (gatingRule && !blockingRuleId) blockingRuleId = gatingRule.id;
const semantic = prechecked.judgeable;
if (judge && semantic.length > 0) {
for (const j of await judgeNode(judge, node, semantic)) {
if (j.failed) {
Expand Down
5 changes: 3 additions & 2 deletions src/schema/metadata-contract.ts
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ const RULE_SCHEMA = {
],
},
description: { type: 'string', minLength: 1 },
/** Expression rules: the test a node must pass. Semantic rules: a precondition evaluated after `when` and before the judge; a failing node is reported without judging. */
check: { type: 'string', minLength: 1 },
/** Optional JSONata applicability test, evaluated per node before the rule runs. The rule applies only when it returns true. */
when: { type: 'string', minLength: 1 },
Expand Down Expand Up @@ -111,12 +112,12 @@ const RULE_SCHEMA = {
required: ['id', 'category', 'description'],
// A global rule has no node to test, so `when` needs a per-node rule.
not: { required: ['when', 'scope'] },
// Semantic rules are judged per node (no `check`, not global) and with custom verdicts must map each failing
// Semantic rules are judged per node (not global, `check` optional) and with custom verdicts must map each failing
// verdict to a severity. Every other rule needs a `check` and takes none of the semantic-only fields.
if: { properties: { mode: { const: 'semantic' } }, required: ['mode'] },
// biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then`/`else` keyword, not a thenable
then: {
not: { anyOf: [{ required: ['check'] }, { required: ['scope'] }] },
not: { required: ['scope'] },
if: { required: ['verdicts'] },
// biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then`/`else` keyword, not a thenable
then: { required: ['severity'], properties: { severity: { type: 'object' } } },
Expand Down
23 changes: 17 additions & 6 deletions src/schema/validate-rules.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,18 @@
import { judgeNode } from '../judge/judge-node';
import type { SemanticJudgeProvider } from '../judge/types';
import type { Rule, RuleViolation, SpaceNode } from '../types';
import { buildEvalContext, expressionErrorViolation, tryEvaluateExpression } from './evaluate-rule';
import {
applySemanticChecks,
buildEvalContext,
expressionErrorViolation,
tryEvaluateExpression,
} from './evaluate-rule';
import { ruleSeverity } from './rule-severity';

export { DEFAULT_SEVERITY_BY_CATEGORY, ruleSeverity } from './rule-severity';

export interface ValidateRulesOptions {
/** Judge for semantic rules. Without one, semantic rules are not evaluated. */
/** Judge for semantic rules. Without one, a semantic rule runs only its `check`. */
judge?: SemanticJudgeProvider;
/** Label of the only node to judge (`validate-file`); other nodes' semantic rules are skipped. */
judgeTarget?: string;
Expand Down Expand Up @@ -108,13 +113,14 @@ export async function validateRules(
}
}

// 2. Evaluate semantic rules only when a judge is supplied
// 2. Evaluate semantic rules. Without a judge only their `check` runs.
const judge = options.judge;
if (judge) {
const runnableSemanticRules = judge ? semanticRules : semanticRules.filter((r) => r.check);
if (runnableSemanticRules.length > 0) {
for (const node of nodes) {
if (options.judgeTarget !== undefined && node.label !== options.judgeTarget) continue;
const applicable: Rule[] = [];
for (const r of semanticRules) {
for (const r of runnableSemanticRules) {
if (r.type && r.type !== node.resolvedType) continue;
if (r.when) {
const when = await tryEvaluateExpression(r.when, buildEvalContext(node, nodes, nodeIndex));
Expand All @@ -131,7 +137,12 @@ export async function validateRules(
// Safe cascading default: gate semantic evaluation if node failed upstream checks
if (failedNodeLabels.has(node.label)) continue;

for (const j of await judgeNode(judge, node, applicable)) {
// A semantic rule's `check` fails fast before the judge.
const prechecked = await applySemanticChecks(applicable, node, nodes, nodeIndex);
violations.push(...prechecked.violations);
if (!judge) continue;

for (const j of await judgeNode(judge, node, prechecked.judgeable)) {
if (!j.failed) continue;
violations.push({
file: node.label,
Expand Down
4 changes: 2 additions & 2 deletions src/validate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ export interface ValidateSpaceOptions {
includeGated?: boolean;
/** Evaluate only rules matching this filter (see `parseRuleFilter`). */
ruleFilter?: RuleFilter;
/** Judge for `mode: semantic` rules. Without one, semantic rules are not evaluated. */
/** Judge for `mode: semantic` rules. Without one, a semantic rule runs only its `check`. */
judge?: SemanticJudgeProvider;
}

Expand Down Expand Up @@ -440,7 +440,7 @@ export interface ValidateFileOptions {
includeGated?: boolean;
/** Evaluate only rules matching this filter (see `parseRuleFilter`). */
ruleFilter?: RuleFilter;
/** Judge for `mode: semantic` rules. Without one, semantic rules are not evaluated. */
/** Judge for `mode: semantic` rules. Without one, a semantic rule runs only its `check`. */
judge?: SemanticJudgeProvider;
}

Expand Down
4 changes: 2 additions & 2 deletions tests/schema/semantic-rule-contract.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,8 @@ describe('rule contract: check and semantic mode', () => {
expect(load('semantic-ok', { ...base, mode: 'semantic' }).rules).toHaveLength(1);
});

it('rejects a semantic rule that also declares check', () => {
expect(() => load('semantic-check', { ...base, mode: 'semantic', check: 'true' })).toThrow();
it('accepts a semantic rule that declares a check precondition', () => {
expect(load('semantic-check', { ...base, mode: 'semantic', check: 'true' }).rules).toHaveLength(1);
});

it('rejects a global semantic rule', () => {
Expand Down
Loading
Loading