Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ There are several places that need reviewing and updating with any new feature o
- `src/schema/metadata-contract.ts` — Single source of truth for the `$metadata` contract
- `schemas/generated/latest/_structured_context_schema_meta.json` — Generated metaschema for the newest dialect (generated on build or with `bun run generate:schema-meta`). `schemas/generated/<version>/` folders are frozen copies, created by the `version` script (`generate:schema-meta --release`) only when a release changes the dialect; never edit them. `src/schema/meta-schemas.ts` loads them at runtime.
- `src/schema/gating.ts` — Cascade gating: computes hierarchy depth and evaluates rules with subtree gating (schema/hierarchy/blocking-rule failures suppress downstream rule evaluation). See [docs/rules.md](docs/rules.md#cascade-gating).
- `src/judge/` — Semantic rule evaluation (the "judge" is its internal name): provider interface and types (`types.ts`), verdict resolution (`verdicts.ts`), per-node and grouped dispatch (`judge-node.ts`), and providers (`provider/anthropic.ts`, `provider/openai.ts`, `provider/jev.ts`, `provider/fake.ts`), provider hydration and the bundled catalogue (`factory.ts`), and `sctx providers` (`src/commands/providers.ts`). See [docs/semantic-rules.md](docs/semantic-rules.md).
- `src/judge/` — Semantic rule evaluation (the "judge" is its internal name): provider interface and types (`types.ts`), verdict resolution (`verdicts.ts`), per-node and grouped dispatch (`judge-node.ts`), related-node selection for `semantic.scope` (`context.ts`), and providers (`provider/anthropic.ts`, `provider/openai.ts`, `provider/jev.ts`, `provider/fake.ts`), provider hydration and the bundled catalogue (`factory.ts`), and `sctx providers` (`src/commands/providers.ts`). See [docs/semantic-rules.md](docs/semantic-rules.md).
- `src/schema/rule-filter.ts` — `--rule-filter` parsing and rule filtering by category, severity, type or id.

## Testing
Expand Down
4 changes: 3 additions & 1 deletion docs/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -171,10 +171,12 @@ See [docs/rules.md](rules.md) for the rules reference, including JSONata express

### Semantic rules

A **semantic rule** is a rule with a `semantic` part. It asks a model to evaluate a node, for example whether an opportunity is a customer need. The `semantic` object can carry a `rubric`, `verdicts` and a verdict `severity` map. A rule can also have an `expression`, a JSONata test that runs before the model.
A **semantic rule** is a rule with a `semantic` part. It asks a model to evaluate a node, for example whether an opportunity is a customer need. The `semantic` object can carry a `rubric`, `verdicts`, a verdict `severity` map, a `scope` and `includeContent`. A rule can also have an `expression`, a JSONata test that runs before the model.

A **model provider** evaluates semantic rules: a decision model (Jev) or a chat model (Claude, OpenAI-compatible). It is a bundled provider or an entry in `modelProviders`. For each rule, the model gives one **verdict** from the rule's `semantic.verdicts`, for example `customer-need`, `vague` or `solution`. The rule's `semantic.severity` says which verdicts report a violation. Semantic rules cost money, so they are evaluated only when a run asks for it (`--semantic`); a rule's `expression` always runs. See [docs/semantic-rules.md](semantic-rules.md).

The **evaluated node** is the node a semantic rule is judged on. A rule's `semantic.scope` adds **related nodes** (parent, siblings, children, ancestors, descendants, or the whole branch) along the main hierarchy's edges, and `semantic.includeContent` chooses whose body is sent. See [Related nodes and content](semantic-rules.md#related-nodes-and-content).

A **node ruleset group** is the set of nodes that share the same semantic rules. It forms within one evaluation step of [cascade gating](#cascade-gating): one hierarchy depth, or the set of non-hierarchy nodes. Nodes are keyed on the rules left after `when` and `expression` filtering. A model provider on the `openrouter-decisions` or `typesafe-systemone` API evaluates one group in as few decision requests as it can.

A **decision request** is one call to a decisions API. It carries the state of one or more nodes and a list of decision questions.
Expand Down
41 changes: 40 additions & 1 deletion docs/semantic-rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,8 @@ A semantic rule has the [common rule fields](rules.md) and a `semantic` object.
| `semantic.rubric` | no | Tells the model what to assess. Default: `description` |
| `semantic.verdicts` | no | The answers the model can give. Default: `pass` and `fail`. See [Verdicts](#verdicts) |
| `semantic.severity` | if `semantic.verdicts` is set | The severity of each verdict. See [Severity](#severity) |
| `semantic.scope` | no | The related nodes to send with the evaluated node. Default: none. See [Related nodes and content](#related-nodes-and-content) |
| `semantic.includeContent` | no | Which node bodies to send. Default: none. See [Related nodes and content](#related-nodes-and-content) |

A semantic rule cannot be `global`. A rule can also have an `expression`; see [Combining an expression](#combining-an-expression).

Expand Down Expand Up @@ -107,13 +109,50 @@ The verdict is a classification. The severity says what the verdict means for th

Gating uses the severity of the verdict. A verdict with `error` severity gates the node's subtree, as for any rule. `gatesDescendants` changes this, as for any rule.

## Related nodes and content

By default the model sees the evaluated node alone. Use `semantic.scope` to send related nodes too, so the model can judge coherence across the tree, for example whether a solution addresses its parent opportunity.

`semantic.scope` is a list. Each entry adds nodes along the edges of the schema's main hierarchy:

| Entry | Adds |
|---|---|
| `parent` | The direct parents |
| `siblings` | The other children of the node's parents |
| `children` | The direct children |
| `ancestors` | The parents, their parents, and so on |
| `descendants` | The children, their children, and so on |
| `branch` | The ancestors and the descendants |

Entries add together, and each node is sent once. An empty or absent `scope` sends the evaluated node alone. Other relationships are not followed.

A wide scope can send many nodes, and every node sends all its schema data. Prefer to attach a rule at the level where `parent`, `children` or `ancestors` covers what it needs, rather than `descendants` or `branch` on a node near the top of a large tree.

Scope sends node data only, never the verdicts of other rules, and [gating](rules.md#cascade-gating) does not change which nodes it selects.

`semantic.includeContent` chooses which node bodies (the markdown `content`) go to the model:

| Value | Sends |
|---|---|
| absent | No bodies, not even the evaluated node's |
| `nodeOnly` | The evaluated node's body |
| `fullScope` | The bodies of the evaluated node and of every related node. `true` means the same |

```json
{ "id": "solution-addresses-opportunity", "category": "coherence", "type": "solution",
"description": "The solution addresses the opportunity it sits under",
"semantic": { "scope": ["parent"], "includeContent": "fullScope" } }
```

A global rule cannot have a `semantic` part, so it has no scope.

## How semantic evaluation works

- **Order.** On each node, structured-context runs each rule's [`expression`](#combining-an-expression) first. It sends the semantic parts of the rules that pass to the model provider.
- **Grouping.** A model provider on the `openrouter-decisions` or `typesafe-systemone` API evaluates each node ruleset group (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. This includes the bundled `openrouter-openai-decisions` provider. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request on tokens is sent whole as its own decision request. A node with more than 40 rules has its rules split across decision requests. If a node's questions are split across several requests and one fails, all of that node's rules are reported as errors. Other providers, including `openai-decisions` and chat models, evaluate one node per request. Within a space, the nodes at one hierarchy depth are evaluated together, because their verdicts decide whether the next depth is evaluated. Grouping can shift verdicts and probabilities slightly compared with evaluating each node alone.
- **Gating.** structured-context does not evaluate the semantic part on a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated.
- **Evaluation errors.** If the model provider fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation with the suffix `(Semantic evaluation error: ...)`. An evaluation error does not gate. If one decision request fails, only the rules on the nodes in that decision request report the error.
- **Input.** The model provider gets the node's title, type, parent titles and schema data, including frontmatter.
- **Input.** The model provider gets the node's title, type, parent titles and schema data, including frontmatter. It gets the node's body, and any [related nodes](#related-nodes-and-content), only as the rule's `semantic.scope` and `semantic.includeContent` ask.
- **Output.** A violation from a semantic verdict also includes the verdict, the model's evidence and any guidance.

`sctx validate-file` evaluates the semantic part only for the file and the rules on its ancestors that can gate it. Thus one file costs at most one model call for each ancestor level, plus one.
Expand Down
2 changes: 1 addition & 1 deletion plugin/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "structured-context",
"version": "0.12.15",
"version": "0.12.17",
"description": "Structured content authoring for Obsidian — automatic validation hooks, slash commands, and agent skills to keep markdown files compliant with your schema.",
"homepage": "https://github.com/mindsocket/structured-context",
"repository": "https://github.com/mindsocket/structured-context",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ Relationships define how sub-entities (nodes inside other files) are parsed and

Rules are a flat array. Each rule's `category` (`validation`, `coherence`, `workflow`, `best-practice`) supplies its default severity: `validation` → `error`, `coherence` and `workflow` → `warning`, `best-practice` → `info`. Only `error` violations fail validation; warnings and info are reported only. An optional per-rule `severity` (`error` | `warning` | `info`) overrides the category default — prefer picking the right category over overriding.

A rule has an `expression` (a JSONata test that must return `true`), a `semantic` object (a model evaluates the node against a `rubric`, which falls back to `description`), or both; a rule with neither is rejected. `global: true` runs an `expression` once for the space (no `when`, `type` or `semantic`). `semantic: {}` takes every default; it may declare `verdicts` (verdict → description, or `{ description, guidance }`; default `pass`/`fail`) with a `semantic.severity` map saying which verdicts are `error`/`warning`/`info`/`ignore` (required with custom verdicts). The top-level `severity` is that of a failed `expression` and of the default `fail` verdict. A rule's `expression` always runs and, when it fails, the `semantic` part is skipped; the `semantic` part runs only with `--semantic`. Run `sctx docs semantic-rules` for the full semantic rule reference. A `when` JSONata test (the applicability condition, any rule) limits a rule to the nodes where it returns true. An optional `gatesDescendants` controls whether a failure suppresses rules on the node's subtree (default: true for `error` severity).
A rule has an `expression` (a JSONata test that must return `true`), a `semantic` object (a model evaluates the node against a `rubric`, which falls back to `description`), or both; a rule with neither is rejected. `global: true` runs an `expression` once for the space (no `when`, `type` or `semantic`). `semantic: {}` takes every default; it may declare `verdicts` (verdict → description, or `{ description, guidance }`; default `pass`/`fail`) with a `semantic.severity` map saying which verdicts are `error`/`warning`/`info`/`ignore` (required with custom verdicts). `semantic.scope` (a list of `parent`, `siblings`, `children`, `ancestors`, `descendants`, `branch` (ancestors plus descendants) along the main hierarchy) sends related nodes with the evaluated node; `semantic.includeContent` (`nodeOnly` or `fullScope`/`true`) sends node bodies, and without it no body is sent. The top-level `severity` is that of a failed `expression` and of the default `fail` verdict. A rule's `expression` always runs and, when it fails, the `semantic` part is skipped; the `semantic` part runs only with `--semantic`. Run `sctx docs semantic-rules` for the full semantic rule reference. A `when` JSONata test (the applicability condition, any rule) limits a rule to the nodes where it returns true. An optional `gatesDescendants` controls whether a failure suppresses rules on the node's subtree (default: true for `error` severity).

## Metadata composition semantics

Expand Down
10 changes: 10 additions & 0 deletions schemas/generated/latest/_structured_context_schema_meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,16 @@
"enum": ["error", "warning", "info", "ignore"]
},
"minProperties": 1
},
"scope": {
"type": "array",
"items": {
"enum": ["parent", "siblings", "children", "ancestors", "descendants", "branch"]
},
"uniqueItems": true
},
"includeContent": {
"enum": ["nodeOnly", "fullScope", true]
}
},
"additionalProperties": false
Expand Down
115 changes: 115 additions & 0 deletions src/judge/context.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
import { SEMANTIC_SCOPES } from '../schema/metadata-contract';
import { ruleSemantic } from '../schema/rule-severity';
import type { SpaceGraph } from '../space-graph';
import type { Rule, SpaceNode } from '../types';
import type { RelatedNode, SemanticJudgeInput } from './types';

export type SemanticScope = Exclude<(typeof SEMANTIC_SCOPES)[number], 'branch'>;
export type ContentMode = 'none' | 'nodeOnly' | 'fullScope';

/** What a rule asks to send besides the evaluated node's data. */
export interface RuleContext {
scope: SemanticScope[];
content: ContentMode;
}

/** The related nodes and bodies a semantic rule asks for. `includeContent: true` means `fullScope`. */
export function ruleContext(rule: Rule): RuleContext {
const semantic = ruleSemantic(rule);
const include = semantic?.includeContent;
const asked = new Set<string>(semantic?.scope ?? []);
if (asked.has('branch')) asked.add('ancestors').add('descendants');
// A wider scope implies the narrower one, so equal node sets give equal inputs and direct relations keep their label.
if (asked.has('ancestors')) asked.add('parent');
if (asked.has('descendants')) asked.add('children');
return {
scope: SEMANTIC_SCOPES.filter((s): s is SemanticScope => s !== 'branch' && asked.has(s)),
content: include === undefined ? 'none' : include === true ? 'fullScope' : include,
};
}

/** All schema data, without the body unless asked for. */
function dataWithBody(node: SpaceNode, withBody: boolean): Record<string, unknown> {
if (withBody) return node.schemaData;
const { content: _content, ...data } = node.schemaData;
return data;
}

/** The evaluated node as a judge sees it. */
function judgeNodeView(node: SpaceNode, withBody: boolean): SemanticJudgeInput['node'] {
return {
label: node.label,
title: node.title,
type: node.type,
resolvedType: node.resolvedType,
schemaData: dataWithBody(node, withBody),
resolvedParents: node.resolvedParents,
};
}

type Relation = RelatedNode['relations'][number];

function relatedView(node: SpaceNode, relation: Relation, withBody: boolean): RelatedNode {
const { title: _title, type: _type, ...data } = dataWithBody(node, withBody);
return { relations: [relation], title: node.title, type: node.resolvedType, data };
}

/**
* The nodes a scope adds for a node, along main-hierarchy edges only. Selection reads the graph's structure and
* nothing else: not verdicts, not gating. Each node appears once, with every relation it has, nearest first.
*/
export function relatedNodes(graph: SpaceGraph, node: SpaceNode, context: RuleContext): RelatedNode[] {
const withBody = context.content === 'fullScope';
const hierarchyName = graph.hierarchy?.name;
const parentsOf = (n: SpaceNode): SpaceNode[] =>
graph.hierarchy === undefined
? []
: n.resolvedParents
.filter((p) => p.source === 'hierarchy' && p.hierarchy === hierarchyName)
.map((p) => graph.nodes.get(p.title))
.filter((p): p is SpaceNode => p !== undefined);
const childrenOf = (n: SpaceNode) => graph.hierarchyChildren.get(n.title) ?? [];

const found = new Map<string, RelatedNode>();
const add = (other: SpaceNode, relation: Relation) => {
if (other.title === node.title) return;
const existing = found.get(other.title);
if (!existing) found.set(other.title, relatedView(other, relation, withBody));
else if (!existing.relations.includes(relation)) existing.relations.push(relation);
};
const reach = (next: (n: SpaceNode) => readonly SpaceNode[], relation: Relation) => {
const seen = new Set<string>([node.title]);
let level = next(node);
while (level.length > 0) {
const following: SpaceNode[] = [];
for (const n of level) {
if (seen.has(n.title)) continue;
seen.add(n.title);
add(n, relation);
following.push(...next(n));
}
level = following;
}
};

const scope = new Set(context.scope);
if (scope.has('parent')) for (const p of parentsOf(node)) add(p, 'parent');
if (scope.has('siblings')) for (const p of parentsOf(node)) for (const s of childrenOf(p)) add(s, 'sibling');
if (scope.has('children')) for (const c of childrenOf(node)) add(c, 'child');
if (scope.has('ancestors')) reach(parentsOf, 'ancestor');
if (scope.has('descendants')) reach(childrenOf, 'descendant');
return [...found.values()];
}

/** The judge input context for rules that share `context`: the node view and its related nodes. */
export function judgeContext(
graph: SpaceGraph,
node: SpaceNode,
context: RuleContext,
): Pick<SemanticJudgeInput, 'node' | 'related'> {
const related = relatedNodes(graph, node, context);
return {
node: judgeNodeView(node, context.content !== 'none'),
...(related.length > 0 && { related }),
};
}
Loading
Loading