Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ See [docs/schemas.md](docs/schemas.md) (`sctx docs schema`) for the dialect, com
### Validate nodes

```bash
sctx validate <space> [--watch] [--depth <n>] [--include-gated]
sctx validate <space> [--watch] [--depth <n>] [--include-gated] [--semantic] [--rule-filter <dim=values>]
```

Validates markdown files against the JSON schema:
Expand All @@ -141,6 +141,7 @@ Validates markdown files against the JSON schema:
- Runs executable rules (`$metadata.rules`) with **cascade gating**: a schema, hierarchy, or blocking-rule failure on a node suppresses rule evaluation for that node's hierarchy subtree. Gated checks are reported in a Coverage section so you know what wasn't evaluated and why. See [docs/rules.md](docs/rules.md#cascade-gating) for details.
- `--depth <n>` limits evaluation to hierarchy nodes within `n` breadth-first levels from the top — useful for staged agentic fix loops (fix the top level, then request the next).
- `--include-gated` evaluates rules even on gated nodes (still reported as gated) — for surfacing cheap bottom-up checks on request.
- `--semantic` also evaluates `mode: semantic` rules with a model judge (Anthropic, OpenAI-compatible, or Jev); `--rule-filter` narrows which rules run. See [docs/rules.md](docs/rules.md#semantic-rules).

### Render space

Expand Down
56 changes: 54 additions & 2 deletions docs/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,11 @@ Rules are a flat array.
| `category` | yes | `validation` \| `coherence` \| `workflow` \| `best-practice` |
| `severity` | no | `error` \| `warning` \| `info`; overrides the category default (see [Severity](#severity)) |
| `description` | yes | Human-readable rule intent |
| `check` | yes | JSONata expression; must evaluate to `true` |
| `check` | yes, unless `mode: semantic` | JSONata expression; must evaluate to `true` |
| `mode` | no | `expression` (default) or `semantic` (judged by a model; see [Semantic rules](#semantic-rules)) |
| `rubric` | semantic only | Detailed prompt for the judge; falls back to `description` |
| `verdicts` | semantic only | Closed set of verdicts the judge must choose from: `{ "<verdict>": "<when it applies>" }`. Default `pass` / `fail`. |
| `failOn` | semantic only | Verdicts that count as a violation. Default `["fail"]`; required when `verdicts` is declared. At least one verdict must pass. |
| `type` | no | Restrict rule to nodes of this `resolvedType` |
| `scope` | no | Use `"global"` to evaluate once for the whole space |
| `override` | no | Only for merge conflicts: allows later duplicate `id` to replace earlier |
Expand Down Expand Up @@ -156,7 +160,7 @@ original blocker), rather than each generating a new one. Non-hierarchy nodes an
`scope: "global"` rules are never gated — cascade gating only applies within the hierarchy
DAG.

This exists so that expensive or semantic downstream checks (see #114) aren't wasted
This exists so that expensive or semantic downstream checks (see [Semantic rules](#semantic-rules)) aren't wasted
evaluating nodes whose ancestors are already known-bad, and so noise doesn't drown out the
actionable failure. Set `gatesDescendants: false` on an error-severity rule whose violations shouldn't suppress
checks further down the tree, or `gatesDescendants: true` on a warning/info rule that should.
Expand Down Expand Up @@ -184,6 +188,54 @@ the top (roots and orphans are depth 1). This is a manual, coarser control than
gating — intended for staged workflows (e.g. an agent fixes the top level, then requests
the next). Excluded nodes are reported under `skippedByDepth`, separate from gated nodes.

## Semantic rules

A rule with `"mode": "semantic"` is judged by a model instead of a JSONata expression. Semantic rules cost money and time, so they never run unless asked: pass `--semantic` (or set `semantic: true` / `judge.autoRun` on a space or in config).

```json
{
"id": "opportunity-customer-voice",
"category": "best-practice",
"mode": "semantic",
"type": "opportunity",
"description": "Opportunity is framed in the customer's voice",
"rubric": "The opportunity reads as a customer need or pain point, not a solution in disguise or a business metric.",
"verdicts": {
"customer-need": "Clearly a customer need or pain point",
"vague": "A need, but too vague to act on",
"solution": "A solution or business goal in disguise"
},
"failOn": ["solution"]
}
```

The judge must pick exactly one verdict from `verdicts` (default `pass`/`fail`); a verdict listed in `failOn` is a violation, everything else passes. A closed set maps directly onto decision models, which return a choice with confidence and probabilities rather than prose.

- **Evaluation order.** For each node, deterministic rules run first, then all of the node's semantic rules go to the judge in one call. Providers that can only answer one question per request (chat LLMs) make one request per rule; decision models such as Jev answer all of a node's rules in a single request.
- **Gating.** Semantic rules follow [cascade gating](#cascade-gating): they are not run on gated nodes (recorded as `gatedRules`), and a failing semantic rule gates its subtree when `gatesDescendants` is true (default for `error` severity). A judge *error* (network, unparseable output) is reported as a violation but never gates.
- **Output.** `--json` includes `semanticEvaluations` (verdict, `pass`, confidence, evidence per node/rule) and `gatedRules`. Violations carry the judge's `evidence`; decision models without prose get a summary of the verdict distribution.
- **Scope.** Semantic rules judge a node in isolation, plus its parent titles. (Wider scopes such as ancestors or siblings are not yet supported.)

### Judge configuration

Set `judge` at the root of `config.json` or per space (the space wins):

```json
{ "judge": { "provider": "anthropic", "model": "claude-haiku-5-5" } }
```

| Field | Description |
|---|---|
| `provider` | `anthropic`, `openai` (any OpenAI-compatible endpoint, including Ollama and OpenRouter chat models), `jev`, or `fake` (tests). Omitted: inferred from `ANTHROPIC_API_KEY`, then `OPENAI_API_KEY`/`OPENAI_BASE_URL`. `jev` is never inferred. |
| `model` | Defaults: `claude-haiku-5-5` (anthropic), `gpt-4o-mini` (openai), `typesafe/jev-1.13` (jev). |
| `endpoint` | Override the API URL. Jev defaults to OpenRouter's `https://openrouter.ai/api/alpha/decisions`. |
| `apiKey` | Prefer environment variables: `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `OPENROUTER_API_KEY` (jev). `SCTX_JUDGE_*` variables override provider, model, endpoint and key. |
| `autoRun` | Run semantic rules without `--semantic`. |

### Filtering rules

`--rule-filter <dimension>=<value>[,<value>]` (repeatable) evaluates only matching rules. Dimensions are `mode`, `category`, `severity` (effective severity), `type` and `id`; different dimensions combine with AND, values within one with OR. For example, `--semantic --rule-filter mode=semantic` runs only the judged rules, and `--rule-filter mode=expression` skips them entirely. Gating is computed from the rules that ran.

## Common patterns

```jsonata
Expand Down
2 changes: 1 addition & 1 deletion plugin/.claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "structured-context",
"version": "0.12.6",
"version": "0.12.7",
"description": "Structured content authoring for Obsidian — automatic validation hooks, slash commands, and agent skills to keep markdown files compliant with your schema.",
"homepage": "https://github.com/mindsocket/structured-context",
"repository": "https://github.com/mindsocket/structured-context",
Expand Down
5 changes: 4 additions & 1 deletion plugin/skills/structured-context/references/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Then use the `sctx` command for all operations. Always pass `--config <path>` wh
## validate

```bash
sctx validate <space> [--watch] [--config <path>] [--depth <n>] [--include-gated]
sctx validate <space> [--watch] [--config <path>] [--depth <n>] [--include-gated] [--semantic] [--rule-filter <dim=values>]
```

Validates all `.md` files in the space against the JSON schema. For each file:
Expand All @@ -29,6 +29,9 @@ Validates all `.md` files in the space against the JSON schema. For each file:
the top — useful for fixing a space level by level (fix the top, re-run with a deeper
`--depth`, repeat). `--include-gated` evaluates rules on gated nodes anyway, still reporting
them as gated — use it to surface cheap bottom-up checks even when an upstream blocker fired.
`--semantic` also runs `mode: semantic` rules through a configured model judge (costly; off by
default), and `--rule-filter <dimension>=<values>` (repeatable; mode, category, severity, type, id)
narrows which rules run. `validate-file` accepts the same two flags.

Exit codes: `0` = clean (warning and info rule violations don't fail), `1` = validation errors found.

Expand Down
8 changes: 8 additions & 0 deletions schemas/_ost_strict.json
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,14 @@
"description": "Explore multiple candidate solutions (aim for at least three) for the target opportunity",
"type": "opportunity",
"check": "(current.status != 'exploring' and current.status != 'active') or $count(nodes[resolvedParentTitle=$$.current.title and resolvedType='solution']) >= 3"
},
{
"id": "opportunity-customer-voice",
"category": "best-practice",
"mode": "semantic",
"type": "opportunity",
"description": "Opportunity must be framed in the customer's voice (something a customer would actually say) as a need or pain point, not a solution or business goal",
"rubric": "Verify that the opportunity is stated from the customer's perspective as a problem, pain point, or desire. It must not be a solution in disguise (e.g. 'build X feature') or a business metric (e.g. 'increase revenue')."
}
]
},
Expand Down
26 changes: 25 additions & 1 deletion schemas/generated/latest/_structured_context_schema_meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,30 @@
"type": "string",
"minLength": 1
},
"mode": {
"enum": ["expression", "semantic"]
},
"rubric": {
"type": "string",
"minLength": 1
},
"verdicts": {
"type": "object",
"additionalProperties": {
"type": "string",
"minLength": 1
},
"minProperties": 2
},
"failOn": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
},
"minItems": 1,
"uniqueItems": true
},
"type": {
"type": "string",
"minLength": 1
Expand All @@ -221,7 +245,7 @@
"type": "boolean"
}
},
"required": ["id", "category", "description", "check"],
"required": ["id", "category", "description"],
"additionalProperties": false
}
}
Expand Down
16 changes: 15 additions & 1 deletion src/api.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,20 @@ export type { AnySchemaObject, SchemaObject, ValidateFunction } from 'ajv';
export type { Config, SpaceConfig } from './config';
export { loadConfig, resolveSchema, resolveSchemaPath, setConfigPath } from './config';
export { PLUGIN_PREFIX } from './constants';
export type {
GatedRuleRecord,
JudgeConfig,
SemanticEvaluationRecord,
SemanticJudgeInput,
SemanticJudgeProvider,
SemanticJudgeVerdict,
} from './judge';
export {
AnthropicJudgeProvider,
createJudgeProvider,
FakeJudgeProvider,
OpenAIJudgeProvider,
} from './judge';
export { assembleSpaceGraph, loadSpaceGraph } from './load-space-graph';
export type {
ParseHook,
Expand Down Expand Up @@ -79,5 +93,5 @@ export type {
UnresolvedRef,
ValidateFileOutput,
} from './types';
export type { ValidationResult } from './validate';
export type { ValidateFileOptions, ValidateSpaceOptions, ValidationResult } from './validate';
export { validateFile, validateSpace } from './validate';
9 changes: 7 additions & 2 deletions src/commands/validate-file.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { loadConfig } from '../config';
import { parseRuleFilter } from '../schema/rule-filter';
import type { FileValidationIssue, FileValidationResult } from '../types';
import { validateFile } from '../validate';

Expand All @@ -11,10 +12,14 @@ export { validateFile } from '../validate';
*/
export async function validateFileCommand(
filePath: string,
options: { json?: boolean; includeGated?: boolean } = {},
options: { json?: boolean; includeGated?: boolean; semantic?: boolean; ruleFilter?: string[] } = {},
): Promise<number> {
const config = loadConfig();
const result = await validateFile(filePath, config, { includeGated: options.includeGated });
const result = await validateFile(filePath, config, {
includeGated: options.includeGated,
semantic: options.semantic,
ruleFilter: options.ruleFilter && parseRuleFilter(options.ruleFilter),
});

if (!result.inSpace) {
if (options.json) {
Expand Down
39 changes: 36 additions & 3 deletions src/commands/validate.ts
Original file line number Diff line number Diff line change
@@ -1,17 +1,32 @@
import { dirname } from 'node:path';
import chokidar from 'chokidar';
import { getConfigSourceFiles } from '../config';
import type { SemanticJudgeProvider } from '../judge/types';
import { resolveSpacePluginSchemas } from '../plugins/util';
import { parseRuleFilter } from '../schema/rule-filter';
import { bundledSchemasDir } from '../schema/schema';
import type { FileValidationIssue, RuleSeverity, SpaceContext } from '../types';
import { formatErrors, validateSpace } from '../validate';

export async function validate(
context: SpaceContext,
options: { json?: boolean; depth?: number; includeGated?: boolean } = {},
options: {
json?: boolean;
depth?: number;
includeGated?: boolean;
semantic?: boolean;
ruleFilter?: string[];
judgeProvider?: SemanticJudgeProvider;
} = {},
): Promise<number> {
const { schema, schemaRefRegistry } = context;
const result = await validateSpace(context, { depth: options.depth, includeGated: options.includeGated });
const result = await validateSpace(context, {
depth: options.depth,
includeGated: options.includeGated,
semantic: options.semantic,
ruleFilter: options.ruleFilter && parseRuleFilter(options.ruleFilter),
judgeProvider: options.judgeProvider,
});
const ruleCounts: Record<RuleSeverity, number> = { error: 0, warning: 0, info: 0 };
for (const v of result.ruleViolations) ruleCounts[v.severity]++;

Expand Down Expand Up @@ -46,7 +61,10 @@ export async function validate(
}
for (const v of result.ruleViolations) {
if (v.file && v.severity === 'error') {
addError(v.file, `rule:${v.ruleId}`, 'rule', `[${v.ruleId}] ${v.description}`, v.severity);
const message = v.evidence
? `[${v.ruleId}] ${v.description} (Evidence: ${v.evidence})`
: `[${v.ruleId}] ${v.description}`;
addError(v.file, `rule:${v.ruleId}`, 'rule', message, v.severity);
}
}
const globalRuleErrorCount = result.ruleViolations.filter((v) => !v.file && v.severity === 'error').length;
Expand Down Expand Up @@ -74,6 +92,7 @@ export async function validate(
coverageGates: result.coverageGates,
skippedByDepth: result.skippedByDepth,
depthLimit: result.depthLimit,
semanticEvaluations: result.semanticEvaluations,
},
null,
2,
Expand Down Expand Up @@ -193,10 +212,24 @@ export async function validate(
console.log(` ${severityColor[severity]}${severity.toUpperCase()}${reset} (${violations.length}):`);
for (const v of violations) {
console.log(` ${v.file ? `${v.file}: ` : ''}${v.description} [${v.category}: ${v.ruleId}]`);
if (v.evidence) {
console.log(` Evidence: ${v.evidence}`);
}
}
}
}

if (result.semanticEvaluations && result.semanticEvaluations.length > 0) {
const passed = result.semanticEvaluations.filter((e) => e.pass).length;
const failed = result.semanticEvaluations.filter((e) => !e.pass).length;
console.log(
`\nSemantic evaluations: ${result.semanticEvaluations.length} total (${passed} passed, ${failed} failed)`,
);
}
if (result.gatedRules && result.gatedRules.length > 0) {
console.log(`Gated semantic rules (skipped due to upstream errors): ${result.gatedRules.length}`);
}

if (result.hierarchyViolations.length > 0) {
console.log(`\nHierarchy violations:`);
for (const v of result.hierarchyViolations) {
Expand Down
25 changes: 25 additions & 0 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { dirname, isAbsolute, join, resolve } from 'node:path';
import Ajv2020 from 'ajv/dist/2020';
import JSON5 from 'json5';
import { ENV_CONFIG_VAR, XDG_CONFIG_DIR } from './constants';
import type { JudgeConfig } from './judge/types';
import {
normalizePluginName,
resolvePluginSchemasDir,
Expand All @@ -12,6 +13,18 @@ import {
} from './plugins/util';
import { bundledSchemasDir } from './schema/schema';

const JUDGE_CONFIG_SCHEMA = {
type: 'object',
properties: {
provider: { type: 'string', enum: ['anthropic', 'openai', 'fake'] },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jev is missing from the provider enum, so a config file with "provider": "jev" fails loadConfig validation with Invalid config. docs/rules.md, JudgeConfig in src/judge/types.ts and createJudgeProvider all support jev; only SCTX_JUDGE_PROVIDER=jev works today.

Suggested change
provider: { type: 'string', enum: ['anthropic', 'openai', 'fake'] },
provider: { type: 'string', enum: ['anthropic', 'openai', 'jev', 'fake'] },

model: { type: 'string' },
apiKey: { type: 'string' },
endpoint: { type: 'string' },
autoRun: { type: 'boolean' },
},
additionalProperties: false,
} as const;

const CONFIG_SCHEMA = {
type: 'object',
properties: {
Expand All @@ -33,13 +46,17 @@ const CONFIG_SCHEMA = {
additionalProperties: false,
},
},
judge: JUDGE_CONFIG_SCHEMA,
semantic: { type: 'boolean' },
},
required: ['name', 'path'],
additionalProperties: false,
},
},
schema: { type: 'string' },
includeSpacesFrom: { type: 'array', items: { type: 'string' } },
judge: JUDGE_CONFIG_SCHEMA,
semantic: { type: 'boolean' },
},
anyOf: [{ required: ['spaces'] }, { required: ['includeSpacesFrom'] }],
additionalProperties: false,
Expand All @@ -53,6 +70,10 @@ export type SpaceConfig = {
plugins?: Record<string, Record<string, unknown>>;
/** Named filter views for this space. Keys are view names; values contain the filter expression. */
views?: Record<string, { expression: string }>;
/** LLM judge provider configuration for semantic rule evaluation. */
judge?: JudgeConfig;
/** Whether to evaluate semantic rules during validation for this space. */
semantic?: boolean;
/**
* Directory of the config file that defined this space, used to anchor relative plugin-config
* paths. Set by loadConfig; absent for a hand-assembled Config, whose callers must pass an
Expand All @@ -65,6 +86,10 @@ export type Config = {
spaces: SpaceConfig[];
schema?: string;
includeSpacesFrom?: string[];
/** Default LLM judge provider configuration for semantic rule evaluation. */
judge?: JudgeConfig;
/** Whether to evaluate semantic rules during validation across spaces. */
semantic?: boolean;
};

let _configPathOverride: string | undefined;
Expand Down
Loading
Loading