diff --git a/README.md b/README.md index efb867a..927834d 100644 --- a/README.md +++ b/README.md @@ -131,7 +131,7 @@ See [docs/schemas.md](docs/schemas.md) (`sctx docs schema`) for the dialect, com ### Validate nodes ```bash -sctx validate [--watch] [--depth ] [--include-gated] +sctx validate [--watch] [--depth ] [--include-gated] [--semantic] [--rule-filter ] ``` Validates markdown files against the JSON schema: @@ -141,6 +141,7 @@ Validates markdown files against the JSON schema: - Runs executable rules (`$metadata.rules`) with **cascade gating**: a schema, hierarchy, or blocking-rule failure on a node suppresses rule evaluation for that node's hierarchy subtree. Gated checks are reported in a Coverage section so you know what wasn't evaluated and why. See [docs/rules.md](docs/rules.md#cascade-gating) for details. - `--depth ` limits evaluation to hierarchy nodes within `n` breadth-first levels from the top — useful for staged agentic fix loops (fix the top level, then request the next). - `--include-gated` evaluates rules even on gated nodes (still reported as gated) — for surfacing cheap bottom-up checks on request. +- `--semantic` also evaluates `mode: semantic` rules with a model judge (Anthropic, OpenAI-compatible, or Jev); `--rule-filter` narrows which rules run. See [docs/rules.md](docs/rules.md#semantic-rules). ### Render space diff --git a/docs/rules.md b/docs/rules.md index ca48f91..ed1863b 100644 --- a/docs/rules.md +++ b/docs/rules.md @@ -26,7 +26,11 @@ Rules are a flat array. | `category` | yes | `validation` \| `coherence` \| `workflow` \| `best-practice` | | `severity` | no | `error` \| `warning` \| `info`; overrides the category default (see [Severity](#severity)) | | `description` | yes | Human-readable rule intent | -| `check` | yes | JSONata expression; must evaluate to `true` | +| `check` | yes, unless `mode: semantic` | JSONata expression; must evaluate to `true` | +| `mode` | no | `expression` (default) or `semantic` (judged by a model; see [Semantic rules](#semantic-rules)) | +| `rubric` | semantic only | Detailed prompt for the judge; falls back to `description` | +| `verdicts` | semantic only | Closed set of verdicts the judge must choose from: `{ "": "" }`. Default `pass` / `fail`. | +| `failOn` | semantic only | Verdicts that count as a violation. Default `["fail"]`; required when `verdicts` is declared. At least one verdict must pass. | | `type` | no | Restrict rule to nodes of this `resolvedType` | | `scope` | no | Use `"global"` to evaluate once for the whole space | | `override` | no | Only for merge conflicts: allows later duplicate `id` to replace earlier | @@ -156,7 +160,7 @@ original blocker), rather than each generating a new one. Non-hierarchy nodes an `scope: "global"` rules are never gated — cascade gating only applies within the hierarchy DAG. -This exists so that expensive or semantic downstream checks (see #114) aren't wasted +This exists so that expensive or semantic downstream checks (see [Semantic rules](#semantic-rules)) aren't wasted evaluating nodes whose ancestors are already known-bad, and so noise doesn't drown out the actionable failure. Set `gatesDescendants: false` on an error-severity rule whose violations shouldn't suppress checks further down the tree, or `gatesDescendants: true` on a warning/info rule that should. @@ -184,6 +188,54 @@ the top (roots and orphans are depth 1). This is a manual, coarser control than gating — intended for staged workflows (e.g. an agent fixes the top level, then requests the next). Excluded nodes are reported under `skippedByDepth`, separate from gated nodes. +## Semantic rules + +A rule with `"mode": "semantic"` is judged by a model instead of a JSONata expression. Semantic rules cost money and time, so they never run unless asked: pass `--semantic` (or set `semantic: true` / `judge.autoRun` on a space or in config). + +```json +{ + "id": "opportunity-customer-voice", + "category": "best-practice", + "mode": "semantic", + "type": "opportunity", + "description": "Opportunity is framed in the customer's voice", + "rubric": "The opportunity reads as a customer need or pain point, not a solution in disguise or a business metric.", + "verdicts": { + "customer-need": "Clearly a customer need or pain point", + "vague": "A need, but too vague to act on", + "solution": "A solution or business goal in disguise" + }, + "failOn": ["solution"] +} +``` + +The judge must pick exactly one verdict from `verdicts` (default `pass`/`fail`); a verdict listed in `failOn` is a violation, everything else passes. A closed set maps directly onto decision models, which return a choice with confidence and probabilities rather than prose. + +- **Evaluation order.** For each node, deterministic rules run first, then all of the node's semantic rules go to the judge in one call. Providers that can only answer one question per request (chat LLMs) make one request per rule; decision models such as Jev answer all of a node's rules in a single request. +- **Gating.** Semantic rules follow [cascade gating](#cascade-gating): they are not run on gated nodes (recorded as `gatedRules`), and a failing semantic rule gates its subtree when `gatesDescendants` is true (default for `error` severity). A judge *error* (network, unparseable output) is reported as a violation but never gates. +- **Output.** `--json` includes `semanticEvaluations` (verdict, `pass`, confidence, evidence per node/rule) and `gatedRules`. Violations carry the judge's `evidence`; decision models without prose get a summary of the verdict distribution. +- **Scope.** Semantic rules judge a node in isolation, plus its parent titles. (Wider scopes such as ancestors or siblings are not yet supported.) + +### Judge configuration + +Set `judge` at the root of `config.json` or per space (the space wins): + +```json +{ "judge": { "provider": "anthropic", "model": "claude-haiku-5-5" } } +``` + +| Field | Description | +|---|---| +| `provider` | `anthropic`, `openai` (any OpenAI-compatible endpoint, including Ollama and OpenRouter chat models), `jev`, or `fake` (tests). Omitted: inferred from `ANTHROPIC_API_KEY`, then `OPENAI_API_KEY`/`OPENAI_BASE_URL`. `jev` is never inferred. | +| `model` | Defaults: `claude-haiku-5-5` (anthropic), `gpt-4o-mini` (openai), `typesafe/jev-1.13` (jev). | +| `endpoint` | Override the API URL. Jev defaults to OpenRouter's `https://openrouter.ai/api/alpha/decisions`. | +| `apiKey` | Prefer environment variables: `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `OPENROUTER_API_KEY` (jev). `SCTX_JUDGE_*` variables override provider, model, endpoint and key. | +| `autoRun` | Run semantic rules without `--semantic`. | + +### Filtering rules + +`--rule-filter =[,]` (repeatable) evaluates only matching rules. Dimensions are `mode`, `category`, `severity` (effective severity), `type` and `id`; different dimensions combine with AND, values within one with OR. For example, `--semantic --rule-filter mode=semantic` runs only the judged rules, and `--rule-filter mode=expression` skips them entirely. Gating is computed from the rules that ran. + ## Common patterns ```jsonata diff --git a/plugin/.claude-plugin/plugin.json b/plugin/.claude-plugin/plugin.json index 037f46f..c1f1c21 100644 --- a/plugin/.claude-plugin/plugin.json +++ b/plugin/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "structured-context", - "version": "0.12.6", + "version": "0.12.7", "description": "Structured content authoring for Obsidian — automatic validation hooks, slash commands, and agent skills to keep markdown files compliant with your schema.", "homepage": "https://github.com/mindsocket/structured-context", "repository": "https://github.com/mindsocket/structured-context", diff --git a/plugin/skills/structured-context/references/commands.md b/plugin/skills/structured-context/references/commands.md index f2bd438..4781908 100644 --- a/plugin/skills/structured-context/references/commands.md +++ b/plugin/skills/structured-context/references/commands.md @@ -11,7 +11,7 @@ Then use the `sctx` command for all operations. Always pass `--config ` wh ## validate ```bash -sctx validate [--watch] [--config ] [--depth ] [--include-gated] +sctx validate [--watch] [--config ] [--depth ] [--include-gated] [--semantic] [--rule-filter ] ``` Validates all `.md` files in the space against the JSON schema. For each file: @@ -29,6 +29,9 @@ Validates all `.md` files in the space against the JSON schema. For each file: the top — useful for fixing a space level by level (fix the top, re-run with a deeper `--depth`, repeat). `--include-gated` evaluates rules on gated nodes anyway, still reporting them as gated — use it to surface cheap bottom-up checks even when an upstream blocker fired. +`--semantic` also runs `mode: semantic` rules through a configured model judge (costly; off by +default), and `--rule-filter =` (repeatable; mode, category, severity, type, id) +narrows which rules run. `validate-file` accepts the same two flags. Exit codes: `0` = clean (warning and info rule violations don't fail), `1` = validation errors found. diff --git a/schemas/_ost_strict.json b/schemas/_ost_strict.json index 86b5acb..08fad84 100644 --- a/schemas/_ost_strict.json +++ b/schemas/_ost_strict.json @@ -35,6 +35,14 @@ "description": "Explore multiple candidate solutions (aim for at least three) for the target opportunity", "type": "opportunity", "check": "(current.status != 'exploring' and current.status != 'active') or $count(nodes[resolvedParentTitle=$$.current.title and resolvedType='solution']) >= 3" + }, + { + "id": "opportunity-customer-voice", + "category": "best-practice", + "mode": "semantic", + "type": "opportunity", + "description": "Opportunity must be framed in the customer's voice (something a customer would actually say) as a need or pain point, not a solution or business goal", + "rubric": "Verify that the opportunity is stated from the customer's perspective as a problem, pain point, or desire. It must not be a solution in disguise (e.g. 'build X feature') or a business metric (e.g. 'increase revenue')." } ] }, diff --git a/schemas/generated/latest/_structured_context_schema_meta.json b/schemas/generated/latest/_structured_context_schema_meta.json index 02d47c2..4d64d61 100644 --- a/schemas/generated/latest/_structured_context_schema_meta.json +++ b/schemas/generated/latest/_structured_context_schema_meta.json @@ -207,6 +207,30 @@ "type": "string", "minLength": 1 }, + "mode": { + "enum": ["expression", "semantic"] + }, + "rubric": { + "type": "string", + "minLength": 1 + }, + "verdicts": { + "type": "object", + "additionalProperties": { + "type": "string", + "minLength": 1 + }, + "minProperties": 2 + }, + "failOn": { + "type": "array", + "items": { + "type": "string", + "minLength": 1 + }, + "minItems": 1, + "uniqueItems": true + }, "type": { "type": "string", "minLength": 1 @@ -221,7 +245,7 @@ "type": "boolean" } }, - "required": ["id", "category", "description", "check"], + "required": ["id", "category", "description"], "additionalProperties": false } } diff --git a/src/api.ts b/src/api.ts index 9362858..1deedf9 100644 --- a/src/api.ts +++ b/src/api.ts @@ -22,6 +22,20 @@ export type { AnySchemaObject, SchemaObject, ValidateFunction } from 'ajv'; export type { Config, SpaceConfig } from './config'; export { loadConfig, resolveSchema, resolveSchemaPath, setConfigPath } from './config'; export { PLUGIN_PREFIX } from './constants'; +export type { + GatedRuleRecord, + JudgeConfig, + SemanticEvaluationRecord, + SemanticJudgeInput, + SemanticJudgeProvider, + SemanticJudgeVerdict, +} from './judge'; +export { + AnthropicJudgeProvider, + createJudgeProvider, + FakeJudgeProvider, + OpenAIJudgeProvider, +} from './judge'; export { assembleSpaceGraph, loadSpaceGraph } from './load-space-graph'; export type { ParseHook, @@ -79,5 +93,5 @@ export type { UnresolvedRef, ValidateFileOutput, } from './types'; -export type { ValidationResult } from './validate'; +export type { ValidateFileOptions, ValidateSpaceOptions, ValidationResult } from './validate'; export { validateFile, validateSpace } from './validate'; diff --git a/src/commands/validate-file.ts b/src/commands/validate-file.ts index e69eef5..61dc7f3 100644 --- a/src/commands/validate-file.ts +++ b/src/commands/validate-file.ts @@ -1,4 +1,5 @@ import { loadConfig } from '../config'; +import { parseRuleFilter } from '../schema/rule-filter'; import type { FileValidationIssue, FileValidationResult } from '../types'; import { validateFile } from '../validate'; @@ -11,10 +12,14 @@ export { validateFile } from '../validate'; */ export async function validateFileCommand( filePath: string, - options: { json?: boolean; includeGated?: boolean } = {}, + options: { json?: boolean; includeGated?: boolean; semantic?: boolean; ruleFilter?: string[] } = {}, ): Promise { const config = loadConfig(); - const result = await validateFile(filePath, config, { includeGated: options.includeGated }); + const result = await validateFile(filePath, config, { + includeGated: options.includeGated, + semantic: options.semantic, + ruleFilter: options.ruleFilter && parseRuleFilter(options.ruleFilter), + }); if (!result.inSpace) { if (options.json) { diff --git a/src/commands/validate.ts b/src/commands/validate.ts index ac9953a..5f94534 100644 --- a/src/commands/validate.ts +++ b/src/commands/validate.ts @@ -1,17 +1,32 @@ import { dirname } from 'node:path'; import chokidar from 'chokidar'; import { getConfigSourceFiles } from '../config'; +import type { SemanticJudgeProvider } from '../judge/types'; import { resolveSpacePluginSchemas } from '../plugins/util'; +import { parseRuleFilter } from '../schema/rule-filter'; import { bundledSchemasDir } from '../schema/schema'; import type { FileValidationIssue, RuleSeverity, SpaceContext } from '../types'; import { formatErrors, validateSpace } from '../validate'; export async function validate( context: SpaceContext, - options: { json?: boolean; depth?: number; includeGated?: boolean } = {}, + options: { + json?: boolean; + depth?: number; + includeGated?: boolean; + semantic?: boolean; + ruleFilter?: string[]; + judgeProvider?: SemanticJudgeProvider; + } = {}, ): Promise { const { schema, schemaRefRegistry } = context; - const result = await validateSpace(context, { depth: options.depth, includeGated: options.includeGated }); + const result = await validateSpace(context, { + depth: options.depth, + includeGated: options.includeGated, + semantic: options.semantic, + ruleFilter: options.ruleFilter && parseRuleFilter(options.ruleFilter), + judgeProvider: options.judgeProvider, + }); const ruleCounts: Record = { error: 0, warning: 0, info: 0 }; for (const v of result.ruleViolations) ruleCounts[v.severity]++; @@ -46,7 +61,10 @@ export async function validate( } for (const v of result.ruleViolations) { if (v.file && v.severity === 'error') { - addError(v.file, `rule:${v.ruleId}`, 'rule', `[${v.ruleId}] ${v.description}`, v.severity); + const message = v.evidence + ? `[${v.ruleId}] ${v.description} (Evidence: ${v.evidence})` + : `[${v.ruleId}] ${v.description}`; + addError(v.file, `rule:${v.ruleId}`, 'rule', message, v.severity); } } const globalRuleErrorCount = result.ruleViolations.filter((v) => !v.file && v.severity === 'error').length; @@ -74,6 +92,7 @@ export async function validate( coverageGates: result.coverageGates, skippedByDepth: result.skippedByDepth, depthLimit: result.depthLimit, + semanticEvaluations: result.semanticEvaluations, }, null, 2, @@ -193,10 +212,24 @@ export async function validate( console.log(` ${severityColor[severity]}${severity.toUpperCase()}${reset} (${violations.length}):`); for (const v of violations) { console.log(` ${v.file ? `${v.file}: ` : ''}${v.description} [${v.category}: ${v.ruleId}]`); + if (v.evidence) { + console.log(` Evidence: ${v.evidence}`); + } } } } + if (result.semanticEvaluations && result.semanticEvaluations.length > 0) { + const passed = result.semanticEvaluations.filter((e) => e.pass).length; + const failed = result.semanticEvaluations.filter((e) => !e.pass).length; + console.log( + `\nSemantic evaluations: ${result.semanticEvaluations.length} total (${passed} passed, ${failed} failed)`, + ); + } + if (result.gatedRules && result.gatedRules.length > 0) { + console.log(`Gated semantic rules (skipped due to upstream errors): ${result.gatedRules.length}`); + } + if (result.hierarchyViolations.length > 0) { console.log(`\nHierarchy violations:`); for (const v of result.hierarchyViolations) { diff --git a/src/config.ts b/src/config.ts index 07bf9b2..ed21616 100644 --- a/src/config.ts +++ b/src/config.ts @@ -4,6 +4,7 @@ import { dirname, isAbsolute, join, resolve } from 'node:path'; import Ajv2020 from 'ajv/dist/2020'; import JSON5 from 'json5'; import { ENV_CONFIG_VAR, XDG_CONFIG_DIR } from './constants'; +import type { JudgeConfig } from './judge/types'; import { normalizePluginName, resolvePluginSchemasDir, @@ -12,6 +13,18 @@ import { } from './plugins/util'; import { bundledSchemasDir } from './schema/schema'; +const JUDGE_CONFIG_SCHEMA = { + type: 'object', + properties: { + provider: { type: 'string', enum: ['anthropic', 'openai', 'fake'] }, + model: { type: 'string' }, + apiKey: { type: 'string' }, + endpoint: { type: 'string' }, + autoRun: { type: 'boolean' }, + }, + additionalProperties: false, +} as const; + const CONFIG_SCHEMA = { type: 'object', properties: { @@ -33,6 +46,8 @@ const CONFIG_SCHEMA = { additionalProperties: false, }, }, + judge: JUDGE_CONFIG_SCHEMA, + semantic: { type: 'boolean' }, }, required: ['name', 'path'], additionalProperties: false, @@ -40,6 +55,8 @@ const CONFIG_SCHEMA = { }, schema: { type: 'string' }, includeSpacesFrom: { type: 'array', items: { type: 'string' } }, + judge: JUDGE_CONFIG_SCHEMA, + semantic: { type: 'boolean' }, }, anyOf: [{ required: ['spaces'] }, { required: ['includeSpacesFrom'] }], additionalProperties: false, @@ -53,6 +70,10 @@ export type SpaceConfig = { plugins?: Record>; /** Named filter views for this space. Keys are view names; values contain the filter expression. */ views?: Record; + /** LLM judge provider configuration for semantic rule evaluation. */ + judge?: JudgeConfig; + /** Whether to evaluate semantic rules during validation for this space. */ + semantic?: boolean; /** * Directory of the config file that defined this space, used to anchor relative plugin-config * paths. Set by loadConfig; absent for a hand-assembled Config, whose callers must pass an @@ -65,6 +86,10 @@ export type Config = { spaces: SpaceConfig[]; schema?: string; includeSpacesFrom?: string[]; + /** Default LLM judge provider configuration for semantic rule evaluation. */ + judge?: JudgeConfig; + /** Whether to evaluate semantic rules during validation across spaces. */ + semantic?: boolean; }; let _configPathOverride: string | undefined; diff --git a/src/index.ts b/src/index.ts index fbe3229..9b571df 100755 --- a/src/index.ts +++ b/src/index.ts @@ -61,8 +61,19 @@ program '--include-gated', 'Evaluate rules even when the file is gated by an upstream schema/hierarchy/rule failure (still reported as gated)', ) + .option( + '--rule-filter ', + 'Evaluate only matching rules; dimensions: mode, category, severity, type, id (repeatable, e.g. mode=semantic)', + (v: string, prev: string[] = []) => [...prev, v], + ) + .option('--semantic', 'Evaluate semantic rules using built-in LLM judge') .action(async (filePath, options) => { - process.exitCode = await validateFileCommand(filePath, { json: options.json, includeGated: options.includeGated }); + process.exitCode = await validateFileCommand(filePath, { + json: options.json, + includeGated: options.includeGated, + semantic: options.semantic, + ruleFilter: options.ruleFilter, + }); }); program @@ -80,6 +91,12 @@ program '--include-gated', 'Evaluate rules even on nodes gated by an upstream schema/hierarchy/rule failure (still reported as gated)', ) + .option( + '--rule-filter ', + 'Evaluate only matching rules; dimensions: mode, category, severity, type, id (repeatable, e.g. mode=semantic)', + (v: string, prev: string[] = []) => [...prev, v], + ) + .option('--semantic', 'Evaluate semantic rules using built-in LLM judge') .action(async (spaceName, options) => { const context = buildSpaceContext(spaceName); if (options.watch) { @@ -90,6 +107,8 @@ program json: options.json, depth: options.depth, includeGated: options.includeGated, + semantic: options.semantic, + ruleFilter: options.ruleFilter, }); } }); diff --git a/src/judge/anthropic.ts b/src/judge/anthropic.ts new file mode 100644 index 0000000..9c581ad --- /dev/null +++ b/src/judge/anthropic.ts @@ -0,0 +1,55 @@ +import { PerRuleJudgeProvider } from './per-rule'; +import { buildJudgeSystemPrompt, buildJudgeUserPrompt, parseJudgeResponse } from './prompt'; +import type { SemanticJudgeInput, SemanticJudgeResult, SemanticRuleSpec } from './types'; + +export interface AnthropicJudgeProviderOptions { + apiKey: string; + model?: string; + endpoint?: string; +} + +export const DEFAULT_ANTHROPIC_JUDGE_MODEL = 'claude-haiku-5-5'; + +/** Built-in judge provider for the Anthropic Messages API. */ +export class AnthropicJudgeProvider extends PerRuleJudgeProvider { + readonly name = 'anthropic'; + private apiKey: string; + private model: string; + private endpoint: string; + + constructor(options: AnthropicJudgeProviderOptions) { + super(); + this.apiKey = options.apiKey; + this.model = options.model || process.env.ANTHROPIC_MODEL || DEFAULT_ANTHROPIC_JUDGE_MODEL; + this.endpoint = options.endpoint || 'https://api.anthropic.com/v1/messages'; + } + + protected async judgeRule(node: SemanticJudgeInput['node'], rule: SemanticRuleSpec): Promise { + const res = await fetch(this.endpoint, { + method: 'POST', + headers: { + 'content-type': 'application/json', + 'x-api-key': this.apiKey, + 'anthropic-version': '2023-06-01', + }, + body: JSON.stringify({ + model: this.model, + max_tokens: 1024, + system: buildJudgeSystemPrompt(), + messages: [{ role: 'user', content: buildJudgeUserPrompt(node, rule) }], + }), + }); + + if (!res.ok) { + const errorText = await res.text().catch(() => ''); + throw new Error(`Anthropic API error (${res.status}): ${errorText}`); + } + + const data = (await res.json()) as { content?: Array<{ type: string; text?: string }> }; + const text = data.content?.find((c) => c.type === 'text')?.text; + if (!text) { + throw new Error('Anthropic API returned empty or missing text block in response.'); + } + return parseJudgeResponse(text, Object.keys(rule.verdicts)); + } +} diff --git a/src/judge/factory.ts b/src/judge/factory.ts new file mode 100644 index 0000000..7f17a5c --- /dev/null +++ b/src/judge/factory.ts @@ -0,0 +1,70 @@ +import { AnthropicJudgeProvider } from './anthropic'; +import { FakeJudgeProvider } from './fake'; +import { JevJudgeProvider } from './jev'; +import { OpenAIJudgeProvider } from './openai'; +import type { JudgeConfig, SemanticJudgeProvider } from './types'; + +/** + * Instantiate a semantic judge provider based on configuration and environment variables. + * + * Precedence: + * 1. Explicitly passed customProvider (if provided) + * 2. config.provider or SCTX_JUDGE_PROVIDER === 'fake' -> FakeJudgeProvider + * 3. config.provider === 'jev' -> JevJudgeProvider (OPENROUTER_API_KEY; always explicit, never auto-detected) + * 4. config.provider === 'anthropic' or ANTHROPIC_API_KEY present -> AnthropicJudgeProvider + * 5. config.provider === 'openai' or OPENAI_API_KEY/OPENAI_BASE_URL present -> OpenAIJudgeProvider + * + * Throws a clear, actionable error if no provider can be resolved. + */ +export function createJudgeProvider( + config?: JudgeConfig, + env: Record = process.env, + customProvider?: SemanticJudgeProvider, +): SemanticJudgeProvider { + if (customProvider) { + return customProvider; + } + + const requestedProvider = config?.provider || env.SCTX_JUDGE_PROVIDER; + const apiKey = config?.apiKey || env.SCTX_JUDGE_API_KEY; + const endpoint = config?.endpoint || env.SCTX_JUDGE_ENDPOINT; + const model = config?.model || env.SCTX_JUDGE_MODEL; + + if (requestedProvider === 'fake') { + return new FakeJudgeProvider(); + } + + if (requestedProvider === 'jev') { + const key = apiKey || env.OPENROUTER_API_KEY; + if (!key) { + throw new Error( + 'Jev judge provider selected, but no API key was found. Set OPENROUTER_API_KEY in the environment or configure apiKey in config.json.', + ); + } + return new JevJudgeProvider({ apiKey: key, model, endpoint }); + } + + // Anthropic resolution + const hasAnthropicKey = Boolean(apiKey || env.ANTHROPIC_API_KEY); + if (requestedProvider === 'anthropic' || (!requestedProvider && hasAnthropicKey)) { + const resolvedKey = apiKey || env.ANTHROPIC_API_KEY; + if (!resolvedKey) { + throw new Error( + 'Anthropic judge provider selected, but no API key was found. Set ANTHROPIC_API_KEY in the environment or configure apiKey in config.json.', + ); + } + return new AnthropicJudgeProvider({ apiKey: resolvedKey, model, endpoint }); + } + + // OpenAI resolution + const hasOpenAiKey = Boolean(apiKey || env.OPENAI_API_KEY); + const hasOpenAiBaseUrl = Boolean(endpoint || env.OPENAI_BASE_URL || env.OPENAI_ENDPOINT); + if (requestedProvider === 'openai' || (!requestedProvider && (hasOpenAiKey || hasOpenAiBaseUrl))) { + const resolvedKey = apiKey || env.OPENAI_API_KEY; + return new OpenAIJudgeProvider({ apiKey: resolvedKey, model, endpoint }); + } + + throw new Error( + 'No semantic judge provider configured. Set ANTHROPIC_API_KEY or OPENAI_API_KEY in the environment, or configure "judge" in config.json.', + ); +} diff --git a/src/judge/fake.ts b/src/judge/fake.ts new file mode 100644 index 0000000..103abf2 --- /dev/null +++ b/src/judge/fake.ts @@ -0,0 +1,69 @@ +import type { SemanticJudgeInput, SemanticJudgeProvider, SemanticJudgeResult, SemanticRuleSpec } from './types'; + +/** A fake answer: a full verdict, or the `{ pass }` shorthand mapped onto the default `pass`/`fail` verdicts. */ +export type FakeAnswer = SemanticJudgeResult | { pass: boolean; evidence?: string }; + +export type FakeJudgeHandler = (call: { + node: SemanticJudgeInput['node']; + rule: SemanticRuleSpec; +}) => Promise | FakeAnswer; + +type FakeConfig = FakeAnswer | FakeJudgeHandler | Map | Record; + +function isHandler(c: FakeConfig): c is FakeJudgeHandler { + return typeof c === 'function'; +} + +function isAnswer(c: FakeConfig): c is FakeAnswer { + return 'pass' in c || 'verdict' in c || 'error' in c; +} + +function normalise(answer: FakeAnswer): SemanticJudgeResult { + if ('pass' in answer) { + return { verdict: answer.pass ? 'pass' : 'fail', evidence: answer.evidence }; + } + return answer; +} + +/** + * In-memory fake judge for offline, deterministic tests. Configure with a single answer, a handler, + * or answers keyed by `ruleId:title`, `ruleId:label` or `ruleId` (most specific wins). Unmatched calls pass. + */ +export class FakeJudgeProvider implements SemanticJudgeProvider { + readonly name = 'fake'; + public callHistory: SemanticJudgeInput[] = []; + + constructor(private config?: FakeConfig) {} + + async evaluate(input: SemanticJudgeInput): Promise> { + this.callHistory.push(input); + const results: Record = {}; + for (const rule of input.rules) { + results[rule.id] = normalise(await this.answer(input.node, rule)); + } + return results; + } + + private async answer(node: SemanticJudgeInput['node'], rule: SemanticRuleSpec): Promise { + const passing = { verdict: Object.keys(rule.verdicts).find((v) => !rule.failOn.includes(v)) ?? 'pass' }; + const fallback = { ...passing, evidence: `Fake passed evaluation for rule "${rule.id}"` }; + const config = this.config; + if (!config) return fallback; + if (isHandler(config)) return await config({ node, rule }); + if (config instanceof Map) { + return ( + config.get(`${rule.id}:${node.title}`) ?? + config.get(`${rule.id}:${node.label}`) ?? + config.get(rule.id) ?? + fallback + ); + } + if (isAnswer(config)) return config; + const rec = config as Record; + return rec[`${rule.id}:${node.title}`] ?? rec[`${rule.id}:${node.label}`] ?? rec[rule.id] ?? fallback; + } + + clearHistory(): void { + this.callHistory = []; + } +} diff --git a/src/judge/index.ts b/src/judge/index.ts new file mode 100644 index 0000000..58a1088 --- /dev/null +++ b/src/judge/index.ts @@ -0,0 +1,22 @@ +export type { AnthropicJudgeProviderOptions } from './anthropic'; +export { AnthropicJudgeProvider } from './anthropic'; +export { createJudgeProvider } from './factory'; +export type { FakeAnswer, FakeJudgeHandler } from './fake'; +export { FakeJudgeProvider } from './fake'; +export type { JevJudgeProviderOptions } from './jev'; +export { JevJudgeProvider } from './jev'; +export type { OpenAIJudgeProviderOptions } from './openai'; +export { OpenAIJudgeProvider } from './openai'; +export { PerRuleJudgeProvider } from './per-rule'; +export { buildJudgeSystemPrompt, buildJudgeUserPrompt, parseJudgeResponse } from './prompt'; +export type { + GatedRuleRecord, + JudgeConfig, + SemanticEvaluationRecord, + SemanticJudgeInput, + SemanticJudgeProvider, + SemanticJudgeResult, + SemanticJudgeVerdict, + SemanticRuleSpec, +} from './types'; +export { DEFAULT_VERDICTS, isJudgeError, semanticRuleSpec } from './verdicts'; diff --git a/src/judge/jev.ts b/src/judge/jev.ts new file mode 100644 index 0000000..46d63f6 --- /dev/null +++ b/src/judge/jev.ts @@ -0,0 +1,94 @@ +import type { SemanticJudgeInput, SemanticJudgeProvider, SemanticJudgeResult } from './types'; + +export interface JevJudgeProviderOptions { + apiKey: string; + model?: string; + endpoint?: string; +} + +export const DEFAULT_JEV_MODEL = 'typesafe/jev-1.13'; +export const DEFAULT_JEV_ENDPOINT = 'https://openrouter.ai/api/alpha/decisions'; + +type JevChoiceAnswer = { + type?: string; + choice?: string; + confidence?: number; + probabilities?: Record; +}; + +/** + * Judge provider for TypeSafe's Jev decision model via OpenRouter's Decisions API. + * + * Each node is sent once as `state`, with one `choice` question per semantic rule whose criteria are + * the rule's closed verdict set. Jev returns the chosen verdict with confidence and probabilities, + * but no prose, so `evidence` is a summary of the distribution. + */ +export class JevJudgeProvider implements SemanticJudgeProvider { + readonly name = 'jev'; + private apiKey: string; + private model: string; + private endpoint: string; + + constructor(options: JevJudgeProviderOptions) { + this.apiKey = options.apiKey; + this.model = options.model || DEFAULT_JEV_MODEL; + this.endpoint = options.endpoint || DEFAULT_JEV_ENDPOINT; + } + + async evaluate(input: SemanticJudgeInput): Promise> { + const { node, rules } = input; + const questions: Record = {}; + for (const rule of rules) { + questions[rule.id] = { + type: 'choice', + instructions: rule.rubric || rule.description, + criteria: rule.verdicts, + }; + } + + const res = await fetch(this.endpoint, { + method: 'POST', + headers: { 'content-type': 'application/json', authorization: `Bearer ${this.apiKey}` }, + body: JSON.stringify({ + model: this.model, + state: { + title: node.title, + type: node.resolvedType, + parents: node.resolvedParents?.map((p) => p.title) ?? [], + ...node.schemaData, + }, + questions, + }), + }); + + if (!res.ok) { + const errorText = await res.text().catch(() => ''); + throw new Error(`Jev API error (${res.status}): ${errorText}`); + } + + const data = (await res.json()) as { answers?: Record }; + const results: Record = {}; + for (const rule of rules) { + const answer = data.answers?.[rule.id]; + if (!answer?.choice || !(answer.choice in rule.verdicts)) { + results[rule.id] = { error: `Jev returned no valid verdict for rule "${rule.id}" (choice: ${answer?.choice})` }; + continue; + } + results[rule.id] = { + verdict: answer.choice, + confidence: answer.confidence, + probabilities: answer.probabilities, + evidence: describeDistribution(answer), + }; + } + return results; + } +} + +function describeDistribution(answer: JevChoiceAnswer): string { + const parts = Object.entries(answer.probabilities ?? {}) + .sort(([, a], [, b]) => b - a) + .map(([verdict, p]) => `${verdict} ${p.toFixed(2)}`); + const confidence = answer.confidence !== undefined ? `confidence ${answer.confidence.toFixed(2)}` : undefined; + return `Jev chose "${answer.choice}"${[confidence, ...parts].filter(Boolean).length ? ` (${[confidence, ...parts].filter(Boolean).join('; ')})` : ''}`; +} diff --git a/src/judge/judge-node.ts b/src/judge/judge-node.ts new file mode 100644 index 0000000..8c17af2 --- /dev/null +++ b/src/judge/judge-node.ts @@ -0,0 +1,68 @@ +import type { Rule, SpaceNode } from '../types'; +import type { SemanticEvaluationRecord, SemanticJudgeProvider } from './types'; +import { isJudgeError, semanticRuleSpec } from './verdicts'; + +export type NodeJudgement = { + rule: Rule; + /** True when the verdict is one of the rule's `failOn` verdicts, or the judge errored. */ + failed: boolean; + evidence?: string; + /** Set when the judge could not produce a verdict for this rule. */ + error?: string; + /** Present when the judge produced a verdict. */ + record?: SemanticEvaluationRecord; +}; + +/** Judge every given semantic rule on a node in one provider call. */ +export async function judgeNode( + judge: SemanticJudgeProvider, + node: SpaceNode, + rules: Rule[], +): Promise { + if (rules.length === 0) return []; + const specs = rules.map(semanticRuleSpec); + let results: Awaited>; + try { + results = await judge.evaluate({ + node: { + label: node.label, + title: node.title, + type: node.type, + resolvedType: node.resolvedType, + schemaData: node.schemaData, + resolvedParents: node.resolvedParents.map((p) => ({ title: p.title, field: p.field })), + }, + rules: specs, + }); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + return rules.map((rule) => ({ + rule, + failed: true, + evidence: `Error during evaluation: ${message}`, + error: message, + })); + } + + return rules.map((rule, i) => { + const result = results[rule.id]; + if (!result || isJudgeError(result)) { + const message = result ? result.error : 'Judge returned no result for this rule'; + return { rule, failed: true, evidence: `Error during evaluation: ${message}`, error: message }; + } + const failed = specs[i]!.failOn.includes(result.verdict); + return { + rule, + failed, + evidence: result.evidence, + record: { + file: node.label, + ruleId: rule.id, + verdict: result.verdict, + pass: !failed, + confidence: result.confidence, + evidence: result.evidence, + }, + }; + }); +} diff --git a/src/judge/openai.ts b/src/judge/openai.ts new file mode 100644 index 0000000..fe044e1 --- /dev/null +++ b/src/judge/openai.ts @@ -0,0 +1,68 @@ +import { PerRuleJudgeProvider } from './per-rule'; +import { buildJudgeSystemPrompt, buildJudgeUserPrompt, parseJudgeResponse } from './prompt'; +import type { SemanticJudgeInput, SemanticJudgeResult, SemanticRuleSpec } from './types'; + +export interface OpenAIJudgeProviderOptions { + apiKey?: string; + model?: string; + endpoint?: string; +} + +/** + * Built-in judge provider for OpenAI-compatible Chat Completions API. + * Supports OpenAI, Ollama, vLLM, Groq, OpenRouter, and other compatible endpoints. + */ +export class OpenAIJudgeProvider extends PerRuleJudgeProvider { + readonly name = 'openai'; + private apiKey?: string; + private model: string; + private endpoint: string; + + constructor(options: OpenAIJudgeProviderOptions) { + super(); + this.apiKey = options.apiKey; + this.model = options.model || process.env.OPENAI_MODEL || 'gpt-4o-mini'; + + const base = + options.endpoint || process.env.OPENAI_BASE_URL || process.env.OPENAI_ENDPOINT || 'https://api.openai.com/v1'; + this.endpoint = base.endsWith('/chat/completions') ? base : `${base.replace(/\/+$/, '')}/chat/completions`; + } + + protected async judgeRule(node: SemanticJudgeInput['node'], rule: SemanticRuleSpec): Promise { + const systemPrompt = buildJudgeSystemPrompt(); + const userPrompt = buildJudgeUserPrompt(node, rule); + + const headers: Record = { + 'content-type': 'application/json', + }; + if (this.apiKey) { + headers.authorization = `Bearer ${this.apiKey}`; + } + + const res = await fetch(this.endpoint, { + method: 'POST', + headers, + body: JSON.stringify({ + model: this.model, + response_format: { type: 'json_object' }, + messages: [ + { role: 'system', content: systemPrompt }, + { role: 'user', content: userPrompt }, + ], + }), + }); + + if (!res.ok) { + const errorText = await res.text().catch(() => ''); + throw new Error(`OpenAI API error (${res.status}): ${errorText}`); + } + + const data = (await res.json()) as { choices?: Array<{ message?: { content?: string } }> }; + const content = data.choices?.[0]?.message?.content; + if (!content) { + throw new Error('OpenAI API returned empty or missing message content in response.'); + } + + return parseJudgeResponse(content, Object.keys(rule.verdicts)); + } +} diff --git a/src/judge/per-rule.ts b/src/judge/per-rule.ts new file mode 100644 index 0000000..244d670 --- /dev/null +++ b/src/judge/per-rule.ts @@ -0,0 +1,23 @@ +import type { SemanticJudgeInput, SemanticJudgeProvider, SemanticJudgeResult, SemanticRuleSpec } from './types'; + +/** + * Base for providers that can only judge one rule per request (chat LLMs). `evaluate` makes one call + * per rule; a failure on one rule is reported for that rule and does not affect the others. + */ +export abstract class PerRuleJudgeProvider implements SemanticJudgeProvider { + abstract readonly name: string; + + protected abstract judgeRule(node: SemanticJudgeInput['node'], rule: SemanticRuleSpec): Promise; + + async evaluate(input: SemanticJudgeInput): Promise> { + const results: Record = {}; + for (const rule of input.rules) { + try { + results[rule.id] = await this.judgeRule(input.node, rule); + } catch (err) { + results[rule.id] = { error: err instanceof Error ? err.message : String(err) }; + } + } + return results; + } +} diff --git a/src/judge/prompt.ts b/src/judge/prompt.ts new file mode 100644 index 0000000..1d62876 --- /dev/null +++ b/src/judge/prompt.ts @@ -0,0 +1,92 @@ +import type { SemanticJudgeInput, SemanticJudgeResult, SemanticRuleSpec } from './types'; + +function verdictList(rule: SemanticRuleSpec): string[] { + return Object.entries(rule.verdicts).map(([name, description]) => `- "${name}": ${description}`); +} + +/** System prompt: a strict judge that answers with one verdict from a closed set, as JSON. */ +export function buildJudgeSystemPrompt(): string { + return [ + 'You are a strict, objective evaluation judge for structured planning and knowledge documents.', + 'Your task is to choose the one verdict that best describes whether a node satisfies a rule.', + '', + 'You must respond ONLY with a JSON object in this exact format:', + '{', + ' "verdict": "",', + ' "confidence": ,', + ' "evidence": "concise explanation citing specific quotes, phrases, or absent details from the node"', + '}', + '', + 'Guidelines:', + '1. "verdict" must be one of the allowed verdicts listed for the rule, spelled exactly.', + '2. "confidence" is how sure you are of the verdict.', + '3. "evidence" must directly cite or quote relevant parts of the node, explaining the reasoning concisely.', + '4. Do not include markdown code block formatting, intro text, or outro text. Respond only with the JSON object.', + ].join('\n'); +} + +/** User prompt for one rule on one node. */ +export function buildJudgeUserPrompt(node: SemanticJudgeInput['node'], rule: SemanticRuleSpec): string { + const rubric = rule.rubric || rule.description; + const lines: string[] = [ + '# Rule to Evaluate', + `- ID: ${rule.id}`, + `- Category: ${rule.category}`, + `- Rubric: ${rubric}`, + '', + '## Allowed verdicts', + ...verdictList(rule), + '', + '# Node Under Evaluation', + `- File / Label: ${node.label}`, + `- Title: ${node.title}`, + `- Type: ${node.type} (resolved: ${node.resolvedType})`, + ]; + + if (node.resolvedParents && node.resolvedParents.length > 0) { + lines.push(`- Parents: ${node.resolvedParents.map((p) => p.title).join(', ')}`); + } + + lines.push('', '## Node Content & Frontmatter:', '```json', JSON.stringify(node.schemaData, null, 2), '```', ''); + lines.push('Evaluate this node against the rule. Return only JSON with "verdict", "confidence" and "evidence".'); + return lines.join('\n'); +} + +/** + * Parse judge output into a verdict from the rule's allowed set. + * Handles clean JSON, markdown-wrapped JSON, and a regex fallback for surrounding text. + */ +export function parseJudgeResponse(rawText: string, allowed: string[]): SemanticJudgeResult { + let cleaned = rawText.trim(); + cleaned = cleaned.replace(/^```(?:json)?\s*/, '').replace(/\s*```$/, ''); + + let verdict: string | undefined; + let confidence: number | undefined; + let evidence: string | undefined; + + try { + const parsed = JSON.parse(cleaned); + if (typeof parsed.verdict === 'string') verdict = parsed.verdict.trim(); + if (typeof parsed.confidence === 'number') confidence = parsed.confidence; + if (typeof parsed.evidence === 'string') evidence = parsed.evidence.trim(); + } catch { + const verdictMatch = /"verdict"\s*:\s*"((?:[^"\\]|\\.)*)"/i.exec(cleaned); + const confidenceMatch = /"confidence"\s*:\s*([0-9.]+)/i.exec(cleaned); + const evidenceMatch = /"evidence"\s*:\s*"((?:[^"\\]|\\.)*)"/i.exec(cleaned); + verdict = verdictMatch?.[1]?.trim(); + confidence = confidenceMatch ? Number(confidenceMatch[1]) : undefined; + evidence = evidenceMatch?.[1]?.replace(/\\"/g, '"').replace(/\\n/g, '\n').trim(); + } + + if (verdict === undefined || !allowed.includes(verdict)) { + return { + error: `Judge returned ${verdict === undefined ? 'no verdict' : `unknown verdict "${verdict}"`} (allowed: ${allowed.join(', ')}): ${rawText.slice(0, 200)}`, + }; + } + return { + verdict, + ...(confidence !== undefined && + Number.isFinite(confidence) && { confidence: Math.min(1, Math.max(0, confidence)) }), + ...(evidence && { evidence }), + }; +} diff --git a/src/judge/types.ts b/src/judge/types.ts new file mode 100644 index 0000000..ef62826 --- /dev/null +++ b/src/judge/types.ts @@ -0,0 +1,73 @@ +import type { RuleCategory } from '../types'; + +/** A semantic rule as presented to a judge: the closed verdict set it must choose from. */ +export interface SemanticRuleSpec { + id: string; + description: string; + rubric?: string; + category: RuleCategory | string; + /** Allowed verdicts, each with a description of when it applies. */ + verdicts: Record; + /** Verdicts that count as a violation. */ + failOn: string[]; +} + +/** One node and every semantic rule to judge on it. Judges may answer all rules in a single call. */ +export interface SemanticJudgeInput { + node: { + label: string; + title: string; + type: string; + resolvedType: string; + schemaData: Record; + resolvedParents?: Array<{ title: string; field: string }>; + }; + rules: SemanticRuleSpec[]; +} + +/** A judge's answer for one rule: a verdict from the rule's closed set. */ +export interface SemanticJudgeVerdict { + verdict: string; + /** 0-1, when the judge reports one. */ + confidence?: number; + /** Probability per verdict, when the judge reports a distribution. */ + probabilities?: Record; + /** Free-text reasoning; decision models may not provide it. */ + evidence?: string; +} + +/** A judge's answer for one rule: a verdict, or an error that applies to that rule only. */ +export type SemanticJudgeResult = SemanticJudgeVerdict | { error: string }; + +/** Judge interface. Results are keyed by rule id; a missing key is treated as an error. */ +export interface SemanticJudgeProvider { + readonly name: string; + evaluate(input: SemanticJudgeInput): Promise>; +} + +/** Judge configuration from config file or options. */ +export interface JudgeConfig { + provider?: 'anthropic' | 'openai' | 'jev' | 'fake' | string; + model?: string; + apiKey?: string; + endpoint?: string; + autoRun?: boolean; +} + +/** Record of a completed semantic evaluation. */ +export interface SemanticEvaluationRecord { + file: string; + ruleId: string; + verdict: string; + /** True when the verdict is not one of the rule's `failOn` verdicts. */ + pass: boolean; + confidence?: number; + evidence?: string; +} + +/** Record of a rule gated/suppressed due to upstream failure. */ +export interface GatedRuleRecord { + file: string; + ruleId: string; + reason: string; +} diff --git a/src/judge/verdicts.ts b/src/judge/verdicts.ts new file mode 100644 index 0000000..6e06c76 --- /dev/null +++ b/src/judge/verdicts.ts @@ -0,0 +1,39 @@ +import type { Rule } from '../types'; +import type { SemanticJudgeResult, SemanticRuleSpec } from './types'; + +/** Verdicts for a semantic rule that declares none. */ +export const DEFAULT_VERDICTS: Record = { + pass: 'The node meets the rubric.', + fail: 'The node does not meet the rubric.', +}; + +/** Build the judge-facing spec for a semantic rule, applying verdict defaults and checking `failOn`. */ +export function semanticRuleSpec(rule: Rule): SemanticRuleSpec { + const verdicts = rule.verdicts ?? DEFAULT_VERDICTS; + const failOn = rule.failOn ?? (rule.verdicts ? [] : ['fail']); + if (failOn.length === 0) { + throw new Error( + `Semantic rule "${rule.id}" declares verdicts but no failOn; list the verdicts that count as a violation.`, + ); + } + const unknown = failOn.filter((v) => !(v in verdicts)); + if (unknown.length > 0) { + throw new Error(`Semantic rule "${rule.id}" has failOn verdicts not in its verdicts: ${unknown.join(', ')}.`); + } + if (failOn.length >= Object.keys(verdicts).length) { + throw new Error(`Semantic rule "${rule.id}" has no passing verdict: every verdict is in failOn.`); + } + return { + id: rule.id, + description: rule.description, + rubric: rule.rubric, + category: rule.category, + verdicts, + failOn, + }; +} + +/** Whether a judge result is an error rather than a verdict. */ +export function isJudgeError(result: SemanticJudgeResult): result is { error: string } { + return 'error' in result; +} diff --git a/src/schema/gating.ts b/src/schema/gating.ts index 3bb416a..576fcec 100644 --- a/src/schema/gating.ts +++ b/src/schema/gating.ts @@ -1,3 +1,5 @@ +import { judgeNode } from '../judge/judge-node'; +import type { GatedRuleRecord, SemanticEvaluationRecord, SemanticJudgeProvider } from '../judge/types'; import type { SpaceGraph } from '../space-graph'; import type { Rule, RuleViolation, SpaceNode } from '../types'; import { buildEvalContext, evaluateExpression } from './evaluate-rule'; @@ -41,8 +43,19 @@ export type CoverageGate = { export type GatedRuleEvaluation = { violations: RuleViolation[]; gates: CoverageGate[]; + /** Semantic judgements made (empty unless a judge was supplied). */ + semanticEvaluations: SemanticEvaluationRecord[]; + /** Semantic rule/node pairs skipped because the node was gated (empty unless a judge was supplied). */ + gatedRules: GatedRuleRecord[]; }; +export interface GatingOptions { + /** Evaluate rules on gated nodes anyway (gates are still reported). */ + includeGated?: boolean; + /** Judge for `mode: semantic` rules. Without it, semantic rules are not evaluated. */ + judge?: SemanticJudgeProvider; +} + /** * Compute breadth-first depth (1-based, from hierarchy roots and orphans) for every * hierarchy-typed node in the graph. Non-hierarchy nodes are not included. @@ -81,7 +94,7 @@ export async function evaluateRulesWithGating( rules: Rule[], graph: SpaceGraph, seeds: GatingSeed[], - options: { includeGated?: boolean } = {}, + options: GatingOptions = {}, ): Promise { const includeGated = options.includeGated ?? false; @@ -98,7 +111,10 @@ export async function evaluateRulesWithGating( } } + const judge = options.judge; const violations: RuleViolation[] = []; + const semanticEvaluations: SemanticEvaluationRecord[] = []; + const gatedRules: GatedRuleRecord[] = []; const gateOrigins = new Map(); function originFor( @@ -113,12 +129,12 @@ export async function evaluateRulesWithGating( return origin; } - const globalRules = rules.filter((r) => r.scope === 'global'); + const globalRules = rules.filter((r) => r.scope === 'global' && r.mode !== 'semantic'); const sentinel = nodes[0]; if (sentinel) { for (const rule of globalRules) { const context = buildEvalContext(sentinel, nodes, nodeIndex); - const result = await evaluateExpression(rule.check, context); + const result = await evaluateExpression(rule.check as string, context); if (result !== true) { violations.push({ file: '', @@ -131,30 +147,51 @@ export async function evaluateRulesWithGating( } } - const perNodeRules = rules.filter((r) => r.scope !== 'global'); + const perNodeRules = rules.filter((r) => r.scope !== 'global' && (r.mode !== 'semantic' || judge)); + + function applicableRules(node: SpaceNode): Rule[] { + return perNodeRules.filter((r) => !r.type || r.type === node.resolvedType); + } async function evaluateApplicableRules(node: SpaceNode): Promise<{ blockingRuleId?: string }> { let blockingRuleId: string | undefined; - const applicable = perNodeRules.filter((r) => !r.type || r.type === node.resolvedType); - for (const rule of applicable) { - const context = buildEvalContext(node, nodes, nodeIndex); - const result = await evaluateExpression(rule.check, context); - if (result !== true) { - violations.push({ - file: node.label, - ruleId: rule.id, - category: rule.category, - severity: ruleSeverity(rule), - description: rule.description, - }); - if (ruleGatesDescendants(rule) && !blockingRuleId) { - blockingRuleId = rule.id; - } + const applicable = applicableRules(node); + + function report(rule: Rule, evidence?: string, error?: string): void { + violations.push({ + file: node.label, + ruleId: rule.id, + category: rule.category, + severity: ruleSeverity(rule), + description: error ? `${rule.description} (Judge error: ${error})` : rule.description, + ...(evidence !== undefined && { evidence }), + }); + // A judge error says nothing about the node, so it never gates. + if (!error && ruleGatesDescendants(rule) && !blockingRuleId) blockingRuleId = rule.id; + } + + // Cheap deterministic rules first; the judge is only consulted afterwards, once per node. + for (const rule of applicable.filter((r) => r.mode !== 'semantic')) { + const result = await evaluateExpression(rule.check as string, buildEvalContext(node, nodes, nodeIndex)); + if (result !== true) report(rule); + } + + const semantic = applicable.filter((r) => r.mode === 'semantic'); + if (judge && semantic.length > 0) { + for (const j of await judgeNode(judge, node, semantic)) { + if (j.record) semanticEvaluations.push(j.record); + if (j.failed) report(j.rule, j.evidence, j.error); } } return { blockingRuleId }; } + function recordGatedSemantic(node: SpaceNode, reason: string): void { + for (const rule of applicableRules(node)) { + if (rule.mode === 'semantic') gatedRules.push({ file: node.label, ruleId: rule.id, reason }); + } + } + // Non-hierarchy nodes sit outside the subtree cascade — always evaluated, never gated. for (const node of graph.nonHierarchy) { await evaluateApplicableRules(node); @@ -179,6 +216,9 @@ export async function evaluateRulesWithGating( const shouldEvaluate = !isGated || includeGated; const { blockingRuleId } = shouldEvaluate ? await evaluateApplicableRules(node) : {}; + if (!shouldEvaluate) { + recordGatedSemantic(node, isGatedByAncestor ? 'Gated by an ancestor failure' : 'Gated by failure on node'); + } let childOrigin: string | undefined = ancestorOrigin; if (isGatedByAncestor) { @@ -199,5 +239,10 @@ export async function evaluateRulesWithGating( } } - return { violations, gates: [...gateOrigins.values()].filter((g) => g.gatedFiles.length > 0) }; + return { + violations, + gates: [...gateOrigins.values()].filter((g) => g.gatedFiles.length > 0), + semanticEvaluations, + gatedRules, + }; } diff --git a/src/schema/metadata-contract.ts b/src/schema/metadata-contract.ts index 41cc0db..aa50e8d 100644 --- a/src/schema/metadata-contract.ts +++ b/src/schema/metadata-contract.ts @@ -63,13 +63,19 @@ const RULE_SCHEMA = { severity: { enum: ['error', 'warning', 'info'] }, description: { type: 'string', minLength: 1 }, check: { type: 'string', minLength: 1 }, + mode: { enum: ['expression', 'semantic'] }, + rubric: { type: 'string', minLength: 1 }, + /** Semantic rules: the closed set of verdicts a judge may return, each with a description. Defaults to `pass`/`fail`. */ + verdicts: { type: 'object', additionalProperties: { type: 'string', minLength: 1 }, minProperties: 2 }, + /** Semantic rules: which verdicts count as a violation. Defaults to `fail`; every other verdict passes. */ + failOn: { type: 'array', items: { type: 'string', minLength: 1 }, minItems: 1, uniqueItems: true }, type: { type: 'string', minLength: 1 }, scope: { enum: ['global'] }, override: { type: 'boolean' }, /** Whether a violation of this rule gates (suppresses) downstream checks on the node's hierarchy subtree. Defaults to true for `error` severity rules and false otherwise. */ gatesDescendants: { type: 'boolean' }, }, - required: ['id', 'category', 'description', 'check'], + required: ['id', 'category', 'description'], additionalProperties: false, } as const; diff --git a/src/schema/rule-filter.ts b/src/schema/rule-filter.ts new file mode 100644 index 0000000..7d58423 --- /dev/null +++ b/src/schema/rule-filter.ts @@ -0,0 +1,55 @@ +import type { Rule } from '../types'; +import { ruleSeverity } from './validate-rules'; + +export const RULE_FILTER_DIMENSIONS = ['mode', 'category', 'severity', 'type', 'id'] as const; +export type RuleFilterDimension = (typeof RULE_FILTER_DIMENSIONS)[number]; + +/** Narrows which rules are evaluated. Dimensions combine with AND; values within a dimension with OR. */ +export type RuleFilter = Partial>; + +/** Parse `dimension=value[,value]` specs (e.g. from repeated `--rule-filter` flags); repeats of a dimension merge. */ +export function parseRuleFilter(specs: string[]): RuleFilter { + const filter: RuleFilter = {}; + for (const spec of specs) { + const eq = spec.indexOf('='); + const dimension = spec.slice(0, eq) as RuleFilterDimension; + const values = spec + .slice(eq + 1) + .split(',') + .map((v) => v.trim()) + .filter(Boolean); + if (eq < 0 || values.length === 0 || !RULE_FILTER_DIMENSIONS.includes(dimension)) { + throw new Error(`Invalid rule filter "${spec}". Use <${RULE_FILTER_DIMENSIONS.join('|')}>=[,...].`); + } + filter[dimension] = [...(filter[dimension] ?? []), ...values]; + } + return filter; +} + +function dimensionValue(rule: Rule, dimension: RuleFilterDimension): string | undefined { + switch (dimension) { + case 'mode': + return rule.mode ?? 'expression'; + case 'severity': + return ruleSeverity(rule); + case 'category': + return rule.category; + case 'type': + return rule.type; + case 'id': + return rule.id; + } +} + +/** Keep the rules matching every dimension of the filter. A rule with no `type` never matches a `type` filter. */ +export function filterRules(rules: Rule[], filter?: RuleFilter): Rule[] { + if (!filter) return rules; + return rules.filter((rule) => + RULE_FILTER_DIMENSIONS.every((dim) => { + const wanted = filter[dim]; + if (!wanted) return true; + const value = dimensionValue(rule, dim); + return value !== undefined && wanted.includes(value); + }), + ); +} diff --git a/src/schema/validate-rules.ts b/src/schema/validate-rules.ts index 25c0786..03b8d35 100644 --- a/src/schema/validate-rules.ts +++ b/src/schema/validate-rules.ts @@ -1,3 +1,5 @@ +import { judgeNode } from '../judge/judge-node'; +import type { GatedRuleRecord, SemanticEvaluationRecord, SemanticJudgeProvider } from '../judge/types'; import type { Rule, RuleCategory, RuleSeverity, RuleViolation, SpaceNode } from '../types'; import { buildEvalContext, evaluateExpression } from './evaluate-rule'; @@ -14,16 +16,40 @@ export function ruleSeverity(rule: Rule): RuleSeverity { return rule.severity ?? DEFAULT_SEVERITY_BY_CATEGORY[rule.category]; } +export interface ValidateRulesOptions { + /** Whether to run semantic rules (default: false). */ + semantic?: boolean; + /** The semantic judge provider to use if semantic is true. */ + judgeProvider?: SemanticJudgeProvider; + /** Set of node labels that have already failed upstream structural checks. */ + gatedNodeLabels?: Set; +} + +export type ValidatedRulesResult = RuleViolation[] & { + semanticEvaluations: SemanticEvaluationRecord[]; + gatedRules: GatedRuleRecord[]; +}; + /** * Validate nodes against rules metadata. - * Returns a list of rule violations found. + * Evaluates deterministic expression rules and (when opt-in) semantic judge rules. * * @param nodes - All nodes in the space * @param rules - Rules metadata list - * @returns Array of rule violations + * @param options - Validation options including semantic opt-in and judge provider + * @returns Array of rule violations with semanticEvaluations and gatedRules metadata attached */ -export async function validateRules(nodes: SpaceNode[], rules: Rule[]): Promise { +export async function validateRules( + nodes: SpaceNode[], + rules: Rule[], + options: ValidateRulesOptions = {}, +): Promise { const violations: RuleViolation[] = []; + const semanticEvaluations: SemanticEvaluationRecord[] = []; + const gatedRules: GatedRuleRecord[] = []; + + // Track nodes that have failed checks (for cascading gating) + const failedNodeLabels = new Set(options.gatedNodeLabels || []); // Build node index for efficient lookups const nodeIndex = new Map(); @@ -34,16 +60,30 @@ export async function validateRules(nodes: SpaceNode[], rules: Rule[]): Promise< } } - // Evaluate each rule against applicable nodes + // Separate rules into expression rules and semantic rules + const expressionRules: Rule[] = []; + const semanticRules: Rule[] = []; + for (const rule of rules) { + if (rule.mode === 'semantic') { + semanticRules.push(rule); + } else if (rule.check) { + expressionRules.push(rule); + } + } + + // 1. Evaluate deterministic expression rules first + for (const rule of expressionRules) { + const check = rule.check as string; const severity = ruleSeverity(rule); + if (rule.scope === 'global') { // Global rules are evaluated once against the full node set. // A sentinel node provides the evaluation context (nodes array is what matters). const sentinel = nodes[0]; if (sentinel) { const context = buildEvalContext(sentinel, nodes, nodeIndex); - const result = await evaluateExpression(rule.check, context); + const result = await evaluateExpression(check, context); if (result !== true) { violations.push({ file: '', @@ -58,7 +98,7 @@ export async function validateRules(nodes: SpaceNode[], rules: Rule[]): Promise< const targetNodes = rule.type ? nodes.filter((n) => n.resolvedType === rule.type) : nodes; for (const node of targetNodes) { const context = buildEvalContext(node, nodes, nodeIndex); - const result = await evaluateExpression(rule.check, context); + const result = await evaluateExpression(check, context); if (result !== true) { violations.push({ file: node.label, @@ -67,10 +107,49 @@ export async function validateRules(nodes: SpaceNode[], rules: Rule[]): Promise< severity, description: rule.description, }); + failedNodeLabels.add(node.label); } } } } - return violations; + // 2. Evaluate semantic rules only if explicitly requested + if (options.semantic) { + const judgeProvider = options.judgeProvider; + if (!judgeProvider) { + throw new Error('Semantic evaluation was requested but no judgeProvider was provided to validateRules.'); + } + + for (const node of nodes) { + const applicable = semanticRules.filter((r) => !r.type || r.type === node.resolvedType); + if (applicable.length === 0) continue; + + // Safe cascading default: gate semantic evaluation if node failed upstream checks + if (failedNodeLabels.has(node.label)) { + for (const rule of applicable) { + gatedRules.push({ file: node.label, ruleId: rule.id, reason: 'Gated by upstream error on node' }); + } + continue; + } + + for (const j of await judgeNode(judgeProvider, node, applicable)) { + if (j.record) semanticEvaluations.push(j.record); + if (!j.failed) continue; + violations.push({ + file: node.label, + ruleId: j.rule.id, + category: j.rule.category, + severity: ruleSeverity(j.rule), + description: j.error ? `${j.rule.description} (Judge error: ${j.error})` : j.rule.description, + evidence: j.evidence, + }); + if (!j.error) failedNodeLabels.add(node.label); + } + } + } + + const result = violations as ValidatedRulesResult; + result.semanticEvaluations = semanticEvaluations; + result.gatedRules = gatedRules; + return result; } diff --git a/src/types.ts b/src/types.ts index 2401a6c..523a62f 100644 --- a/src/types.ts +++ b/src/types.ts @@ -201,6 +201,8 @@ export type RuleViolation = { category: RuleCategory; severity: RuleSeverity; description: string; + /** Cited evidence or reasoning from evaluation (for semantic judge rules). */ + evidence?: string; }; export type GraphViolation = { diff --git a/src/validate.ts b/src/validate.ts index 26bdc62..4ac28d3 100644 --- a/src/validate.ts +++ b/src/validate.ts @@ -1,8 +1,11 @@ import { isAbsolute, relative, resolve } from 'node:path'; import type { ErrorObject } from 'ajv'; import type { Config, SpaceConfig } from './config'; +import { createJudgeProvider } from './judge/factory'; +import type { GatedRuleRecord, JudgeConfig, SemanticEvaluationRecord, SemanticJudgeProvider } from './judge/types'; import { readSpace } from './read/read-space'; import { computeHierarchyDepths, evaluateRulesWithGating, type GatingSeed } from './schema/gating'; +import { filterRules, type RuleFilter } from './schema/rule-filter'; import { extractEntityInfo } from './schema/schema'; import { validateGraph } from './schema/validate-graph'; import { validateRules } from './schema/validate-rules'; @@ -52,6 +55,10 @@ export interface ValidationResult { skippedByDepth: Array<{ file: string; title: string }>; /** The depth limit applied, if any (undefined means the full space was validated). */ depthLimit?: number; + /** Details on semantic rule evaluations performed (if semantic mode enabled). */ + semanticEvaluations?: SemanticEvaluationRecord[]; + /** Semantic rule/node pairs skipped because the node was gated by an upstream failure. */ + gatedRules?: GatedRuleRecord[]; } /** @@ -159,21 +166,71 @@ export function formatErrors( return formatted; } +export interface ValidateSpaceOptions { + readResult?: ReadSpaceResult; + /** Limit validation to hierarchy nodes within N breadth-first levels from the top. */ + depth?: number; + /** Evaluate rules on gated nodes anyway (gates are still reported). */ + includeGated?: boolean; + /** Evaluate only rules matching this filter (see `parseRuleFilter`). */ + ruleFilter?: RuleFilter; + /** Whether to evaluate semantic rules (default: false, or true if configured in space/config). */ + semantic?: boolean; + /** Optional custom judge provider (for testing or injection). */ + judgeProvider?: SemanticJudgeProvider; +} + +/** + * Resolve the semantic judge for a run: none unless semantic evaluation is requested (option, else + * space/config `semantic` or `judge.autoRun`). An injected provider wins over the configured one. + */ +function resolveJudge( + requested: boolean | undefined, + injected: SemanticJudgeProvider | undefined, + space: SpaceConfig, + config: Config, +): SemanticJudgeProvider | undefined { + const enabled = + requested ?? space.semantic ?? space.judge?.autoRun ?? config.semantic ?? config.judge?.autoRun ?? false; + if (!enabled) return undefined; + return injected ?? createJudgeProvider((space.judge ?? config.judge) as JudgeConfig | undefined); +} + +type SpaceRuleEvaluation = { + violations: RuleViolation[]; + gates: CoverageGate[]; + semanticEvaluations: SemanticEvaluationRecord[]; + gatedRules: GatedRuleRecord[]; +}; + /** - * Evaluate a space's executable rules. With a main hierarchy, evaluation cascades: schema, hierarchy + * Evaluate a space's rules. With a main hierarchy, evaluation cascades: schema, hierarchy * and gating-rule failures (the `seeds`) suppress rule evaluation for the failing node's subtree. * Shared by `validateSpace` and `validateFile` so both report identical rule results for a node. + * Semantic rules run only when a `judge` is supplied. */ async function evaluateSpaceRules( nodes: SpaceNode[], metadata: SchemaMetadata, seeds: GatingSeed[], - options: { includeGated?: boolean } = {}, -): Promise<{ violations: RuleViolation[]; gates: CoverageGate[] }> { - if (!metadata.rules) return { violations: [], gates: [] }; - if (!metadata.hierarchy) return { violations: await validateRules(nodes, metadata.rules), gates: [] }; + options: { includeGated?: boolean; judge?: SemanticJudgeProvider; ruleFilter?: RuleFilter } = {}, +): Promise { + const rules = metadata.rules && filterRules(metadata.rules, options.ruleFilter); + if (!rules) return { violations: [], gates: [], semanticEvaluations: [], gatedRules: [] }; + if (!metadata.hierarchy) { + const result = await validateRules(nodes, rules, { + semantic: options.judge !== undefined, + judgeProvider: options.judge, + }); + return { + violations: result, + gates: [], + semanticEvaluations: result.semanticEvaluations, + gatedRules: result.gatedRules, + }; + } const graph = buildSpaceGraph(nodes, metadata.hierarchy); - return evaluateRulesWithGating(nodes, metadata.rules, graph, seeds, options); + return evaluateRulesWithGating(nodes, rules, graph, seeds, options); } /** @@ -182,7 +239,7 @@ async function evaluateSpaceRules( */ export async function validateSpace( context: SpaceContext, - options: { readResult?: ReadSpaceResult; depth?: number; includeGated?: boolean } = {}, + options: ValidateSpaceOptions = {}, ): Promise { const { schema, schemaRefRegistry, schemaValidator } = context; const metadata = schema.metadata; @@ -299,11 +356,18 @@ export async function validateSpace( ), ...result.hierarchyViolations.map((v): GatingSeed => ({ file: v.file, title: v.nodeTitle, reason: 'hierarchy' })), ]; - const { violations, gates } = await evaluateSpaceRules(nodes, metadata, seeds, { + const judge = resolveJudge(options.semantic, options.judgeProvider, context.space, context.config); + const { violations, gates, semanticEvaluations, gatedRules } = await evaluateSpaceRules(nodes, metadata, seeds, { includeGated: options.includeGated, + ruleFilter: options.ruleFilter, + judge, }); result.ruleViolations = violations; result.coverageGates = gates; + if (judge) { + result.semanticEvaluations = semanticEvaluations; + result.gatedRules = gatedRules; + } } // Collect unresolved content links (warnings — not errors) @@ -338,6 +402,18 @@ function resolveFileSpace(filePath: string, config: Config): { space: SpaceConfi return null; } +export interface ValidateFileOptions { + configDir?: string; + /** Evaluate rules on gated nodes anyway (gates are still reported). */ + includeGated?: boolean; + /** Evaluate only rules matching this filter (see `parseRuleFilter`). */ + ruleFilter?: RuleFilter; + /** Whether to evaluate semantic rules (default: false, or true if configured in space/config). */ + semantic?: boolean; + /** Optional custom judge provider (for testing or injection). */ + judgeProvider?: SemanticJudgeProvider; +} + /** * Validate a single file within its space. * @@ -350,7 +426,7 @@ function resolveFileSpace(filePath: string, config: Config): { space: SpaceConfi export async function validateFile( filePath: string, config: Config, - options: { configDir?: string; includeGated?: boolean } = {}, + options: ValidateFileOptions = {}, ): Promise { const resolved = resolveFileSpace(filePath, config); @@ -459,13 +535,22 @@ export async function validateFile( (v): GatingSeed => ({ file: v.file, title: v.nodeTitle, reason: 'hierarchy' }), ), ]; + const judge = resolveJudge(options.semantic, options.judgeProvider, space, config); const { violations, gates } = await evaluateSpaceRules(nodes, metadata, seeds, { includeGated: options.includeGated, + ruleFilter: options.ruleFilter, + judge, }); for (const v of violations) { if (v.file === label) { const target = v.severity === 'error' ? errors : warnings; - target[`rule:${v.ruleId}`] = { kind: 'rule', message: `[${v.ruleId}] ${v.description}`, severity: v.severity }; + target[`rule:${v.ruleId}`] = { + kind: 'rule', + message: v.evidence + ? `[${v.ruleId}] ${v.description} (Evidence: ${v.evidence})` + : `[${v.ruleId}] ${v.description}`, + severity: v.severity, + }; } } // A gated file with no violations would otherwise look clean — say that its rules were not checked. diff --git a/tests/fixtures/strict_ost/valid-directory/Progressive sign-up form.md b/tests/fixtures/strict_ost/valid-directory/Progressive sign-up form.md new file mode 100644 index 0000000..b5df6e6 --- /dev/null +++ b/tests/fixtures/strict_ost/valid-directory/Progressive sign-up form.md @@ -0,0 +1,7 @@ +--- +type: solution +status: exploring +parent: "[[Sign-up is too complex]]" +--- + +Break the sign-up into a progressive multi-step form that asks for one thing at a time. diff --git a/tests/fixtures/strict_ost/valid-directory/Social login option.md b/tests/fixtures/strict_ost/valid-directory/Social login option.md new file mode 100644 index 0000000..2b8ccb2 --- /dev/null +++ b/tests/fixtures/strict_ost/valid-directory/Social login option.md @@ -0,0 +1,7 @@ +--- +type: solution +status: exploring +parent: "[[Sign-up is too complex]]" +--- + +Offer social login so new users can skip manual account creation entirely. diff --git a/tests/judge/factory.test.ts b/tests/judge/factory.test.ts new file mode 100644 index 0000000..8283112 --- /dev/null +++ b/tests/judge/factory.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from 'bun:test'; +import { AnthropicJudgeProvider } from '../../src/judge/anthropic'; +import { createJudgeProvider } from '../../src/judge/factory'; +import { FakeJudgeProvider } from '../../src/judge/fake'; +import { OpenAIJudgeProvider } from '../../src/judge/openai'; + +describe('createJudgeProvider factory', () => { + it('returns custom provider when passed directly', () => { + const custom = new FakeJudgeProvider(); + const provider = createJudgeProvider(undefined, {}, custom); + expect(provider).toBe(custom); + }); + + it('creates FakeJudgeProvider when provider is "fake"', () => { + const provider = createJudgeProvider({ provider: 'fake' }, {}); + expect(provider.name).toBe('fake'); + expect(provider instanceof FakeJudgeProvider).toBe(true); + }); + + it('creates AnthropicJudgeProvider when ANTHROPIC_API_KEY is in env', () => { + const provider = createJudgeProvider(undefined, { ANTHROPIC_API_KEY: 'test-key' }); + expect(provider.name).toBe('anthropic'); + expect(provider instanceof AnthropicJudgeProvider).toBe(true); + }); + + it('creates AnthropicJudgeProvider from config apiKey', () => { + const provider = createJudgeProvider({ provider: 'anthropic', apiKey: 'test-key-config' }, {}); + expect(provider.name).toBe('anthropic'); + }); + + it('creates OpenAIJudgeProvider when OPENAI_API_KEY is in env', () => { + const provider = createJudgeProvider(undefined, { OPENAI_API_KEY: 'test-openai-key' }); + expect(provider.name).toBe('openai'); + expect(provider instanceof OpenAIJudgeProvider).toBe(true); + }); + + it('creates OpenAIJudgeProvider from OPENAI_BASE_URL (for local endpoints like Ollama)', () => { + const provider = createJudgeProvider(undefined, { OPENAI_BASE_URL: 'http://localhost:11434/v1' }); + expect(provider.name).toBe('openai'); + }); + + it('throws descriptive error when no provider or key is configured', () => { + expect(() => createJudgeProvider(undefined, {})).toThrow('No semantic judge provider configured'); + }); +}); diff --git a/tests/judge/fake-provider.test.ts b/tests/judge/fake-provider.test.ts new file mode 100644 index 0000000..ab93f16 --- /dev/null +++ b/tests/judge/fake-provider.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from 'bun:test'; +import { FakeJudgeProvider } from '../../src/judge/fake'; +import type { SemanticJudgeInput } from '../../src/judge/types'; +import { graded, node, passFailRule } from './helpers'; + +const input: SemanticJudgeInput = { node, rules: [passFailRule, graded] }; + +describe('FakeJudgeProvider', () => { + it("passes every rule by default using each rule's first non-failing verdict", async () => { + const fake = new FakeJudgeProvider(); + const results = await fake.evaluate(input); + expect(results['customer-voice']).toMatchObject({ verdict: 'pass' }); + expect(results.framing).toMatchObject({ verdict: 'good' }); + expect(fake.callHistory).toHaveLength(1); + }); + + it('maps the { pass } shorthand onto pass/fail', async () => { + const results = await new FakeJudgeProvider({ pass: false, evidence: 'nope' }).evaluate({ + node, + rules: [passFailRule], + }); + expect(results['customer-voice']).toEqual({ verdict: 'fail', evidence: 'nope' }); + }); + + it('returns full verdicts and errors as configured', async () => { + const fake = new FakeJudgeProvider({ + framing: { verdict: 'solution', confidence: 0.8 }, + 'customer-voice': { error: 'boom' }, + }); + const results = await fake.evaluate(input); + expect(results.framing).toEqual({ verdict: 'solution', confidence: 0.8 }); + expect(results['customer-voice']).toEqual({ error: 'boom' }); + }); + + it('prefers rule:title over rule keys, and supports Map and handler configs', async () => { + const map = new Map([ + ['framing', { verdict: 'weak' }], + ['framing:I cannot find my receipts', { verdict: 'solution' }], + ]); + expect((await new FakeJudgeProvider(map).evaluate(input)).framing).toEqual({ verdict: 'solution' }); + + const handler = new FakeJudgeProvider(({ rule }) => ({ verdict: rule.id === 'framing' ? 'weak' : 'pass' })); + expect((await handler.evaluate(input)).framing).toEqual({ verdict: 'weak' }); + }); +}); diff --git a/tests/judge/helpers.ts b/tests/judge/helpers.ts new file mode 100644 index 0000000..0ceb608 --- /dev/null +++ b/tests/judge/helpers.ts @@ -0,0 +1,27 @@ +import type { SemanticJudgeInput, SemanticRuleSpec } from '../../src/judge/types'; + +export const node: SemanticJudgeInput['node'] = { + label: 'opp1.md', + title: 'I cannot find my receipts', + type: 'opportunity', + resolvedType: 'opportunity', + schemaData: { title: 'I cannot find my receipts', type: 'opportunity', source: 'User interview with Alice' }, + resolvedParents: [{ title: 'Faster Tax Filing', field: 'parent' }], +}; + +export const passFailRule: SemanticRuleSpec = { + id: 'customer-voice', + description: 'Must be in customer voice', + rubric: 'Needs to sound like a customer quote, not a solution or business goal', + category: 'best-practice', + verdicts: { pass: 'Customer voice.', fail: 'Not customer voice.' }, + failOn: ['fail'], +}; + +export const graded: SemanticRuleSpec = { + id: 'framing', + description: 'Framing quality', + category: 'best-practice', + verdicts: { good: 'Clear need', weak: 'Vague need', solution: 'A solution in disguise' }, + failOn: ['solution'], +}; diff --git a/tests/judge/jev.test.ts b/tests/judge/jev.test.ts new file mode 100644 index 0000000..c220ec3 --- /dev/null +++ b/tests/judge/jev.test.ts @@ -0,0 +1,77 @@ +import { afterEach, describe, expect, it } from 'bun:test'; +import { createJudgeProvider } from '../../src/judge/factory'; +import { JevJudgeProvider } from '../../src/judge/jev'; +import { graded, node, passFailRule } from './helpers'; + +const realFetch = globalThis.fetch; +afterEach(() => { + globalThis.fetch = realFetch; +}); + +function mockFetch(body: unknown, status = 200) { + const calls: Array<{ url: string; init: RequestInit }> = []; + globalThis.fetch = (async (url: string, init: RequestInit) => { + calls.push({ url, init }); + return new Response(JSON.stringify(body), { status }); + }) as unknown as typeof fetch; + return calls; +} + +describe('JevJudgeProvider', () => { + it('sends one request per node with a choice question per rule', async () => { + const calls = mockFetch({ + answers: { + 'customer-voice': { + type: 'choice', + choice: 'pass', + confidence: 0.9, + probabilities: { pass: 0.95, fail: 0.05 }, + }, + framing: { + type: 'choice', + choice: 'solution', + confidence: 0.6, + probabilities: { good: 0.1, weak: 0.3, solution: 0.6 }, + }, + }, + }); + const jev = new JevJudgeProvider({ apiKey: 'k' }); + const results = await jev.evaluate({ node, rules: [passFailRule, graded] }); + + expect(calls).toHaveLength(1); + expect(calls[0]?.url).toBe('https://openrouter.ai/api/alpha/decisions'); + const sent = JSON.parse(calls[0]?.init.body as string); + expect(sent.model).toBe('typesafe/jev-1.13'); + expect(sent.state).toMatchObject({ title: 'I cannot find my receipts', parents: ['Faster Tax Filing'] }); + expect(sent.questions.framing).toEqual({ + type: 'choice', + instructions: 'Framing quality', + criteria: graded.verdicts, + }); + expect(sent.questions['customer-voice'].instructions).toContain('customer quote'); + expect(((calls[0]?.init.headers ?? {}) as Record).authorization).toBe('Bearer k'); + + expect(results.framing).toMatchObject({ verdict: 'solution', confidence: 0.6 }); + expect((results.framing as { evidence: string }).evidence).toContain('solution 0.60'); + }); + + it('reports an error for a rule with a missing or unknown choice, leaving others intact', async () => { + mockFetch({ answers: { framing: { choice: 'nonsense' }, 'customer-voice': { choice: 'fail' } } }); + const results = await new JevJudgeProvider({ apiKey: 'k' }).evaluate({ node, rules: [passFailRule, graded] }); + expect(results.framing).toHaveProperty('error'); + expect(results['customer-voice']).toMatchObject({ verdict: 'fail' }); + }); + + it('throws on an HTTP error', async () => { + mockFetch({ error: 'bad' }, 401); + await expect(new JevJudgeProvider({ apiKey: 'k' }).evaluate({ node, rules: [passFailRule] })).rejects.toThrow( + 'Jev API error (401)', + ); + }); + + it('is created explicitly via provider "jev" with OPENROUTER_API_KEY, never auto-detected', () => { + expect(createJudgeProvider({ provider: 'jev' }, { OPENROUTER_API_KEY: 'k' }).name).toBe('jev'); + expect(() => createJudgeProvider({ provider: 'jev' }, {})).toThrow('OPENROUTER_API_KEY'); + expect(() => createJudgeProvider(undefined, { OPENROUTER_API_KEY: 'k' })).toThrow('No semantic judge provider'); + }); +}); diff --git a/tests/judge/prompt.test.ts b/tests/judge/prompt.test.ts new file mode 100644 index 0000000..efce894 --- /dev/null +++ b/tests/judge/prompt.test.ts @@ -0,0 +1,62 @@ +import { describe, expect, it } from 'bun:test'; +import { buildJudgeSystemPrompt, buildJudgeUserPrompt, parseJudgeResponse } from '../../src/judge/prompt'; +import { graded, node, passFailRule } from './helpers'; + +describe('judge prompt and response parser', () => { + it('system prompt asks for a JSON verdict, confidence and evidence', () => { + const prompt = buildJudgeSystemPrompt(); + expect(prompt).toContain('"verdict"'); + expect(prompt).toContain('"confidence"'); + expect(prompt).toContain('"evidence"'); + }); + + it('user prompt lists the rule, its allowed verdicts and the node', () => { + const prompt = buildJudgeUserPrompt(node, graded); + expect(prompt).toContain('framing'); + expect(prompt).toContain('"weak": Vague need'); + expect(prompt).toContain('"solution": A solution in disguise'); + expect(prompt).toContain('I cannot find my receipts'); + expect(prompt).toContain('Faster Tax Filing'); + expect(prompt).toContain('User interview with Alice'); + }); + + it('user prompt prefers the rubric over the description', () => { + expect(buildJudgeUserPrompt(node, passFailRule)).toContain('Needs to sound like a customer quote'); + expect(buildJudgeUserPrompt(node, graded)).toContain('Rubric: Framing quality'); + }); + + const allowed = ['pass', 'fail']; + + it('parses clean JSON', () => { + expect(parseJudgeResponse('{"verdict":"pass","confidence":0.9,"evidence":"clear"}', allowed)).toEqual({ + verdict: 'pass', + confidence: 0.9, + evidence: 'clear', + }); + }); + + it('parses fenced JSON and clamps confidence', () => { + const raw = '```json\n{"verdict":"fail","confidence":4,"evidence":"solution"}\n```'; + expect(parseJudgeResponse(raw, allowed)).toEqual({ verdict: 'fail', confidence: 1, evidence: 'solution' }); + }); + + it('allows confidence and evidence to be absent', () => { + expect(parseJudgeResponse('{"verdict":"pass"}', allowed)).toEqual({ verdict: 'pass' }); + }); + + it('extracts fields from text surrounding malformed JSON', () => { + const raw = 'Sure! Here you go: {"verdict": "fail", "confidence": 0.4, "evidence": "says \\"build X\\"" trailing'; + const result = parseJudgeResponse(raw, allowed); + expect(result).toMatchObject({ verdict: 'fail', confidence: 0.4 }); + }); + + it('returns an error for a verdict outside the allowed set', () => { + const result = parseJudgeResponse('{"verdict":"maybe"}', allowed); + expect(result).toHaveProperty('error'); + expect((result as { error: string }).error).toContain('unknown verdict "maybe"'); + }); + + it('returns an error when nothing parseable is present', () => { + expect(parseJudgeResponse('I cannot decide', allowed)).toHaveProperty('error'); + }); +}); diff --git a/tests/judge/verdicts.test.ts b/tests/judge/verdicts.test.ts new file mode 100644 index 0000000..4578f0c --- /dev/null +++ b/tests/judge/verdicts.test.ts @@ -0,0 +1,47 @@ +import { describe, expect, it } from 'bun:test'; +import { PerRuleJudgeProvider } from '../../src/judge/per-rule'; +import type { SemanticJudgeResult } from '../../src/judge/types'; +import { semanticRuleSpec } from '../../src/judge/verdicts'; +import type { Rule } from '../../src/types'; +import { graded, node, passFailRule } from './helpers'; + +const base: Rule = { id: 'r', category: 'best-practice', description: 'd', mode: 'semantic' }; + +describe('semanticRuleSpec', () => { + it('defaults to pass/fail with fail as the violation', () => { + const spec = semanticRuleSpec(base); + expect(Object.keys(spec.verdicts)).toEqual(['pass', 'fail']); + expect(spec.failOn).toEqual(['fail']); + }); + + it('uses declared verdicts and failOn', () => { + const spec = semanticRuleSpec({ ...base, verdicts: { a: 'A', b: 'B', c: 'C' }, failOn: ['b', 'c'] }); + expect(spec.failOn).toEqual(['b', 'c']); + }); + + it('rejects declared verdicts without failOn, unknown failOn, and all-failing sets', () => { + expect(() => semanticRuleSpec({ ...base, verdicts: { a: 'A', b: 'B' } })).toThrow('no failOn'); + expect(() => semanticRuleSpec({ ...base, verdicts: { a: 'A', b: 'B' }, failOn: ['z'] })).toThrow( + 'not in its verdicts', + ); + expect(() => semanticRuleSpec({ ...base, verdicts: { a: 'A', b: 'B' }, failOn: ['a', 'b'] })).toThrow( + 'no passing verdict', + ); + }); +}); + +describe('PerRuleJudgeProvider', () => { + class Stub extends PerRuleJudgeProvider { + readonly name = 'stub'; + protected async judgeRule(_n: unknown, rule: { id: string }): Promise { + if (rule.id === 'framing') throw new Error('network down'); + return { verdict: 'pass' }; + } + } + + it('isolates a failing rule from the others', async () => { + const results = await new Stub().evaluate({ node, rules: [passFailRule, graded] }); + expect(results['customer-voice']).toEqual({ verdict: 'pass' }); + expect(results.framing).toEqual({ error: 'network down' }); + }); +}); diff --git a/tests/schema/rule-filter.test.ts b/tests/schema/rule-filter.test.ts new file mode 100644 index 0000000..37054f9 --- /dev/null +++ b/tests/schema/rule-filter.test.ts @@ -0,0 +1,39 @@ +import { describe, expect, it } from 'bun:test'; +import { filterRules, parseRuleFilter } from '../../src/schema/rule-filter'; +import type { Rule } from '../../src/types'; + +const rules: Rule[] = [ + { id: 'a', category: 'validation', description: 'a', check: 'true', type: 'goal' }, + { id: 'b', category: 'best-practice', description: 'b', mode: 'semantic', type: 'opportunity' }, + { id: 'c', category: 'coherence', description: 'c', check: 'true', severity: 'error' }, +]; +const ids = (r: Rule[]) => r.map((x) => x.id); + +describe('rule filter', () => { + it('parses repeated specs and merges values per dimension', () => { + expect(parseRuleFilter(['mode=semantic', 'category=coherence,validation', 'category=workflow'])).toEqual({ + mode: ['semantic'], + category: ['coherence', 'validation', 'workflow'], + }); + }); + + it('rejects malformed specs and unknown dimensions', () => { + expect(() => parseRuleFilter(['semantic'])).toThrow('Invalid rule filter'); + expect(() => parseRuleFilter(['colour=red'])).toThrow('Invalid rule filter'); + expect(() => parseRuleFilter(['mode='])).toThrow('Invalid rule filter'); + }); + + it('filters by mode (defaulting to expression), severity (effective), category, type and id', () => { + expect(ids(filterRules(rules, { mode: ['semantic'] }))).toEqual(['b']); + expect(ids(filterRules(rules, { mode: ['expression'] }))).toEqual(['a', 'c']); + expect(ids(filterRules(rules, { severity: ['error'] }))).toEqual(['a', 'c']); + expect(ids(filterRules(rules, { category: ['coherence'] }))).toEqual(['c']); + expect(ids(filterRules(rules, { type: ['goal', 'opportunity'] }))).toEqual(['a', 'b']); + expect(ids(filterRules(rules, { id: ['c'] }))).toEqual(['c']); + }); + + it('ANDs dimensions and treats an undefined filter as everything', () => { + expect(ids(filterRules(rules, { severity: ['error'], mode: ['semantic'] }))).toEqual([]); + expect(filterRules(rules, undefined)).toBe(rules); + }); +}); diff --git a/tests/schema/validate-rules-semantic.test.ts b/tests/schema/validate-rules-semantic.test.ts new file mode 100644 index 0000000..6ed1cac --- /dev/null +++ b/tests/schema/validate-rules-semantic.test.ts @@ -0,0 +1,168 @@ +import { describe, expect, it } from 'bun:test'; +import { FakeJudgeProvider } from '../../src/judge/fake'; +import { validateRules } from '../../src/schema/validate-rules'; +import type { Rule, SpaceNode } from '../../src/types'; +import { makeParentRef } from '../test-helpers'; + +describe('validateRules - semantic evaluation', () => { + const sampleNodes: SpaceNode[] = [ + { + label: 'outcome.md', + title: 'Outcome Metric', + type: 'outcome', + resolvedType: 'outcome', + schemaData: { title: 'Outcome Metric', type: 'outcome', status: 'active', metric: 'Increase conversions' }, + linkTargets: ['Outcome Metric'], + resolvedParents: [], + resolvedLinks: [], + }, + { + label: 'valid-opp.md', + title: 'I struggle to finish signup on mobile', + type: 'opportunity', + resolvedType: 'opportunity', + schemaData: { + title: 'I struggle to finish signup on mobile', + type: 'opportunity', + status: 'active', + parent: '[[Outcome Metric]]', + source: 'User interview with Jane', + }, + linkTargets: ['I struggle to finish signup on mobile'], + resolvedParents: [makeParentRef('Outcome Metric')], + resolvedLinks: [], + }, + { + label: 'bad-opp.md', + title: 'Build social login integration', + type: 'opportunity', + resolvedType: 'opportunity', + schemaData: { + title: 'Build social login integration', + type: 'opportunity', + status: 'active', + parent: '[[Outcome Metric]]', + source: 'Engineering suggestion', + }, + linkTargets: ['Build social login integration'], + resolvedParents: [makeParentRef('Outcome Metric')], + resolvedLinks: [], + }, + ]; + + const rules: Rule[] = [ + { + id: 'active-opportunity-count', + category: 'workflow', + description: 'At most 2 active opportunities', + check: '$count(nodes[resolvedType="opportunity" and status="active"]) <= 2', + }, + { + id: 'opportunity-customer-voice', + category: 'best-practice', + mode: 'semantic', + type: 'opportunity', + description: 'Opportunity must be framed in customer voice, not a solution or business goal', + rubric: 'Must describe a customer pain point or need in customer language, not a solution to build.', + }, + ]; + + it('skips semantic rules when semantic option is false / omitted (default validate)', async () => { + // Default validate should only run expression rules and remain deterministic & free + const violations = await validateRules(sampleNodes, rules); + expect(violations).toHaveLength(0); + expect(violations.semanticEvaluations).toHaveLength(0); + expect(violations.gatedRules).toHaveLength(0); + }); + + it('evaluates semantic rules when semantic option is true', async () => { + const fake = new FakeJudgeProvider( + new Map([ + [ + 'opportunity-customer-voice:I struggle to finish signup on mobile', + { pass: true, evidence: 'Clearly stated from customer perspective as a pain point' }, + ], + [ + 'opportunity-customer-voice:Build social login integration', + { pass: false, evidence: 'Stated as a solution feature ("Build social login") rather than a customer need' }, + ], + ]), + ); + + const violations = await validateRules(sampleNodes, rules, { + semantic: true, + judgeProvider: fake, + }); + + expect(violations).toHaveLength(1); + expect(violations[0]?.ruleId).toBe('opportunity-customer-voice'); + expect(violations[0]?.file).toBe('bad-opp.md'); + expect(violations[0]?.category).toBe('best-practice'); + expect(violations[0]?.evidence).toBe( + 'Stated as a solution feature ("Build social login") rather than a customer need', + ); + + expect(violations.semanticEvaluations).toHaveLength(2); + expect(violations.semanticEvaluations.find((e) => e.file === 'valid-opp.md')?.pass).toBe(true); + expect(violations.semanticEvaluations.find((e) => e.file === 'bad-opp.md')?.pass).toBe(false); + }); + + it('gates semantic rules when node has upstream structural error (cascading)', async () => { + const fake = new FakeJudgeProvider(); + + // Mark bad-opp.md as having an upstream structural error via gatedNodeLabels + const gatedNodeLabels = new Set(['bad-opp.md']); + + const violations = await validateRules(sampleNodes, rules, { + semantic: true, + judgeProvider: fake, + gatedNodeLabels, + }); + + // bad-opp.md was gated; only valid-opp.md was sent to the judge + expect(fake.callHistory).toHaveLength(1); + expect(fake.callHistory[0]?.node.label).toBe('valid-opp.md'); + + expect(violations.gatedRules).toHaveLength(1); + expect(violations.gatedRules[0]?.file).toBe('bad-opp.md'); + expect(violations.gatedRules[0]?.ruleId).toBe('opportunity-customer-voice'); + }); + + it('gates semantic rules when node fails an upstream expression rule', async () => { + const strictRules: Rule[] = [ + { + id: 'require-interviews-source', + category: 'validation', + type: 'opportunity', + check: '$contains(current.source, "interview") = true', + description: 'Source must mention interview', + }, + { + id: 'opportunity-customer-voice', + category: 'best-practice', + mode: 'semantic', + type: 'opportunity', + description: 'Must be in customer voice', + }, + ]; + + const fake = new FakeJudgeProvider(); + + const violations = await validateRules(sampleNodes, strictRules, { + semantic: true, + judgeProvider: fake, + }); + + // bad-opp.md source is "Engineering suggestion", so it failed require-interviews-source + // Therefore it should be gated from opportunity-customer-voice + const badOppViolations = violations.filter((v) => v.file === 'bad-opp.md'); + expect(badOppViolations).toHaveLength(1); + expect(badOppViolations[0]?.ruleId).toBe('require-interviews-source'); + + // Only valid-opp.md was evaluated by the semantic judge + expect(fake.callHistory).toHaveLength(1); + expect(fake.callHistory[0]?.node.label).toBe('valid-opp.md'); + + expect(violations.gatedRules.some((g) => g.file === 'bad-opp.md')).toBe(true); + }); +}); diff --git a/tests/validate/gating.test.ts b/tests/validate/gating.test.ts index 01a48e2..be8113a 100644 --- a/tests/validate/gating.test.ts +++ b/tests/validate/gating.test.ts @@ -43,14 +43,11 @@ describe('validateSpace cascade gating', () => { expect(gate!.gatedFiles).toContain('Child Opportunity.md'); }); - it('does not gate on a non-error rule violation by default (solution-quantity is best-practice)', async () => { - // The fixture opportunity has only one solution below it, violating solution-quantity. - // That rule is info severity, so it doesn't block and the solution's own rules still run. + it('a clean space with only non-blocking rules reports no gates', async () => { const context = makeSpaceContext(VALID_DIR, SCHEMA_PATH); const result = await validateSpace(context); expect(result.nodeErrorCount).toBe(0); - expect(result.ruleViolations.some((v) => v.ruleId === 'solution-quantity')).toBe(true); expect(result.coverageGates).toEqual([]); }); }); diff --git a/tests/validate/semantic-validation.test.ts b/tests/validate/semantic-validation.test.ts new file mode 100644 index 0000000..3c75dee --- /dev/null +++ b/tests/validate/semantic-validation.test.ts @@ -0,0 +1,191 @@ +import { describe, expect, it } from 'bun:test'; +import { join } from 'node:path'; +import type { Config } from '../../src/config'; +import { FakeJudgeProvider } from '../../src/judge/fake'; +import { bundledSchemasDir } from '../../src/schema/schema'; +import { createSpaceContext } from '../../src/space-context'; +import type { FileValidationResult } from '../../src/types'; +import { validateFile, validateSpace } from '../../src/validate'; + +const STRICT_SCHEMA_PATH = join(bundledSchemasDir, 'strict_ost.json'); +const VALID_DIR = join(import.meta.dir, '../fixtures/strict_ost/valid-directory'); +const INVALID_DIR = join(import.meta.dir, '../fixtures/strict_ost/invalid'); + +describe('Semantic rule evaluation integration (validateSpace & validateFile)', () => { + const validConfig: Config = { + spaces: [ + { + name: 'strict-valid', + path: VALID_DIR, + schema: STRICT_SCHEMA_PATH, + }, + ], + }; + + const validContext = createSpaceContext('strict-valid', validConfig, { configDir: import.meta.dir }); + + describe('validateSpace', () => { + it('skips semantic rules by default (deterministic & free)', async () => { + const result = await validateSpace(validContext); + expect(result.semanticEvaluations ?? []).toHaveLength(0); + expect(result.gatedRules ?? []).toHaveLength(0); + expect(result.ruleViolations.some((v) => v.ruleId === 'opportunity-customer-voice')).toBe(false); + }); + + it('evaluates semantic rules when opt-in flag semantic: true is passed', async () => { + const fakeProvider = new FakeJudgeProvider({ + pass: true, + evidence: 'Opportunity is framed in clear customer language describing difficulty with sign-up.', + }); + + const result = await validateSpace(validContext, { + semantic: true, + judgeProvider: fakeProvider, + }); + + expect(result.semanticEvaluations).toBeDefined(); + expect(result.semanticEvaluations!.length).toBeGreaterThan(0); + + const oppEval = result.semanticEvaluations!.find((e) => e.ruleId === 'opportunity-customer-voice'); + expect(oppEval).toBeDefined(); + expect(oppEval?.pass).toBe(true); + expect(oppEval?.evidence).toContain('framed in clear customer language'); + + // No violation when pass is true + expect(result.ruleViolations.some((v) => v.ruleId === 'opportunity-customer-voice')).toBe(false); + }); + + it('records violation with evidence when judge returns pass: false', async () => { + const fakeProvider = new FakeJudgeProvider({ + pass: false, + evidence: 'The opportunity reads like a feature request rather than a problem statement.', + }); + + const result = await validateSpace(validContext, { + semantic: true, + judgeProvider: fakeProvider, + }); + + const violation = result.ruleViolations.find((v) => v.ruleId === 'opportunity-customer-voice'); + expect(violation).toBeDefined(); + expect(violation?.category).toBe('best-practice'); + expect(violation?.evidence).toBe('The opportunity reads like a feature request rather than a problem statement.'); + expect(violation?.file).toBe('Sign-up is too complex.md'); + }); + + it('gates semantic rules when node has upstream structural errors (cascading)', async () => { + const invalidConfig: Config = { + spaces: [ + { + name: 'strict-invalid', + path: INVALID_DIR, + schema: STRICT_SCHEMA_PATH, + }, + ], + }; + const invalidContext = createSpaceContext('strict-invalid', invalidConfig, { configDir: import.meta.dir }); + const fakeProvider = new FakeJudgeProvider(); + + const result = await validateSpace(invalidContext, { + semantic: true, + judgeProvider: fakeProvider, + }); + + // opportunity-no-source.md fails schema validation (missing required 'source' field) + expect(result.nodeErrors.some((e) => e.file === 'opportunity-no-source.md')).toBe(true); + + // Therefore it must be gated and never evaluated by the judge + expect(result.gatedRules).toBeDefined(); + const gated = result.gatedRules!.find( + (g) => g.file === 'opportunity-no-source.md' && g.ruleId === 'opportunity-customer-voice', + ); + expect(gated).toBeDefined(); + expect(gated?.reason).toMatch(/^Gated by/); + + // Judge was never called for opportunity-no-source.md + expect(fakeProvider.callHistory.some((c) => c.node.label === 'opportunity-no-source.md')).toBe(false); + }); + }); + + describe('validateFile', () => { + const targetFile = join(VALID_DIR, 'Sign-up is too complex.md'); + + it('skips semantic rule evaluation for single file by default', async () => { + const result = (await validateFile(targetFile, validConfig, { + configDir: import.meta.dir, + })) as FileValidationResult; + + expect(result.inSpace).toBe(true); + expect(result.errors['rule:opportunity-customer-voice']).toBeUndefined(); + }); + + it('evaluates semantic rule and reports evidence on violation when semantic: true', async () => { + const failingProvider = new FakeJudgeProvider({ + pass: false, + evidence: 'Opportunity is framed in solution space.', + }); + + const result = (await validateFile(targetFile, validConfig, { + semantic: true, + judgeProvider: failingProvider, + configDir: import.meta.dir, + })) as FileValidationResult; + + expect(result.inSpace).toBe(true); + // best-practice rules default to non-error severity, so they land in warnings + const ruleError = result.warnings['rule:opportunity-customer-voice']; + expect(ruleError).toBeDefined(); + expect(ruleError?.kind).toBe('rule'); + expect(ruleError?.message).toContain('Evidence: Opportunity is framed in solution space.'); + }); + }); +}); + +describe('Semantic rules: verdicts, batching and rule filter', () => { + const config: Config = { + spaces: [{ name: 'strict-valid', path: VALID_DIR, schema: STRICT_SCHEMA_PATH }], + }; + const context = createSpaceContext('strict-valid', config, { configDir: import.meta.dir }); + + it('judges each node once with all of its semantic rules', async () => { + const fake = new FakeJudgeProvider(); + await validateSpace(context, { semantic: true, judgeProvider: fake }); + const labels = fake.callHistory.map((c) => c.node.label); + expect(new Set(labels).size).toBe(labels.length); + expect(fake.callHistory.every((c) => c.rules.length >= 1)).toBe(true); + }); + + it('records the verdict and confidence on evaluations and fails only on failOn verdicts', async () => { + const fake = new FakeJudgeProvider({ verdict: 'fail', confidence: 0.7, evidence: 'solution in disguise' }); + const result = await validateSpace(context, { semantic: true, judgeProvider: fake }); + const record = result.semanticEvaluations?.find((e) => e.ruleId === 'opportunity-customer-voice'); + expect(record).toMatchObject({ verdict: 'fail', pass: false, confidence: 0.7 }); + expect(result.ruleViolations.some((v) => v.ruleId === 'opportunity-customer-voice')).toBe(true); + }); + + it('reports a per-rule judge error as a violation without gating the subtree', async () => { + const fake = new FakeJudgeProvider({ error: 'rate limited' }); + const result = await validateSpace(context, { semantic: true, judgeProvider: fake }); + const violation = result.ruleViolations.find((v) => v.ruleId === 'opportunity-customer-voice'); + expect(violation?.description).toContain('Judge error: rate limited'); + expect(result.coverageGates.filter((g) => g.ruleId === 'opportunity-customer-voice')).toEqual([]); + }); + + it('--rule-filter narrows evaluation: semantic-only runs no expression rules, expression-only runs no judge', async () => { + const semanticOnly = await validateSpace(context, { + semantic: true, + judgeProvider: new FakeJudgeProvider({ pass: false }), + ruleFilter: { mode: ['semantic'] }, + }); + expect(semanticOnly.ruleViolations.every((v) => v.ruleId === 'opportunity-customer-voice')).toBe(true); + + const fake = new FakeJudgeProvider(); + const expressionOnly = await validateSpace(context, { + semantic: true, + judgeProvider: fake, + ruleFilter: { mode: ['expression'] }, + }); + expect(fake.callHistory).toHaveLength(0); + expect(expressionOnly.semanticEvaluations ?? []).toHaveLength(0); + }); +}); diff --git a/tests/validate/strict.test.ts b/tests/validate/strict.test.ts index 753ef52..3f0e2ff 100644 --- a/tests/validate/strict.test.ts +++ b/tests/validate/strict.test.ts @@ -198,8 +198,8 @@ describe('Strict OST schema validation', () => { ({ nodes } = await readSpaceDirectory(makePluginContext(VALID_DIR, STRICT_SCHEMA_PATH))); }); - it('reads all 4 nodes from valid-directory', () => { - expect(nodes).toHaveLength(4); + it('reads all 6 nodes from valid-directory', () => { + expect(nodes).toHaveLength(6); }); it('all nodes pass strict_ost schema validation', () => {