diff --git a/AGENTS.md b/AGENTS.md index 2b9f439..7a7d763 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -9,7 +9,9 @@ Before starting new work, review [docs/concepts.md](docs/concepts.md) for canoni - `bun run src/index.ts --help` — list CLI commands - `bun run typecheck` — type-check all code (`tsconfig.json`) - `bun run lint` — biome +- `bun run lint:fix` — biome with fixes - `bun run build` — compile `src/` to `dist/` (`tsconfig.build.json`) +- see `package.json` - Releases go through the `/release` skill. Space names (e.g. `personal`, `politics`) are resolved via a config file - `$SCTX_CONFIG`, `$XDG_CONFIG_HOME/structured-context/config.json`, `--config ` param, or `./config.json` @@ -38,14 +40,27 @@ There are several places that need reviewing and updating with any new feature o - **Simplicity first:** Strive for the simplest design that completely solves the problem. Prefer conventions and established filesystem patterns over adding new configuration, programmatic APIs, or abstractions. - **Explicitly state alternatives:** When planning or making architectural decisions, do not anchor to the first familiar mechanism. Formulate and state at least one alternative approach, weighing trade-offs, complexity, and ergonomic impact. - **Minimise blast radius:** Protect established boundaries and interfaces. Treat unnecessary ripple effects (such as making synchronous APIs asynchronous or adding dual APIs) as a design smell indicating the approach should be re-evaluated. +- **Public API surface:** Do not add exports to `src/api.ts`, `src/types.ts` or `src/testing.ts` (the `structured-context/api`, `/types` and `/testing` entry points), or add types or hooks to the plugin interface, without the maintainer's explicit agreement on that specific surface. Exporting something to make docs complete, to tidy up, or because a feature has a public-looking class is not a reason. If a change seems to need a new export, stop and propose it, with the consumer that needs it. Internal modules can change freely. +- **Where a setting lives:** Config says *what* to use, CLI flags override it for one run, and the environment holds secrets and the config location only (`apiKeyEnv` names a secret; `SCTX_CONFIG` locates the config). Do not add env vars that set config values, and never infer a choice (such as a judge provider) from a variable that happens to be set: a key present for another tool must not be spent silently. Config also never switches on costly or outward-facing behaviour (such as semantic judging): each run asks for it, through a flag or an API option. +- **Failures are visible:** A rule or expression that cannot be evaluated is reported as a violation, never skipped or treated as a pass. Likewise a misconfiguration fails before any work or spend. + +## Documentation Principles + +- **One idea lives in one place:** Decide which doc owns each concept, document it there once, and link instead of repeating. Small, non-obvious tidbits may be repeated; explanations may not. +- **Document what it does, not what it doesn't:** Describe behaviour and purpose. State a limit only when a new user would realistically assume otherwise. Specification and decision history do not leak into docs. +- **Proper name in docs:** `structured-context` is the project name for prose; `sctx` is only the CLI command in command examples. +- **Approachability:** Write short, declarative sentences in active voice, one idea per sentence — about 80% of the way to ASD-STE100. Introduce a concept once, before sections that need it. Frame options by purpose (what each is for, and for whom). +- **Front-load the happy path:** Quick starts give the fewest steps that run. Tuning, alternatives and edge cases come after. ## Key Files -- `src/api.ts` — Public library entry point (`structured-context/api`). Re-exports the supported programmatic surface. Keep CLI-only concerns out of it. +- `src/api.ts` — Public library entry point (`structured-context/api`). Re-exports the supported programmatic surface. Keep CLI-only concerns out of it. Do not extend it without agreement (see **Public API surface** above). - `schemas/` — Bundled default schema files (JSON5) using the structured-context schema dialect and top-level `$metadata`. Files starting with `_` are "partials" (fragments for `$ref`). - `src/schema/metadata-contract.ts` — Single source of truth for the `$metadata` contract - `schemas/generated/latest/_structured_context_schema_meta.json` — Generated metaschema for the newest dialect (generated on build or with `bun run generate:schema-meta`). `schemas/generated//` folders are frozen copies, created by the `version` script (`generate:schema-meta --release`) only when a release changes the dialect; never edit them. `src/schema/meta-schemas.ts` loads them at runtime. - `src/schema/gating.ts` — Cascade gating: computes hierarchy depth and evaluates rules with subtree gating (schema/hierarchy/blocking-rule failures suppress downstream rule checks). See [docs/rules.md](docs/rules.md#cascade-gating). +- `src/judge/` — Semantic rule judges: provider interface and types (`types.ts`), verdict resolution (`verdicts.ts`), per-node dispatch (`judge-node.ts`), and providers (`provider/anthropic.ts`, `provider/openai.ts`, `provider/jev.ts`, `provider/fake.ts`), provider hydration and the bundled catalogue (`factory.ts`), and `sctx providers` (`src/commands/providers.ts`). See [docs/semantic-rules.md](docs/semantic-rules.md). +- `src/schema/rule-filter.ts` — `--rule-filter` parsing and rule filtering by mode, category, severity, type or id. ## Testing For most development only the main unit tests need re-running regularly. @@ -53,6 +68,8 @@ For most development only the main unit tests need re-running regularly. - `bun run test:hook` — unit test plugin hooks (`hook-test/unit/`) - hook development only - `bun run test:hook:e2e` — test plugin hooks in Claude Code (`hook-test/`) - hook development only - `bun run test:smoke` — smoke tests run against locally configured spaces - only use when changes could affect compatibility. +- `bun run test:judge` — live judge tests (`tests/judge/live.test.ts`): one real provider call per provider whose API key is set. Costs money; use sparingly when changing judges or semantic rule behaviour. +- `bun run test:models` — check that every bundled provider's configured model exists (no API keys) ## Debugging diff --git a/README.md b/README.md index efb867a..1834bb4 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ Rule violations: High tech juice press.md: Explore multiple candidate solutions (aim for at least three) for the target opportunity [best-practice: solution-quantity] ``` -Rule violations carry a severity (`error`, `warning` or `info`) that defaults from the rule's category. Only errors fail validation — see [docs/rules.md](docs/rules.md#categories-and-severity). +Rule violations carry a severity (`error`, `warning` or `info`) that defaults from the rule's category. Only errors fail validation — see [docs/rules.md](docs/rules.md#rule-shape). ## Installation @@ -131,7 +131,7 @@ See [docs/schemas.md](docs/schemas.md) (`sctx docs schema`) for the dialect, com ### Validate nodes ```bash -sctx validate [--watch] [--depth ] [--include-gated] +sctx validate [--watch] [--depth ] [--include-gated] [--semantic] [--judge ] [--rule-filter ] ``` Validates markdown files against the JSON schema: @@ -141,6 +141,7 @@ Validates markdown files against the JSON schema: - Runs executable rules (`$metadata.rules`) with **cascade gating**: a schema, hierarchy, or blocking-rule failure on a node suppresses rule evaluation for that node's hierarchy subtree. Gated checks are reported in a Coverage section so you know what wasn't evaluated and why. See [docs/rules.md](docs/rules.md#cascade-gating) for details. - `--depth ` limits evaluation to hierarchy nodes within `n` breadth-first levels from the top — useful for staged agentic fix loops (fix the top level, then request the next). - `--include-gated` evaluates rules even on gated nodes (still reported as gated) — for surfacing cheap bottom-up checks on request. +- `--semantic` also evaluates `mode: semantic` rules with the configured judge (Jev, Anthropic or OpenAI-compatible); `--judge ` uses that configured model provider instead and implies `--semantic`; `--rule-filter` narrows which rules run. See [docs/semantic-rules.md](docs/semantic-rules.md) (`sctx docs semantic-rules`). ### Render space diff --git a/config.example.json b/config.example.json index 77796c3..10b0d94 100644 --- a/config.example.json +++ b/config.example.json @@ -17,5 +17,6 @@ } } ], - "schema": "/path/to/custom/schemas/my-general-schema.json" + "schema": "/path/to/custom/schemas/my-general-schema.json", + "judge": { "provider": "openrouter-jev" } } diff --git a/docs/concepts.md b/docs/concepts.md index 3eaaec3..04e584b 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -151,9 +151,15 @@ Each executable rule has a **category** (`validation`, `coherence`, `workflow`, See [docs/rules.md](rules.md) for the rules reference, including JSONata expression syntax and the full `$metadata` field reference. +### Semantic rules + +A **semantic rule** (`mode: semantic`) asks a model to judge a node, for example whether an opportunity is a customer need. It has a `rubric` instead of a `check`. + +A **judge** is the model that judges semantic rules: a decision model (Jev) or a chat model (Claude, OpenAI-compatible). For each rule, the judge gives one **verdict** from the rule's `verdicts`, for example `customer-need`, `vague` or `solution`. The rule's `severity` says which verdicts report a violation. Semantic rules cost money, so they run only when a run asks for them (`--semantic` or `--judge`). The judge is a configured **model provider**: a bundled provider or an entry in `modelProviders`. See [docs/semantic-rules.md](semantic-rules.md). + ### Cascade gating -**Cascade gating** is the mechanism by which a schema failure, hierarchy structure violation, or blocking rule violation on a node suppresses rule evaluation for that node's `hierarchy` subtree. It exists so downstream checks — especially future evaluative or LLM-judged layers — aren't run against inputs already known to be invalid. +**Cascade gating** is the mechanism by which a schema failure, hierarchy structure violation, or blocking rule violation on a node suppresses rule evaluation for that node's `hierarchy` subtree. It exists so downstream checks — especially costly semantic (model-judged) rules — aren't run against inputs already known to be invalid. A **coverage gate** records one instance of this: the gating node, what kind of failure triggered it (`schema`, `hierarchy`, or a specific rule `id`), and the set of nodes whose rule checks were suppressed as a result. `sctx validate` reports coverage gates so both humans and agents can see the validation "edge" — where evaluation stopped and why. See [docs/rules.md](rules.md#cascade-gating) for the full model, including the per-rule `gatesDescendants` field and the `--include-gated` / `--depth` CLI options. diff --git a/docs/config.md b/docs/config.md index 664dd9d..240989f 100644 --- a/docs/config.md +++ b/docs/config.md @@ -65,6 +65,71 @@ All plugin names must start with `sctx-` (the prefix is optional in config and n Plugins can also contribute schemas and partials via a `/schemas/` directory (or `"sctx": { "schemas": "..." }` in `package.json`). If a space specifies a plugin-prefixed schema (e.g. `"sctx-wardley-mapping/wardley_map.json"`), schemas are resolved statically without executing plugin code, and the plugin's code hooks are loaded when parsing or rendering. +## Model providers and judge + +structured-context judges [semantic rules](semantic-rules.md) with a **model provider**: a named definition that says which wire API to speak, which model to use and where. A provider definition combines an `api`, explicit values or environment-variable names, and the key source. + +**Bundled providers** work without any configuration: a fixed model per vendor chosen by this project, plus `-env` variants that also read the vendor's usual model and endpoint variables (`OPENAI_MODEL`, `OPENAI_BASE_URL`, `ANTHROPIC_MODEL`), falling back to the bundle's defaults where a variable is unset. Bundles without `-env` read no environment beyond the key: they behave the same on every machine. + +Run `sctx providers` for the authoritative list of bundled and configured providers with their effective API, endpoint, model and key variable. Keys are never shown. `bun run test:models` checks that each bundled model resolves on its endpoint. + +### Wire APIs and bundled-provider conventions + +A provider's `api` names the **request shape**, potentially supported by multiple vendors. + +| API | Shape | Bundled providers | +|---|---|---| +| `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities | `openrouter-jev`, `openrouter-openai-decisions` | +| `typesafe-systemone` | Same decisions shape (currently) | `typesafe-jev` | +| `openai-decisions` | OpenAI's Decisions API: same concept, its own request/response shape | `openai-decisions` | +| `anthropic-messages` | Chat: one request per rule, structured JSON verdict | `anthropic`, `anthropic-haiku-5-5`, `anthropic-env` | +| `openai-completions` | Chat: OpenAI-compatible; works with Ollama, vLLM and OpenRouter | `openai`, `openai-env`, `openrouter-openai` | + +Bundled provider names follow the pattern: + +- `` — this project's recommended model for that vendor; can change between releases. +- `-` (e.g. `anthropic-haiku-5-5`) — pins a model id, so it does not drift when the `` bundle's recommendation moves. +- `-env` — hydrates model, endpoint etc from the vendor's usual environment variables. +- `openrouter--` (e.g. `openrouter-openai-decisions`) — aggregating vendor's endpoint variant + +**Configured providers** live in `modelProviders` at the top level, and replace a bundled provider of the same name. Fields a configured provider leaves out still take the API defaults: + +```json +{ + "modelProviders": { + "mine": { "api": "openrouter-decisions", "apiKeyEnv": "MY_JEV_KEY" }, + "local": { "api": "openai-completions", "endpoint": "http://localhost:11434/v1" } + } +} +``` + +| Field | Description | +|---|---| +| `api` | **Required.** The wire API (see the table above and [Choosing a judge](semantic-rules.md#choosing-a-judge)), or `fake` (for tests). | +| `model` | A fixed model. Its default depends on the API. | +| `modelEnv` | Environment variable holding a model override. An explicit `model` wins; a variable set but empty is an error. | +| `endpoint` | A fixed API URL. | +| `endpointEnv` | Environment variable holding an endpoint override. Same precedence as `modelEnv`. | +| `apiKeyEnv` | The environment variable that holds the key. Defaults to the API's usual variable (see the table above). A local `openai-completions` endpoint can need no key. | +| `apiKey` | The key itself. Use `apiKeyEnv` instead, so that keys stay out of config files. | + +Resolution per field: the explicit value wins, then the named environment variable, then the API default. Provider implementations read no environment; all hydration happens before one runs. + +The environment holds only keys and the variables a `-env` bundle names. No environment variable selects a provider. + +The top-level `judge` names the model provider that judges semantic rules: a bundled provider or an entry in `modelProviders`. A space can set its own `judge`, which replaces the top-level one for that space, for example to keep a private space on a local model: + +```json +{ + "judge": { "provider": "openrouter-jev" }, + "spaces": [ + { "name": "private-notes", "path": "/path/to/notes", "judge": { "provider": "local" } } + ] +} +``` + +Config does not turn semantic rules on. Each run asks for them with `--semantic`, or with `--judge `, which also selects a different provider for that run. + ## Markdown plugin config Set under `plugins.markdown` per space. diff --git a/docs/rules.md b/docs/rules.md index d788963..d7c1890 100644 --- a/docs/rules.md +++ b/docs/rules.md @@ -4,7 +4,10 @@ title: "Executable Rules" # Executable Rules -Rules are JSONata expressions in schema metadata (`$metadata.rules`). They run after structural JSON Schema validation and let you enforce cross-node checks and workflow constraints. +Rules check content after JSON Schema validation, for example cross-node checks and workflow constraints. A schema declares rules in `$metadata.rules`. There are two kinds of rule: + +- An **expression rule** (the default) is a JSONata `check` that must return `true`. +- A **semantic rule** asks a model to judge a node. See [Semantic rules](semantic-rules.md). For metadata structure and composition behavior, see [docs/schemas.md](schemas.md). @@ -24,21 +27,7 @@ Rules are a flat array. ] ``` -| Field | Required | Description | -|---|---|---| -| `id` | yes | Unique rule identifier | -| `category` | yes | `validation` \| `coherence` \| `workflow` \| `best-practice` | -| `severity` | no | `error` \| `warning` \| `info`; overrides the category default (see [Severity](#severity)) | -| `description` | yes | Human-readable rule intent | -| `check` | yes | JSONata expression; must evaluate to `true` | -| `type` | no | Restrict rule to nodes of this `resolvedType` | -| `scope` | no | Use `"global"` to evaluate once for the whole space | -| `override` | no | Only for merge conflicts: allows later duplicate `id` to replace earlier | -| `gatesDescendants` | no | Whether a violation gates downstream checks on the node's hierarchy subtree. Defaults to `true` for rules with `error` severity, `false` for `warning` and `info`. Set explicitly to override either way. | - -## Categories and severity - -Categories describe a rule's intent and supply its default severity. They do not change expression execution semantics. +Every rule has a category, which describes its intent and supplies a default severity. | Category | Typical use | Default severity | |---|---|---| @@ -47,9 +36,7 @@ Categories describe a rule's intent and supply its default severity. They do not | `workflow` | Process/operating-discipline checks | `warning` | | `best-practice` | Advisory quality checks | `info` | -### Severity - -Every rule violation is reported with a severity: +Severity can also be set per rule. | Severity | Meaning | Fails validation | |---|---|---| @@ -74,19 +61,32 @@ Prefer choosing the right category and relying on its default. Set `severity` on `sctx schemas show` lists each category with its default severity and marks rules that override it. -### Output +## Selecting nodes -- **`sctx validate`** — the summary counts rule violations per severity, and details are grouped by severity with each line showing the rule's category and id. -- **`sctx validate --json`** — `errors` includes only `error` severity rule violations (keyed `rule:`, with a `severity` field). A `ruleViolations` array lists every violation with `file`, `ruleId`, `category`, `severity` and `description` (`file` is empty for global rules). -- **`sctx validate-file`** — `error` severity violations appear under `errors`; `warning` and `info` violations appear under `warnings`. Rule entries carry a `severity` field. +- A rule with `scope: "global"` runs once for the space. +- Other rules run once for each node. +- `type` limits a rule to nodes with that `resolvedType`. +- `when` limits a rule to nodes where a test returns `true`. See [Conditional rules](#conditional-rules-when). -## Evaluation model +### Conditional rules (`when`) -- Rules with `scope: "global"` run once. -- Other rules run per applicable node. -- If `type` is set, only nodes with matching `resolvedType` are evaluated. +`when` is a JSONata test that determines whether a rule runs on a node. It has the same context as `check` and runs before it. Use `when` when `type` cannot select the nodes a rule applies to: -## Expression context +```json +{ "id": "ready-has-owner", "category": "workflow", "type": "task", + "description": "Ready tasks have an owner", + "when": "current.status = 'ready'", "check": "$exists(current.owner)" } +``` + +`when` costs almost nothing, so use it to stop an expensive rule from running where it does not apply. For a semantic rule, this saves a model call on each node that `when` excludes. + +An error in a `when` or `check` is reported as a non-gating `error` violation. + +## Writing expressions + +An expression rule's `check` (and `when`) is JSONata. This section covers what an expression can see and how to write checks. + +### Expression context Each evaluation receives: @@ -104,7 +104,7 @@ Useful resolved fields on nodes: Prefer `resolvedType` over raw `type` so aliases are handled correctly. -## Predicate scoping (`$$`) +### Predicate scoping (`$$`) Inside `nodes[...]`, bare names refer to each candidate node. Use `$$` to reference outer variables: @@ -112,9 +112,33 @@ Inside `nodes[...]`, bare names refer to each candidate node. Use `$$` to refere $count(nodes[resolvedParentTitle=$$.current.title and resolvedType='solution']) ``` -## Reusing rules +### Common patterns + +```jsonata +$exists(current.metric) = true +$count(current.sources) >= 1 +$count(nodes[resolvedType='outcome' and status='active']) <= 1 +current.status != 'active' or $exists(parent) = false or parent.status = 'active' +``` + +#### Dates and times + +Dates and datetimes reach rules as ISO 8601 strings. Convert with `$toMillis()` to compare them, including date against datetime, and use `$millis()` for the current time: + +```jsonata +$not($exists(current.due)) or $toMillis(current.due) > $millis() +$toMillis(current.stale_after) > $toMillis(current.generated.at) +``` + +A datetime without an offset (`2026-07-01T00:00:00`) is read in the machine's local time zone, so require an offset with `format: "date-time"` when results must not depend on where validation runs. `$toMillis()` also needs seconds: it cannot parse a minute-precision datetime such as `2026-07-01T09:15`, which `format: "date-or-date-time"` accepts. A value `$toMillis()` cannot parse makes the rule fail. + +The same applies to [filter expressions](concepts.md#filter-expressions): compare dates with `$toMillis()`, not as strings. -To share rules, put them in a schema's `$metadata.rules` and list that schema's `$id` in `$metadata.imports`. An import brings the whole schema's metadata, including its own imports: +## Sharing rules + +To share rules, put them in a schema's `$metadata.rules` and list that schema's `$id` in `$metadata.imports`. An import brings the whole schema's metadata, including its own imports. Merged rules combine by `id`; a duplicate `id` is an error, and a later rule replaces an earlier one only with `override: true`. + +### Overriding rules ```json5 "$metadata": { @@ -122,13 +146,6 @@ To share rules, put them in a schema's `$metadata.rules` and list that schema's } ``` -## Merge and conflict behavior - -When metadata is merged from imports (imports in order, then the schema's own metadata): -- Rules are merged by `id`. -- A duplicate `id` is an error. -- A later rule may replace an earlier one only with `override: true`. - Example override: ```json5 @@ -152,18 +169,16 @@ in a way that makes its descendants' checks untrustworthy or wasteful to run: suppressed too, since its data is structurally invalid. - **Hierarchy structure violation** on a node (e.g. an invalid parent) — always gates the node's subtree, though the node's own rules still run. -- **A rule violation with `gatesDescendants: true`** (the default for `error`-severity rules; `warning` and `info` rules don't gate unless they set `gatesDescendants: true`) — gates the node's subtree. The - violating node's own other rules still run (they're the same cheap deterministic layer). +- **A rule violation with `gatesDescendants: true`** — gates the node's subtree, though the + violating node's own other rules still run. -Gating is per-subtree: once a node is gated, its descendants inherit the same gate (the -original blocker), rather than each generating a new one. Non-hierarchy nodes and -`scope: "global"` rules are never gated — cascade gating only applies within the hierarchy -DAG. - -This exists so that expensive or semantic downstream checks (see #114) aren't wasted +Gating exists so that expensive or semantic downstream checks (see [Semantic rules](semantic-rules.md)) aren't wasted evaluating nodes whose ancestors are already known-bad, and so noise doesn't drown out the -actionable failure. Set `gatesDescendants: false` on an error-severity rule whose violations shouldn't suppress -checks further down the tree, or `gatesDescendants: true` on a warning/info rule that should. +actionable failure. It is per-subtree: once a node is gated, its descendants inherit the same +gate (the original blocker), rather than each generating a new one. Cascade gating only applies to nodes within the +hierarchy DAG. + +The default is `gatesDescendants: true` for `error`-severity rules and `false` for `warning` and `info` rules. Set it explicitly when that default doesn't fit. ### Coverage output @@ -179,7 +194,26 @@ don't depend on an ancestor being valid. ### Single-file validation -`sctx validate-file ` applies the same gating as `sctx validate`, so a file's rule results match its entry in a full run. Rules are evaluated for the whole space and the output is restricted to the file. If the file is gated, its rule checks are skipped and a `gated:` entry (kind `gated`, severity `info`) under `warnings` names the blocker, so a gated file doesn't read as clean. `--include-gated` evaluates the file's rules anyway. `--depth` is not supported. +`sctx validate-file ` applies the same gating as `sctx validate`, so a file's rule results match its entry in a full run. + +- Expression rules are evaluated for the whole space; the output is restricted to the file. +- Semantic rules are judged only for the file and its gating ancestors (see [Semantic rules](semantic-rules.md#single-file-validation)). +- If the file is gated, its rule checks are skipped and a `gated:` entry under `warnings` names the blocker, so a gated file doesn't read as clean. `--include-gated` evaluates the file's rules anyway. + +## Controlling validation runs + +These options restrict which rules run, without changing rule definitions. + +### Filtering rules + +`--rule-filter =[,]` runs only the rules that match. It works on `sctx validate` and `sctx validate-file`. + +- Dimensions: `mode`, `category`, `severity`, `type` and `id`. +- Values in one dimension combine with OR. Repeat the option to add a dimension; dimensions combine with AND. +- A semantic rule with a severity map matches each severity that it can report. +- Gating comes only from the rules that ran. + +For example, `--rule-filter category=workflow,coherence --rule-filter severity=error` runs only workflow and coherence rules that can report an error. ### Depth-limited evaluation @@ -188,24 +222,11 @@ the top (roots and orphans are depth 1). This is a manual, coarser control than gating — intended for staged workflows (e.g. an agent fixes the top level, then requests the next). Excluded nodes are reported under `skippedByDepth`, separate from gated nodes. -## Common patterns - -```jsonata -$exists(current.metric) = true -$count(current.sources) >= 1 -$count(nodes[resolvedType='outcome' and status='active']) <= 1 -current.status != 'active' or $exists(parent) = false or parent.status = 'active' -``` - -### Dates and times - -Dates and datetimes reach rules as ISO 8601 strings. Convert with `$toMillis()` to compare them, including date against datetime, and use `$millis()` for the current time: +## Output -```jsonata -$not($exists(current.due)) or $toMillis(current.due) > $millis() -$toMillis(current.stale_after) > $toMillis(current.generated.at) -``` +- **`sctx validate`** — the summary counts rule violations per severity, and details are grouped by severity with each line showing the rule's category and id. +- **`sctx validate --json`** — `errors` includes only `error` severity rule violations (keyed `rule:`, with a `severity` field). A `ruleViolations` array lists every violation with `file`, `ruleId`, `category`, `severity` and `description` (`file` is empty for global rules). +- **`sctx validate-file`** — `error` severity violations appear under `errors`; `warning` and `info` violations appear under `warnings`. Rule entries carry a `severity` field. -A datetime without an offset (`2026-07-01T00:00:00`) is read in the machine's local time zone, so require an offset with `format: "date-time"` when results must not depend on where validation runs. `$toMillis()` also needs seconds: it cannot parse a minute-precision datetime such as `2026-07-01T09:15`, which `format: "date-or-date-time"` accepts. A value `$toMillis()` cannot parse makes the rule fail. +Gated files: see [Single-file validation](#single-file-validation). -The same applies to [filter expressions](concepts.md#filter-expressions): compare dates with `$toMillis()`, not as strings. diff --git a/docs/schemas.md b/docs/schemas.md index b4ebc05..3fce5d6 100644 --- a/docs/schemas.md +++ b/docs/schemas.md @@ -159,7 +159,7 @@ Top-level metadata shape: | `hierarchy.allowSkipLevels` | `boolean` | Optional; allows parent to be any ancestor level | | `relationships` | `Relationship[]` | Optional; defines related node links outside the primary hierarchy. A relationship may set `override: true` | | `aliases` | `Record` | Optional type alias map. Use `{ "type": "goal", "override": true }` to replace an imported alias | -| `rules` | `Rule[]` | Optional flat rule array | +| `rules` | `Rule[]` | Optional flat rule array. Rules are `expression` (JSONata `check`, the default) or `semantic` (judged against a rubric); see [Rules](rules.md) | ### Relationships diff --git a/docs/semantic-rules.md b/docs/semantic-rules.md new file mode 100644 index 0000000..9911045 --- /dev/null +++ b/docs/semantic-rules.md @@ -0,0 +1,106 @@ +# Semantic Rules + +A semantic rule asks a model to judge a node, for example "Is this opportunity a customer need, or a solution in disguise?". Use a semantic rule for a check that an expression cannot do. + +This page assumes you know [Rules](rules.md). Semantic rules use the same fields, severities, [`when`](rules.md#conditional-rules-when) and [cascade gating](rules.md#cascade-gating) as expression rules. This page gives only the differences. + +## Quick start + +1. Set `OPENROUTER_API_KEY` in your environment. +2. Choose Jev as the judge in `config.json`. Jev is a decision model and the recommended judge (see [Choosing a judge](#choosing-a-judge)): + + ```json + { "judge": { "provider": "openrouter-jev" } } + ``` + + `openrouter-jev` is a bundled provider. To use Claude instead, set `"provider": "anthropic"` and `ANTHROPIC_API_KEY`. +3. Add a rule to the schema's `$metadata.rules`: + + ```json + { + "id": "opportunity-customer-need", + "category": "best-practice", + "mode": "semantic", + "type": "opportunity", + "description": "The opportunity is a customer need or pain point, not a solution or a business metric" + } + ``` + +4. Run `sctx validate --semantic`. + +## Run `sctx validate` or `sctx validate-file` with semantic rule checking + +Semantic rules cost money and send node content to the judge's provider. Thus they run only when you ask, on each run: + +- `--semantic` runs them with the configured judge. +- `--judge ` runs them with a specific model provider instead. + +## Rule fields + +| Field | Required | Description | +|---|---|---| +| `mode` | yes | `"semantic"` | +| `rubric` | no | Tells the judge what to assess. Default: `description` | +| `verdicts` | no | The answers the judge can give. Default: `pass` and `fail`. See [Verdicts](#verdicts) | +| `severity` | if `verdicts` is set | The severity of each verdict. See [Severity](#severity) | + +A semantic rule cannot have `check` or `scope: "global"`. + +## Verdicts + +For each node, the judge gives exactly one verdict for each rule. The verdict must come from the rule's `verdicts`. A fixed set of verdicts makes results comparable between runs. + +Each verdict has a purpose for one audience: + +- The **description** tells the judge when to choose that verdict. +- The **guidance** (optional) tells the user what to do when the judge chooses that verdict. structured-context shows it with the violation and does not send it to the judge. + +A verdict is a description string, or an object with `description` and `guidance`: + +```json +"verdicts": { + "customer-need": "A customer need or pain point", + "vague": { + "description": "A need, but too vague to act on", + "guidance": "Say who has the need and what they are trying to do." + }, + "solution": "A solution or business goal in disguise" +}, +"severity": { "vague": "info", "solution": "warning" } +``` + +## Severity + +The verdict is a classification. The severity says what the verdict means for the user. + +- With custom verdicts, `severity` is a map from verdict to `error`, `warning`, `info` or `ignore`. +- A verdict with `error`, `warning` or `info` reports a violation at that severity. +- A verdict that is not in the map reports no violation. At least one verdict must report no violation. +- `ignore` also reports no violation. Use it to show that you chose to ignore a verdict. +- With the default verdicts, `severity` is one value for `fail`, the same as for an expression rule. + +Gating uses the severity of the verdict. A verdict with `error` severity gates the node's subtree, as for any rule. `gatesDescendants` changes this, as for any rule. + +## How judging works + +- **Order.** On each node, expression rules run first. Then structured-context sends all of the node's semantic rules to the judge in one call. +- **Gating.** structured-context does not judge a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated. +- **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. +- **Input.** The judge gets the node's title, type, parent titles and schema data, including frontmatter. +- **Output.** A violation from a semantic rule also includes the verdict, the judge's evidence and any guidance. + +`sctx validate-file` judges only the file and the rules on its ancestors that can gate it. Thus one file costs at most one judge call for each ancestor level, plus one. + +## Choosing a judge + +Use a decision model when you can. Its results are repeatable and cheap, and they include a confidence. Use a chat model when the rubric needs open reasoning or a written explanation. + +Bundled providers give you a model with no configuration; the decision models among them speak a decisions API. Run `sctx providers` for the full list with effective models, endpoints and key variables. + +structured-context uses only the provider that you configure. It does not choose a provider from the keys in your environment. To override models, endpoints or keys, add an entry of the same name to `modelProviders`, or define a new one (see [Model providers and judge](config.md#model-providers-and-judge)). + +A rule does not choose its judge: that is configuration, not schema. To use a different provider for one run, name it with `--judge`. For example, to get a second opinion from a chat model on one rule and one file: + +```bash +sctx validate-file --rule-filter id= --judge claude +``` diff --git a/package.json b/package.json index 9915c6a..9c5d6d7 100644 --- a/package.json +++ b/package.json @@ -41,6 +41,8 @@ "diagram": "bun run src/index.ts diagram", "test": "bun test tests/", "test:smoke": "bun test smoke/", + "test:judge": "SCTX_LIVE_JUDGE_TESTS=1 bun test tests/judge/live.test.ts", + "test:models": "SCTX_MODEL_VALIDATION=1 bun test tests/judge/model-validation.test.ts", "test:hook": "bun test hook-test/unit/", "test:hook:e2e": "bun test hook-test/hooks.test.ts", "test:all": "bun test", diff --git a/plugin/.claude-plugin/plugin.json b/plugin/.claude-plugin/plugin.json index 037f46f..09abbb9 100644 --- a/plugin/.claude-plugin/plugin.json +++ b/plugin/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "structured-context", - "version": "0.12.6", + "version": "0.12.13", "description": "Structured content authoring for Obsidian — automatic validation hooks, slash commands, and agent skills to keep markdown files compliant with your schema.", "homepage": "https://github.com/mindsocket/structured-context", "repository": "https://github.com/mindsocket/structured-context", diff --git a/plugin/skills/structured-context/SKILL.md b/plugin/skills/structured-context/SKILL.md index 1cb308d..8e049b7 100644 --- a/plugin/skills/structured-context/SKILL.md +++ b/plugin/skills/structured-context/SKILL.md @@ -134,4 +134,4 @@ sctx docs config - **`references/schema-authoring.md`** — schema file structure, `$metadata`, JSONata rules (run `sctx docs schema` for schema dialect reference) - **`references/schema-design.md`** — process for designing a schema from existing content -For CLI and config reference, use `sctx docs ` (topics: `concepts`, `config`, `schema`, `rules`). +For CLI and config reference, use `sctx docs ` (topics: `concepts`, `config`, `schema`, `rules`, `semantic-rules`). diff --git a/plugin/skills/structured-context/references/commands.md b/plugin/skills/structured-context/references/commands.md index f2bd438..0e160a6 100644 --- a/plugin/skills/structured-context/references/commands.md +++ b/plugin/skills/structured-context/references/commands.md @@ -11,7 +11,7 @@ Then use the `sctx` command for all operations. Always pass `--config ` wh ## validate ```bash -sctx validate [--watch] [--config ] [--depth ] [--include-gated] +sctx validate [--watch] [--config ] [--depth ] [--include-gated] [--semantic] [--judge ] [--rule-filter ] ``` Validates all `.md` files in the space against the JSON schema. For each file: @@ -29,6 +29,9 @@ Validates all `.md` files in the space against the JSON schema. For each file: the top — useful for fixing a space level by level (fix the top, re-run with a deeper `--depth`, repeat). `--include-gated` evaluates rules on gated nodes anyway, still reporting them as gated — use it to surface cheap bottom-up checks even when an upstream blocker fired. +`--semantic` also runs `mode: semantic` rules through a configured model judge (costly; off by +default; `--judge ` judges with that configured model provider and implies `--semantic`), and `--rule-filter =` (repeatable; mode, category, severity, type, id) +narrows which rules run. `validate-file` accepts the same two flags. Exit codes: `0` = clean (warning and info rule violations don't fail), `1` = validation errors found. diff --git a/plugin/skills/structured-context/references/schema-authoring.md b/plugin/skills/structured-context/references/schema-authoring.md index 2909ea7..5a6095b 100644 --- a/plugin/skills/structured-context/references/schema-authoring.md +++ b/plugin/skills/structured-context/references/schema-authoring.md @@ -66,6 +66,8 @@ Relationships define how sub-entities (nodes inside other files) are parsed and Rules are a flat array. Each rule's `category` (`validation`, `coherence`, `workflow`, `best-practice`) supplies its default severity: `validation` → `error`, `coherence` and `workflow` → `warning`, `best-practice` → `info`. Only `error` violations fail validation; warnings and info are reported only. An optional per-rule `severity` (`error` | `warning` | `info`) overrides the category default — prefer picking the right category over overriding. +Rules default to `mode: "expression"` (a JSONata `check`). Set `mode: "semantic"` for rules an LLM or decision model judges against a `rubric` (falls back to `description`): they take no `check` or `scope`, may declare `verdicts` (verdict → description, or `{ description, guidance }`; default `pass`/`fail`) with a `severity` map saying which verdicts are `error`/`warning`/`info`/`ignore`, and only run with `--semantic`. Run `sctx docs semantic-rules` for the full semantic rule reference. A `when` JSONata test (any rule) limits a rule to the nodes where it returns true. An optional `gatesDescendants` controls whether a violation suppresses rule checks on the node's subtree (default: true for `error` severity). + ## Metadata composition semantics `$ref`/`$defs` are validation-only: a `$ref` brings no metadata. Metadata travels only via `$metadata.imports` (schema `$id`s): diff --git a/schemas/_okf.json b/schemas/_okf.json index ea2049f..cd5410e 100644 --- a/schemas/_okf.json +++ b/schemas/_okf.json @@ -17,8 +17,7 @@ }, { "id": "okf-stale-after-generated", - "category": "coherence", - "severity": "error", + "category": "validation", "description": "stale_after must be later than generated.at", "check": "$not($exists(current.stale_after)) or $not($exists(current.generated.at)) or $toMillis(current.stale_after) > $toMillis(current.generated.at)" }, @@ -34,7 +33,6 @@ // from other body text: any body counts. "id": "okf-computation-present", "category": "validation", - "severity": "error", "type": "Attested Computation", "description": "An Attested Computation needs its computation: a `computation` path or a body `# Computation` section", "check": "$exists(current.computation) or $exists(current.content)" diff --git a/schemas/_ost_strict.json b/schemas/_ost_strict.json index 86b5acb..08fad84 100644 --- a/schemas/_ost_strict.json +++ b/schemas/_ost_strict.json @@ -35,6 +35,14 @@ "description": "Explore multiple candidate solutions (aim for at least three) for the target opportunity", "type": "opportunity", "check": "(current.status != 'exploring' and current.status != 'active') or $count(nodes[resolvedParentTitle=$$.current.title and resolvedType='solution']) >= 3" + }, + { + "id": "opportunity-customer-voice", + "category": "best-practice", + "mode": "semantic", + "type": "opportunity", + "description": "Opportunity must be framed in the customer's voice (something a customer would actually say) as a need or pain point, not a solution or business goal", + "rubric": "Verify that the opportunity is stated from the customer's perspective as a problem, pain point, or desire. It must not be a solution in disguise (e.g. 'build X feature') or a business metric (e.g. 'increase revenue')." } ] }, diff --git a/schemas/generated/latest/_structured_context_schema_meta.json b/schemas/generated/latest/_structured_context_schema_meta.json index 02d47c2..feb8506 100644 --- a/schemas/generated/latest/_structured_context_schema_meta.json +++ b/schemas/generated/latest/_structured_context_schema_meta.json @@ -197,7 +197,18 @@ "enum": ["validation", "coherence", "workflow", "best-practice"] }, "severity": { - "enum": ["error", "warning", "info"] + "oneOf": [ + { + "enum": ["error", "warning", "info"] + }, + { + "type": "object", + "additionalProperties": { + "enum": ["error", "warning", "info", "ignore"] + }, + "minProperties": 1 + } + ] }, "description": { "type": "string", @@ -207,6 +218,44 @@ "type": "string", "minLength": 1 }, + "when": { + "type": "string", + "minLength": 1 + }, + "mode": { + "enum": ["expression", "semantic"] + }, + "rubric": { + "type": "string", + "minLength": 1 + }, + "verdicts": { + "type": "object", + "additionalProperties": { + "oneOf": [ + { + "type": "string", + "minLength": 1 + }, + { + "type": "object", + "properties": { + "description": { + "type": "string", + "minLength": 1 + }, + "guidance": { + "type": "string", + "minLength": 1 + } + }, + "required": ["description"], + "additionalProperties": false + } + ] + }, + "minProperties": 2 + }, "type": { "type": "string", "minLength": 1 @@ -221,7 +270,73 @@ "type": "boolean" } }, - "required": ["id", "category", "description", "check"], + "required": ["id", "category", "description"], + "not": { + "required": ["when", "scope"] + }, + "if": { + "properties": { + "mode": { + "const": "semantic" + } + }, + "required": ["mode"] + }, + "then": { + "not": { + "anyOf": [ + { + "required": ["check"] + }, + { + "required": ["scope"] + } + ] + }, + "if": { + "required": ["verdicts"] + }, + "then": { + "required": ["severity"], + "properties": { + "severity": { + "type": "object" + } + } + }, + "else": { + "properties": { + "severity": { + "oneOf": [ + { + "enum": ["error", "warning", "info"] + }, + { + "type": "object" + } + ] + } + } + } + }, + "else": { + "required": ["check"], + "not": { + "anyOf": [ + { + "required": ["verdicts"] + }, + { + "required": ["rubric"] + } + ] + }, + "properties": { + "severity": { + "enum": ["error", "warning", "info"] + } + } + }, "additionalProperties": false } } diff --git a/src/commands/docs.ts b/src/commands/docs.ts index a4a85a5..339579c 100644 --- a/src/commands/docs.ts +++ b/src/commands/docs.ts @@ -6,6 +6,7 @@ const TOPICS: Record = { config: 'config.md', schema: 'schemas.md', rules: 'rules.md', + 'semantic-rules': 'semantic-rules.md', }; export function docs(topic?: string): void { diff --git a/src/commands/providers.ts b/src/commands/providers.ts new file mode 100644 index 0000000..3808504 --- /dev/null +++ b/src/commands/providers.ts @@ -0,0 +1,66 @@ +import { loadConfig } from '../config'; +import { BUNDLED_PROVIDERS, hydrateProvider } from '../judge/factory'; +import type { ModelProviderConfig } from '../judge/types'; + +interface ProviderRow { + name: string; + source: 'bundled' | 'config'; + api: string; + model: string | undefined; + endpoint: string | undefined; + keyEnv: string | undefined; + /** Whether the provider would resolve with a key right now. */ + valid: boolean; + error?: string; +} + +function row(name: string, def: ModelProviderConfig, source: 'bundled' | 'config'): ProviderRow { + try { + const h = hydrateProvider(name, def); + const needsKey = h.api !== 'openai-completions' && h.api !== 'fake'; + return { + name, + source, + api: h.api, + model: h.model, + endpoint: h.endpoint, + keyEnv: h.keyEnv, + valid: !needsKey || h.apiKey !== undefined, + error: needsKey && h.apiKey === undefined ? `no API key; set ${h.keyEnv}` : undefined, + }; + } catch (err) { + return { + name, + source, + api: def.api, + model: def.model, + endpoint: def.endpoint, + keyEnv: def.apiKeyEnv, + valid: false, + error: err instanceof Error ? err.message : String(err), + }; + } +} + +/** List bundled and configured model providers with their effective config; keys are never shown. */ +export function listProviders(options: { json?: boolean } = {}): void { + const config = loadConfig(); + const configured = config.modelProviders ?? {}; + const rows: ProviderRow[] = [ + ...Object.entries(configured).map(([name, def]) => row(name, def, 'config')), + ...Object.entries(BUNDLED_PROVIDERS) + .filter(([name]) => !(name in configured)) + .map(([name, def]) => row(name, def, 'bundled')), + ]; + if (options.json) { + console.log(JSON.stringify(rows, null, 2)); + return; + } + for (const r of rows) { + const origin = r.source === 'bundled' ? '' : ' (config)'; + const model = r.model ?? '(api default)'; + const endpoint = r.endpoint ?? '(api default)'; + const key = r.valid ? `key via ${r.keyEnv}` : (r.error ?? 'no key'); + console.log(`${r.name}${origin}\n api=${r.api}\n endpoint=${endpoint}\n model=${model}\n ${key}`); + } +} diff --git a/src/commands/schemas.ts b/src/commands/schemas.ts index 7fca269..d6cb99f 100644 --- a/src/commands/schemas.ts +++ b/src/commands/schemas.ts @@ -136,8 +136,7 @@ function showMetadata(metadata: SchemaMetadata): void { for (const [category, rules] of groups) { console.log(` ${category} (default severity: ${DEFAULT_SEVERITY_BY_CATEGORY[category]}):`); for (const rule of rules) { - const override = rule.severity ? ` [severity: ${rule.severity}]` : ''; - console.log(` ${rule.id}: ${rule.description}${override}`); + console.log(` ${rule.id}: ${rule.description}${ruleAnnotations(rule)}`); } } } @@ -227,6 +226,23 @@ function generateMermaidErd(metadata: SchemaMetadata, entities: EntityInfo[]): s return mmd; } +/** Bracketed notes on a rule: mode, severity (or verdict severities) and `when`. Empty for a plain rule. */ +function ruleAnnotations(rule: Rule): string { + const notes: string[] = []; + if (rule.mode === 'semantic') notes.push('semantic'); + if (typeof rule.severity === 'string') { + notes.push(`severity: ${rule.severity}`); + } else if (rule.severity) { + notes.push( + `severity: ${Object.entries(rule.severity) + .map(([verdict, sev]) => `${verdict}=${sev}`) + .join(', ')}`, + ); + } + if (rule.when) notes.push(`when: ${rule.when}`); + return notes.length > 0 ? ` [${notes.join('] [')}]` : ''; +} + export async function listSchemas(): Promise { const config = loadConfig(); diff --git a/src/commands/validate-file.ts b/src/commands/validate-file.ts index e69eef5..b1d2c6a 100644 --- a/src/commands/validate-file.ts +++ b/src/commands/validate-file.ts @@ -1,6 +1,8 @@ import { loadConfig } from '../config'; +import { judgeForRun } from '../judge/factory'; +import { parseRuleFilter } from '../schema/rule-filter'; import type { FileValidationIssue, FileValidationResult } from '../types'; -import { validateFile } from '../validate'; +import { resolveFileSpace, validateFile } from '../validate'; export { validateFile } from '../validate'; @@ -11,10 +13,15 @@ export { validateFile } from '../validate'; */ export async function validateFileCommand( filePath: string, - options: { json?: boolean; includeGated?: boolean } = {}, + options: { json?: boolean; includeGated?: boolean; semantic?: boolean; judge?: string; ruleFilter?: string[] } = {}, ): Promise { const config = loadConfig(); - const result = await validateFile(filePath, config, { includeGated: options.includeGated }); + const space = resolveFileSpace(filePath, config)?.space; + const result = await validateFile(filePath, config, { + includeGated: options.includeGated, + judge: space && judgeForRun(config, space, options), + ruleFilter: options.ruleFilter && parseRuleFilter(options.ruleFilter), + }); if (!result.inSpace) { if (options.json) { diff --git a/src/commands/validate.ts b/src/commands/validate.ts index ac9953a..ce9722b 100644 --- a/src/commands/validate.ts +++ b/src/commands/validate.ts @@ -1,17 +1,31 @@ import { dirname } from 'node:path'; import chokidar from 'chokidar'; import { getConfigSourceFiles } from '../config'; +import { judgeForRun } from '../judge/factory'; import { resolveSpacePluginSchemas } from '../plugins/util'; +import { parseRuleFilter } from '../schema/rule-filter'; import { bundledSchemasDir } from '../schema/schema'; import type { FileValidationIssue, RuleSeverity, SpaceContext } from '../types'; -import { formatErrors, validateSpace } from '../validate'; +import { formatErrors, ruleViolationMessage, validateSpace } from '../validate'; export async function validate( context: SpaceContext, - options: { json?: boolean; depth?: number; includeGated?: boolean } = {}, + options: { + json?: boolean; + depth?: number; + includeGated?: boolean; + semantic?: boolean; + judge?: string; + ruleFilter?: string[]; + } = {}, ): Promise { const { schema, schemaRefRegistry } = context; - const result = await validateSpace(context, { depth: options.depth, includeGated: options.includeGated }); + const result = await validateSpace(context, { + depth: options.depth, + includeGated: options.includeGated, + judge: judgeForRun(context.config, context.space, options), + ruleFilter: options.ruleFilter && parseRuleFilter(options.ruleFilter), + }); const ruleCounts: Record = { error: 0, warning: 0, info: 0 }; for (const v of result.ruleViolations) ruleCounts[v.severity]++; @@ -46,7 +60,7 @@ export async function validate( } for (const v of result.ruleViolations) { if (v.file && v.severity === 'error') { - addError(v.file, `rule:${v.ruleId}`, 'rule', `[${v.ruleId}] ${v.description}`, v.severity); + addError(v.file, `rule:${v.ruleId}`, 'rule', ruleViolationMessage(v), v.severity); } } const globalRuleErrorCount = result.ruleViolations.filter((v) => !v.file && v.severity === 'error').length; @@ -193,6 +207,12 @@ export async function validate( console.log(` ${severityColor[severity]}${severity.toUpperCase()}${reset} (${violations.length}):`); for (const v of violations) { console.log(` ${v.file ? `${v.file}: ` : ''}${v.description} [${v.category}: ${v.ruleId}]`); + if (v.evidence) { + console.log(` Evidence: ${v.evidence}`); + } + if (v.guidance) { + console.log(` Guidance: ${v.guidance}`); + } } } } diff --git a/src/config.ts b/src/config.ts index 07bf9b2..7f90c84 100644 --- a/src/config.ts +++ b/src/config.ts @@ -4,6 +4,7 @@ import { dirname, isAbsolute, join, resolve } from 'node:path'; import Ajv2020 from 'ajv/dist/2020'; import JSON5 from 'json5'; import { ENV_CONFIG_VAR, XDG_CONFIG_DIR } from './constants'; +import { type JudgeConfig, MODEL_PROVIDER_APIS, type ModelProviderConfig } from './judge/types'; import { normalizePluginName, resolvePluginSchemasDir, @@ -12,6 +13,27 @@ import { } from './plugins/util'; import { bundledSchemasDir } from './schema/schema'; +const MODEL_PROVIDER_SCHEMA = { + type: 'object', + properties: { + api: { type: 'string', enum: MODEL_PROVIDER_APIS }, + model: { type: 'string' }, + modelEnv: { type: 'string' }, + endpoint: { type: 'string' }, + endpointEnv: { type: 'string' }, + apiKey: { type: 'string' }, + apiKeyEnv: { type: 'string' }, + }, + required: ['api'], + additionalProperties: false, +} as const; + +const JUDGE_CONFIG_SCHEMA = { + type: 'object', + properties: { provider: { type: 'string', minLength: 1 } }, + additionalProperties: false, +} as const; + const CONFIG_SCHEMA = { type: 'object', properties: { @@ -33,6 +55,7 @@ const CONFIG_SCHEMA = { additionalProperties: false, }, }, + judge: JUDGE_CONFIG_SCHEMA, }, required: ['name', 'path'], additionalProperties: false, @@ -40,6 +63,8 @@ const CONFIG_SCHEMA = { }, schema: { type: 'string' }, includeSpacesFrom: { type: 'array', items: { type: 'string' } }, + modelProviders: { type: 'object', additionalProperties: MODEL_PROVIDER_SCHEMA }, + judge: JUDGE_CONFIG_SCHEMA, }, anyOf: [{ required: ['spaces'] }, { required: ['includeSpacesFrom'] }], additionalProperties: false, @@ -53,6 +78,8 @@ export type SpaceConfig = { plugins?: Record>; /** Named filter views for this space. Keys are view names; values contain the filter expression. */ views?: Record; + /** Judge for this space's semantic rules; overrides the top-level `judge`. */ + judge?: JudgeConfig; /** * Directory of the config file that defined this space, used to anchor relative plugin-config * paths. Set by loadConfig; absent for a hand-assembled Config, whose callers must pass an @@ -65,6 +92,10 @@ export type Config = { spaces: SpaceConfig[]; schema?: string; includeSpacesFrom?: string[]; + /** Model providers by name: how to reach each model. */ + modelProviders?: Record; + /** Judge for semantic rules (a model provider name). A space's `judge` overrides it. */ + judge?: JudgeConfig; }; let _configPathOverride: string | undefined; diff --git a/src/index.ts b/src/index.ts index fbe3229..3c69d75 100755 --- a/src/index.ts +++ b/src/index.ts @@ -16,6 +16,7 @@ import { validate, watchValidate } from './commands/validate'; import { validateFileCommand } from './commands/validate-file'; import { loadConfig, setConfigPath } from './config'; import { CLI_NAME } from './constants'; +import { JudgeConfigError } from './judge/factory'; import { createSpaceContext, SpaceNotFoundError } from './space-context'; import type { SpaceContext } from './types'; @@ -37,6 +38,17 @@ export function buildSpaceContext(spaceName: string): SpaceContext { } } +/** Run a validate command, reporting a misconfigured judge as a plain error rather than a stack trace. */ +async function reportJudgeConfigErrors(run: () => Promise): Promise { + try { + return await run(); + } catch (err) { + if (!(err instanceof JudgeConfigError)) throw err; + console.error(`Error: ${err.message}`); + return 1; + } +} + const require = createRequire(import.meta.url); const packageJson = require('../package.json'); @@ -61,8 +73,23 @@ program '--include-gated', 'Evaluate rules even when the file is gated by an upstream schema/hierarchy/rule failure (still reported as gated)', ) + .option( + '--rule-filter ', + 'Evaluate only matching rules; dimensions: mode, category, severity, type, id (repeatable, e.g. mode=semantic)', + (v: string, prev: string[] = []) => [...prev, v], + ) + .option('--semantic', 'Evaluate semantic rules with the configured judge') + .option('--judge ', 'Judge semantic rules with this model provider (implies --semantic)') .action(async (filePath, options) => { - process.exitCode = await validateFileCommand(filePath, { json: options.json, includeGated: options.includeGated }); + process.exitCode = await reportJudgeConfigErrors(() => + validateFileCommand(filePath, { + json: options.json, + includeGated: options.includeGated, + semantic: options.semantic, + judge: options.judge, + ruleFilter: options.ruleFilter, + }), + ); }); program @@ -80,17 +107,29 @@ program '--include-gated', 'Evaluate rules even on nodes gated by an upstream schema/hierarchy/rule failure (still reported as gated)', ) + .option( + '--rule-filter ', + 'Evaluate only matching rules; dimensions: mode, category, severity, type, id (repeatable, e.g. mode=semantic)', + (v: string, prev: string[] = []) => [...prev, v], + ) + .option('--semantic', 'Evaluate semantic rules with the configured judge') + .option('--judge ', 'Judge semantic rules with this model provider (implies --semantic)') .action(async (spaceName, options) => { const context = buildSpaceContext(spaceName); if (options.watch) { await watchValidate(context); } else { // exitCode rather than process.exit(): exiting straight away can cut off large piped output - process.exitCode = await validate(context, { - json: options.json, - depth: options.depth, - includeGated: options.includeGated, - }); + process.exitCode = await reportJudgeConfigErrors(() => + validate(context, { + json: options.json, + depth: options.depth, + includeGated: options.includeGated, + semantic: options.semantic, + judge: options.judge, + ruleFilter: options.ruleFilter, + }), + ); } }); @@ -152,6 +191,19 @@ spacesCmd .action((options) => listSpaces(options)); program.addCommand(spacesCmd); +const providersCmd = new Command('providers').description( + 'List model providers (bundled and configured) with their effective config', +); +providersCmd + .command('list', { isDefault: true }) + .description('List bundled and configured model providers; keys are never shown') + .option('--json', 'Output as JSON') + .action(async (options) => { + const { listProviders } = await import('./commands/providers'); + listProviders(options); + }); +program.addCommand(providersCmd); + const schemasCmd = new Command('schemas').alias('schema').description('List and inspect schemas'); schemasCmd .command('list', { isDefault: true }) @@ -176,7 +228,7 @@ program.addCommand(schemasCmd); program .command('docs [topic]') - .description('Show documentation (no arg: README; topics: concepts, config, schema, rules)') + .description('Show documentation (no arg: README; topics: concepts, config, schema, rules, semantic-rules)') .action((topic?: string) => docs(topic)); const renderCmd = new Command('render').description('Render a space in a given format'); diff --git a/src/judge/factory.ts b/src/judge/factory.ts new file mode 100644 index 0000000..66f5a34 --- /dev/null +++ b/src/judge/factory.ts @@ -0,0 +1,189 @@ +import type { AnthropicJudgeProviderOptions } from './provider/anthropic'; +import { AnthropicJudgeProvider, RECOMMENDED_ANTHROPIC_MODEL } from './provider/anthropic'; +import { FakeJudgeProvider } from './provider/fake'; +import type { JevJudgeProviderOptions } from './provider/jev'; +import { JEV_LATEST, JevJudgeProvider } from './provider/jev'; +import { OpenAIJudgeProvider, RECOMMENDED_OPENAI_MODEL } from './provider/openai'; +import type { OpenAIDecisionsJudgeProviderOptions } from './provider/openai-decisions'; +import { OpenAIDecisionsJudgeProvider, RECOMMENDED_OPENAI_DECISIONS_MODEL } from './provider/openai-decisions'; +import { + type HydratedProvider, + MODEL_PROVIDER_APIS, + type ModelProviderConfig, + type SemanticJudgeProvider, +} from './types'; + +type Env = Record; + +const OPENROUTER_COMPLETIONS_ENDPOINT = 'https://openrouter.ai/api/v1'; + +const RECOMMENDED_OPENROUTER_OPENAI_DECISIONS_MODEL = `openai/${RECOMMENDED_OPENAI_DECISIONS_MODEL}-decisions`; +const RECOMMENDED_OPENROUTER_OPENAI_MODEL = `openai/${RECOMMENDED_OPENAI_MODEL}`; + +const ANTHROPIC_KEY_ENV = 'ANTHROPIC_API_KEY'; +const OPENAI_KEY_ENV = 'OPENAI_API_KEY'; +const OPENROUTER_KEY_ENV = 'OPENROUTER_API_KEY'; + +const ANTHROPIC_BASE: ModelProviderConfig = { + api: 'anthropic-messages', + model: RECOMMENDED_ANTHROPIC_MODEL, + apiKeyEnv: ANTHROPIC_KEY_ENV, +}; +const OPENAI_BASE: ModelProviderConfig = { + api: 'openai-completions', + model: RECOMMENDED_OPENAI_MODEL, + apiKeyEnv: OPENAI_KEY_ENV, +}; + +/** + * Bundled providers: usable as a judge by name, without a `modelProviders` entry. A user entry of the + * same name replaces one wholesale; fields it leaves out take the type defaults in the constructors. + * Every value here is explicit: no bundle reads the environment. + */ +export const BUNDLED_PROVIDERS: Record = { + 'typesafe-jev': { api: 'typesafe-systemone', model: JEV_LATEST, apiKeyEnv: 'TYPESAFE_API_KEY' }, + 'openrouter-jev': { api: 'openrouter-decisions', model: `~typesafe/${JEV_LATEST}`, apiKeyEnv: OPENROUTER_KEY_ENV }, + 'openrouter-openai-decisions': { + api: 'openrouter-decisions', + model: RECOMMENDED_OPENROUTER_OPENAI_DECISIONS_MODEL, + apiKeyEnv: OPENROUTER_KEY_ENV, + }, + 'openai-decisions': { + api: 'openai-decisions', + model: RECOMMENDED_OPENAI_DECISIONS_MODEL, + apiKeyEnv: OPENAI_KEY_ENV, + }, + anthropic: ANTHROPIC_BASE, + 'anthropic-haiku-5-5': { api: 'anthropic-messages', model: 'claude-haiku-5-5', apiKeyEnv: ANTHROPIC_KEY_ENV }, + 'anthropic-env': { + ...ANTHROPIC_BASE, + modelEnv: 'ANTHROPIC_MODEL', + }, + openai: OPENAI_BASE, + 'openai-env': { + ...OPENAI_BASE, + modelEnv: 'OPENAI_MODEL', + endpointEnv: 'OPENAI_BASE_URL', + }, + 'openrouter-openai': { + api: 'openai-completions', + endpoint: OPENROUTER_COMPLETIONS_ENDPOINT, + model: RECOMMENDED_OPENROUTER_OPENAI_MODEL, + apiKeyEnv: OPENROUTER_KEY_ENV, + }, +} as const; + +/** Each API reads its key from this variable unless `apiKeyEnv` names another one. Model and endpoint never read env vars. */ +const DEFAULT_KEY_ENV: Record = { + 'anthropic-messages': 'ANTHROPIC_API_KEY', + 'openai-completions': 'OPENAI_API_KEY', + 'openai-decisions': 'OPENAI_API_KEY', + 'openrouter-decisions': 'OPENROUTER_API_KEY', + 'typesafe-systemone': 'TYPESAFE_API_KEY', +} as const; + +/** The judge is misconfigured (no provider, unknown provider, missing key). Reported as a plain CLI error. */ +export class JudgeConfigError extends Error {} + +/** + * Hydrate a provider definition into concrete values, in one place. Per field: the explicit value + * wins, then the named environment variable if it is set (a set-but-empty variable is an error, not + * a silent fall-through), then the definition's own default. The key is special: it must resolve. + */ +export function hydrateProvider(name: string, def: ModelProviderConfig, env: Env = process.env): HydratedProvider { + if (!(MODEL_PROVIDER_APIS as readonly string[]).includes(def.api)) { + throw new JudgeConfigError( + `Model provider "${name}" has unknown api "${def.api}". Use one of: ${MODEL_PROVIDER_APIS.join(', ')}.`, + ); + } + const fromEnv = (value: string | undefined, envName: string | undefined): string | undefined => { + if (value !== undefined) return value; + if (!envName) return undefined; + const envValue = env[envName]; + if (envValue !== undefined && envValue.trim() === '') { + throw new JudgeConfigError( + `Model provider "${name}" has ${envName}=${envName} set but empty; set a value or unset the variable.`, + ); + } + return envValue; + }; + const model = fromEnv(def.model, def.modelEnv); + const endpoint = fromEnv(def.endpoint, def.endpointEnv); + const keyEnv = def.apiKeyEnv ?? DEFAULT_KEY_ENV[def.api]; + const apiKey = def.apiKey ?? (keyEnv ? env[keyEnv] : undefined); + return { api: def.api, model, endpoint, apiKey, keyEnv }; +} + +/** + * Instantiate one provider from a definition. The API is always explicit and is never inferred from + * API keys in the environment, so a key set for another tool is never spent on judging. Constructors + * apply their own API defaults for fields left undefined after hydration. + */ +export function createModelProvider( + name: string, + def: ModelProviderConfig, + env: Env = process.env, +): SemanticJudgeProvider { + const h = hydrateProvider(name, def, env); + // OpenAI-compatible endpoints (Ollama, vLLM, ...) may need no key; the others must resolve one. + // A key never borrows another provider's variable: only the named apiKeyEnv (or literal apiKey) counts. + if (!h.apiKey && h.api !== 'openai-completions' && h.api !== 'fake') { + throw new JudgeConfigError( + `Model provider "${name}" (${h.api}) has no API key. Set ${h.keyEnv} in the environment, or "apiKey" / "apiKeyEnv" on the provider.`, + ); + } + switch (h.api) { + case 'fake': + return new FakeJudgeProvider(); + case 'anthropic-messages': + return new AnthropicJudgeProvider(h as AnthropicJudgeProviderOptions); + case 'openrouter-decisions': + case 'typesafe-systemone': + return new JevJudgeProvider(h as JevJudgeProviderOptions); + case 'openai-decisions': + return new OpenAIDecisionsJudgeProvider(h as OpenAIDecisionsJudgeProviderOptions); + case 'openai-completions': + return new OpenAIJudgeProvider(h); + } +} + +/** + * The judge a CLI run asks for: none without `--semantic` or `--judge `; the named provider with + * `--judge`; else the configured one. + */ +export function judgeForRun( + config: Parameters[0], + space: Parameters[1], + options: { semantic?: boolean; judge?: string }, +): SemanticJudgeProvider | undefined { + if (!options.semantic && options.judge === undefined) return undefined; + return createJudge(config, space, options.judge); +} + +/** + * The judge for a run: the model provider named by `name` (the `--judge` override), else the space's + * `judge.provider`, else the top-level one. The name resolves against `modelProviders` first, then + * the bundled providers. Throws {@link JudgeConfigError} when it cannot be built. + */ +export function createJudge( + config: { modelProviders?: Record; judge?: { provider?: string } }, + space: { judge?: { provider?: string } } = {}, + name?: string, + env: Env = process.env, +): SemanticJudgeProvider { + const chosen = name ?? space.judge?.provider ?? config.judge?.provider; + if (!chosen) { + throw new JudgeConfigError( + 'No judge configured. Set "judge.provider" in config.json to a bundled provider (e.g. "openrouter-jev") or to the name of an entry in "modelProviders".', + ); + } + const def = config.modelProviders?.[chosen] ?? BUNDLED_PROVIDERS[chosen]; + if (!def) { + const known = [...Object.keys(BUNDLED_PROVIDERS), ...Object.keys(config.modelProviders ?? {})]; + + throw new JudgeConfigError( + `Model provider "${chosen}" is not configured.${known.length ? ` Known providers: ${known.join(', ')}.` : ''}`, + ); + } + return createModelProvider(chosen, def, env); +} diff --git a/src/judge/index.ts b/src/judge/index.ts new file mode 100644 index 0000000..2068eb3 --- /dev/null +++ b/src/judge/index.ts @@ -0,0 +1,21 @@ +export { createJudge, createModelProvider, JudgeConfigError } from './factory'; +export { buildJudgeSystemPrompt, buildJudgeUserPrompt, parseJudgeResponse } from './prompt'; +export type { AnthropicJudgeProviderOptions } from './provider/anthropic'; +export { AnthropicJudgeProvider } from './provider/anthropic'; +export type { FakeAnswer, FakeJudgeHandler } from './provider/fake'; +export { FakeJudgeProvider } from './provider/fake'; +export type { JevJudgeProviderOptions } from './provider/jev'; +export { JevJudgeProvider } from './provider/jev'; +export type { OpenAIJudgeProviderOptions } from './provider/openai'; +export { OpenAIJudgeProvider } from './provider/openai'; +export { PerRuleJudgeProvider } from './provider/per-rule'; +export type { + JudgeConfig, + ModelProviderConfig, + SemanticJudgeInput, + SemanticJudgeProvider, + SemanticJudgeResult, + SemanticJudgeVerdict, + SemanticRuleSpec, +} from './types'; +export { DEFAULT_VERDICTS, isJudgeError, semanticRuleSpec } from './verdicts'; diff --git a/src/judge/judge-node.ts b/src/judge/judge-node.ts new file mode 100644 index 0000000..7ab7edb --- /dev/null +++ b/src/judge/judge-node.ts @@ -0,0 +1,65 @@ +import type { Rule, RuleSeverity, SpaceNode } from '../types'; +import type { SemanticJudgeProvider } from './types'; +import { isJudgeError, resolveSemanticRule } from './verdicts'; + +export type NodeJudgement = { + rule: Rule; + /** True when the verdict fails the rule (it has a severity), or the judge errored. */ + failed: boolean; + /** Severity of the failing verdict; unset for a judge error, which takes the rule's own severity. */ + severity?: RuleSeverity; + verdict?: string; + /** User-facing guidance declared for the failing verdict. */ + guidance?: string; + evidence?: string; + /** Set when the judge could not produce a verdict for this rule. */ + error?: string; +}; + +/** Judge every given semantic rule on a node in one provider call. */ +export async function judgeNode( + judge: SemanticJudgeProvider, + node: SpaceNode, + rules: Rule[], +): Promise { + if (rules.length === 0) return []; + const resolved = rules.map(resolveSemanticRule); + const specs = resolved.map((r) => r.spec); + let results: Awaited>; + try { + results = await judge.evaluate({ + node, + rules: specs, + }); + } catch (err) { + const message = err instanceof Error ? err.message : String(err); + return rules.map((rule) => ({ + rule, + failed: true, + evidence: `Error during evaluation: ${message}`, + error: message, + })); + } + + return rules.map((rule, i) => { + const result = results[rule.id]; + if (!result || isJudgeError(result)) { + const message = result ? result.error : 'Judge returned no result for this rule'; + return { rule, failed: true, evidence: `Error during evaluation: ${message}`, error: message }; + } + const severity = resolved[i]!.severities[result.verdict]; + if (severity === undefined && !(result.verdict in resolved[i]!.spec.verdicts)) { + const message = `Judge returned unknown verdict "${result.verdict}" (allowed: ${Object.keys(resolved[i]!.spec.verdicts).join(', ')})`; + return { rule, failed: true, evidence: `Error during evaluation: ${message}`, error: message }; + } + const failed = severity !== undefined; + return { + rule, + failed, + severity, + verdict: result.verdict, + guidance: failed ? resolved[i]!.guidance[result.verdict] : undefined, + evidence: result.evidence, + }; + }); +} diff --git a/src/judge/prompt.ts b/src/judge/prompt.ts new file mode 100644 index 0000000..1d62876 --- /dev/null +++ b/src/judge/prompt.ts @@ -0,0 +1,92 @@ +import type { SemanticJudgeInput, SemanticJudgeResult, SemanticRuleSpec } from './types'; + +function verdictList(rule: SemanticRuleSpec): string[] { + return Object.entries(rule.verdicts).map(([name, description]) => `- "${name}": ${description}`); +} + +/** System prompt: a strict judge that answers with one verdict from a closed set, as JSON. */ +export function buildJudgeSystemPrompt(): string { + return [ + 'You are a strict, objective evaluation judge for structured planning and knowledge documents.', + 'Your task is to choose the one verdict that best describes whether a node satisfies a rule.', + '', + 'You must respond ONLY with a JSON object in this exact format:', + '{', + ' "verdict": "",', + ' "confidence": ,', + ' "evidence": "concise explanation citing specific quotes, phrases, or absent details from the node"', + '}', + '', + 'Guidelines:', + '1. "verdict" must be one of the allowed verdicts listed for the rule, spelled exactly.', + '2. "confidence" is how sure you are of the verdict.', + '3. "evidence" must directly cite or quote relevant parts of the node, explaining the reasoning concisely.', + '4. Do not include markdown code block formatting, intro text, or outro text. Respond only with the JSON object.', + ].join('\n'); +} + +/** User prompt for one rule on one node. */ +export function buildJudgeUserPrompt(node: SemanticJudgeInput['node'], rule: SemanticRuleSpec): string { + const rubric = rule.rubric || rule.description; + const lines: string[] = [ + '# Rule to Evaluate', + `- ID: ${rule.id}`, + `- Category: ${rule.category}`, + `- Rubric: ${rubric}`, + '', + '## Allowed verdicts', + ...verdictList(rule), + '', + '# Node Under Evaluation', + `- File / Label: ${node.label}`, + `- Title: ${node.title}`, + `- Type: ${node.type} (resolved: ${node.resolvedType})`, + ]; + + if (node.resolvedParents && node.resolvedParents.length > 0) { + lines.push(`- Parents: ${node.resolvedParents.map((p) => p.title).join(', ')}`); + } + + lines.push('', '## Node Content & Frontmatter:', '```json', JSON.stringify(node.schemaData, null, 2), '```', ''); + lines.push('Evaluate this node against the rule. Return only JSON with "verdict", "confidence" and "evidence".'); + return lines.join('\n'); +} + +/** + * Parse judge output into a verdict from the rule's allowed set. + * Handles clean JSON, markdown-wrapped JSON, and a regex fallback for surrounding text. + */ +export function parseJudgeResponse(rawText: string, allowed: string[]): SemanticJudgeResult { + let cleaned = rawText.trim(); + cleaned = cleaned.replace(/^```(?:json)?\s*/, '').replace(/\s*```$/, ''); + + let verdict: string | undefined; + let confidence: number | undefined; + let evidence: string | undefined; + + try { + const parsed = JSON.parse(cleaned); + if (typeof parsed.verdict === 'string') verdict = parsed.verdict.trim(); + if (typeof parsed.confidence === 'number') confidence = parsed.confidence; + if (typeof parsed.evidence === 'string') evidence = parsed.evidence.trim(); + } catch { + const verdictMatch = /"verdict"\s*:\s*"((?:[^"\\]|\\.)*)"/i.exec(cleaned); + const confidenceMatch = /"confidence"\s*:\s*([0-9.]+)/i.exec(cleaned); + const evidenceMatch = /"evidence"\s*:\s*"((?:[^"\\]|\\.)*)"/i.exec(cleaned); + verdict = verdictMatch?.[1]?.trim(); + confidence = confidenceMatch ? Number(confidenceMatch[1]) : undefined; + evidence = evidenceMatch?.[1]?.replace(/\\"/g, '"').replace(/\\n/g, '\n').trim(); + } + + if (verdict === undefined || !allowed.includes(verdict)) { + return { + error: `Judge returned ${verdict === undefined ? 'no verdict' : `unknown verdict "${verdict}"`} (allowed: ${allowed.join(', ')}): ${rawText.slice(0, 200)}`, + }; + } + return { + verdict, + ...(confidence !== undefined && + Number.isFinite(confidence) && { confidence: Math.min(1, Math.max(0, confidence)) }), + ...(evidence && { evidence }), + }; +} diff --git a/src/judge/provider/anthropic.ts b/src/judge/provider/anthropic.ts new file mode 100644 index 0000000..9f3bd6c --- /dev/null +++ b/src/judge/provider/anthropic.ts @@ -0,0 +1,55 @@ +import { buildJudgeSystemPrompt, buildJudgeUserPrompt, parseJudgeResponse } from '../prompt'; +import type { HydratedProvider, SemanticJudgeInput, SemanticJudgeResult, SemanticRuleSpec } from '../types'; +import { PerRuleJudgeProvider } from './per-rule'; + +export interface AnthropicJudgeProviderOptions extends HydratedProvider { + /** The factory requires a resolved key before constructing a non-OpenAI provider. */ + apiKey?: string; +} + +export const RECOMMENDED_ANTHROPIC_MODEL = 'claude-haiku-5-5'; +export const ANTHROPIC_MESSAGES_ENDPOINT = 'https://api.anthropic.com/v1/messages'; + +/** Judge provider for the Anthropic Messages API. Reads no environment: it takes hydrated values. */ +export class AnthropicJudgeProvider extends PerRuleJudgeProvider { + readonly name = 'anthropic'; + private apiKey: string; + private model: string; + private endpoint: string; + + constructor(options: AnthropicJudgeProviderOptions) { + super(); + this.apiKey = options.apiKey ?? ''; + this.model = options.model || RECOMMENDED_ANTHROPIC_MODEL; + this.endpoint = options.endpoint || ANTHROPIC_MESSAGES_ENDPOINT; + } + + protected async judgeRule(node: SemanticJudgeInput['node'], rule: SemanticRuleSpec): Promise { + const res = await fetch(this.endpoint, { + method: 'POST', + headers: { + 'content-type': 'application/json', + 'x-api-key': this.apiKey, + 'anthropic-version': '2023-06-01', + }, + body: JSON.stringify({ + model: this.model, + max_tokens: 1024, + system: buildJudgeSystemPrompt(), + messages: [{ role: 'user', content: buildJudgeUserPrompt(node, rule) }], + }), + }); + + if (!res.ok) { + const errorText = await res.text().catch(() => ''); + throw new Error(`Anthropic API error (${res.status}): ${errorText}`); + } + + const data = (await res.json()) as { content?: Array<{ type: string; text?: string }> }; + const text = data.content?.find((c) => c.type === 'text')?.text; + if (!text) { + throw new Error('Anthropic API returned empty or missing text block in response.'); + } + return parseJudgeResponse(text, Object.keys(rule.verdicts)); + } +} diff --git a/src/judge/provider/fake.ts b/src/judge/provider/fake.ts new file mode 100644 index 0000000..863c4fc --- /dev/null +++ b/src/judge/provider/fake.ts @@ -0,0 +1,71 @@ +import type { SemanticJudgeInput, SemanticJudgeProvider, SemanticJudgeResult, SemanticRuleSpec } from '../types'; + +/** A fake answer: a full verdict, or the `{ pass }` shorthand mapped onto the default `pass`/`fail` verdicts. */ +export type FakeAnswer = SemanticJudgeResult | { pass: boolean; evidence?: string }; + +export type FakeJudgeHandler = (call: { + node: SemanticJudgeInput['node']; + rule: SemanticRuleSpec; +}) => Promise | FakeAnswer; + +type FakeConfig = FakeAnswer | FakeJudgeHandler | Map | Record; + +function isHandler(c: FakeConfig): c is FakeJudgeHandler { + return typeof c === 'function'; +} + +function isAnswer(c: FakeConfig): c is FakeAnswer { + return 'pass' in c || 'verdict' in c || 'error' in c; +} + +function normalise(answer: FakeAnswer): SemanticJudgeResult { + if ('pass' in answer) { + return { verdict: answer.pass ? 'pass' : 'fail', evidence: answer.evidence }; + } + return answer; +} + +/** + * In-memory fake judge for offline, deterministic tests. Configure with a single answer, a handler, + * or answers keyed by `ruleId:title`, `ruleId:label` or `ruleId` (most specific wins). Unmatched calls get `pass`, + * or the rule's first declared verdict. + */ +export class FakeJudgeProvider implements SemanticJudgeProvider { + readonly name = 'fake'; + public callHistory: SemanticJudgeInput[] = []; + + constructor(private config?: FakeConfig) {} + + async evaluate(input: SemanticJudgeInput): Promise> { + this.callHistory.push(input); + const results: Record = {}; + for (const rule of input.rules) { + results[rule.id] = normalise(await this.answer(input.node, rule)); + } + return results; + } + + private async answer(node: SemanticJudgeInput['node'], rule: SemanticRuleSpec): Promise { + // The fake does not know severities: default to `pass`, else the first declared verdict. + const passing = { verdict: 'pass' in rule.verdicts ? 'pass' : (Object.keys(rule.verdicts)[0] ?? 'pass') }; + const fallback = { ...passing, evidence: `Fake passed evaluation for rule "${rule.id}"` }; + const config = this.config; + if (!config) return fallback; + if (isHandler(config)) return await config({ node, rule }); + if (config instanceof Map) { + return ( + config.get(`${rule.id}:${node.title}`) ?? + config.get(`${rule.id}:${node.label}`) ?? + config.get(rule.id) ?? + fallback + ); + } + if (isAnswer(config)) return config; + const rec = config as Record; + return rec[`${rule.id}:${node.title}`] ?? rec[`${rule.id}:${node.label}`] ?? rec[rule.id] ?? fallback; + } + + clearHistory(): void { + this.callHistory = []; + } +} diff --git a/src/judge/provider/jev.ts b/src/judge/provider/jev.ts new file mode 100644 index 0000000..6ab9b10 --- /dev/null +++ b/src/judge/provider/jev.ts @@ -0,0 +1,93 @@ +import type { HydratedProvider, SemanticJudgeInput, SemanticJudgeProvider, SemanticJudgeResult } from '../types'; + +export interface JevJudgeProviderOptions extends HydratedProvider { + apiKey?: string; +} + +export const JEV_LATEST = 'jev-latest'; + +export const OPENROUTER_DECISIONS_ENDPOINT = 'https://openrouter.ai/api/alpha/decisions'; + +type JevChoiceAnswer = { + type?: string; + choice?: string; + confidence?: number; + probabilities?: Record; +}; + +/** + * Judge provider for TypeSafe's Jev decision model via OpenRouter's Decisions API. + * + * Each node is sent once as `state`, with one `choice` question per semantic rule whose criteria are + * the rule's closed verdict set. Jev returns the chosen verdict with confidence and probabilities, + * but no prose, so `evidence` is a summary of the distribution. + */ +export class JevJudgeProvider implements SemanticJudgeProvider { + readonly name = 'jev'; + private apiKey: string; + private model: string; + private endpoint: string; + + constructor(options: JevJudgeProviderOptions) { + this.apiKey = options.apiKey ?? ''; + this.model = options.model || JEV_LATEST; + this.endpoint = options.endpoint || OPENROUTER_DECISIONS_ENDPOINT; + } + + async evaluate(input: SemanticJudgeInput): Promise> { + const { node, rules } = input; + const questions: Record = {}; + for (const rule of rules) { + questions[rule.id] = { + type: 'choice', + instructions: rule.rubric || rule.description, + criteria: rule.verdicts, + }; + } + + const res = await fetch(this.endpoint, { + method: 'POST', + headers: { 'content-type': 'application/json', authorization: `Bearer ${this.apiKey}` }, + body: JSON.stringify({ + model: this.model, + state: { + title: node.title, + type: node.resolvedType, + parents: node.resolvedParents?.map((p) => p.title) ?? [], + ...node.schemaData, + }, + questions, + }), + }); + + if (!res.ok) { + const errorText = await res.text().catch(() => ''); + throw new Error(`Jev API error (${res.status}): ${errorText}`); + } + + const data = (await res.json()) as { answers?: Record }; + const results: Record = {}; + for (const rule of rules) { + const answer = data.answers?.[rule.id]; + if (!answer?.choice || !(answer.choice in rule.verdicts)) { + results[rule.id] = { error: `Jev returned no valid verdict for rule "${rule.id}" (choice: ${answer?.choice})` }; + continue; + } + results[rule.id] = { + verdict: answer.choice, + confidence: answer.confidence, + probabilities: answer.probabilities, + evidence: describeDistribution(answer), + }; + } + return results; + } +} + +function describeDistribution(answer: JevChoiceAnswer): string { + const parts = Object.entries(answer.probabilities ?? {}) + .sort(([, a], [, b]) => b - a) + .map(([verdict, p]) => `${verdict} ${p.toFixed(2)}`); + const confidence = answer.confidence !== undefined ? `confidence ${answer.confidence.toFixed(2)}` : undefined; + return `Jev chose "${answer.choice}"${[confidence, ...parts].filter(Boolean).length ? ` (${[confidence, ...parts].filter(Boolean).join('; ')})` : ''}`; +} diff --git a/src/judge/provider/openai-decisions.ts b/src/judge/provider/openai-decisions.ts new file mode 100644 index 0000000..072c999 --- /dev/null +++ b/src/judge/provider/openai-decisions.ts @@ -0,0 +1,101 @@ +import type { HydratedProvider, SemanticJudgeInput, SemanticJudgeProvider, SemanticJudgeResult } from '../types'; + +export interface OpenAIDecisionsJudgeProviderOptions extends HydratedProvider { + apiKey?: string; +} + +export const RECOMMENDED_OPENAI_DECISIONS_MODEL = 'gpt-6-luna'; +export const OPENAI_DECISIONS_ENDPOINT = 'https://api.openai.com/v1/decisions'; + +type OpenAIDecisionAnswer = { + type?: string; + name?: string; + choice?: string; + confidence?: number; + probabilities?: Array<{ value: string; probability: number }>; +}; + +/** + * Judge provider for OpenAI's Decisions API (https://developers.openai.com/api/docs/guides/decisions). + * + * Each node is sent once as the `input` text, with one `choice` question per semantic rule whose + * choices are the rule's closed verdict set. The API returns the chosen value with per-choice + * probabilities and a confidence, but no prose, so `evidence` is a summary of the distribution. + */ +export class OpenAIDecisionsJudgeProvider implements SemanticJudgeProvider { + readonly name = 'openai-decisions'; + private apiKey: string; + private model: string; + private endpoint: string; + + constructor(options: OpenAIDecisionsJudgeProviderOptions) { + this.apiKey = options.apiKey ?? ''; + this.model = options.model || RECOMMENDED_OPENAI_DECISIONS_MODEL; + this.endpoint = options.endpoint || OPENAI_DECISIONS_ENDPOINT; + } + + async evaluate(input: SemanticJudgeInput): Promise> { + const { node, rules } = input; + + const stateLines = [ + `Title: ${node.title}`, + `Type: ${node.resolvedType}`, + ...(node.resolvedParents?.length ? [`Parents: ${node.resolvedParents.map((p) => p.title).join(', ')}`] : []), + ...Object.entries(node.schemaData ?? {}) + .filter(([field]) => field !== 'title' && field !== 'type') + .map(([field, value]) => `${field}: ${JSON.stringify(value) ?? String(value)}`), + ]; + + const res = await fetch(this.endpoint, { + method: 'POST', + headers: { 'content-type': 'application/json', authorization: `Bearer ${this.apiKey}` }, + body: JSON.stringify({ + model: this.model, + input: stateLines.join('\n'), + questions: rules.map((rule) => ({ + type: 'choice', + name: rule.id, + instructions: rule.rubric || rule.description, + choices: Object.entries(rule.verdicts).map(([value, description]) => ({ value, description })), + })), + }), + }); + + if (!res.ok) { + const errorText = await res.text().catch(() => ''); + throw new Error(`OpenAI Decisions API error (${res.status}): ${errorText}`); + } + + const data = (await res.json()) as { answers?: OpenAIDecisionAnswer[] }; + const answers = new Map((data.answers ?? []).filter((a) => a.name).map((a) => [a.name as string, a])); + const results: Record = {}; + for (const rule of rules) { + const answer = answers.get(rule.id); + if (!answer || answer.type === 'refusal' || !answer.choice || !(answer.choice in rule.verdicts)) { + results[rule.id] = { error: `OpenAI Decisions returned no valid verdict for rule "${rule.id}"` }; + continue; + } + const probabilities = Object.fromEntries((answer.probabilities ?? []).map((p) => [p.value, p.probability])); + results[rule.id] = { + verdict: answer.choice, + confidence: answer.confidence, + probabilities, + evidence: describeDistribution(answer.choice, answer.confidence, probabilities), + }; + } + return results; + } +} + +function describeDistribution( + choice: string, + confidence: number | undefined, + probabilities: Record, +): string { + const parts = Object.entries(probabilities) + .sort(([, a], [, b]) => b - a) + .map(([verdict, p]) => `${verdict} ${p.toFixed(2)}`); + const conf = confidence !== undefined ? `confidence ${confidence.toFixed(2)}` : undefined; + const detail = [conf, ...parts].filter(Boolean).join('; '); + return `OpenAI Decisions chose "${choice}"${detail ? ` (${detail})` : ''}`; +} diff --git a/src/judge/provider/openai.ts b/src/judge/provider/openai.ts new file mode 100644 index 0000000..a624306 --- /dev/null +++ b/src/judge/provider/openai.ts @@ -0,0 +1,69 @@ +import { buildJudgeSystemPrompt, buildJudgeUserPrompt, parseJudgeResponse } from '../prompt'; +import type { HydratedProvider, SemanticJudgeInput, SemanticJudgeResult, SemanticRuleSpec } from '../types'; +import { PerRuleJudgeProvider } from './per-rule'; + +export interface OpenAIJudgeProviderOptions extends HydratedProvider { + /** Optional: local OpenAI-compatible servers need no key. */ + apiKey?: string; +} + +export const RECOMMENDED_OPENAI_MODEL = 'gpt-6-luna'; +export const OPENAI_COMPLETIONS_ENDPOINT = 'https://api.openai.com/v1'; + +/** + * Judge provider for the OpenAI-compatible Chat Completions API (OpenAI, Ollama, vLLM, ...). + * Reads no environment: it takes hydrated values. + */ +export class OpenAIJudgeProvider extends PerRuleJudgeProvider { + readonly name = 'openai'; + private apiKey?: string; + private model: string; + private endpoint: string; + + constructor(options: OpenAIJudgeProviderOptions) { + super(); + this.apiKey = options.apiKey; + this.model = options.model || RECOMMENDED_OPENAI_MODEL; + + const base = options.endpoint || OPENAI_COMPLETIONS_ENDPOINT; + this.endpoint = base.endsWith('/chat/completions') ? base : `${base.replace(/\/+$/, '')}/chat/completions`; + } + + protected async judgeRule(node: SemanticJudgeInput['node'], rule: SemanticRuleSpec): Promise { + const systemPrompt = buildJudgeSystemPrompt(); + const userPrompt = buildJudgeUserPrompt(node, rule); + + const headers: Record = { + 'content-type': 'application/json', + }; + if (this.apiKey) { + headers.authorization = `Bearer ${this.apiKey}`; + } + + const res = await fetch(this.endpoint, { + method: 'POST', + headers, + body: JSON.stringify({ + model: this.model, + response_format: { type: 'json_object' }, + messages: [ + { role: 'system', content: systemPrompt }, + { role: 'user', content: userPrompt }, + ], + }), + }); + + if (!res.ok) { + const errorText = await res.text().catch(() => ''); + throw new Error(`OpenAI API error (${res.status}): ${errorText}`); + } + + const data = (await res.json()) as { choices?: Array<{ message?: { content?: string } }> }; + const content = data.choices?.[0]?.message?.content; + if (!content) { + throw new Error('OpenAI API returned empty or missing message content in response.'); + } + + return parseJudgeResponse(content, Object.keys(rule.verdicts)); + } +} diff --git a/src/judge/provider/per-rule.ts b/src/judge/provider/per-rule.ts new file mode 100644 index 0000000..a309c49 --- /dev/null +++ b/src/judge/provider/per-rule.ts @@ -0,0 +1,23 @@ +import type { SemanticJudgeInput, SemanticJudgeProvider, SemanticJudgeResult, SemanticRuleSpec } from '../types'; + +/** + * Base for providers that can only judge one rule per request (chat LLMs). `evaluate` makes one call + * per rule; a failure on one rule is reported for that rule and does not affect the others. + */ +export abstract class PerRuleJudgeProvider implements SemanticJudgeProvider { + abstract readonly name: string; + + protected abstract judgeRule(node: SemanticJudgeInput['node'], rule: SemanticRuleSpec): Promise; + + async evaluate(input: SemanticJudgeInput): Promise> { + const results: Record = {}; + for (const rule of input.rules) { + try { + results[rule.id] = await this.judgeRule(input.node, rule); + } catch (err) { + results[rule.id] = { error: err instanceof Error ? err.message : String(err) }; + } + } + return results; + } +} diff --git a/src/judge/types.ts b/src/judge/types.ts new file mode 100644 index 0000000..79417d1 --- /dev/null +++ b/src/judge/types.ts @@ -0,0 +1,82 @@ +import type { RuleCategory, SpaceNode } from '../types'; + +/** A semantic rule as presented to a judge: the closed verdict set it must choose from. */ +export interface SemanticRuleSpec { + id: string; + description: string; + rubric?: string; + category: RuleCategory | string; + /** Allowed verdicts, each with a description of when it applies. */ + verdicts: Record; +} + +/** One node and every semantic rule to judge on it. Judges may answer all rules in a single call. */ +export interface SemanticJudgeInput { + node: Pick; + rules: SemanticRuleSpec[]; +} + +/** A judge's answer for one rule: a verdict from the rule's closed set. */ +export interface SemanticJudgeVerdict { + verdict: string; + /** 0-1, when the judge reports one. */ + confidence?: number; + /** Probability per verdict, when the judge reports a distribution. */ + probabilities?: Record; + /** Free-text reasoning; decision models may not provide it. */ + evidence?: string; +} + +/** A judge's answer for one rule: a verdict, or an error that applies to that rule only. */ +export type SemanticJudgeResult = SemanticJudgeVerdict | { error: string }; + +/** Judge interface. Results are keyed by rule id; a missing key is treated as an error. */ +export interface SemanticJudgeProvider { + readonly name: string; + evaluate(input: SemanticJudgeInput): Promise>; +} + +/** Wire APIs a provider can speak. Names the request shape, never the vendor or model. */ +export const MODEL_PROVIDER_APIS = [ + 'openai-completions', + 'openai-decisions', + 'anthropic-messages', + 'openrouter-decisions', + 'typesafe-systemone', + 'fake', +] as const; +export type ModelProviderApi = (typeof MODEL_PROVIDER_APIS)[number]; + +/** How to reach a model: a bundled provider or config `modelProviders.`, before hydration. */ +export interface ModelProviderConfig { + /** The wire API to speak. */ + api: ModelProviderApi; + /** Fixed model override. Beats `modelEnv` and the bundled default. */ + model?: string; + /** Environment variable that holds the model override. */ + modelEnv?: string; + /** Fixed endpoint override. Beats `endpointEnv` and the bundled default. */ + endpoint?: string; + /** Environment variable that holds the endpoint override. */ + endpointEnv?: string; + /** Prefer `apiKeyEnv`: keys do not belong in config files. */ + apiKey?: string; + /** Environment variable holding the key. */ + apiKeyEnv?: string; +} + +/** A provider after hydration: every field is a concrete value; the implementation reads no environment. */ +export interface HydratedProvider { + api: ModelProviderApi; + model?: string; + endpoint?: string; + apiKey?: string; + /** The environment variable the key was read from, when one was named or defaulted. */ + keyEnv?: string; +} + +/** Which model provider judges semantic rules: config `judge`, at the top level or on a space. */ +export interface JudgeConfig { + /** Name of an entry in `modelProviders`, or a bundled provider. */ + provider?: string; +} diff --git a/src/judge/verdicts.ts b/src/judge/verdicts.ts new file mode 100644 index 0000000..1f593c8 --- /dev/null +++ b/src/judge/verdicts.ts @@ -0,0 +1,82 @@ +import { ruleSeverity } from '../schema/rule-severity'; +import type { Rule, RuleSeverity } from '../types'; +import type { SemanticJudgeResult, SemanticRuleSpec } from './types'; + +/** Verdicts for a semantic rule that declares none. */ +export const DEFAULT_VERDICTS: Record = { + pass: 'The node meets the rubric.', + fail: 'The node does not meet the rubric.', +}; + +/** A semantic rule resolved for evaluation: what the judge sees, and what each verdict means for the user. */ +export interface ResolvedSemanticRule { + /** What the judge sees: the closed verdict set. */ + spec: SemanticRuleSpec; + /** Severity of each verdict that fails the rule. A verdict absent here passes (including `ignore`d ones). */ + severities: Record; + /** User-facing guidance for verdicts that declare it. */ + guidance: Record; +} + +/** + * Resolve a semantic rule: apply verdict defaults, split each verdict into its judge description and user guidance, + * and work out which verdicts fail at what severity. Throws on an inconsistent rule. + */ +export function resolveSemanticRule(rule: Rule): ResolvedSemanticRule { + const declared = rule.verdicts ?? DEFAULT_VERDICTS; + const verdicts: Record = {}; + const guidance: Record = {}; + for (const [name, value] of Object.entries(declared)) { + if (typeof value === 'string') { + verdicts[name] = value; + } else { + verdicts[name] = value.description; + if (value.guidance) guidance[name] = value.guidance; + } + } + + let mapping: Record; + if (typeof rule.severity === 'object') { + mapping = rule.severity; + } else if (rule.verdicts) { + throw new Error( + `Semantic rule "${rule.id}" declares verdicts, so its severity must map the failing verdicts, e.g. { "${Object.keys(verdicts)[0]}": "warning" }.`, + ); + } else { + mapping = { fail: ruleSeverity(rule) }; + } + + const unknown = Object.keys(mapping).filter((v) => !(v in verdicts)); + if (unknown.length > 0) { + throw new Error(`Semantic rule "${rule.id}" has severity for verdicts not in its verdicts: ${unknown.join(', ')}.`); + } + const severities: Record = {}; + for (const [verdict, severity] of Object.entries(mapping)) { + if (severity !== 'ignore') severities[verdict] = severity; + } + if (Object.keys(severities).length >= Object.keys(verdicts).length) { + throw new Error(`Semantic rule "${rule.id}" has no passing verdict: every verdict has a severity.`); + } + + return { + spec: { + id: rule.id, + description: rule.description, + rubric: rule.rubric, + category: rule.category, + verdicts, + }, + severities, + guidance, + }; +} + +/** The judge-facing spec for a semantic rule. */ +export function semanticRuleSpec(rule: Rule): SemanticRuleSpec { + return resolveSemanticRule(rule).spec; +} + +/** Whether a judge result is an error rather than a verdict. */ +export function isJudgeError(result: SemanticJudgeResult): result is { error: string } { + return 'error' in result; +} diff --git a/src/schema/evaluate-rule.ts b/src/schema/evaluate-rule.ts index dc2dc49..4db5554 100644 --- a/src/schema/evaluate-rule.ts +++ b/src/schema/evaluate-rule.ts @@ -1,5 +1,5 @@ import jsonata from 'jsonata'; -import type { SpaceNode } from '../types'; +import type { Rule, RuleViolation, SpaceNode } from '../types'; const expressionCache = new Map>(); @@ -17,7 +17,7 @@ export interface EvalContext { /** * Evaluate a JSONata expression against a context. - * Returns the result of the expression (boolean for rule checks). + * Returns the result of the expression (boolean for rule checks). Throws {@link ExpressionError} if it fails. * * @param expr - JSONata expression string * @param context - Evaluation context with $, $$, parent, $lookup @@ -51,9 +51,31 @@ export async function evaluateExpression(expr: string, context: EvalContext): Pr const result = await expression.evaluate(input); return result as boolean | string | number; } catch (error) { - // Log warning and return false (fail safe) - console.warn(`Warning: Error evaluating expression "${expr}":`, error); - return false; + throw new ExpressionError(expr, error); + } +} + +/** A JSONata expression could not be compiled or evaluated. */ +export class ExpressionError extends Error { + constructor( + readonly expression: string, + cause: unknown, + ) { + // JSONata throws plain objects ({ code, message, position }), not Error instances. + const reason = (cause as { message?: unknown } | null)?.message ?? cause; + super(`Expression "${expression}" failed: ${String(reason)}`); + } +} + +/** Result of evaluating a rule's `check` or `when`: a value, or the error that stopped it (never thrown). */ +export type ExpressionOutcome = { value: boolean | string | number } | { error: string }; + +/** Like {@link evaluateExpression} but reports a failure as data, so callers can turn it into a violation. */ +export async function tryEvaluateExpression(expr: string, context: EvalContext): Promise { + try { + return { value: await evaluateExpression(expr, context) }; + } catch (error) { + return { error: error instanceof ExpressionError ? error.message : String(error) }; } } @@ -106,3 +128,14 @@ export function buildEvalContext( parents: flattenedParents.length > 0 ? flattenedParents : undefined, }; } + +/** The violation reported when a rule's `check` or `when` cannot be evaluated: an authoring error, so always `error`. */ +export function expressionErrorViolation(rule: Rule, file: string, error: string): RuleViolation { + return { + file, + ruleId: rule.id, + category: rule.category, + severity: 'error', + description: `${rule.description} (Rule error: ${error})`, + }; +} diff --git a/src/schema/gating.ts b/src/schema/gating.ts index 3bb416a..0c9588f 100644 --- a/src/schema/gating.ts +++ b/src/schema/gating.ts @@ -1,11 +1,21 @@ +import { judgeNode } from '../judge/judge-node'; +import type { SemanticJudgeProvider } from '../judge/types'; import type { SpaceGraph } from '../space-graph'; -import type { Rule, RuleViolation, SpaceNode } from '../types'; -import { buildEvalContext, evaluateExpression } from './evaluate-rule'; -import { ruleSeverity } from './validate-rules'; +import type { Rule, RuleSeverity, RuleViolation, SpaceNode } from '../types'; +import { buildEvalContext, expressionErrorViolation, tryEvaluateExpression } from './evaluate-rule'; +import { ruleSeverities, ruleSeverity } from './rule-severity'; -/** Whether a violation of this rule gates its subtree: explicit `gatesDescendants`, else true only for `error` severity. */ -export function ruleGatesDescendants(rule: Rule): boolean { - return rule.gatesDescendants ?? ruleSeverity(rule) === 'error'; +/** + * Whether a violation of this rule gates its subtree: explicit `gatesDescendants`, else true only for `error` severity. + * `severity` is that of the violation; a semantic rule's verdicts can differ. + */ +export function ruleGatesDescendants(rule: Rule, severity: RuleSeverity = ruleSeverity(rule)): boolean { + return rule.gatesDescendants ?? severity === 'error'; +} + +/** Whether any violation of this rule could gate its subtree. */ +function ruleCanGate(rule: Rule): boolean { + return rule.gatesDescendants ?? ruleSeverities(rule).includes('error'); } /** @@ -43,6 +53,19 @@ export type GatedRuleEvaluation = { gates: CoverageGate[]; }; +export interface GatingOptions { + /** Evaluate rules on gated nodes anyway (gates are still reported). */ + includeGated?: boolean; + /** Judge for `mode: semantic` rules. Without it, semantic rules are not evaluated. */ + provider?: SemanticJudgeProvider; + /** + * Label of the one node whose result is wanted (`validate-file`). Gating is still computed over the + * whole space, but the judge is only consulted for that node and for its hierarchy ancestors' gate-capable + * semantic rules (top-down, so a gated ancestor stops the walk). Other nodes' semantic rules are not judged. + */ + judgeTarget?: string; +} + /** * Compute breadth-first depth (1-based, from hierarchy roots and orphans) for every * hierarchy-typed node in the graph. Non-hierarchy nodes are not included. @@ -81,7 +104,7 @@ export async function evaluateRulesWithGating( rules: Rule[], graph: SpaceGraph, seeds: GatingSeed[], - options: { includeGated?: boolean } = {}, + options: GatingOptions = {}, ): Promise { const includeGated = options.includeGated ?? false; @@ -98,6 +121,7 @@ export async function evaluateRulesWithGating( } } + const judge = options.provider; const violations: RuleViolation[] = []; const gateOrigins = new Map(); @@ -113,13 +137,15 @@ export async function evaluateRulesWithGating( return origin; } - const globalRules = rules.filter((r) => r.scope === 'global'); + const globalRules = rules.filter((r) => r.scope === 'global' && r.mode !== 'semantic'); const sentinel = nodes[0]; if (sentinel) { for (const rule of globalRules) { const context = buildEvalContext(sentinel, nodes, nodeIndex); - const result = await evaluateExpression(rule.check, context); - if (result !== true) { + const outcome = await tryEvaluateExpression(rule.check as string, context); + if ('error' in outcome) { + violations.push(expressionErrorViolation(rule, '', outcome.error)); + } else if (outcome.value !== true) { violations.push({ file: '', ruleId: rule.id, @@ -131,24 +157,92 @@ export async function evaluateRulesWithGating( } } - const perNodeRules = rules.filter((r) => r.scope !== 'global'); + const perNodeRules = rules.filter((r) => r.scope !== 'global' && (r.mode !== 'semantic' || judge)); + + // With a judge target, only the target and its hierarchy ancestors can need judging. + const targetAncestors = new Set(); + if (options.judgeTarget !== undefined) { + const parentsOf = new Map(); + for (const [parent, children] of graph.hierarchyChildren) { + for (const child of children) parentsOf.set(child.title, [...(parentsOf.get(child.title) ?? []), parent]); + } + const target = nodes.find((n) => n.label === options.judgeTarget); + const pending = target ? [target.title] : []; + while (pending.length > 0) { + for (const parent of parentsOf.get(pending.pop()!) ?? []) { + if (!targetAncestors.has(parent)) { + targetAncestors.add(parent); + pending.push(parent); + } + } + } + } + + /** Rules that apply to this node: matching `type`, passing their `when` test, and (for a judge target) worth judging. */ + async function applicableRules(node: SpaceNode, onWhenError?: (rule: Rule, error: string) => void): Promise { + const applicable: Rule[] = []; + for (const r of perNodeRules) { + if (r.type && r.type !== node.resolvedType) continue; + if (r.mode === 'semantic' && options.judgeTarget !== undefined && node.label !== options.judgeTarget) { + // Ancestors only matter through the gates they could raise. + if (!(targetAncestors.has(node.title) && ruleCanGate(r))) continue; + } + if (r.when) { + const outcome = await tryEvaluateExpression(r.when, buildEvalContext(node, nodes, nodeIndex)); + if ('error' in outcome) { + onWhenError?.(r, outcome.error); + continue; + } + if (outcome.value !== true) continue; + } + applicable.push(r); + } + return applicable; + } async function evaluateApplicableRules(node: SpaceNode): Promise<{ blockingRuleId?: string }> { let blockingRuleId: string | undefined; - const applicable = perNodeRules.filter((r) => !r.type || r.type === node.resolvedType); - for (const rule of applicable) { - const context = buildEvalContext(node, nodes, nodeIndex); - const result = await evaluateExpression(rule.check, context); - if (result !== true) { - violations.push({ - file: node.label, - ruleId: rule.id, - category: rule.category, - severity: ruleSeverity(rule), - description: rule.description, - }); - if (ruleGatesDescendants(rule) && !blockingRuleId) { - blockingRuleId = rule.id; + const applicable = await applicableRules(node, (rule, error) => + violations.push(expressionErrorViolation(rule, node.label, error)), + ); + + function report( + rule: Rule, + detail: { severity?: RuleSeverity; verdict?: string; guidance?: string; evidence?: string; error?: string } = {}, + ): void { + const severity = detail.severity ?? ruleSeverity(rule); + violations.push({ + file: node.label, + ruleId: rule.id, + category: rule.category, + severity, + description: detail.error ? `${rule.description} (Judge error: ${detail.error})` : rule.description, + ...(detail.evidence !== undefined && { evidence: detail.evidence }), + ...(detail.verdict !== undefined && { verdict: detail.verdict }), + ...(detail.guidance !== undefined && { guidance: detail.guidance }), + }); + // A judge error says nothing about the node, so it never gates. + if (!detail.error && ruleGatesDescendants(rule, severity) && !blockingRuleId) blockingRuleId = rule.id; + } + + // Cheap deterministic rules first; the judge is only consulted afterwards, once per node. + for (const rule of applicable.filter((r) => r.mode !== 'semantic')) { + const outcome = await tryEvaluateExpression(rule.check as string, buildEvalContext(node, nodes, nodeIndex)); + if ('error' in outcome) violations.push(expressionErrorViolation(rule, node.label, outcome.error)); + else if (outcome.value !== true) report(rule); + } + + const semantic = applicable.filter((r) => r.mode === 'semantic'); + if (judge && semantic.length > 0) { + for (const j of await judgeNode(judge, node, semantic)) { + if (j.failed) { + report(j.rule, { + severity: j.severity, + verdict: j.verdict, + guidance: j.guidance, + evidence: j.evidence, + error: j.error, + }); } } } @@ -199,5 +293,8 @@ export async function evaluateRulesWithGating( } } - return { violations, gates: [...gateOrigins.values()].filter((g) => g.gatedFiles.length > 0) }; + return { + violations, + gates: [...gateOrigins.values()].filter((g) => g.gatedFiles.length > 0), + }; } diff --git a/src/schema/metadata-contract.ts b/src/schema/metadata-contract.ts index 41cc0db..08ac537 100644 --- a/src/schema/metadata-contract.ts +++ b/src/schema/metadata-contract.ts @@ -55,21 +55,78 @@ const HIERARCHY_LEVEL_SCHEMA = { additionalProperties: false, } as const; +const SEVERITY_SCHEMA = { enum: ['error', 'warning', 'info'] } as const; + const RULE_SCHEMA = { type: 'object', properties: { id: { type: 'string', minLength: 1 }, category: { enum: ['validation', 'coherence', 'workflow', 'best-practice'] }, - severity: { enum: ['error', 'warning', 'info'] }, + /** + * A severity, or (semantic rules only) a map of verdict to severity. In a map, a verdict mapped to + * `ignore` is judged but not reported, and a verdict left out passes. Defaults from the category. + */ + severity: { + oneOf: [ + SEVERITY_SCHEMA, + { + type: 'object', + additionalProperties: { enum: ['error', 'warning', 'info', 'ignore'] }, + minProperties: 1, + }, + ], + }, description: { type: 'string', minLength: 1 }, check: { type: 'string', minLength: 1 }, + /** Optional JSONata applicability test, evaluated per node before the rule runs. The rule applies only when it returns true. */ + when: { type: 'string', minLength: 1 }, + mode: { enum: ['expression', 'semantic'] }, + rubric: { type: 'string', minLength: 1 }, + /** + * Semantic rules: the closed set of verdicts a judge may return. Each is a description (what the verdict + * means, for the judge) or `{ description, guidance }`, where `guidance` is shown to the user when it fires. + * Defaults to `pass`/`fail`. + */ + verdicts: { + type: 'object', + additionalProperties: { + oneOf: [ + { type: 'string', minLength: 1 }, + { + type: 'object', + properties: { description: { type: 'string', minLength: 1 }, guidance: { type: 'string', minLength: 1 } }, + required: ['description'], + additionalProperties: false, + }, + ], + }, + minProperties: 2, + }, type: { type: 'string', minLength: 1 }, scope: { enum: ['global'] }, override: { type: 'boolean' }, /** Whether a violation of this rule gates (suppresses) downstream checks on the node's hierarchy subtree. Defaults to true for `error` severity rules and false otherwise. */ gatesDescendants: { type: 'boolean' }, }, - required: ['id', 'category', 'description', 'check'], + required: ['id', 'category', 'description'], + // A global rule has no node to test, so `when` needs a per-node rule. + not: { required: ['when', 'scope'] }, + // Semantic rules are judged per node (no `check`, not global) and with custom verdicts must map each failing + // verdict to a severity. Every other rule needs a `check` and takes none of the semantic-only fields. + if: { properties: { mode: { const: 'semantic' } }, required: ['mode'] }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then`/`else` keyword, not a thenable + then: { + not: { anyOf: [{ required: ['check'] }, { required: ['scope'] }] }, + if: { required: ['verdicts'] }, + // biome-ignore lint/suspicious/noThenProperty: JSON Schema `if`/`then`/`else` keyword, not a thenable + then: { required: ['severity'], properties: { severity: { type: 'object' } } }, + else: { properties: { severity: { oneOf: [SEVERITY_SCHEMA, { type: 'object' }] } } }, + }, + else: { + required: ['check'], + not: { anyOf: [{ required: ['verdicts'] }, { required: ['rubric'] }] }, + properties: { severity: SEVERITY_SCHEMA }, + }, additionalProperties: false, } as const; diff --git a/src/schema/rule-filter.ts b/src/schema/rule-filter.ts new file mode 100644 index 0000000..4393e8a --- /dev/null +++ b/src/schema/rule-filter.ts @@ -0,0 +1,57 @@ +import type { Rule } from '../types'; +import { ruleSeverities } from './rule-severity'; + +export const RULE_FILTER_DIMENSIONS = ['mode', 'category', 'severity', 'type', 'id'] as const; +export type RuleFilterDimension = (typeof RULE_FILTER_DIMENSIONS)[number]; + +/** Narrows which rules are evaluated. Dimensions combine with AND; values within a dimension with OR. */ +export type RuleFilter = Partial>; + +/** Parse `dimension=value[,value]` specs (e.g. from repeated `--rule-filter` flags); repeats of a dimension merge. */ +export function parseRuleFilter(specs: string[]): RuleFilter { + const filter: RuleFilter = {}; + for (const spec of specs) { + const eq = spec.indexOf('='); + const dimension = spec.slice(0, eq) as RuleFilterDimension; + const values = spec + .slice(eq + 1) + .split(',') + .map((v) => v.trim()) + .filter(Boolean); + if (eq < 0 || values.length === 0 || !RULE_FILTER_DIMENSIONS.includes(dimension)) { + throw new Error(`Invalid rule filter "${spec}". Use <${RULE_FILTER_DIMENSIONS.join('|')}>=[,...].`); + } + filter[dimension] = [...(filter[dimension] ?? []), ...values]; + } + return filter; +} + +function dimensionValues(rule: Rule, dimension: RuleFilterDimension): string[] { + switch (dimension) { + case 'mode': + return [rule.mode ?? 'expression']; + case 'severity': + return ruleSeverities(rule); + case 'category': + return [rule.category]; + case 'type': + return rule.type ? [rule.type] : []; + case 'id': + return [rule.id]; + } +} + +/** + * Keep the rules matching every dimension of the filter. A rule with no `type` never matches a `type` filter, + * and a rule with a verdict map matches a `severity` filter on any severity it can report. + */ +export function filterRules(rules: Rule[], filter?: RuleFilter): Rule[] { + if (!filter) return rules; + return rules.filter((rule) => + RULE_FILTER_DIMENSIONS.every((dim) => { + const wanted = filter[dim]; + if (!wanted) return true; + return dimensionValues(rule, dim).some((value) => wanted.includes(value)); + }), + ); +} diff --git a/src/schema/rule-severity.ts b/src/schema/rule-severity.ts new file mode 100644 index 0000000..969fd30 --- /dev/null +++ b/src/schema/rule-severity.ts @@ -0,0 +1,24 @@ +import type { Rule, RuleCategory, RuleSeverity } from '../types'; + +/** Default severity for each rule category. A rule's own `severity` field overrides this. */ +export const DEFAULT_SEVERITY_BY_CATEGORY: Record = { + validation: 'error', + coherence: 'warning', + workflow: 'warning', + 'best-practice': 'info', +}; + +const SEVERITY_ORDER: RuleSeverity[] = ['error', 'warning', 'info']; + +/** Every severity this rule can report. A verdict map lists several; `ignore` entries report nothing. */ +export function ruleSeverities(rule: Rule): RuleSeverity[] { + if (rule.severity === undefined) return [DEFAULT_SEVERITY_BY_CATEGORY[rule.category]]; + if (typeof rule.severity === 'string') return [rule.severity]; + const mapped = new Set(Object.values(rule.severity).filter((s): s is RuleSeverity => s !== 'ignore')); + return SEVERITY_ORDER.filter((s) => mapped.has(s)); +} + +/** Effective severity of a rule: its explicit `severity`, else its category default. With a verdict map, the most severe verdict. */ +export function ruleSeverity(rule: Rule): RuleSeverity { + return ruleSeverities(rule)[0] ?? 'info'; +} diff --git a/src/schema/validate-rules.ts b/src/schema/validate-rules.ts index 25c0786..30afe3f 100644 --- a/src/schema/validate-rules.ts +++ b/src/schema/validate-rules.ts @@ -1,30 +1,39 @@ -import type { Rule, RuleCategory, RuleSeverity, RuleViolation, SpaceNode } from '../types'; -import { buildEvalContext, evaluateExpression } from './evaluate-rule'; +import { judgeNode } from '../judge/judge-node'; +import type { SemanticJudgeProvider } from '../judge/types'; +import type { Rule, RuleViolation, SpaceNode } from '../types'; +import { buildEvalContext, expressionErrorViolation, tryEvaluateExpression } from './evaluate-rule'; +import { ruleSeverity } from './rule-severity'; -/** Default severity for each rule category. A rule's own `severity` field overrides this. */ -export const DEFAULT_SEVERITY_BY_CATEGORY: Record = { - validation: 'error', - coherence: 'warning', - workflow: 'warning', - 'best-practice': 'info', -}; +export { DEFAULT_SEVERITY_BY_CATEGORY, ruleSeverity } from './rule-severity'; -/** Effective severity of a rule: its explicit `severity`, else its category default. */ -export function ruleSeverity(rule: Rule): RuleSeverity { - return rule.severity ?? DEFAULT_SEVERITY_BY_CATEGORY[rule.category]; +export interface ValidateRulesOptions { + /** Judge for semantic rules. Without one, semantic rules are not evaluated. */ + judge?: SemanticJudgeProvider; + /** Label of the only node to judge (`validate-file`); other nodes' semantic rules are skipped. */ + judgeTarget?: string; + /** Set of node labels that have already failed upstream structural checks. */ + gatedNodeLabels?: Set; } /** * Validate nodes against rules metadata. - * Returns a list of rule violations found. + * Evaluates deterministic expression rules and (when opt-in) semantic judge rules. * * @param nodes - All nodes in the space * @param rules - Rules metadata list - * @returns Array of rule violations + * @param options - Validation options, including the judge for semantic rules + * @returns Rule violations */ -export async function validateRules(nodes: SpaceNode[], rules: Rule[]): Promise { +export async function validateRules( + nodes: SpaceNode[], + rules: Rule[], + options: ValidateRulesOptions = {}, +): Promise { const violations: RuleViolation[] = []; + // Track nodes that have failed checks (for cascading gating) + const failedNodeLabels = new Set(options.gatedNodeLabels || []); + // Build node index for efficient lookups const nodeIndex = new Map(); for (const node of nodes) { @@ -34,17 +43,33 @@ export async function validateRules(nodes: SpaceNode[], rules: Rule[]): Promise< } } - // Evaluate each rule against applicable nodes + // Separate rules into expression rules and semantic rules + const expressionRules: Rule[] = []; + const semanticRules: Rule[] = []; + for (const rule of rules) { + if (rule.mode === 'semantic') { + semanticRules.push(rule); + } else if (rule.check) { + expressionRules.push(rule); + } + } + + // 1. Evaluate deterministic expression rules first + for (const rule of expressionRules) { + const check = rule.check as string; const severity = ruleSeverity(rule); + if (rule.scope === 'global') { // Global rules are evaluated once against the full node set. // A sentinel node provides the evaluation context (nodes array is what matters). const sentinel = nodes[0]; if (sentinel) { const context = buildEvalContext(sentinel, nodes, nodeIndex); - const result = await evaluateExpression(rule.check, context); - if (result !== true) { + const outcome = await tryEvaluateExpression(check, context); + if ('error' in outcome) { + violations.push(expressionErrorViolation(rule, '', outcome.error)); + } else if (outcome.value !== true) { violations.push({ file: '', ruleId: rule.id, @@ -58,8 +83,18 @@ export async function validateRules(nodes: SpaceNode[], rules: Rule[]): Promise< const targetNodes = rule.type ? nodes.filter((n) => n.resolvedType === rule.type) : nodes; for (const node of targetNodes) { const context = buildEvalContext(node, nodes, nodeIndex); - const result = await evaluateExpression(rule.check, context); - if (result !== true) { + if (rule.when) { + const when = await tryEvaluateExpression(rule.when, context); + if ('error' in when) { + violations.push(expressionErrorViolation(rule, node.label, when.error)); + continue; + } + if (when.value !== true) continue; + } + const outcome = await tryEvaluateExpression(check, context); + if ('error' in outcome) { + violations.push(expressionErrorViolation(rule, node.label, outcome.error)); + } else if (outcome.value !== true) { violations.push({ file: node.label, ruleId: rule.id, @@ -67,10 +102,51 @@ export async function validateRules(nodes: SpaceNode[], rules: Rule[]): Promise< severity, description: rule.description, }); + failedNodeLabels.add(node.label); } } } } + // 2. Evaluate semantic rules only when a judge is supplied + const judge = options.judge; + if (judge) { + for (const node of nodes) { + if (options.judgeTarget !== undefined && node.label !== options.judgeTarget) continue; + const applicable: Rule[] = []; + for (const r of semanticRules) { + if (r.type && r.type !== node.resolvedType) continue; + if (r.when) { + const when = await tryEvaluateExpression(r.when, buildEvalContext(node, nodes, nodeIndex)); + if ('error' in when) { + violations.push(expressionErrorViolation(r, node.label, when.error)); + continue; + } + if (when.value !== true) continue; + } + applicable.push(r); + } + if (applicable.length === 0) continue; + + // Safe cascading default: gate semantic evaluation if node failed upstream checks + if (failedNodeLabels.has(node.label)) continue; + + for (const j of await judgeNode(judge, node, applicable)) { + if (!j.failed) continue; + violations.push({ + file: node.label, + ruleId: j.rule.id, + category: j.rule.category, + severity: j.severity ?? ruleSeverity(j.rule), + description: j.error ? `${j.rule.description} (Judge error: ${j.error})` : j.rule.description, + evidence: j.evidence, + ...(j.verdict !== undefined && { verdict: j.verdict }), + ...(j.guidance !== undefined && { guidance: j.guidance }), + }); + if (!j.error) failedNodeLabels.add(node.label); + } + } + } + return violations; } diff --git a/src/types.ts b/src/types.ts index 2401a6c..03fc923 100644 --- a/src/types.ts +++ b/src/types.ts @@ -201,6 +201,12 @@ export type RuleViolation = { category: RuleCategory; severity: RuleSeverity; description: string; + /** Cited evidence or reasoning from evaluation (for semantic judge rules). */ + evidence?: string; + /** The verdict that failed the rule (semantic rules). */ + verdict?: string; + /** User-facing guidance declared for that verdict (semantic rules). */ + guidance?: string; }; export type GraphViolation = { diff --git a/src/validate.ts b/src/validate.ts index 26bdc62..546c149 100644 --- a/src/validate.ts +++ b/src/validate.ts @@ -1,8 +1,10 @@ import { isAbsolute, relative, resolve } from 'node:path'; import type { ErrorObject } from 'ajv'; import type { Config, SpaceConfig } from './config'; +import type { SemanticJudgeProvider } from './judge/types'; import { readSpace } from './read/read-space'; import { computeHierarchyDepths, evaluateRulesWithGating, type GatingSeed } from './schema/gating'; +import { filterRules, type RuleFilter } from './schema/rule-filter'; import { extractEntityInfo } from './schema/schema'; import { validateGraph } from './schema/validate-graph'; import { validateRules } from './schema/validate-rules'; @@ -159,21 +161,63 @@ export function formatErrors( return formatted; } +export interface ValidateSpaceOptions { + readResult?: ReadSpaceResult; + /** Limit validation to hierarchy nodes within N breadth-first levels from the top. */ + depth?: number; + /** Evaluate rules on gated nodes anyway (gates are still reported). */ + includeGated?: boolean; + /** Evaluate only rules matching this filter (see `parseRuleFilter`). */ + ruleFilter?: RuleFilter; + /** Judge for `mode: semantic` rules. Without one, semantic rules are not evaluated. */ + judge?: SemanticJudgeProvider; +} + +/** One-line text for a rule violation: rule, description, and any judge evidence and verdict guidance. */ +export function ruleViolationMessage(v: RuleViolation): string { + const evidence = v.evidence ? ` (Evidence: ${v.evidence})` : ''; + const guidance = v.guidance ? ` Guidance: ${v.guidance}` : ''; + return `[${v.ruleId}] ${v.description}${evidence}${guidance}`; +} + +type SpaceRuleEvaluation = { + violations: RuleViolation[]; + gates: CoverageGate[]; +}; + /** - * Evaluate a space's executable rules. With a main hierarchy, evaluation cascades: schema, hierarchy + * Evaluate a space's rules. With a main hierarchy, evaluation cascades: schema, hierarchy * and gating-rule failures (the `seeds`) suppress rule evaluation for the failing node's subtree. * Shared by `validateSpace` and `validateFile` so both report identical rule results for a node. + * Semantic rules run only when a `judge` is supplied. */ async function evaluateSpaceRules( nodes: SpaceNode[], metadata: SchemaMetadata, seeds: GatingSeed[], - options: { includeGated?: boolean } = {}, -): Promise<{ violations: RuleViolation[]; gates: CoverageGate[] }> { - if (!metadata.rules) return { violations: [], gates: [] }; - if (!metadata.hierarchy) return { violations: await validateRules(nodes, metadata.rules), gates: [] }; + options: { + includeGated?: boolean; + judge?: SemanticJudgeProvider; + judgeTarget?: string; + ruleFilter?: RuleFilter; + } = {}, +): Promise { + const rules = metadata.rules && filterRules(metadata.rules, options.ruleFilter); + if (!rules) return { violations: [], gates: [] }; + const provider = options.judge; + if (!metadata.hierarchy) { + const result = await validateRules(nodes, rules, { + judge: provider, + judgeTarget: options.judgeTarget, + }); + return { violations: result, gates: [] }; + } const graph = buildSpaceGraph(nodes, metadata.hierarchy); - return evaluateRulesWithGating(nodes, metadata.rules, graph, seeds, options); + return evaluateRulesWithGating(nodes, rules, graph, seeds, { + includeGated: options.includeGated, + provider, + judgeTarget: options.judgeTarget, + }); } /** @@ -182,7 +226,7 @@ async function evaluateSpaceRules( */ export async function validateSpace( context: SpaceContext, - options: { readResult?: ReadSpaceResult; depth?: number; includeGated?: boolean } = {}, + options: ValidateSpaceOptions = {}, ): Promise { const { schema, schemaRefRegistry, schemaValidator } = context; const metadata = schema.metadata; @@ -301,6 +345,8 @@ export async function validateSpace( ]; const { violations, gates } = await evaluateSpaceRules(nodes, metadata, seeds, { includeGated: options.includeGated, + ruleFilter: options.ruleFilter, + judge: options.judge, }); result.ruleViolations = violations; result.coverageGates = gates; @@ -324,7 +370,7 @@ export async function validateSpace( } /** Find which space a file belongs to by checking directory containment. */ -function resolveFileSpace(filePath: string, config: Config): { space: SpaceConfig; label: string } | null { +export function resolveFileSpace(filePath: string, config: Config): { space: SpaceConfig; label: string } | null { const absFile = isAbsolute(filePath) ? filePath : resolve(process.cwd(), filePath); for (const space of config.spaces) { @@ -338,6 +384,16 @@ function resolveFileSpace(filePath: string, config: Config): { space: SpaceConfi return null; } +export interface ValidateFileOptions { + configDir?: string; + /** Evaluate rules on gated nodes anyway (gates are still reported). */ + includeGated?: boolean; + /** Evaluate only rules matching this filter (see `parseRuleFilter`). */ + ruleFilter?: RuleFilter; + /** Judge for `mode: semantic` rules. Without one, semantic rules are not evaluated. */ + judge?: SemanticJudgeProvider; +} + /** * Validate a single file within its space. * @@ -350,7 +406,7 @@ function resolveFileSpace(filePath: string, config: Config): { space: SpaceConfi export async function validateFile( filePath: string, config: Config, - options: { configDir?: string; includeGated?: boolean } = {}, + options: ValidateFileOptions = {}, ): Promise { const resolved = resolveFileSpace(filePath, config); @@ -461,11 +517,18 @@ export async function validateFile( ]; const { violations, gates } = await evaluateSpaceRules(nodes, metadata, seeds, { includeGated: options.includeGated, + ruleFilter: options.ruleFilter, + judge: options.judge, + judgeTarget: label, }); for (const v of violations) { if (v.file === label) { const target = v.severity === 'error' ? errors : warnings; - target[`rule:${v.ruleId}`] = { kind: 'rule', message: `[${v.ruleId}] ${v.description}`, severity: v.severity }; + target[`rule:${v.ruleId}`] = { + kind: 'rule', + message: ruleViolationMessage(v), + severity: v.severity, + }; } } // A gated file with no violations would otherwise look clean — say that its rules were not checked. diff --git a/tests/commands/validate-judge-flag.test.ts b/tests/commands/validate-judge-flag.test.ts new file mode 100644 index 0000000..e88d6f6 --- /dev/null +++ b/tests/commands/validate-judge-flag.test.ts @@ -0,0 +1,41 @@ +import { afterAll, describe, expect, it } from 'bun:test'; +import { writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { makeTempSpace } from '../helpers/temp-space'; + +const ROOT = join(import.meta.dir, '..', '..'); +const space = makeTempSpace( + [{ id: 'step-fit', type: 'step', mode: 'semantic', category: 'coherence', description: 'Step fits its task' }], + { judging: { modelProviders: { f: { api: 'fake' } } } }, +); +const configPath = join(space.dir, 'config.json'); +writeFileSync(configPath, JSON.stringify(space.config)); +afterAll(() => space.cleanup()); + +function spawn(...args: string[]) { + const result = Bun.spawnSync(['bun', 'run', 'src/index.ts', ...args, '--json'], { + cwd: ROOT, + env: { ...process.env, SCTX_CONFIG: configPath }, + stdout: 'pipe', + stderr: 'pipe', + }); + return { ...result, stderr: new TextDecoder().decode(result.stderr) }; +} + +describe('--judge on the CLI', () => { + // An unknown name fails only if the flag reaches the judge. + for (const args of [ + ['validate', 'ts'], + ['validate-file', space.file('S.md')], + ]) { + it(`${args[0]} passes --judge to the judge`, () => { + const result = spawn(...args, '--judge', 'missing'); + expect(result.exitCode).toBe(1); + expect(result.stderr).toContain('missing'); + }); + } + + it('runs no judge without --semantic or --judge', () => { + expect(spawn('validate', 'ts').exitCode).toBe(0); + }); +}); diff --git a/tests/config.test.ts b/tests/config.test.ts index b2eb463..96f1993 100644 --- a/tests/config.test.ts +++ b/tests/config.test.ts @@ -386,4 +386,24 @@ describe('loadConfig with includeSpacesFrom', () => { const mainConfig = JSON5.parse(readFileSync(mainConfigPath, 'utf-8')) as Config; expect(mainConfig.spaces[0]!.name).toBe('main-space'); }); + + it('accepts model providers, a top-level judge and a space judge override', () => { + const configPath = join(testDir, 'judge-config.json'); + writeFileSync( + configPath, + JSON.stringify({ + modelProviders: { + jev: { api: 'openrouter-decisions' }, + claude: { api: 'anthropic-messages', model: 'claude-haiku-5-5' }, + }, + judge: { provider: 'jev' }, + spaces: [{ name: 's', path: '/p', judge: { provider: 'claude' } }], + }), + ); + setConfigPath(configPath); + const config = loadConfig(); + expect(config.judge?.provider).toBe('jev'); + expect(config.modelProviders?.claude?.model).toBe('claude-haiku-5-5'); + expect(config.spaces?.[0]?.judge?.provider).toBe('claude'); + }); }); diff --git a/tests/helpers/temp-space.ts b/tests/helpers/temp-space.ts new file mode 100644 index 0000000..4cde0ae --- /dev/null +++ b/tests/helpers/temp-space.ts @@ -0,0 +1,58 @@ +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import type { Config } from '../../src/config'; + +const META = + 'https://raw.githubusercontent.com/mindsocket/structured-context/main/schemas/generated/latest/_structured_context_schema_meta.json'; + +const typeSchema = (type: string, parent: boolean) => ({ + type: 'object', + properties: { type: { const: type }, title: { type: 'string' }, ...(parent && { parent: { type: 'string' } }) }, + required: ['type', 'title'], +}); + +export interface TempSpace { + dir: string; + config: Config; + /** Absolute path of a file in the space. */ + file: (name: string) => string; + cleanup: () => void; +} + +/** + * A throwaway goal -> task -> step space (files G, T, S, plus an unrelated goal Other) with the given rules. + * Each file may add frontmatter via `frontmatter`, e.g. `{ T: 'status: ready' }`. + */ +export function makeTempSpace( + rules: object[], + options: { frontmatter?: Record; judging?: Pick } = {}, +): TempSpace { + const dir = mkdtempSync(join(tmpdir(), 'sctx-temp-space-')); + const schemaPath = join(dir, 'schema.json'); + writeFileSync( + schemaPath, + JSON.stringify({ + $schema: META, + $id: 'sctx://temp-space.test', + $metadata: { hierarchy: { levels: ['goal', 'task', 'step'] }, rules }, + oneOf: [typeSchema('goal', false), typeSchema('task', true), typeSchema('step', true)], + }), + ); + const space = join(dir, 'space'); + mkdirSync(space); + const extra = (name: string) => (options.frontmatter?.[name] ? `${options.frontmatter[name]}\n` : ''); + writeFileSync(join(space, 'G.md'), `---\ntype: goal\n${extra('G')}---\n`); + writeFileSync(join(space, 'T.md'), `---\ntype: task\nparent: "[[G]]"\n${extra('T')}---\n`); + writeFileSync(join(space, 'S.md'), `---\ntype: step\nparent: "[[T]]"\n${extra('S')}---\n`); + writeFileSync(join(space, 'Other.md'), `---\ntype: goal\n${extra('Other')}---\n`); + return { + dir, + config: { + spaces: [{ name: 'ts', path: space, schema: schemaPath }], + ...options.judging, + }, + file: (name) => join(space, name), + cleanup: () => rmSync(dir, { recursive: true, force: true }), + }; +} diff --git a/tests/judge/factory.test.ts b/tests/judge/factory.test.ts new file mode 100644 index 0000000..4742146 --- /dev/null +++ b/tests/judge/factory.test.ts @@ -0,0 +1,184 @@ +import { describe, expect, it } from 'bun:test'; +import { + createJudge, + createModelProvider, + hydrateProvider, + JudgeConfigError, + judgeForRun, +} from '../../src/judge/factory'; +import { AnthropicJudgeProvider } from '../../src/judge/provider/anthropic'; +import { FakeJudgeProvider } from '../../src/judge/provider/fake'; +import { JevJudgeProvider } from '../../src/judge/provider/jev'; +import { OpenAIJudgeProvider } from '../../src/judge/provider/openai'; + +describe('hydrateProvider', () => { + it('prefers the explicit value over the env var and leaves unset fields undefined', () => { + const env = { MY_MODEL: 'from-env', MY_URL: 'https://from-env' }; + const h = hydrateProvider( + 'p', + { api: 'openai-completions', model: 'explicit', modelEnv: 'MY_MODEL', endpointEnv: 'MY_URL' }, + env, + ); + expect(h).toMatchObject({ + api: 'openai-completions', + model: 'explicit', + endpoint: 'https://from-env', + apiKey: undefined, + }); + expect(hydrateProvider('p', { api: 'openai-completions' }, env).model).toBeUndefined(); + }); + + it('errors when a named env var is set but empty, instead of falling through silently', () => { + expect(() => hydrateProvider('p', { api: 'openai-completions', modelEnv: 'MY_MODEL' }, { MY_MODEL: ' ' })).toThrow( + 'MY_MODEL set but empty', + ); + }); + + it('reads the key from apiKeyEnv, or takes the literal apiKey', () => { + expect(hydrateProvider('p', { api: 'anthropic-messages', apiKeyEnv: 'MY_KEY' }, { MY_KEY: 'k' }).apiKey).toBe('k'); + expect(hydrateProvider('p', { api: 'anthropic-messages', apiKey: 'lit' }, {}).apiKey).toBe('lit'); + }); + + it('rejects an unknown api', () => { + expect(() => hydrateProvider('p', { api: 'gemini' as never }, {})).toThrow('unknown api "gemini"'); + }); +}); + +describe('createModelProvider', () => { + it('the decisions API defaults to the OpenRouter decisions endpoint', async () => { + const calls: string[] = []; + const realFetch = globalThis.fetch; + globalThis.fetch = (async (url: string | URL) => { + calls.push(String(url)); + throw new Error('stop'); + }) as unknown as typeof fetch; + try { + const judge = createJudge({ judge: { provider: 'openrouter-jev' } }, {}, undefined, { OPENROUTER_API_KEY: 'k' }); + await judge + .evaluate({ + node: { + label: 'n', + title: 't', + type: 'opportunity', + resolvedType: 'opportunity', + schemaData: {}, + resolvedParents: [], + }, + rules: [], + }) + .catch(() => {}); + expect(calls[0]).toBe('https://openrouter.ai/api/alpha/decisions'); + } finally { + globalThis.fetch = realFetch; + } + }); + + it('creates FakeJudgeProvider for api "fake"', () => { + expect(createModelProvider('f', { api: 'fake' }, {}) instanceof FakeJudgeProvider).toBe(true); + }); + + it.each([ + ['anthropic-messages', 'ANTHROPIC_API_KEY', AnthropicJudgeProvider], + ['openai-completions', 'OPENAI_API_KEY', OpenAIJudgeProvider], + ['openrouter-decisions', 'OPENROUTER_API_KEY', JevJudgeProvider], + ['typesafe-systemone', 'TYPESAFE_API_KEY', JevJudgeProvider], + ] as const)('creates %s with its key from the given env (%s)', (api, keyEnv, Class) => { + expect(createModelProvider('p', { api }, { [keyEnv]: 'k' }) instanceof Class).toBe(true); + }); + + it("does not borrow another provider's key", () => { + expect(() => createModelProvider('p', { api: 'anthropic-messages' }, { OPENAI_API_KEY: 'o' })).toThrow( + 'ANTHROPIC_API_KEY', + ); + expect(() => createModelProvider('p', { api: 'openrouter-decisions' }, { ANTHROPIC_API_KEY: 'a' })).toThrow( + 'OPENROUTER_API_KEY', + ); + }); + + it('lets openai run without a key for local endpoints', () => { + const provider = createModelProvider( + 'local', + { api: 'openai-completions', endpoint: 'http://localhost:11434/v1' }, + {}, + ); + expect(provider instanceof OpenAIJudgeProvider).toBe(true); + }); +}); + +describe('createJudge', () => { + const config = { + modelProviders: { mine: { api: 'openrouter-decisions' }, claude: { api: 'anthropic-messages' } } as const, + judge: { provider: 'claude' }, + }; + const env = { OPENROUTER_API_KEY: 'r', ANTHROPIC_API_KEY: 'a', TYPESAFE_API_KEY: 't' }; + + it('uses the top-level judge provider', () => { + expect(createJudge(config, {}, undefined, env).name).toBe('anthropic'); + }); + + it("uses the space's judge over the top-level one, and a run's over both", () => { + const space = { judge: { provider: 'mine' } }; + expect(createJudge(config, space, undefined, env).name).toBe('jev'); + expect(createJudge(config, space, 'claude', env).name).toBe('anthropic'); + }); + + it('accepts a bundled provider as the judge, with the type defaults', () => { + expect(createJudge({}, { judge: { provider: 'openrouter-jev' } }, undefined, env).name).toBe('jev'); + expect(createJudge({ judge: { provider: 'anthropic' } }, {}, undefined, env).name).toBe('anthropic'); + }); + + it('ships the pinned and env variants of each vendor bundle', () => { + expect(createJudge({}, { judge: { provider: 'anthropic-haiku-5-5' } }, undefined, env).name).toBe('anthropic'); + expect(createJudge({}, { judge: { provider: 'anthropic-env' } }, undefined, env).name).toBe('anthropic'); + expect(createJudge({}, { judge: { provider: 'openai-env' } }, undefined, env).name).toBe('openai'); + expect(createJudge({}, { judge: { provider: 'typesafe-jev' } }, undefined, env).name).toBe('jev'); + }); + + it('lets a configured model provider replace a bundled provider of the same name, keeping type defaults for fields it leaves out', () => { + const augmented = { + modelProviders: { anthropic: { api: 'anthropic-messages', model: 'claude-sonnet-5-5' } } as const, + judge: { provider: 'anthropic' }, + }; + expect(createJudge(augmented, {}, undefined, env).name).toBe('anthropic'); + }); + + it('never infers a provider from API keys in the environment', () => { + const keys = { ANTHROPIC_API_KEY: 'a', OPENAI_API_KEY: 'o', OPENROUTER_API_KEY: 'r' }; + expect(() => createJudge({}, {}, undefined, keys)).toThrow(JudgeConfigError); + expect(() => createJudge({ modelProviders: config.modelProviders }, {}, undefined, keys)).toThrow( + 'No judge configured', + ); + }); + + it('rejects a judge that is neither a bundled provider nor a configured one', () => { + expect(() => createJudge(config, {}, 'nope', env)).toThrow('not configured'); + expect(() => createJudge(config, {}, 'nope', env)).toThrow('Known providers:'); + }); + + it('fails when the chosen provider has no key', () => { + expect(() => createJudge(config, {}, undefined, { OPENROUTER_API_KEY: 'r' })).toThrow('ANTHROPIC_API_KEY'); + }); + + it('does not bundle the test-only fake provider', () => { + expect(() => createJudge({}, { judge: { provider: 'fake' } }, undefined, env)).toThrow('not configured'); + }); +}); + +describe('judgeForRun (CLI)', () => { + const config = { + modelProviders: { f: { api: 'fake' }, g: { api: 'fake' } } as const, + judge: { provider: 'f' }, + }; + + it('gives no judge unless the run asks with --semantic or --judge', () => { + expect(judgeForRun(config, {}, {})).toBeUndefined(); + expect(judgeForRun({}, {}, {})).toBeUndefined(); // no judge config needed when not asked + expect(judgeForRun(config, {}, { semantic: true }) instanceof FakeJudgeProvider).toBe(true); + expect(judgeForRun(config, {}, { judge: 'g' }) instanceof FakeJudgeProvider).toBe(true); + }); + + it('fails when asked and no judge is configured, or the named provider is missing', () => { + expect(() => judgeForRun({}, {}, { semantic: true })).toThrow('No judge configured'); + expect(() => judgeForRun(config, {}, { judge: 'nope' })).toThrow('Model provider "nope" is not configured'); + }); +}); diff --git a/tests/judge/helpers.ts b/tests/judge/helpers.ts new file mode 100644 index 0000000..5e9d054 --- /dev/null +++ b/tests/judge/helpers.ts @@ -0,0 +1,27 @@ +import type { SemanticJudgeInput, SemanticRuleSpec } from '../../src/judge/types'; + +export const node: SemanticJudgeInput['node'] = { + label: 'opp1.md', + title: 'I cannot find my receipts', + type: 'opportunity', + resolvedType: 'opportunity', + schemaData: { title: 'I cannot find my receipts', type: 'opportunity', source: 'User interview with Alice' }, + resolvedParents: [ + { title: 'Faster Tax Filing', field: 'parent', source: 'hierarchy', selfRef: false, fieldOn: 'child' }, + ], +}; + +export const passFailRule: SemanticRuleSpec = { + id: 'customer-voice', + description: 'Must be in customer voice', + rubric: 'Needs to sound like a customer quote, not a solution or business goal', + category: 'best-practice', + verdicts: { pass: 'Customer voice.', fail: 'Not customer voice.' }, +}; + +export const graded: SemanticRuleSpec = { + id: 'framing', + description: 'Framing quality', + category: 'best-practice', + verdicts: { good: 'Clear need', weak: 'Vague need', solution: 'A solution in disguise' }, +}; diff --git a/tests/judge/judge-node.test.ts b/tests/judge/judge-node.test.ts new file mode 100644 index 0000000..1948012 --- /dev/null +++ b/tests/judge/judge-node.test.ts @@ -0,0 +1,38 @@ +import { describe, expect, it } from 'bun:test'; +import { judgeNode, type NodeJudgement } from '../../src/judge/judge-node'; +import type { SemanticJudgeProvider, SemanticJudgeResult } from '../../src/judge/types'; +import type { Rule } from '../../src/schema/metadata-contract'; +import type { SpaceNode } from '../../src/types'; +import { node, passFailRule } from './helpers'; + +/** Provider returning canned results keyed by rule id, bypassing response parsing. */ +function canned(results: Record): SemanticJudgeProvider { + return { + name: 'canned', + evaluate: (input) => + Promise.resolve(Object.fromEntries(input.rules.map((r) => [r.id, results[r.id] ?? { verdict: 'pass' }]))), + }; +} + +describe('judgeNode', () => { + it('treats an unknown verdict as a judge error, not a pass', async () => { + const rule = { + id: passFailRule.id, + description: passFailRule.description, + category: passFailRule.category, + mode: 'semantic', + severity: { fail: 'error' }, + verdicts: passFailRule.verdicts, + } as unknown as Rule; + const judgements: NodeJudgement[] = await judgeNode( + canned({ 'customer-voice': { verdict: 'unexpected', evidence: 'hmm' } }), + node as SpaceNode, + [rule], + ); + expect(judgements).toHaveLength(1); + expect(judgements[0]).toMatchObject({ + failed: true, + error: expect.stringContaining('unknown verdict "unexpected"'), + }); + }); +}); diff --git a/tests/judge/live.test.ts b/tests/judge/live.test.ts new file mode 100644 index 0000000..315a79c --- /dev/null +++ b/tests/judge/live.test.ts @@ -0,0 +1,40 @@ +/** + * Live judge tests: call real providers on one inline node. They cost money, so they run only with + * SCTX_LIVE_JUDGE_TESTS=1 (`bun run test:judge`), and each provider only when its API key is set. + */ +import { describe, expect, it } from 'bun:test'; +import { createModelProvider } from '../../src/judge/factory'; +import type { SemanticJudgeInput, SemanticRuleSpec } from '../../src/judge/types'; +import { node, passFailRule } from './helpers'; + +const CASES = [ + { name: 'openrouter-jev', api: 'openrouter-decisions' as const, keyEnv: 'OPENROUTER_API_KEY' }, + { name: 'anthropic', api: 'anthropic-messages' as const, keyEnv: 'ANTHROPIC_API_KEY' }, + { name: 'openai', api: 'openai-completions' as const, keyEnv: 'OPENAI_API_KEY' }, +]; + +describe.skipIf(process.env.SCTX_LIVE_JUDGE_TESTS !== '1')('live judges', () => { + for (const { name, api, keyEnv } of CASES) { + it.skipIf(!process.env[keyEnv])( + `${name}: returns a verdict from the rule's closed set`, + async () => { + const provider = createModelProvider(name, { api }, { [keyEnv]: process.env[keyEnv] }); + const rule: SemanticRuleSpec = { + ...passFailRule, + description: 'The node title names a customer need or pain point', + }; + const input: SemanticJudgeInput = { node, rules: [rule] }; + const results = await provider.evaluate(input); + const result = results[rule.id]; + expect(result).toBeDefined(); + if (!result || 'error' in result) { + expect(result).not.toHaveProperty('error'); + return; + } + expect(result.verdict).toBeOneOf(['pass', 'fail']); + expect(typeof result.evidence).toBe('string'); + }, + 60_000, + ); + } +}); diff --git a/tests/judge/model-validation.test.ts b/tests/judge/model-validation.test.ts new file mode 100644 index 0000000..3f1b0c2 --- /dev/null +++ b/tests/judge/model-validation.test.ts @@ -0,0 +1,98 @@ +/** + * Model-validation smoke tests: check that every bundled provider's configured model actually + * exists on its endpoint. Unlike live.test.ts these make no judgement call — they only prove the + * model id resolves, so the check stays cheap and never sends real content. + * + * Opt in with SCTX_MODEL_VALIDATION=1 (`bun run test:models`); each check runs only when its + * provider's key is set. OpenRouter chat models validate keylessly against the public models + * list; decisions endpoints expose no keyless catalog, so they take a minimal probe. + */ +import { describe, expect, it } from 'bun:test'; +import { BUNDLED_PROVIDERS, hydrateProvider } from '../../src/judge/factory'; +import type { HydratedProvider } from '../../src/judge/types'; + +const env = process.env as Record; + +function resolved(name: string): { model?: string; api: string } { + const h: HydratedProvider = hydrateProvider(name, BUNDLED_PROVIDERS[name]!, env); + return { model: h.model, api: h.api }; +} + +const CHECKABLE = Object.keys(BUNDLED_PROVIDERS).filter((name) => resolved(name).model !== undefined); + +describe.skipIf(process.env.SCTX_MODEL_VALIDATION !== '1')('bundled models exist', () => { + it.each(CHECKABLE)( + '%s: model resolves on its endpoint', + async (name) => { + const { model, api } = resolved(name); + + if (api === 'openai-completions' && name.startsWith('openrouter')) { + // OpenRouter's models list is public and keyless; chat models appear in it. + const res = await fetch('https://openrouter.ai/api/v1/models'); + expect(res.ok).toBe(true); + const data = (await res.json()) as { data?: Array<{ id: string }> }; + expect(new Set((data.data ?? []).map((m) => m.id)).has(model!)).toBe(true); + return; + } + + const keyEnv = BUNDLED_PROVIDERS[name]!.apiKeyEnv!; + const key = env[keyEnv]; + if (!key) { + console.log(`skip ${name}: ${keyEnv} not set`); + return; + } + + if (api === 'openrouter-decisions') { + // The decisions catalog is not the public models list (verified: live slugs are absent from + // it), so probe the endpoint: an unknown model is a 400 naming the model, a known one is not. + const res = await fetch('https://openrouter.ai/api/alpha/decisions', { + method: 'POST', + headers: { 'content-type': 'application/json', authorization: `Bearer ${key}` }, + body: JSON.stringify({ + model, + state: {}, + questions: { probe: { type: 'choice', instructions: 'Answer a.', criteria: { a: 'always' } } }, + }), + }); + const text = await res.text(); + expect(res.ok || !/does not exist/i.test(text)).toBe(true); + return; + } + + if (api === 'openai-decisions') { + // No keyless catalog; an unknown model is 404 model_not_found, a known one is not. + const res = await fetch('https://api.openai.com/v1/decisions', { + method: 'POST', + headers: { 'content-type': 'application/json', authorization: `Bearer ${key}` }, + body: JSON.stringify({ + model, + input: 'model existence check', + questions: [{ type: 'predicate', name: 'probe', instructions: 'Answer yes.' }], + }), + }); + expect(res.status).not.toBe(404); + return; + } + + if (api === 'anthropic-messages' || api === 'openai-completions') { + if (api === 'anthropic-messages') { + const res = await fetch('https://api.anthropic.com/v1/models', { + headers: { 'x-api-key': key, 'anthropic-version': '2023-06-01' }, + }); + expect(res.ok).toBe(true); + const data = (await res.json()) as { data?: Array<{ id: string }> }; + expect(new Set((data.data ?? []).map((m) => m.id)).has(model!)).toBe(true); + } else { + const res = await fetch('https://api.openai.com/v1/models', { headers: { authorization: `Bearer ${key}` } }); + expect(res.ok).toBe(true); + const data = (await res.json()) as { data?: Array<{ id: string }> }; + expect(new Set((data.data ?? []).map((m) => m.id)).has(model!)).toBe(true); + } + return; + } + + console.log(`skip ${name}: no validation mechanism for api "${api}"`); + }, + 30_000, + ); +}); diff --git a/tests/judge/prompt.test.ts b/tests/judge/prompt.test.ts new file mode 100644 index 0000000..efce894 --- /dev/null +++ b/tests/judge/prompt.test.ts @@ -0,0 +1,62 @@ +import { describe, expect, it } from 'bun:test'; +import { buildJudgeSystemPrompt, buildJudgeUserPrompt, parseJudgeResponse } from '../../src/judge/prompt'; +import { graded, node, passFailRule } from './helpers'; + +describe('judge prompt and response parser', () => { + it('system prompt asks for a JSON verdict, confidence and evidence', () => { + const prompt = buildJudgeSystemPrompt(); + expect(prompt).toContain('"verdict"'); + expect(prompt).toContain('"confidence"'); + expect(prompt).toContain('"evidence"'); + }); + + it('user prompt lists the rule, its allowed verdicts and the node', () => { + const prompt = buildJudgeUserPrompt(node, graded); + expect(prompt).toContain('framing'); + expect(prompt).toContain('"weak": Vague need'); + expect(prompt).toContain('"solution": A solution in disguise'); + expect(prompt).toContain('I cannot find my receipts'); + expect(prompt).toContain('Faster Tax Filing'); + expect(prompt).toContain('User interview with Alice'); + }); + + it('user prompt prefers the rubric over the description', () => { + expect(buildJudgeUserPrompt(node, passFailRule)).toContain('Needs to sound like a customer quote'); + expect(buildJudgeUserPrompt(node, graded)).toContain('Rubric: Framing quality'); + }); + + const allowed = ['pass', 'fail']; + + it('parses clean JSON', () => { + expect(parseJudgeResponse('{"verdict":"pass","confidence":0.9,"evidence":"clear"}', allowed)).toEqual({ + verdict: 'pass', + confidence: 0.9, + evidence: 'clear', + }); + }); + + it('parses fenced JSON and clamps confidence', () => { + const raw = '```json\n{"verdict":"fail","confidence":4,"evidence":"solution"}\n```'; + expect(parseJudgeResponse(raw, allowed)).toEqual({ verdict: 'fail', confidence: 1, evidence: 'solution' }); + }); + + it('allows confidence and evidence to be absent', () => { + expect(parseJudgeResponse('{"verdict":"pass"}', allowed)).toEqual({ verdict: 'pass' }); + }); + + it('extracts fields from text surrounding malformed JSON', () => { + const raw = 'Sure! Here you go: {"verdict": "fail", "confidence": 0.4, "evidence": "says \\"build X\\"" trailing'; + const result = parseJudgeResponse(raw, allowed); + expect(result).toMatchObject({ verdict: 'fail', confidence: 0.4 }); + }); + + it('returns an error for a verdict outside the allowed set', () => { + const result = parseJudgeResponse('{"verdict":"maybe"}', allowed); + expect(result).toHaveProperty('error'); + expect((result as { error: string }).error).toContain('unknown verdict "maybe"'); + }); + + it('returns an error when nothing parseable is present', () => { + expect(parseJudgeResponse('I cannot decide', allowed)).toHaveProperty('error'); + }); +}); diff --git a/tests/judge/provider/fake.test.ts b/tests/judge/provider/fake.test.ts new file mode 100644 index 0000000..5bcfd13 --- /dev/null +++ b/tests/judge/provider/fake.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from 'bun:test'; +import { FakeJudgeProvider } from '../../../src/judge/provider/fake'; +import type { SemanticJudgeInput } from '../../../src/judge/types'; +import { graded, node, passFailRule } from '../helpers'; + +const input: SemanticJudgeInput = { node, rules: [passFailRule, graded] }; + +describe('FakeJudgeProvider', () => { + it("passes every rule by default using each rule's first non-failing verdict", async () => { + const fake = new FakeJudgeProvider(); + const results = await fake.evaluate(input); + expect(results['customer-voice']).toMatchObject({ verdict: 'pass' }); + expect(results.framing).toMatchObject({ verdict: 'good' }); + expect(fake.callHistory).toHaveLength(1); + }); + + it('maps the { pass } shorthand onto pass/fail', async () => { + const results = await new FakeJudgeProvider({ pass: false, evidence: 'nope' }).evaluate({ + node, + rules: [passFailRule], + }); + expect(results['customer-voice']).toEqual({ verdict: 'fail', evidence: 'nope' }); + }); + + it('returns full verdicts and errors as configured', async () => { + const fake = new FakeJudgeProvider({ + framing: { verdict: 'solution', confidence: 0.8 }, + 'customer-voice': { error: 'boom' }, + }); + const results = await fake.evaluate(input); + expect(results.framing).toEqual({ verdict: 'solution', confidence: 0.8 }); + expect(results['customer-voice']).toEqual({ error: 'boom' }); + }); + + it('prefers rule:title over rule keys, and supports Map and handler configs', async () => { + const map = new Map([ + ['framing', { verdict: 'weak' }], + ['framing:I cannot find my receipts', { verdict: 'solution' }], + ]); + expect((await new FakeJudgeProvider(map).evaluate(input)).framing).toEqual({ verdict: 'solution' }); + + const handler = new FakeJudgeProvider(({ rule }) => ({ verdict: rule.id === 'framing' ? 'weak' : 'pass' })); + expect((await handler.evaluate(input)).framing).toEqual({ verdict: 'weak' }); + }); +}); diff --git a/tests/judge/provider/jev.test.ts b/tests/judge/provider/jev.test.ts new file mode 100644 index 0000000..da3ae03 --- /dev/null +++ b/tests/judge/provider/jev.test.ts @@ -0,0 +1,83 @@ +import { afterEach, describe, expect, it } from 'bun:test'; +import { createModelProvider } from '../../../src/judge/factory'; +import { JevJudgeProvider } from '../../../src/judge/provider/jev'; +import { graded, node, passFailRule } from '../helpers'; + +const realFetch = globalThis.fetch; +afterEach(() => { + globalThis.fetch = realFetch; +}); + +function mockFetch(body: unknown, status = 200) { + const calls: Array<{ url: string; init: RequestInit }> = []; + globalThis.fetch = (async (url: string, init: RequestInit) => { + calls.push({ url, init }); + return new Response(JSON.stringify(body), { status }); + }) as unknown as typeof fetch; + return calls; +} + +describe('JevJudgeProvider', () => { + it('sends one request per node with a choice question per rule', async () => { + const calls = mockFetch({ + answers: { + 'customer-voice': { + type: 'choice', + choice: 'pass', + confidence: 0.9, + probabilities: { pass: 0.95, fail: 0.05 }, + }, + framing: { + type: 'choice', + choice: 'solution', + confidence: 0.6, + probabilities: { good: 0.1, weak: 0.3, solution: 0.6 }, + }, + }, + }); + const jev = new JevJudgeProvider({ api: 'openrouter-decisions', apiKey: 'k' }); + const results = await jev.evaluate({ node, rules: [passFailRule, graded] }); + + expect(calls).toHaveLength(1); + expect(calls[0]?.url).toBe('https://openrouter.ai/api/alpha/decisions'); + const sent = JSON.parse(calls[0]?.init.body as string); + expect(sent.model).toBe('jev-latest'); + expect(sent.state).toMatchObject({ title: 'I cannot find my receipts', parents: ['Faster Tax Filing'] }); + expect(sent.questions.framing).toEqual({ + type: 'choice', + instructions: 'Framing quality', + criteria: graded.verdicts, + }); + expect(sent.questions['customer-voice'].instructions).toContain('customer quote'); + expect(((calls[0]?.init.headers ?? {}) as Record).authorization).toBe('Bearer k'); + + expect(results.framing).toMatchObject({ verdict: 'solution', confidence: 0.6 }); + expect((results.framing as { evidence: string }).evidence).toContain('solution 0.60'); + }); + + it('reports an error for a rule with a missing or unknown choice, leaving others intact', async () => { + mockFetch({ answers: { framing: { choice: 'nonsense' }, 'customer-voice': { choice: 'fail' } } }); + const results = await new JevJudgeProvider({ api: 'openrouter-decisions', apiKey: 'k' }).evaluate({ + node, + rules: [passFailRule, graded], + }); + expect(results.framing).toHaveProperty('error'); + expect(results['customer-voice']).toMatchObject({ verdict: 'fail' }); + }); + + it('throws on an HTTP error', async () => { + mockFetch({ error: 'bad' }, 401); + await expect( + new JevJudgeProvider({ api: 'openrouter-decisions', apiKey: 'k' }).evaluate({ node, rules: [passFailRule] }), + ).rejects.toThrow('Jev API error (401)'); + }); + + it('is created explicitly via a bundled provider with its key env, never auto-detected', () => { + expect( + createModelProvider('openrouter-jev', { api: 'openrouter-decisions' }, { OPENROUTER_API_KEY: 'k' }).name, + ).toBe('jev'); + expect(() => createModelProvider('openrouter-jev', { api: 'openrouter-decisions' }, {})).toThrow( + 'OPENROUTER_API_KEY', + ); + }); +}); diff --git a/tests/judge/provider/openai-decisions.test.ts b/tests/judge/provider/openai-decisions.test.ts new file mode 100644 index 0000000..2c4e0be --- /dev/null +++ b/tests/judge/provider/openai-decisions.test.ts @@ -0,0 +1,114 @@ +import { afterEach, describe, expect, it } from 'bun:test'; +import { createModelProvider } from '../../../src/judge/factory'; +import { OPENAI_DECISIONS_ENDPOINT, OpenAIDecisionsJudgeProvider } from '../../../src/judge/provider/openai-decisions'; +import { graded, node, passFailRule } from '../helpers'; + +const realFetch = globalThis.fetch; +afterEach(() => { + globalThis.fetch = realFetch; +}); + +function mockFetch(body: unknown, status = 200) { + const calls: Array<{ url: string; init: RequestInit }> = []; + globalThis.fetch = (async (url: string, init: RequestInit) => { + calls.push({ url, init }); + return new Response(JSON.stringify(body), { status }); + }) as unknown as typeof fetch; + return calls; +} + +describe('OpenAIDecisionsJudgeProvider', () => { + it('sends one request per node with a choice question per rule', async () => { + const calls = mockFetch({ + answers: [ + { + type: 'choice', + name: 'customer-voice', + choice: 'pass', + confidence: 0.9, + probabilities: [ + { value: 'pass', probability: 0.95 }, + { value: 'fail', probability: 0.05 }, + ], + }, + { + type: 'choice', + name: 'framing', + choice: 'solution', + confidence: 0.6, + probabilities: [ + { value: 'good', probability: 0.1 }, + { value: 'weak', probability: 0.3 }, + { value: 'solution', probability: 0.6 }, + ], + }, + ], + }); + const provider = new OpenAIDecisionsJudgeProvider({ api: 'openai-decisions', apiKey: 'k' }); + const results = await provider.evaluate({ node, rules: [passFailRule, graded] }); + + expect(calls).toHaveLength(1); + expect(calls[0]?.url).toBe(OPENAI_DECISIONS_ENDPOINT); + const sent = JSON.parse(calls[0]?.init.body as string); + expect(sent.model).toBe('gpt-6-luna'); + expect(sent.input).toContain('Title: I cannot find my receipts'); + expect(sent.input).toContain('Parents: Faster Tax Filing'); + expect(sent.questions).toEqual([ + { + type: 'choice', + name: 'customer-voice', + instructions: 'Needs to sound like a customer quote, not a solution or business goal', + choices: [ + { value: 'pass', description: 'Customer voice.' }, + { value: 'fail', description: 'Not customer voice.' }, + ], + }, + { + type: 'choice', + name: 'framing', + instructions: 'Framing quality', + choices: [ + { value: 'good', description: 'Clear need' }, + { value: 'weak', description: 'Vague need' }, + { value: 'solution', description: 'A solution in disguise' }, + ], + }, + ]); + expect(((calls[0]?.init.headers ?? {}) as Record).authorization).toBe('Bearer k'); + + expect(results.framing).toMatchObject({ verdict: 'solution', confidence: 0.6, probabilities: { solution: 0.6 } }); + expect((results.framing as { evidence: string }).evidence).toContain('solution 0.60'); + }); + + it('reports an error for a rule with a missing, refused or unknown choice, leaving others intact', async () => { + mockFetch({ + answers: [ + { type: 'choice', name: 'framing', choice: 'nonsense' }, + { type: 'refusal', name: 'customer-voice' }, + ], + }); + const results = await new OpenAIDecisionsJudgeProvider({ api: 'openai-decisions', apiKey: 'k' }).evaluate({ + node, + rules: [passFailRule, graded], + }); + expect(results.framing).toHaveProperty('error'); + expect(results['customer-voice']).toHaveProperty('error'); + }); + + it('throws on an HTTP error', async () => { + mockFetch({ error: 'bad' }, 401); + await expect( + new OpenAIDecisionsJudgeProvider({ api: 'openai-decisions', apiKey: 'k' }).evaluate({ + node, + rules: [passFailRule], + }), + ).rejects.toThrow('OpenAI Decisions API error (401)'); + }); + + it('is created explicitly via a bundled provider with its key env, never auto-detected', () => { + expect(createModelProvider('openai-decisions', { api: 'openai-decisions' }, { OPENAI_API_KEY: 'k' }).name).toBe( + 'openai-decisions', + ); + expect(() => createModelProvider('openai-decisions', { api: 'openai-decisions' }, {})).toThrow('OPENAI_API_KEY'); + }); +}); diff --git a/tests/judge/verdicts.test.ts b/tests/judge/verdicts.test.ts new file mode 100644 index 0000000..15b62e0 --- /dev/null +++ b/tests/judge/verdicts.test.ts @@ -0,0 +1,67 @@ +import { describe, expect, it } from 'bun:test'; +import { PerRuleJudgeProvider } from '../../src/judge/provider/per-rule'; +import type { SemanticJudgeResult } from '../../src/judge/types'; +import { resolveSemanticRule } from '../../src/judge/verdicts'; +import type { Rule } from '../../src/types'; +import { graded, node, passFailRule } from './helpers'; + +const base: Rule = { id: 'r', category: 'best-practice', description: 'd', mode: 'semantic' }; + +describe('resolveSemanticRule', () => { + it('defaults to pass/fail with fail at the rule severity', () => { + const resolved = resolveSemanticRule(base); + expect(Object.keys(resolved.spec.verdicts)).toEqual(['pass', 'fail']); + expect(resolved.severities).toEqual({ fail: 'info' }); // best-practice default + expect(resolveSemanticRule({ ...base, severity: 'error' }).severities).toEqual({ fail: 'error' }); + }); + + it('maps declared verdicts to severities; unmapped and ignored verdicts pass', () => { + const resolved = resolveSemanticRule({ + ...base, + verdicts: { ok: 'O', minor: 'M', major: 'J', skip: 'S' }, + severity: { minor: 'warning', major: 'error', skip: 'ignore' }, + }); + expect(resolved.severities).toEqual({ minor: 'warning', major: 'error' }); + }); + + it('splits a verdict into the judge description and user guidance', () => { + const resolved = resolveSemanticRule({ + ...base, + verdicts: { + ok: 'Fine', + vague: { description: 'Too vague to act on', guidance: 'Name the customer and the pain.' }, + }, + severity: { vague: 'warning' }, + }); + expect(resolved.spec.verdicts).toEqual({ ok: 'Fine', vague: 'Too vague to act on' }); + expect(resolved.guidance).toEqual({ vague: 'Name the customer and the pain.' }); + }); + + it('rejects custom verdicts without a severity map, unknown verdicts, and sets with no passing verdict', () => { + expect(() => resolveSemanticRule({ ...base, verdicts: { a: 'A', b: 'B' } })).toThrow( + 'must map the failing verdicts', + ); + expect(() => resolveSemanticRule({ ...base, verdicts: { a: 'A', b: 'B' }, severity: { z: 'error' } })).toThrow( + 'not in its verdicts', + ); + expect(() => + resolveSemanticRule({ ...base, verdicts: { a: 'A', b: 'B' }, severity: { a: 'error', b: 'warning' } }), + ).toThrow('no passing verdict'); + }); +}); + +describe('PerRuleJudgeProvider', () => { + class Stub extends PerRuleJudgeProvider { + readonly name = 'stub'; + protected async judgeRule(_n: unknown, rule: { id: string }): Promise { + if (rule.id === 'framing') throw new Error('network down'); + return { verdict: 'pass' }; + } + } + + it('isolates a failing rule from the others', async () => { + const results = await new Stub().evaluate({ node, rules: [passFailRule, graded] }); + expect(results['customer-voice']).toEqual({ verdict: 'pass' }); + expect(results.framing).toEqual({ error: 'network down' }); + }); +}); diff --git a/tests/schema/evaluate-rule.test.ts b/tests/schema/evaluate-rule.test.ts index 20e8124..1e09bc4 100644 --- a/tests/schema/evaluate-rule.test.ts +++ b/tests/schema/evaluate-rule.test.ts @@ -1,5 +1,10 @@ import { describe, expect, it } from 'bun:test'; -import { buildEvalContext, evaluateExpression } from '../../src/schema/evaluate-rule'; +import { + buildEvalContext, + ExpressionError, + evaluateExpression, + tryEvaluateExpression, +} from '../../src/schema/evaluate-rule'; import type { SpaceNode } from '../../src/types'; import { makeParentRef } from '../test-helpers'; @@ -111,18 +116,10 @@ describe('evaluate-rule', () => { expect(result).toBe(1); }); - it('returns false on expression evaluation error', async () => { - // Suppress expected warning for invalid syntax - const originalWarn = console.warn; - console.warn = () => {}; - - try { - const context = buildEvalContext(mockNode, allNodes, mockNodeIndex); - const result = await evaluateExpression('invalid.syntax.('!, context); - expect(result).toBe(false); - } finally { - console.warn = originalWarn; - } + it('throws on an expression evaluation error, which tryEvaluateExpression reports as data', async () => { + const context = buildEvalContext(mockNode, allNodes, mockNodeIndex); + await expect(evaluateExpression('invalid.syntax.('!, context)).rejects.toThrow(ExpressionError); + expect(await tryEvaluateExpression('invalid.syntax.('!, context)).toHaveProperty('error'); }); }); diff --git a/tests/schema/rule-filter.test.ts b/tests/schema/rule-filter.test.ts new file mode 100644 index 0000000..37054f9 --- /dev/null +++ b/tests/schema/rule-filter.test.ts @@ -0,0 +1,39 @@ +import { describe, expect, it } from 'bun:test'; +import { filterRules, parseRuleFilter } from '../../src/schema/rule-filter'; +import type { Rule } from '../../src/types'; + +const rules: Rule[] = [ + { id: 'a', category: 'validation', description: 'a', check: 'true', type: 'goal' }, + { id: 'b', category: 'best-practice', description: 'b', mode: 'semantic', type: 'opportunity' }, + { id: 'c', category: 'coherence', description: 'c', check: 'true', severity: 'error' }, +]; +const ids = (r: Rule[]) => r.map((x) => x.id); + +describe('rule filter', () => { + it('parses repeated specs and merges values per dimension', () => { + expect(parseRuleFilter(['mode=semantic', 'category=coherence,validation', 'category=workflow'])).toEqual({ + mode: ['semantic'], + category: ['coherence', 'validation', 'workflow'], + }); + }); + + it('rejects malformed specs and unknown dimensions', () => { + expect(() => parseRuleFilter(['semantic'])).toThrow('Invalid rule filter'); + expect(() => parseRuleFilter(['colour=red'])).toThrow('Invalid rule filter'); + expect(() => parseRuleFilter(['mode='])).toThrow('Invalid rule filter'); + }); + + it('filters by mode (defaulting to expression), severity (effective), category, type and id', () => { + expect(ids(filterRules(rules, { mode: ['semantic'] }))).toEqual(['b']); + expect(ids(filterRules(rules, { mode: ['expression'] }))).toEqual(['a', 'c']); + expect(ids(filterRules(rules, { severity: ['error'] }))).toEqual(['a', 'c']); + expect(ids(filterRules(rules, { category: ['coherence'] }))).toEqual(['c']); + expect(ids(filterRules(rules, { type: ['goal', 'opportunity'] }))).toEqual(['a', 'b']); + expect(ids(filterRules(rules, { id: ['c'] }))).toEqual(['c']); + }); + + it('ANDs dimensions and treats an undefined filter as everything', () => { + expect(ids(filterRules(rules, { severity: ['error'], mode: ['semantic'] }))).toEqual([]); + expect(filterRules(rules, undefined)).toBe(rules); + }); +}); diff --git a/tests/schema/semantic-rule-contract.test.ts b/tests/schema/semantic-rule-contract.test.ts new file mode 100644 index 0000000..9c349a1 --- /dev/null +++ b/tests/schema/semantic-rule-contract.test.ts @@ -0,0 +1,79 @@ +import { afterAll, beforeAll, describe, expect, it } from 'bun:test'; +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { loadMetadata } from '../../src/schema/schema'; + +const META = + 'https://raw.githubusercontent.com/mindsocket/structured-context/main/schemas/generated/latest/_structured_context_schema_meta.json'; +const base = { id: 'r', category: 'validation', description: 'd' }; + +describe('rule contract: check and semantic mode', () => { + let dir: string; + beforeAll(() => { + dir = mkdtempSync(join(tmpdir(), 'sctx-rule-contract-')); + }); + afterAll(() => rmSync(dir, { recursive: true, force: true })); + + const load = (name: string, rule: object) => { + const path = join(dir, `${name}.json`); + writeFileSync( + path, + JSON.stringify({ + $schema: META, + $id: `sctx://rule-contract.${name}`, + $metadata: { rules: [rule] }, + type: 'object', + }), + ); + return loadMetadata(path); + }; + + it('requires check on expression rules', () => { + expect(() => load('no-check', base)).toThrow(); + }); + + it('accepts a semantic rule without check', () => { + expect(load('semantic-ok', { ...base, mode: 'semantic' }).rules).toHaveLength(1); + }); + + it('rejects a semantic rule that also declares check', () => { + expect(() => load('semantic-check', { ...base, mode: 'semantic', check: 'true' })).toThrow(); + }); + + it('rejects a global semantic rule', () => { + expect(() => load('semantic-global', { ...base, mode: 'semantic', scope: 'global' })).toThrow(); + }); + + it('accepts a verdict map with guidance, a severity map (including ignore) and a when', () => { + const rule = { + ...base, + mode: 'semantic', + when: "current.status = 'ready'", + verdicts: { ok: 'Fine', vague: { description: 'Too vague', guidance: 'Be specific.' } }, + severity: { vague: 'warning' }, + }; + expect(load('shape-ok', rule).rules).toHaveLength(1); + expect(load('ignore-ok', { ...rule, severity: { vague: 'ignore' } }).rules).toHaveLength(1); + }); + + it('requires a severity map when custom verdicts are declared', () => { + const verdicts = { ok: 'Fine', bad: 'Bad' }; + expect(() => load('verdicts-no-severity', { ...base, mode: 'semantic', verdicts })).toThrow(); + expect(() => load('verdicts-scalar', { ...base, mode: 'semantic', verdicts, severity: 'error' })).toThrow(); + }); + + it('keeps semantic-only fields off expression rules', () => { + const expr = { ...base, check: 'true' }; + expect(() => load('expr-map', { ...expr, severity: { fail: 'error' } })).toThrow(); + // Rules do not choose a judge: that is config, not schema. + expect(() => load('expr-judge', { ...expr, judge: 'llm' })).toThrow(); + expect(() => load('semantic-judge', { ...base, mode: 'semantic', judge: 'llm' })).toThrow(); + expect(() => load('expr-verdicts', { ...expr, verdicts: { a: 'A', b: 'B' } })).toThrow(); + expect(load('expr-when', { ...expr, when: 'true' }).rules).toHaveLength(1); + }); + + it('rejects `when` on a global rule', () => { + expect(() => load('global-when', { ...base, check: 'true', scope: 'global', when: 'true' })).toThrow(); + }); +}); diff --git a/tests/schema/validate-rules-semantic.test.ts b/tests/schema/validate-rules-semantic.test.ts new file mode 100644 index 0000000..7b29f56 --- /dev/null +++ b/tests/schema/validate-rules-semantic.test.ts @@ -0,0 +1,155 @@ +import { describe, expect, it } from 'bun:test'; +import { FakeJudgeProvider } from '../../src/judge/provider/fake'; +import { validateRules } from '../../src/schema/validate-rules'; +import type { Rule, SpaceNode } from '../../src/types'; +import { makeParentRef } from '../test-helpers'; + +describe('validateRules - semantic evaluation', () => { + const sampleNodes: SpaceNode[] = [ + { + label: 'outcome.md', + title: 'Outcome Metric', + type: 'outcome', + resolvedType: 'outcome', + schemaData: { title: 'Outcome Metric', type: 'outcome', status: 'active', metric: 'Increase conversions' }, + linkTargets: ['Outcome Metric'], + resolvedParents: [], + resolvedLinks: [], + }, + { + label: 'valid-opp.md', + title: 'I struggle to finish signup on mobile', + type: 'opportunity', + resolvedType: 'opportunity', + schemaData: { + title: 'I struggle to finish signup on mobile', + type: 'opportunity', + status: 'active', + parent: '[[Outcome Metric]]', + source: 'User interview with Jane', + }, + linkTargets: ['I struggle to finish signup on mobile'], + resolvedParents: [makeParentRef('Outcome Metric')], + resolvedLinks: [], + }, + { + label: 'bad-opp.md', + title: 'Build social login integration', + type: 'opportunity', + resolvedType: 'opportunity', + schemaData: { + title: 'Build social login integration', + type: 'opportunity', + status: 'active', + parent: '[[Outcome Metric]]', + source: 'Engineering suggestion', + }, + linkTargets: ['Build social login integration'], + resolvedParents: [makeParentRef('Outcome Metric')], + resolvedLinks: [], + }, + ]; + + const rules: Rule[] = [ + { + id: 'active-opportunity-count', + category: 'workflow', + description: 'At most 2 active opportunities', + check: '$count(nodes[resolvedType="opportunity" and status="active"]) <= 2', + }, + { + id: 'opportunity-customer-voice', + category: 'best-practice', + mode: 'semantic', + type: 'opportunity', + description: 'Opportunity must be framed in customer voice, not a solution or business goal', + rubric: 'Must describe a customer pain point or need in customer language, not a solution to build.', + }, + ]; + + it('skips semantic rules without a judge (default validate)', async () => { + // Default validate should only run expression rules and remain deterministic & free + const violations = await validateRules(sampleNodes, rules); + expect(violations).toHaveLength(0); + }); + + it('evaluates semantic rules with a judge', async () => { + const fake = new FakeJudgeProvider( + new Map([ + [ + 'opportunity-customer-voice:I struggle to finish signup on mobile', + { pass: true, evidence: 'Clearly stated from customer perspective as a pain point' }, + ], + [ + 'opportunity-customer-voice:Build social login integration', + { pass: false, evidence: 'Stated as a solution feature ("Build social login") rather than a customer need' }, + ], + ]), + ); + + const violations = await validateRules(sampleNodes, rules, { + judge: fake, + }); + + expect(violations).toHaveLength(1); + expect(violations[0]?.ruleId).toBe('opportunity-customer-voice'); + expect(violations[0]?.file).toBe('bad-opp.md'); + expect(violations[0]?.category).toBe('best-practice'); + expect(violations[0]?.evidence).toBe( + 'Stated as a solution feature ("Build social login") rather than a customer need', + ); + expect(fake.callHistory.map((c) => c.node.label).sort()).toEqual(['bad-opp.md', 'valid-opp.md']); + }); + + it('gates semantic rules when node has upstream structural error (cascading)', async () => { + const fake = new FakeJudgeProvider(); + + // Mark bad-opp.md as having an upstream structural error via gatedNodeLabels + const gatedNodeLabels = new Set(['bad-opp.md']); + + const violations = await validateRules(sampleNodes, rules, { + judge: fake, + gatedNodeLabels, + }); + + // bad-opp.md was gated; only valid-opp.md was sent to the judge + expect(fake.callHistory).toHaveLength(1); + expect(fake.callHistory[0]?.node.label).toBe('valid-opp.md'); + expect(violations).toHaveLength(0); + }); + + it('gates semantic rules when node fails an upstream expression rule', async () => { + const strictRules: Rule[] = [ + { + id: 'require-interviews-source', + category: 'validation', + type: 'opportunity', + check: '$contains(current.source, "interview") = true', + description: 'Source must mention interview', + }, + { + id: 'opportunity-customer-voice', + category: 'best-practice', + mode: 'semantic', + type: 'opportunity', + description: 'Must be in customer voice', + }, + ]; + + const fake = new FakeJudgeProvider(); + + const violations = await validateRules(sampleNodes, strictRules, { + judge: fake, + }); + + // bad-opp.md source is "Engineering suggestion", so it failed require-interviews-source + // Therefore it should be gated from opportunity-customer-voice + const badOppViolations = violations.filter((v) => v.file === 'bad-opp.md'); + expect(badOppViolations).toHaveLength(1); + expect(badOppViolations[0]?.ruleId).toBe('require-interviews-source'); + + // Only valid-opp.md was evaluated by the semantic judge + expect(fake.callHistory).toHaveLength(1); + expect(fake.callHistory[0]?.node.label).toBe('valid-opp.md'); + }); +}); diff --git a/tests/validate/dates.test.ts b/tests/validate/dates.test.ts index 2e1d5f2..dc11f9c 100644 --- a/tests/validate/dates.test.ts +++ b/tests/validate/dates.test.ts @@ -54,13 +54,12 @@ describe('dates and format validation', () => { }); describe('format: "date-time" schema validation', () => { - it.each([ - '2026-03-31T09:15:00Z', - '2026-03-31T09:15:00.250Z', - '2026-03-31T09:15:00+10:00', - ])('accepts %s', (value) => { - expect(validateNode({ type: 'note', title: 'Test', updated: value })).toBe(true); - }); + it.each(['2026-03-31T09:15:00Z', '2026-03-31T09:15:00.250Z', '2026-03-31T09:15:00+10:00'])( + 'accepts %s', + (value) => { + expect(validateNode({ type: 'note', title: 'Test', updated: value })).toBe(true); + }, + ); it.each([ '2026-03-31', diff --git a/tests/validate/semantic-rule-shape.test.ts b/tests/validate/semantic-rule-shape.test.ts new file mode 100644 index 0000000..829c85f --- /dev/null +++ b/tests/validate/semantic-rule-shape.test.ts @@ -0,0 +1,169 @@ +import { afterAll, describe, expect, it } from 'bun:test'; +import { FakeJudgeProvider } from '../../src/judge/provider/fake'; +import { createSpaceContext } from '../../src/space-context'; +import type { FileValidationResult } from '../../src/types'; +import { validateFile, validateSpace } from '../../src/validate'; +import { makeTempSpace } from '../helpers/temp-space'; + +const spaces: ReturnType[] = []; +afterAll(() => { + for (const s of spaces) s.cleanup(); +}); + +/** A semantic rule on steps with minor/major verdicts mapped to warning/error and one ignored verdict. */ +const stepRule = { + id: 'step-fit', + type: 'step', + mode: 'semantic', + category: 'coherence', + description: 'Step fits its task', + verdicts: { + fits: 'The step fits', + minor: { description: 'Slightly off', guidance: 'Tighten the wording.' }, + major: { description: 'Unrelated', guidance: 'Move it under a better task.' }, + unsure: 'Cannot tell', + }, + severity: { minor: 'warning', major: 'error', unsure: 'ignore' }, +}; + +function setup(rules: object[], options?: Parameters[1]) { + const space = makeTempSpace(rules, options); + spaces.push(space); + return { + space, + context: () => createSpaceContext('ts', space.config, { configDir: space.dir }), + }; +} + +describe('semantic rules: verdict severity, ignore and guidance', () => { + it('reports each verdict at its mapped severity with its guidance', async () => { + const { context } = setup([stepRule]); + const minor = await validateSpace(context(), { + judge: new FakeJudgeProvider({ 'step-fit': { verdict: 'minor' } }), + }); + expect(minor.ruleViolations).toEqual([ + expect.objectContaining({ + ruleId: 'step-fit', + severity: 'warning', + verdict: 'minor', + guidance: 'Tighten the wording.', + }), + ]); + + const major = await validateSpace(context(), { + judge: new FakeJudgeProvider({ 'step-fit': { verdict: 'major' } }), + }); + expect(major.ruleViolations[0]).toMatchObject({ severity: 'error', verdict: 'major' }); + }); + + it('judges an ignored verdict without reporting it', async () => { + const { context } = setup([stepRule]); + const fake = new FakeJudgeProvider({ 'step-fit': { verdict: 'unsure' } }); + const result = await validateSpace(context(), { judge: fake }); + expect(fake.callHistory).toHaveLength(1); + expect(result.ruleViolations).toEqual([]); + }); + + it('gates the subtree per verdict: an error-mapped verdict gates, a warning-mapped one does not', async () => { + const taskRule = { ...stepRule, id: 'task-fit', type: 'task' }; + const { context } = setup([taskRule, { ...stepRule, id: 'step-fit' }]); + + const minor = new FakeJudgeProvider({ 'task-fit': { verdict: 'minor' } }); + const ungated = await validateSpace(context(), { judge: minor }); + expect(ungated.coverageGates).toEqual([]); + expect(minor.callHistory.some((c) => c.node.label === 'S.md')).toBe(true); + + const major = new FakeJudgeProvider({ 'task-fit': { verdict: 'major' } }); + const gated = await validateSpace(context(), { judge: major }); + expect(gated.coverageGates[0]).toMatchObject({ reason: 'rule', ruleId: 'task-fit' }); + expect(major.callHistory.some((c) => c.node.label === 'S.md')).toBe(false); + }); + + it('shows guidance in validate-file output', async () => { + const { space } = setup([stepRule]); + const result = (await validateFile(space.file('S.md'), space.config, { + judge: new FakeJudgeProvider({ 'step-fit': { verdict: 'minor' } }), + configDir: space.dir, + })) as FileValidationResult; + expect(result.warnings['rule:step-fit']?.message).toContain('Guidance: Tighten the wording.'); + }); + + it('--rule-filter severity matches any severity a verdict map can report', async () => { + const { context } = setup([stepRule]); + const judge = new FakeJudgeProvider({ 'step-fit': { verdict: 'minor' } }); + await validateSpace(context(), { judge: judge, ruleFilter: { severity: ['warning'] } }); + expect(judge.callHistory.length).toBeGreaterThan(0); + const none = new FakeJudgeProvider(); + await validateSpace(context(), { judge: none, ruleFilter: { severity: ['info'] } }); + expect(none.callHistory).toHaveLength(0); + }); +}); + +describe('rule `when` precondition', () => { + const when = "current.status = 'ready'"; + + it('skips an expression rule on nodes where `when` is not true', async () => { + const rule = { + id: 'needs-owner', + category: 'validation', + description: 'owner', + check: '$exists(current.owner)', + when, + }; + const { context } = setup([rule], { frontmatter: { T: 'status: ready', S: 'status: draft' } }); + const result = await validateSpace(context()); + expect(result.ruleViolations.map((v) => v.file).sort()).toEqual(['T.md']); + }); + + it('does not call the judge for nodes where `when` is not true', async () => { + const rule = { ...stepRule, id: 'task-fit', type: 'task', when }; + const { context } = setup([rule], { frontmatter: { T: 'status: ready' } }); + const judge = new FakeJudgeProvider(); + await validateSpace(context(), { judge: judge }); + expect(judge.callHistory.map((c) => c.node.label)).toEqual(['T.md']); + + const none = setup([rule]); + const idle = new FakeJudgeProvider(); + const result = await validateSpace(none.context(), { judge: idle }); + expect(idle.callHistory).toHaveLength(0); + expect(result.coverageGates).toEqual([]); // not applicable is not gated + }); +}); + +describe('judge option', () => { + it('evaluates no semantic rules without a judge', async () => { + const { context } = setup([stepRule]); + const result = await validateSpace(context(), {}); + expect(result.ruleViolations).toEqual([]); + }); + + it('judges with the judge it is given', async () => { + const { context } = setup([stepRule]); + const judge = new FakeJudgeProvider({ 'step-fit': { verdict: 'minor' } }); + const result = await validateSpace(context(), { judge }); + expect(judge.callHistory.length).toBeGreaterThan(0); + expect(result.ruleViolations[0]).toMatchObject({ verdict: 'minor' }); + }); +}); + +describe('rule expression errors', () => { + const broken = { id: 'broken', category: 'validation', description: 'Broken check', check: '$notAFunction(' }; + const brokenWhen = { + id: 'bad-when', + category: 'coherence', + description: 'Task rule', + type: 'task', + when: '1 +', + check: 'true', + }; + + it('reports a failing check or when as an error violation instead of passing silently', async () => { + const { context } = setup([broken, brokenWhen]); + const result = await validateSpace(context(), {}); + const byId = (id: string) => result.ruleViolations.filter((v) => v.ruleId === id); + expect(byId('broken').length).toBeGreaterThan(0); + expect(byId('broken')[0]).toMatchObject({ severity: 'error' }); + expect(byId('broken')[0]?.description).toContain('Rule error'); + expect(byId('bad-when')).toEqual([expect.objectContaining({ file: 'T.md', severity: 'error' })]); + }); +}); diff --git a/tests/validate/semantic-validation.test.ts b/tests/validate/semantic-validation.test.ts new file mode 100644 index 0000000..845a47c --- /dev/null +++ b/tests/validate/semantic-validation.test.ts @@ -0,0 +1,231 @@ +import { afterAll, beforeAll, describe, expect, it } from 'bun:test'; +import { join } from 'node:path'; +import type { Config } from '../../src/config'; +import { FakeJudgeProvider } from '../../src/judge/provider/fake'; +import { bundledSchemasDir } from '../../src/schema/schema'; +import { createSpaceContext } from '../../src/space-context'; +import type { FileValidationResult } from '../../src/types'; +import { validateFile, validateSpace } from '../../src/validate'; +import { makeTempSpace, type TempSpace } from '../helpers/temp-space'; + +const STRICT_SCHEMA_PATH = join(bundledSchemasDir, 'strict_ost.json'); +const VALID_DIR = join(import.meta.dir, '../fixtures/strict_ost/valid-directory'); +const INVALID_DIR = join(import.meta.dir, '../fixtures/strict_ost/invalid'); + +describe('Semantic rule evaluation integration (validateSpace & validateFile)', () => { + const validConfig: Config = { + spaces: [ + { + name: 'strict-valid', + path: VALID_DIR, + schema: STRICT_SCHEMA_PATH, + }, + ], + }; + + const validContext = createSpaceContext('strict-valid', validConfig, { configDir: import.meta.dir }); + + describe('validateSpace', () => { + it('skips semantic rules without a judge (deterministic & free)', async () => { + const result = await validateSpace(validContext); + expect(result.ruleViolations.some((v) => v.ruleId === 'opportunity-customer-voice')).toBe(false); + }); + + it('evaluates semantic rules when a judge is passed', async () => { + const fakeProvider = new FakeJudgeProvider({ + pass: true, + evidence: 'Opportunity is framed in clear customer language describing difficulty with sign-up.', + }); + + const result = await validateSpace(validContext, { + judge: fakeProvider, + }); + + expect(fakeProvider.callHistory.some((c) => c.rules.some((r) => r.id === 'opportunity-customer-voice'))).toBe( + true, + ); + + // No violation when the verdict has no severity + expect(result.ruleViolations.some((v) => v.ruleId === 'opportunity-customer-voice')).toBe(false); + }); + + it('records violation with evidence when judge returns pass: false', async () => { + const fakeProvider = new FakeJudgeProvider({ + pass: false, + evidence: 'The opportunity reads like a feature request rather than a problem statement.', + }); + + const result = await validateSpace(validContext, { + judge: fakeProvider, + }); + + const violation = result.ruleViolations.find((v) => v.ruleId === 'opportunity-customer-voice'); + expect(violation).toBeDefined(); + expect(violation?.category).toBe('best-practice'); + expect(violation?.evidence).toBe('The opportunity reads like a feature request rather than a problem statement.'); + expect(violation?.file).toBe('Sign-up is too complex.md'); + }); + + it('gates semantic rules when node has upstream structural errors (cascading)', async () => { + const invalidConfig: Config = { + spaces: [ + { + name: 'strict-invalid', + path: INVALID_DIR, + schema: STRICT_SCHEMA_PATH, + }, + ], + }; + const invalidContext = createSpaceContext('strict-invalid', invalidConfig, { configDir: import.meta.dir }); + const fakeProvider = new FakeJudgeProvider(); + + const result = await validateSpace(invalidContext, { + judge: fakeProvider, + }); + + // opportunity-no-source.md fails schema validation (missing required 'source' field) + expect(result.nodeErrors.some((e) => e.file === 'opportunity-no-source.md')).toBe(true); + + // Therefore it must be gated and never evaluated by the judge + expect(result.coverageGates.some((g) => g.file === 'opportunity-no-source.md' && g.reason === 'schema')).toBe( + true, + ); + // Judge was never called for opportunity-no-source.md + expect(fakeProvider.callHistory.some((c) => c.node.label === 'opportunity-no-source.md')).toBe(false); + }); + }); + + describe('validateFile', () => { + const targetFile = join(VALID_DIR, 'Sign-up is too complex.md'); + + it('skips semantic rule evaluation for single file by default', async () => { + const result = (await validateFile(targetFile, validConfig, { + configDir: import.meta.dir, + })) as FileValidationResult; + + expect(result.inSpace).toBe(true); + expect(result.errors['rule:opportunity-customer-voice']).toBeUndefined(); + }); + + it('evaluates semantic rule and reports evidence on violation when semantic: true', async () => { + const failingProvider = new FakeJudgeProvider({ + pass: false, + evidence: 'Opportunity is framed in solution space.', + }); + + const result = (await validateFile(targetFile, validConfig, { + judge: failingProvider, + configDir: import.meta.dir, + })) as FileValidationResult; + + expect(result.inSpace).toBe(true); + // best-practice rules default to non-error severity, so they land in warnings + const ruleError = result.warnings['rule:opportunity-customer-voice']; + expect(ruleError).toBeDefined(); + expect(ruleError?.kind).toBe('rule'); + expect(ruleError?.message).toContain('Evidence: Opportunity is framed in solution space.'); + }); + }); +}); + +describe('validateFile judges only the target and its gate-capable ancestor rules', () => { + let space: TempSpace; + const rule = (id: string, type: string, severity: string) => ({ + id, + type, + mode: 'semantic', + category: 'validation', + severity, + description: id, + }); + + beforeAll(() => { + space = makeTempSpace([ + rule('goal-sound', 'goal', 'error'), // gates descendants + rule('task-sound', 'task', 'warning'), // does not gate + rule('step-sound', 'step', 'warning'), + ]); + }); + afterAll(() => space.cleanup()); + + const run = (judge: FakeJudgeProvider, file: string, extra = {}) => + validateFile(space.file(file), space.config, { + judge: judge, + configDir: space.dir, + ...extra, + }) as Promise; + + it('judges the target plus only the gate-capable rules of its ancestors', async () => { + const judge = new FakeJudgeProvider({ pass: true }); + await run(judge, 'S.md'); + const judged = judge.callHistory.map((c) => `${c.node.label}:${c.rules.map((r) => r.id).join(',')}`).sort(); + // G's rule can gate; T's cannot and Other is unrelated. + expect(judged).toEqual(['G.md:goal-sound', 'S.md:step-sound']); + }); + + it('still lets a failing ancestor rule gate the target', async () => { + const judge = new FakeJudgeProvider({ 'goal-sound': { verdict: 'fail' } }); + const result = await run(judge, 'S.md'); + expect(result.warnings['rule:step-sound']).toBeUndefined(); + expect(Object.keys(result.warnings).some((k) => k.startsWith('gated:'))).toBe(true); + expect(judge.callHistory.some((c) => c.node.label === 'S.md')).toBe(false); + }); + + it('with includeGated judges the target even when an ancestor fails', async () => { + const judge = new FakeJudgeProvider({ 'goal-sound': { verdict: 'fail' }, 'step-sound': { verdict: 'fail' } }); + const result = await run(judge, 'S.md', { includeGated: true }); + expect(result.warnings['rule:step-sound']).toBeDefined(); + }); + + it('judges nothing when the filter excludes every semantic rule', async () => { + const judge = new FakeJudgeProvider({ pass: true }); + await run(judge, 'S.md', { ruleFilter: { id: ['no-such-rule'] } }); + expect(judge.callHistory).toHaveLength(0); + }); +}); + +describe('Semantic rules: verdicts, batching and rule filter', () => { + const config: Config = { + spaces: [{ name: 'strict-valid', path: VALID_DIR, schema: STRICT_SCHEMA_PATH }], + }; + const context = createSpaceContext('strict-valid', config, { configDir: import.meta.dir }); + + it('judges each node once with all of its semantic rules', async () => { + const fake = new FakeJudgeProvider(); + await validateSpace(context, { judge: fake }); + const labels = fake.callHistory.map((c) => c.node.label); + expect(new Set(labels).size).toBe(labels.length); + expect(fake.callHistory.every((c) => c.rules.length >= 1)).toBe(true); + }); + + it('reports a verdict that has a severity as a violation carrying the verdict and evidence', async () => { + const fake = new FakeJudgeProvider({ verdict: 'fail', confidence: 0.7, evidence: 'solution in disguise' }); + const result = await validateSpace(context, { judge: fake }); + const violation = result.ruleViolations.find((v) => v.ruleId === 'opportunity-customer-voice'); + expect(violation).toMatchObject({ verdict: 'fail', evidence: 'solution in disguise' }); + }); + + it('reports a per-rule judge error as a violation without gating the subtree', async () => { + const fake = new FakeJudgeProvider({ error: 'rate limited' }); + const result = await validateSpace(context, { judge: fake }); + const violation = result.ruleViolations.find((v) => v.ruleId === 'opportunity-customer-voice'); + expect(violation?.description).toContain('Judge error: rate limited'); + expect(result.coverageGates.filter((g) => g.ruleId === 'opportunity-customer-voice')).toEqual([]); + }); + + it('--rule-filter narrows evaluation: semantic-only runs no expression rules, expression-only runs no judge', async () => { + const semanticOnly = await validateSpace(context, { + judge: new FakeJudgeProvider({ pass: false }), + ruleFilter: { mode: ['semantic'] }, + }); + expect(semanticOnly.ruleViolations.every((v) => v.ruleId === 'opportunity-customer-voice')).toBe(true); + + const fake = new FakeJudgeProvider(); + const expressionOnly = await validateSpace(context, { + judge: fake, + ruleFilter: { mode: ['expression'] }, + }); + expect(fake.callHistory).toHaveLength(0); + expect(expressionOnly.ruleViolations.some((v) => v.verdict !== undefined)).toBe(false); + }); +});