Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
c27080e
feat(rules): semantic rules judged by model or decision model (#114)
mindsocket Sep 18, 2026
c931432
fix(rules): require check unless semantic, reject global semantic rul…
mindsocket Oct 13, 2026
0f28d3d
docs: document judge config, semantic rule terms and API exports
mindsocket Oct 13, 2026
1c6141f
docs: point schema docs and plugin authoring reference at semantic rules
mindsocket Oct 13, 2026
f12511f
feat(validate-file): judge only the target and its gate-capable ances…
mindsocket Oct 13, 2026
a96732b
refactor(judge): require an explicit provider, move providers to judg…
mindsocket Oct 13, 2026
b93f7e5
feat(cli): --no-semantic override, plain error for misconfigured judg…
mindsocket Oct 13, 2026
403bfbc
feat(rules): verdict severity map with ignore and guidance, when prec…
mindsocket Oct 14, 2026
d7cbe7a
feat(judge): type/default config keys, drop SCTX_JUDGE_* env, report …
mindsocket Oct 14, 2026
a5fade2
docs(rules): restructure rule and semantic rule docs; drop judge.auto…
mindsocket Oct 14, 2026
69d4fb5
test(judge): opt-in live judge tests; fix --judge ignored by validate…
mindsocket Oct 14, 2026
b2571b3
refactor(api): keep judge and rule-filter internals out of the public…
mindsocket Oct 14, 2026
595e7a9
refactor(judge): drop semanticEvaluations and gatedRules, slim judge …
mindsocket Oct 14, 2026
d7549df
refactor(judge): model providers in config, judge picks one; runs ask…
mindsocket Oct 14, 2026
18564df
refactor(validate): one judge option on the API; the CLI decides when…
mindsocket Oct 14, 2026
828c04d
feat(judge): judge shorthand by bundled type; docs pass
mindsocket Oct 15, 2026
6f83b03
refactor(judge): bundled providers, not judge.type; live tests inline
mindsocket Oct 15, 2026
7f8f330
refactor(judge): provider.api with vendor bundles; sctx providers
mindsocket Oct 15, 2026
a112497
fix(judge): pin the openrouter bundle's endpoint; show endpoints in s…
mindsocket Oct 15, 2026
898e96a
feat(judge): OpenAI Decisions API provider; tighter bundled-provider …
mindsocket Oct 15, 2026
77eced9
docs: clarify rules.md structure — gating defaults, overrides, single…
mindsocket Oct 15, 2026
d997be9
style: format dates.test.ts per biome
mindsocket Oct 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ Before starting new work, review [docs/concepts.md](docs/concepts.md) for canoni
- `bun run src/index.ts --help` — list CLI commands
- `bun run typecheck` — type-check all code (`tsconfig.json`)
- `bun run lint` — biome
- `bun run lint:fix` — biome with fixes
- `bun run build` — compile `src/` to `dist/` (`tsconfig.build.json`)
- see `package.json`
- Releases go through the `/release` skill.

Space names (e.g. `personal`, `politics`) are resolved via a config file - `$SCTX_CONFIG`, `$XDG_CONFIG_HOME/structured-context/config.json`, `--config <file>` param, or `./config.json`
Expand Down Expand Up @@ -38,21 +40,36 @@ There are several places that need reviewing and updating with any new feature o
- **Simplicity first:** Strive for the simplest design that completely solves the problem. Prefer conventions and established filesystem patterns over adding new configuration, programmatic APIs, or abstractions.
- **Explicitly state alternatives:** When planning or making architectural decisions, do not anchor to the first familiar mechanism. Formulate and state at least one alternative approach, weighing trade-offs, complexity, and ergonomic impact.
- **Minimise blast radius:** Protect established boundaries and interfaces. Treat unnecessary ripple effects (such as making synchronous APIs asynchronous or adding dual APIs) as a design smell indicating the approach should be re-evaluated.
- **Public API surface:** Do not add exports to `src/api.ts`, `src/types.ts` or `src/testing.ts` (the `structured-context/api`, `/types` and `/testing` entry points), or add types or hooks to the plugin interface, without the maintainer's explicit agreement on that specific surface. Exporting something to make docs complete, to tidy up, or because a feature has a public-looking class is not a reason. If a change seems to need a new export, stop and propose it, with the consumer that needs it. Internal modules can change freely.
- **Where a setting lives:** Config says *what* to use, CLI flags override it for one run, and the environment holds secrets and the config location only (`apiKeyEnv` names a secret; `SCTX_CONFIG` locates the config). Do not add env vars that set config values, and never infer a choice (such as a judge provider) from a variable that happens to be set: a key present for another tool must not be spent silently. Config also never switches on costly or outward-facing behaviour (such as semantic judging): each run asks for it, through a flag or an API option.
- **Failures are visible:** A rule or expression that cannot be evaluated is reported as a violation, never skipped or treated as a pass. Likewise a misconfiguration fails before any work or spend.

## Documentation Principles

- **One idea lives in one place:** Decide which doc owns each concept, document it there once, and link instead of repeating. Small, non-obvious tidbits may be repeated; explanations may not.
- **Document what it does, not what it doesn't:** Describe behaviour and purpose. State a limit only when a new user would realistically assume otherwise. Specification and decision history do not leak into docs.
- **Proper name in docs:** `structured-context` is the project name for prose; `sctx` is only the CLI command in command examples.
- **Approachability:** Write short, declarative sentences in active voice, one idea per sentence — about 80% of the way to ASD-STE100. Introduce a concept once, before sections that need it. Frame options by purpose (what each is for, and for whom).
- **Front-load the happy path:** Quick starts give the fewest steps that run. Tuning, alternatives and edge cases come after.

## Key Files

- `src/api.ts` — Public library entry point (`structured-context/api`). Re-exports the supported programmatic surface. Keep CLI-only concerns out of it.
- `src/api.ts` — Public library entry point (`structured-context/api`). Re-exports the supported programmatic surface. Keep CLI-only concerns out of it. Do not extend it without agreement (see **Public API surface** above).
- `schemas/` — Bundled default schema files (JSON5) using the structured-context schema dialect and top-level `$metadata`. Files starting with `_` are "partials" (fragments for `$ref`).
- `src/schema/metadata-contract.ts` — Single source of truth for the `$metadata` contract
- `schemas/generated/latest/_structured_context_schema_meta.json` — Generated metaschema for the newest dialect (generated on build or with `bun run generate:schema-meta`). `schemas/generated/<version>/` folders are frozen copies, created by the `version` script (`generate:schema-meta --release`) only when a release changes the dialect; never edit them. `src/schema/meta-schemas.ts` loads them at runtime.
- `src/schema/gating.ts` — Cascade gating: computes hierarchy depth and evaluates rules with subtree gating (schema/hierarchy/blocking-rule failures suppress downstream rule checks). See [docs/rules.md](docs/rules.md#cascade-gating).
- `src/judge/` — Semantic rule judges: provider interface and types (`types.ts`), verdict resolution (`verdicts.ts`), per-node dispatch (`judge-node.ts`), and providers (`provider/anthropic.ts`, `provider/openai.ts`, `provider/jev.ts`, `provider/fake.ts`), provider hydration and the bundled catalogue (`factory.ts`), and `sctx providers` (`src/commands/providers.ts`). See [docs/semantic-rules.md](docs/semantic-rules.md).
- `src/schema/rule-filter.ts` — `--rule-filter` parsing and rule filtering by mode, category, severity, type or id.

## Testing
For most development only the main unit tests need re-running regularly.
- `bun run test` — unit tests (fixtures in `tests/`)
- `bun run test:hook` — unit test plugin hooks (`hook-test/unit/`) - hook development only
- `bun run test:hook:e2e` — test plugin hooks in Claude Code (`hook-test/`) - hook development only
- `bun run test:smoke` — smoke tests run against locally configured spaces - only use when changes could affect compatibility.
- `bun run test:judge` — live judge tests (`tests/judge/live.test.ts`): one real provider call per provider whose API key is set. Costs money; use sparingly when changing judges or semantic rule behaviour.
- `bun run test:models` — check that every bundled provider's configured model exists (no API keys)

## Debugging

Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ Rule violations:
High tech juice press.md: Explore multiple candidate solutions (aim for at least three) for the target opportunity [best-practice: solution-quantity]
```

Rule violations carry a severity (`error`, `warning` or `info`) that defaults from the rule's category. Only errors fail validation — see [docs/rules.md](docs/rules.md#categories-and-severity).
Rule violations carry a severity (`error`, `warning` or `info`) that defaults from the rule's category. Only errors fail validation — see [docs/rules.md](docs/rules.md#rule-shape).

## Installation

Expand Down Expand Up @@ -131,7 +131,7 @@ See [docs/schemas.md](docs/schemas.md) (`sctx docs schema`) for the dialect, com
### Validate nodes

```bash
sctx validate <space> [--watch] [--depth <n>] [--include-gated]
sctx validate <space> [--watch] [--depth <n>] [--include-gated] [--semantic] [--judge <provider>] [--rule-filter <dim=values>]
```

Validates markdown files against the JSON schema:
Expand All @@ -141,6 +141,7 @@ Validates markdown files against the JSON schema:
- Runs executable rules (`$metadata.rules`) with **cascade gating**: a schema, hierarchy, or blocking-rule failure on a node suppresses rule evaluation for that node's hierarchy subtree. Gated checks are reported in a Coverage section so you know what wasn't evaluated and why. See [docs/rules.md](docs/rules.md#cascade-gating) for details.
- `--depth <n>` limits evaluation to hierarchy nodes within `n` breadth-first levels from the top — useful for staged agentic fix loops (fix the top level, then request the next).
- `--include-gated` evaluates rules even on gated nodes (still reported as gated) — for surfacing cheap bottom-up checks on request.
- `--semantic` also evaluates `mode: semantic` rules with the configured judge (Jev, Anthropic or OpenAI-compatible); `--judge <provider>` uses that configured model provider instead and implies `--semantic`; `--rule-filter` narrows which rules run. See [docs/semantic-rules.md](docs/semantic-rules.md) (`sctx docs semantic-rules`).

### Render space

Expand Down
3 changes: 2 additions & 1 deletion config.example.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,6 @@
}
}
],
"schema": "/path/to/custom/schemas/my-general-schema.json"
"schema": "/path/to/custom/schemas/my-general-schema.json",
"judge": { "provider": "openrouter-jev" }
}
8 changes: 7 additions & 1 deletion docs/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,9 +151,15 @@ Each executable rule has a **category** (`validation`, `coherence`, `workflow`,

See [docs/rules.md](rules.md) for the rules reference, including JSONata expression syntax and the full `$metadata` field reference.

### Semantic rules

A **semantic rule** (`mode: semantic`) asks a model to judge a node, for example whether an opportunity is a customer need. It has a `rubric` instead of a `check`.

A **judge** is the model that judges semantic rules: a decision model (Jev) or a chat model (Claude, OpenAI-compatible). For each rule, the judge gives one **verdict** from the rule's `verdicts`, for example `customer-need`, `vague` or `solution`. The rule's `severity` says which verdicts report a violation. Semantic rules cost money, so they run only when a run asks for them (`--semantic` or `--judge`). The judge is a configured **model provider**: a bundled provider or an entry in `modelProviders`. See [docs/semantic-rules.md](semantic-rules.md).

### Cascade gating

**Cascade gating** is the mechanism by which a schema failure, hierarchy structure violation, or blocking rule violation on a node suppresses rule evaluation for that node's `hierarchy` subtree. It exists so downstream checks — especially future evaluative or LLM-judged layers — aren't run against inputs already known to be invalid.
**Cascade gating** is the mechanism by which a schema failure, hierarchy structure violation, or blocking rule violation on a node suppresses rule evaluation for that node's `hierarchy` subtree. It exists so downstream checks — especially costly semantic (model-judged) rules — aren't run against inputs already known to be invalid.

A **coverage gate** records one instance of this: the gating node, what kind of failure triggered it (`schema`, `hierarchy`, or a specific rule `id`), and the set of nodes whose rule checks were suppressed as a result. `sctx validate` reports coverage gates so both humans and agents can see the validation "edge" — where evaluation stopped and why. See [docs/rules.md](rules.md#cascade-gating) for the full model, including the per-rule `gatesDescendants` field and the `--include-gated` / `--depth` CLI options.

Expand Down
65 changes: 65 additions & 0 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,71 @@ All plugin names must start with `sctx-` (the prefix is optional in config and n

Plugins can also contribute schemas and partials via a `<pluginRoot>/schemas/` directory (or `"sctx": { "schemas": "..." }` in `package.json`). If a space specifies a plugin-prefixed schema (e.g. `"sctx-wardley-mapping/wardley_map.json"`), schemas are resolved statically without executing plugin code, and the plugin's code hooks are loaded when parsing or rendering.

## Model providers and judge

structured-context judges [semantic rules](semantic-rules.md) with a **model provider**: a named definition that says which wire API to speak, which model to use and where. A provider definition combines an `api`, explicit values or environment-variable names, and the key source.

**Bundled providers** work without any configuration: a fixed model per vendor chosen by this project, plus `-env` variants that also read the vendor's usual model and endpoint variables (`OPENAI_MODEL`, `OPENAI_BASE_URL`, `ANTHROPIC_MODEL`), falling back to the bundle's defaults where a variable is unset. Bundles without `-env` read no environment beyond the key: they behave the same on every machine.

Run `sctx providers` for the authoritative list of bundled and configured providers with their effective API, endpoint, model and key variable. Keys are never shown. `bun run test:models` checks that each bundled model resolves on its endpoint.

### Wire APIs and bundled-provider conventions

A provider's `api` names the **request shape**, potentially supported by multiple vendors.

| API | Shape | Bundled providers |
|---|---|---|
| `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities | `openrouter-jev`, `openrouter-openai-decisions` |
| `typesafe-systemone` | Same decisions shape (currently) | `typesafe-jev` |
| `openai-decisions` | OpenAI's Decisions API: same concept, its own request/response shape | `openai-decisions` |
| `anthropic-messages` | Chat: one request per rule, structured JSON verdict | `anthropic`, `anthropic-haiku-5-5`, `anthropic-env` |
| `openai-completions` | Chat: OpenAI-compatible; works with Ollama, vLLM and OpenRouter | `openai`, `openai-env`, `openrouter-openai` |

Bundled provider names follow the pattern:

- `<vendor>` — this project's recommended model for that vendor; can change between releases.
- `<vendor>-<model>` (e.g. `anthropic-haiku-5-5`) — pins a model id, so it does not drift when the `<vendor>` bundle's recommendation moves.
- `<vendor>-env` — hydrates model, endpoint etc from the vendor's usual environment variables.
- `openrouter-<vendor>-<purpose>` (e.g. `openrouter-openai-decisions`) — aggregating vendor's endpoint variant

**Configured providers** live in `modelProviders` at the top level, and replace a bundled provider of the same name. Fields a configured provider leaves out still take the API defaults:

```json
{
"modelProviders": {
"mine": { "api": "openrouter-decisions", "apiKeyEnv": "MY_JEV_KEY" },
"local": { "api": "openai-completions", "endpoint": "http://localhost:11434/v1" }
}
}
```

| Field | Description |
|---|---|
| `api` | **Required.** The wire API (see the table above and [Choosing a judge](semantic-rules.md#choosing-a-judge)), or `fake` (for tests). |
| `model` | A fixed model. Its default depends on the API. |
| `modelEnv` | Environment variable holding a model override. An explicit `model` wins; a variable set but empty is an error. |
| `endpoint` | A fixed API URL. |
| `endpointEnv` | Environment variable holding an endpoint override. Same precedence as `modelEnv`. |
| `apiKeyEnv` | The environment variable that holds the key. Defaults to the API's usual variable (see the table above). A local `openai-completions` endpoint can need no key. |
| `apiKey` | The key itself. Use `apiKeyEnv` instead, so that keys stay out of config files. |

Resolution per field: the explicit value wins, then the named environment variable, then the API default. Provider implementations read no environment; all hydration happens before one runs.

The environment holds only keys and the variables a `-env` bundle names. No environment variable selects a provider.

The top-level `judge` names the model provider that judges semantic rules: a bundled provider or an entry in `modelProviders`. A space can set its own `judge`, which replaces the top-level one for that space, for example to keep a private space on a local model:

```json
{
"judge": { "provider": "openrouter-jev" },
"spaces": [
{ "name": "private-notes", "path": "/path/to/notes", "judge": { "provider": "local" } }
Comment thread
mindsocket marked this conversation as resolved.
Resolved
]
}
```

Config does not turn semantic rules on. Each run asks for them with `--semantic`, or with `--judge <provider>`, which also selects a different provider for that run.
Comment thread
mindsocket marked this conversation as resolved.
Resolved

## Markdown plugin config

Set under `plugins.markdown` per space.
Expand Down
Loading
Loading