Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ There are several places that need reviewing and updating with any new feature o
- `src/schema/metadata-contract.ts` — Single source of truth for the `$metadata` contract
- `schemas/generated/latest/_structured_context_schema_meta.json` — Generated metaschema for the newest dialect (generated on build or with `bun run generate:schema-meta`). `schemas/generated/<version>/` folders are frozen copies, created by the `version` script (`generate:schema-meta --release`) only when a release changes the dialect; never edit them. `src/schema/meta-schemas.ts` loads them at runtime.
- `src/schema/gating.ts` — Cascade gating: computes hierarchy depth and evaluates rules with subtree gating (schema/hierarchy/blocking-rule failures suppress downstream rule checks). See [docs/rules.md](docs/rules.md#cascade-gating).
- `src/judge/` — Semantic rule judges: provider interface and types (`types.ts`), verdict resolution (`verdicts.ts`), per-node dispatch (`judge-node.ts`), and providers (`provider/anthropic.ts`, `provider/openai.ts`, `provider/jev.ts`, `provider/fake.ts`), provider hydration and the bundled catalogue (`factory.ts`), and `sctx providers` (`src/commands/providers.ts`). See [docs/semantic-rules.md](docs/semantic-rules.md).
- `src/judge/` — Semantic rule judges: provider interface and types (`types.ts`), verdict resolution (`verdicts.ts`), per-node and grouped dispatch (`judge-node.ts`), and providers (`provider/anthropic.ts`, `provider/openai.ts`, `provider/jev.ts`, `provider/fake.ts`), provider hydration and the bundled catalogue (`factory.ts`), and `sctx providers` (`src/commands/providers.ts`). See [docs/semantic-rules.md](docs/semantic-rules.md).
- `src/schema/rule-filter.ts` — `--rule-filter` parsing and rule filtering by mode, category, severity, type or id.

## Testing
Expand Down
6 changes: 6 additions & 0 deletions docs/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,12 @@ A **semantic rule** (`mode: semantic`) asks a model to judge a node, for example

A **judge** is the model that judges semantic rules: a decision model (Jev) or a chat model (Claude, OpenAI-compatible). For each rule, the judge gives one **verdict** from the rule's `verdicts`, for example `customer-need`, `vague` or `solution`. The rule's `severity` says which verdicts report a violation. Semantic rules cost money, so they are judged only when a run asks for it (`--semantic` or `--judge`); a rule's `check` always runs. The judge is a configured **model provider**: a bundled provider or an entry in `modelProviders`. See [docs/semantic-rules.md](semantic-rules.md).

A **node ruleset group** is the set of nodes that share the same semantic rules. It forms within one evaluation step of [cascade gating](#cascade-gating): one hierarchy depth, or the set of non-hierarchy nodes. Nodes are keyed on the rules left after `when` and `check` filtering. A judge on the `openrouter-decisions` or `typesafe-systemone` API judges one group in as few decision requests as it can.

A **decision request** is one call to a decisions API. It carries the state of one or more nodes and a list of decision questions.

A **decision question** is one choice question about one node under one semantic rule, inside a decision request.

### Cascade gating

**Cascade gating** is the mechanism by which a schema failure, hierarchy structure violation, or blocking rule violation on a node suppresses rule evaluation for that node's `hierarchy` subtree. It exists so downstream checks — especially costly semantic (model-judged) rules — aren't run against inputs already known to be invalid.
Expand Down
6 changes: 3 additions & 3 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,9 @@ A provider's `api` names the **request shape**, potentially supported by multipl

| API | Shape | Bundled providers |
|---|---|---|
| `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities | `openrouter-jev`, `openrouter-openai-decisions` |
| `typesafe-systemone` | Same decisions shape (currently) | `typesafe-jev` |
| `openai-decisions` | OpenAI's Decisions API: same concept, its own request/response shape | `openai-decisions` |
| `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities; nodes with the same rules share a decision request | `openrouter-jev`, `openrouter-openai-decisions` |

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not relevant to config. Don't add trivia to docs.

| `typesafe-systemone` | Same decisions shape (currently), including shared decision requests | `typesafe-jev` |
| `openai-decisions` | OpenAI's Decisions API: same concept, its own request/response shape; one node per request | `openai-decisions` |
| `anthropic-messages` | Chat: one request per rule, structured JSON verdict | `anthropic`, `anthropic-haiku-5-5`, `anthropic-env` |
| `openai-completions` | Chat: OpenAI-compatible; works with Ollama, vLLM and OpenRouter | `openai`, `openai-env`, `openrouter-openai` |

Expand Down
5 changes: 3 additions & 2 deletions docs/semantic-rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,9 +108,10 @@ Gating uses the severity of the verdict. A verdict with `error` severity gates t

## How judging works

- **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge in one call.
- **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge.
- **Grouping.** A judge on the `openrouter-decisions` or `typesafe-systemone` API judges each node ruleset group (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. This includes the bundled `openrouter-openai-decisions` provider. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request on tokens is sent whole as its own decision request. A node with more than 40 rules has its rules split across decision requests. If a node's questions are split across several requests and one fails, all of that node's rules are reported as errors. Other providers, including `openai-decisions` and chat models, judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Grouping can shift verdicts and probabilities slightly compared with judging each node alone.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is way too much detail for user docs. Don't hardcode and repeat so much implementation detail and focus on "need to know" stuff that people using the project might care about. Fine to mention the high level approach in case someone's curious, but they can dig into the code themselves beyond that.

- **Gating.** structured-context does not judge a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated.
- **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate.
- **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. If one decision request fails, only the rules on the nodes in that decision request report the error.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, irrelevant. The output will tell people this if it happens.

- **Input.** The judge gets the node's title, type, parent titles and schema data, including frontmatter.
- **Output.** A violation from a semantic rule also includes the verdict, the judge's evidence and any guidance.

Expand Down
91 changes: 67 additions & 24 deletions src/judge/judge-node.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,31 +16,23 @@ export type NodeJudgement = {
error?: string;
};

/** Judge every given semantic rule on a node in one provider call. */
export async function judgeNode(
judge: SemanticJudgeProvider,
node: SpaceNode,
rules: Rule[],
): Promise<NodeJudgement[]> {
if (rules.length === 0) return [];
const resolved = rules.map(resolveSemanticRule);
const specs = resolved.map((r) => r.spec);
let results: Awaited<ReturnType<SemanticJudgeProvider['evaluate']>>;
try {
results = await judge.evaluate({
node,
rules: specs,
});
} catch (err) {
const message = err instanceof Error ? err.message : String(err);
return rules.map((rule) => ({
rule,
failed: true,
evidence: `Error during evaluation: ${message}`,
error: message,
}));
}
type JudgeResults = Awaited<ReturnType<SemanticJudgeProvider['evaluate']>>;
type ResolvedRules = ReturnType<typeof resolveAll>;

function resolveAll(rules: Rule[]) {
return rules.map(resolveSemanticRule);
}

function errorJudgements(rules: Rule[], message: string): NodeJudgement[] {
return rules.map((rule) => ({
rule,
failed: true,
evidence: `Error during evaluation: ${message}`,
error: message,
}));
}

function interpret(rules: Rule[], resolved: ResolvedRules, results: JudgeResults): NodeJudgement[] {
return rules.map((rule, i) => {
const result = results[rule.id];
if (!result || isJudgeError(result)) {
Expand All @@ -63,3 +55,54 @@ export async function judgeNode(
};
});
}

const errorMessage = (err: unknown) => (err instanceof Error ? err.message : String(err));

/** Judge every given semantic rule on a node in one provider call. */
export async function judgeNode(
judge: SemanticJudgeProvider,
node: SpaceNode,
rules: Rule[],
): Promise<NodeJudgement[]> {
if (rules.length === 0) return [];
const resolved = resolveAll(rules);
let results: JudgeResults;
try {
results = await judge.evaluate({ node, rules: resolved.map((r) => r.spec) });
} catch (err) {
return errorJudgements(rules, errorMessage(err));
}
return interpret(rules, resolved, results);
}

/**
* Judge several nodes at once. A judge with `evaluateNodes` gets every node in one call and may fold them into
* fewer requests; any other judge is called once per node. Results align with `items`. A failure marks only the
* affected node's rules as errors.
*/
export async function judgeNodes(
judge: SemanticJudgeProvider,
items: Array<{ node: SpaceNode; rules: Rule[] }>,
): Promise<NodeJudgement[][]> {
if (!judge.evaluateNodes) {
const out: NodeJudgement[][] = [];
for (const item of items) out.push(await judgeNode(judge, item.node, item.rules));
return out;
}
const active = items
.map((item, index) => ({ ...item, index, resolved: resolveAll(item.rules) }))
.filter((a) => a.rules.length > 0);
const out: NodeJudgement[][] = items.map(() => []);
if (active.length === 0) return out;
let perNode: JudgeResults[];
try {
perNode = await judge.evaluateNodes(active.map((a) => ({ node: a.node, rules: a.resolved.map((r) => r.spec) })));
} catch (err) {
for (const a of active) out[a.index] = errorJudgements(a.rules, errorMessage(err));
return out;
}
active.forEach((a, k) => {
out[a.index] = interpret(a.rules, a.resolved, perNode[k] ?? {});
});
return out;
}
179 changes: 144 additions & 35 deletions src/judge/provider/jev.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
import type { HydratedProvider, SemanticJudgeInput, SemanticJudgeProvider, SemanticJudgeResult } from '../types';
import type {
HydratedProvider,
SemanticJudgeInput,
SemanticJudgeProvider,
SemanticJudgeResult,
SemanticRuleSpec,
} from '../types';

export interface JevJudgeProviderOptions extends HydratedProvider {
apiKey?: string;
Expand All @@ -8,6 +14,12 @@ export const JEV_LATEST = 'jev-latest';

export const OPENROUTER_DECISIONS_ENDPOINT = 'https://openrouter.ai/api/alpha/decisions';

/** Estimated input-token cap (characters / 4) for one DecisionRequest's `state` plus `questions`: a conservative cost and latency guard. */
export const JEV_DECISION_REQUEST_MAX_ESTIMATED_TOKENS = 32_000;

/** Cap on the number of DecisionQuestions in one DecisionRequest. */
export const JEV_DECISION_REQUEST_MAX_QUESTIONS = 40;

type JevChoiceAnswer = {
type?: string;
choice?: string;
Expand All @@ -19,7 +31,7 @@ type JevChoiceAnswer = {
* Judge provider for TypeSafe's Jev decision model via OpenRouter's Decisions API.
*
* Each node is sent once as `state`, with one `choice` question per semantic rule whose criteria are
* the rule's closed verdict set. Jev returns the chosen verdict with confidence and probabilities,
* the rule's closed verdict set. `evaluateNodes` shares one DecisionRequest between nodes with the same rule set. Jev returns the chosen verdict with confidence and probabilities,
* but no prose, so `evidence` is a summary of the distribution.
*/
export class JevJudgeProvider implements SemanticJudgeProvider {
Expand All @@ -36,52 +48,149 @@ export class JevJudgeProvider implements SemanticJudgeProvider {

async evaluate(input: SemanticJudgeInput): Promise<Record<string, SemanticJudgeResult>> {
const { node, rules } = input;
const questions: Record<string, unknown> = {};
for (const rule of rules) {
questions[rule.id] = {
type: 'choice',
instructions: rule.rubric || rule.description,
criteria: rule.verdicts,
};
const results: Record<string, SemanticJudgeResult> = {};
// A node with more rules than one DecisionRequest allows is split by rule; the state is sent whole each time.
for (let start = 0; start < rules.length; start += JEV_DECISION_REQUEST_MAX_QUESTIONS) {
const part = rules.slice(start, start + JEV_DECISION_REQUEST_MAX_QUESTIONS);
const questions: Record<string, unknown> = {};
for (const rule of part) questions[rule.id] = decisionQuestion(rule, rule.rubric || rule.description);
const data = await this.post(nodeState(node), questions);
for (const rule of part) results[rule.id] = toResult(rule, data.answers?.[rule.id]);
}
return results;
}

/**
* Judge many nodes with fewer DecisionRequests. Nodes that share the same semantic rule set form a
* NodeRulesetGroup, sent as one DecisionRequest,
* with every node under `state.nodes` and one DecisionQuestion per node and rule. A NodeRulesetGroup is split into DecisionRequests by
* `JEV_DECISION_REQUEST_MAX_ESTIMATED_TOKENS` and `JEV_DECISION_REQUEST_MAX_QUESTIONS`, never within a node. A node alone in its DecisionRequest, even
* one over the token cap, is sent whole as a plain single-node DecisionRequest; one with more rules than the question cap has its rules split
* across DecisionRequests. A failed DecisionRequest marks only its own nodes' rules
* as errors.
*/
async evaluateNodes(inputs: SemanticJudgeInput[]): Promise<Array<Record<string, SemanticJudgeResult>>> {
const results: Array<Record<string, SemanticJudgeResult>> = inputs.map(() => ({}));
const nodeRulesetGroups = new Map<string, number[]>();
inputs.forEach((input, i) => {
if (input.rules.length === 0) return;
const key = JSON.stringify(input.rules);
nodeRulesetGroups.set(key, [...(nodeRulesetGroups.get(key) ?? []), i]);
});

for (const indices of nodeRulesetGroups.values()) {
for (const requestInputs of planDecisionRequests(indices, inputs)) {
try {
if (requestInputs.length === 1) {
results[requestInputs[0]!] = await this.evaluate(inputs[requestInputs[0]!]!);
} else {
await this.evaluateDecisionRequest(requestInputs, inputs, results);
}
} catch (err) {
const error = err instanceof Error ? err.message : String(err);
for (const i of requestInputs) {
results[i] = Object.fromEntries(inputs[i]!.rules.map((r) => [r.id, { error }]));
}
}
}
}
return results;
}

private async evaluateDecisionRequest(
requestInputs: number[],
inputs: SemanticJudgeInput[],
results: Array<Record<string, SemanticJudgeResult>>,
): Promise<void> {
const nodes: Record<string, unknown> = {};
const questions: Record<string, unknown> = {};
requestInputs.forEach((inputIndex, k) => {
const input = inputs[inputIndex]!;
nodes[`n${k}`] = nodeState(input.node);
for (const rule of input.rules) {
questions[`${rule.id}__n${k}`] = decisionQuestion(
rule,
`For node \`nodes.n${k}\`: ${rule.rubric || rule.description}`,
);
}
});
const data = await this.post({ nodes }, questions);
requestInputs.forEach((inputIndex, k) => {
for (const rule of inputs[inputIndex]!.rules) {
results[inputIndex]![rule.id] = toResult(rule, data.answers?.[`${rule.id}__n${k}`]);
}
});
}

private async post(state: unknown, questions: Record<string, unknown>): Promise<JevResponse> {
const res = await fetch(this.endpoint, {
method: 'POST',
headers: { 'content-type': 'application/json', authorization: `Bearer ${this.apiKey}` },
body: JSON.stringify({
model: this.model,
state: {
title: node.title,
type: node.resolvedType,
parents: node.resolvedParents?.map((p) => p.title) ?? [],
...node.schemaData,
},
questions,
}),
body: JSON.stringify({ model: this.model, state, questions }),
});

if (!res.ok) {
const errorText = await res.text().catch(() => '');
throw new Error(`Jev API error (${res.status}): ${errorText}`);
}
return (await res.json()) as JevResponse;
}
}

const data = (await res.json()) as { answers?: Record<string, JevChoiceAnswer> };
const results: Record<string, SemanticJudgeResult> = {};
for (const rule of rules) {
const answer = data.answers?.[rule.id];
if (!answer?.choice || !(answer.choice in rule.verdicts)) {
results[rule.id] = { error: `Jev returned no valid verdict for rule "${rule.id}" (choice: ${answer?.choice})` };
continue;
}
results[rule.id] = {
verdict: answer.choice,
confidence: answer.confidence,
probabilities: answer.probabilities,
evidence: describeDistribution(answer),
};
type JevResponse = { answers?: Record<string, JevChoiceAnswer> };

function nodeState(node: SemanticJudgeInput['node']) {
return {
title: node.title,
type: node.resolvedType,
parents: node.resolvedParents?.map((p) => p.title) ?? [],
...node.schemaData,
};
}

function decisionQuestion(rule: SemanticRuleSpec, instructions: string) {
return { type: 'choice', instructions, criteria: rule.verdicts };
}

function toResult(rule: SemanticRuleSpec, answer: JevChoiceAnswer | undefined): SemanticJudgeResult {
if (!answer?.choice || !(answer.choice in rule.verdicts)) {
return { error: `Jev returned no valid verdict for rule "${rule.id}" (choice: ${answer?.choice})` };
}
return {
verdict: answer.choice,
confidence: answer.confidence,
probabilities: answer.probabilities,
evidence: describeDistribution(answer),
};
}

const estimateTokens = (value: unknown) => Math.ceil(JSON.stringify(value).length / 4);

/** Plan the DecisionRequests for one NodeRulesetGroup (inputs sharing a rule set): runs of nodes, in order, never splitting a node. */
function planDecisionRequests(indices: number[], inputs: SemanticJudgeInput[]): number[][] {
const requests: number[][] = [];
let current: number[] = [];
let tokens = 0;
let questions = 0;
for (const i of indices) {
const input = inputs[i]!;
const cost = estimateTokens(nodeState(input.node)) + estimateTokens(input.rules);
const count = input.rules.length;
if (
current.length > 0 &&
(tokens + cost > JEV_DECISION_REQUEST_MAX_ESTIMATED_TOKENS ||
questions + count > JEV_DECISION_REQUEST_MAX_QUESTIONS)
) {
requests.push(current);
current = [];
tokens = 0;
questions = 0;
}
return results;
current.push(i);
tokens += cost;
questions += count;
}
if (current.length > 0) requests.push(current);
return requests;
}

function describeDistribution(answer: JevChoiceAnswer): string {
Expand Down
Loading
Loading