From 3fef8486483638b6d7568f1a4e4d2f1c10e01830 Mon Sep 17 00:00:00 2001 From: Roger Barnes Date: Fri, 16 Oct 2026 09:05:16 +1100 Subject: [PATCH 1/9] feat(judge): batch Jev semantic questions across nodes that share a rule set Nodes with the same semantic rule set now share one Jev request: all nodes go under state.nodes, with one question per node and rule. A group is split by an estimated-token cap and a question cap, never within a node; a lone or oversized node is sent whole as a plain single-node request. A failed request marks only its own nodes' rules as errors. Chat providers are unchanged. Gating now judges one hierarchy depth at a time so a batch never needs a verdict that is still pending; violation order per node is unchanged. measured: probe against openrouter ~typesafe/jev-latest (jev-1.14-20261014) - one request with 3 nodes in state.nodes and 3 questions returned 3 answers keyed by question id (HTTP 200). - drift, same question alone vs in the 3-node batch: mean |dp| 0.047, max 0.12 (n1 0.98 -> 1.00, n2 unchanged, n3 0.97 -> 0.85 on fail; no verdict flipped). 3 nodes, 1 rule: indicative only. - size: 45k-token state and a 250k-token state were both accepted (HTTP 200); no 32k limit was reached, so JEV_BATCH_MAX_TOKENS (32k, from issue #152, unverified) is a conservative guard, not a hard limit. --- AGENTS.md | 2 +- docs/config.md | 2 +- docs/semantic-rules.md | 5 +- src/judge/judge-node.ts | 91 +++++++++++----- src/judge/provider/jev.ts | 172 +++++++++++++++++++++++++------ src/judge/types.ts | 6 ++ src/schema/gating.ts | 128 ++++++++++++++--------- src/schema/validate-rules.ts | 24 +++-- tests/judge/judge-node.test.ts | 69 ++++++++++++- tests/judge/provider/jev.test.ts | 99 +++++++++++++++++- tests/schema/gating.test.ts | 40 +++++++ 11 files changed, 520 insertions(+), 118 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 7a7d7635..36016498 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -59,7 +59,7 @@ There are several places that need reviewing and updating with any new feature o - `src/schema/metadata-contract.ts` — Single source of truth for the `$metadata` contract - `schemas/generated/latest/_structured_context_schema_meta.json` — Generated metaschema for the newest dialect (generated on build or with `bun run generate:schema-meta`). `schemas/generated//` folders are frozen copies, created by the `version` script (`generate:schema-meta --release`) only when a release changes the dialect; never edit them. `src/schema/meta-schemas.ts` loads them at runtime. - `src/schema/gating.ts` — Cascade gating: computes hierarchy depth and evaluates rules with subtree gating (schema/hierarchy/blocking-rule failures suppress downstream rule checks). See [docs/rules.md](docs/rules.md#cascade-gating). -- `src/judge/` — Semantic rule judges: provider interface and types (`types.ts`), verdict resolution (`verdicts.ts`), per-node dispatch (`judge-node.ts`), and providers (`provider/anthropic.ts`, `provider/openai.ts`, `provider/jev.ts`, `provider/fake.ts`), provider hydration and the bundled catalogue (`factory.ts`), and `sctx providers` (`src/commands/providers.ts`). See [docs/semantic-rules.md](docs/semantic-rules.md). +- `src/judge/` — Semantic rule judges: provider interface and types (`types.ts`), verdict resolution (`verdicts.ts`), per-node and batched dispatch (`judge-node.ts`), and providers (`provider/anthropic.ts`, `provider/openai.ts`, `provider/jev.ts`, `provider/fake.ts`), provider hydration and the bundled catalogue (`factory.ts`), and `sctx providers` (`src/commands/providers.ts`). See [docs/semantic-rules.md](docs/semantic-rules.md). - `src/schema/rule-filter.ts` — `--rule-filter` parsing and rule filtering by mode, category, severity, type or id. ## Testing diff --git a/docs/config.md b/docs/config.md index 3245a3f3..6214076d 100644 --- a/docs/config.md +++ b/docs/config.md @@ -79,7 +79,7 @@ A provider's `api` names the **request shape**, potentially supported by multipl | API | Shape | Bundled providers | |---|---|---| -| `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities | `openrouter-jev`, `openrouter-openai-decisions` | +| `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities; nodes with the same rules share a request | `openrouter-jev`, `openrouter-openai-decisions` | | `typesafe-systemone` | Same decisions shape (currently) | `typesafe-jev` | | `openai-decisions` | OpenAI's Decisions API: same concept, its own request/response shape | `openai-decisions` | | `anthropic-messages` | Chat: one request per rule, structured JSON verdict | `anthropic`, `anthropic-haiku-5-5`, `anthropic-env` | diff --git a/docs/semantic-rules.md b/docs/semantic-rules.md index fbe3c8e6..fc856f3f 100644 --- a/docs/semantic-rules.md +++ b/docs/semantic-rules.md @@ -108,9 +108,10 @@ Gating uses the severity of the verdict. A verdict with `error` severity gates t ## How judging works -- **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge in one call. +- **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge. +- **Batching.** A decision model judges nodes that share the same set of semantic rules in one request, with one question per node and rule. A request is limited to about 32,000 estimated input tokens and 40 questions, so a larger group is split into several requests, never within a node. A node that fills a request alone is sent whole as its own request. Chat models judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Batching can shift probabilities slightly compared with judging each node alone. - **Gating.** structured-context does not judge a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated. -- **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. +- **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. If one batched request fails, only the rules on the nodes in that request report the error. - **Input.** The judge gets the node's title, type, parent titles and schema data, including frontmatter. - **Output.** A violation from a semantic rule also includes the verdict, the judge's evidence and any guidance. diff --git a/src/judge/judge-node.ts b/src/judge/judge-node.ts index 7ab7edb2..146e2210 100644 --- a/src/judge/judge-node.ts +++ b/src/judge/judge-node.ts @@ -16,31 +16,23 @@ export type NodeJudgement = { error?: string; }; -/** Judge every given semantic rule on a node in one provider call. */ -export async function judgeNode( - judge: SemanticJudgeProvider, - node: SpaceNode, - rules: Rule[], -): Promise { - if (rules.length === 0) return []; - const resolved = rules.map(resolveSemanticRule); - const specs = resolved.map((r) => r.spec); - let results: Awaited>; - try { - results = await judge.evaluate({ - node, - rules: specs, - }); - } catch (err) { - const message = err instanceof Error ? err.message : String(err); - return rules.map((rule) => ({ - rule, - failed: true, - evidence: `Error during evaluation: ${message}`, - error: message, - })); - } +type JudgeResults = Awaited>; +type ResolvedRules = ReturnType; +function resolveAll(rules: Rule[]) { + return rules.map(resolveSemanticRule); +} + +function errorJudgements(rules: Rule[], message: string): NodeJudgement[] { + return rules.map((rule) => ({ + rule, + failed: true, + evidence: `Error during evaluation: ${message}`, + error: message, + })); +} + +function interpret(rules: Rule[], resolved: ResolvedRules, results: JudgeResults): NodeJudgement[] { return rules.map((rule, i) => { const result = results[rule.id]; if (!result || isJudgeError(result)) { @@ -63,3 +55,54 @@ export async function judgeNode( }; }); } + +const errorMessage = (err: unknown) => (err instanceof Error ? err.message : String(err)); + +/** Judge every given semantic rule on a node in one provider call. */ +export async function judgeNode( + judge: SemanticJudgeProvider, + node: SpaceNode, + rules: Rule[], +): Promise { + if (rules.length === 0) return []; + const resolved = resolveAll(rules); + let results: JudgeResults; + try { + results = await judge.evaluate({ node, rules: resolved.map((r) => r.spec) }); + } catch (err) { + return errorJudgements(rules, errorMessage(err)); + } + return interpret(rules, resolved, results); +} + +/** + * Judge several nodes at once. A judge with `evaluateBatch` gets every node in one call and may fold them into + * fewer requests; any other judge is called once per node. Results align with `items`. A failure marks only the + * affected node's rules as errors. + */ +export async function judgeNodes( + judge: SemanticJudgeProvider, + items: Array<{ node: SpaceNode; rules: Rule[] }>, +): Promise { + if (!judge.evaluateBatch) { + const out: NodeJudgement[][] = []; + for (const item of items) out.push(await judgeNode(judge, item.node, item.rules)); + return out; + } + const active = items + .map((item, index) => ({ ...item, index, resolved: resolveAll(item.rules) })) + .filter((a) => a.rules.length > 0); + const out: NodeJudgement[][] = items.map(() => []); + if (active.length === 0) return out; + let batch: JudgeResults[]; + try { + batch = await judge.evaluateBatch(active.map((a) => ({ node: a.node, rules: a.resolved.map((r) => r.spec) }))); + } catch (err) { + for (const a of active) out[a.index] = errorJudgements(a.rules, errorMessage(err)); + return out; + } + active.forEach((a, k) => { + out[a.index] = interpret(a.rules, a.resolved, batch[k] ?? {}); + }); + return out; +} diff --git a/src/judge/provider/jev.ts b/src/judge/provider/jev.ts index 6ab9b105..cb963605 100644 --- a/src/judge/provider/jev.ts +++ b/src/judge/provider/jev.ts @@ -1,4 +1,10 @@ -import type { HydratedProvider, SemanticJudgeInput, SemanticJudgeProvider, SemanticJudgeResult } from '../types'; +import type { + HydratedProvider, + SemanticJudgeInput, + SemanticJudgeProvider, + SemanticJudgeResult, + SemanticRuleSpec, +} from '../types'; export interface JevJudgeProviderOptions extends HydratedProvider { apiKey?: string; @@ -8,6 +14,17 @@ export const JEV_LATEST = 'jev-latest'; export const OPENROUTER_DECISIONS_ENDPOINT = 'https://openrouter.ai/api/alpha/decisions'; +/** + * Estimated input-token cap (characters / 4) for one batched request. Conservative: issue #152 cites a + * "32k token limit for state plus questions" for Jev, which is unverified. measured: a live probe accepted a + * 250k-token state through OpenRouter with HTTP 200, so the real limit is higher and this cap is a cost and + * latency guard, not a hard limit. + */ +export const JEV_BATCH_MAX_TOKENS = 32_000; + +/** Cap on questions per batched request. Sourced from the jev-rank reference client (20 pairs x 2 = 40 per request). */ +export const JEV_BATCH_MAX_QUESTIONS = 40; + type JevChoiceAnswer = { type?: string; choice?: string; @@ -19,7 +36,7 @@ type JevChoiceAnswer = { * Judge provider for TypeSafe's Jev decision model via OpenRouter's Decisions API. * * Each node is sent once as `state`, with one `choice` question per semantic rule whose criteria are - * the rule's closed verdict set. Jev returns the chosen verdict with confidence and probabilities, + * the rule's closed verdict set. `evaluateBatch` shares one request between nodes with the same rule set. Jev returns the chosen verdict with confidence and probabilities, * but no prose, so `evidence` is a summary of the distribution. */ export class JevJudgeProvider implements SemanticJudgeProvider { @@ -37,51 +54,138 @@ export class JevJudgeProvider implements SemanticJudgeProvider { async evaluate(input: SemanticJudgeInput): Promise> { const { node, rules } = input; const questions: Record = {}; - for (const rule of rules) { - questions[rule.id] = { - type: 'choice', - instructions: rule.rubric || rule.description, - criteria: rule.verdicts, - }; + for (const rule of rules) questions[rule.id] = choiceQuestion(rule, rule.rubric || rule.description); + const data = await this.post(nodeState(node), questions); + const results: Record = {}; + for (const rule of rules) results[rule.id] = toResult(rule, data.answers?.[rule.id]); + return results; + } + + /** + * Judge many nodes with fewer requests. Nodes that share the same semantic rule set go into one request, + * with every node under `state.nodes` and one question per node and rule. A group is split into chunks by + * `JEV_BATCH_MAX_TOKENS` and `JEV_BATCH_MAX_QUESTIONS`, never within a node. A node alone in its chunk, even + * one over the cap, is sent as a plain single-node request. A failed request marks only its own nodes' rules + * as errors. + */ + async evaluateBatch(inputs: SemanticJudgeInput[]): Promise>> { + const results: Array> = inputs.map(() => ({})); + const groups = new Map(); + inputs.forEach((input, i) => { + if (input.rules.length === 0) return; + const key = JSON.stringify(input.rules); + groups.set(key, [...(groups.get(key) ?? []), i]); + }); + + for (const indices of groups.values()) { + for (const chunk of chunkGroup(indices, inputs)) { + try { + if (chunk.length === 1) { + results[chunk[0]!] = await this.evaluate(inputs[chunk[0]!]!); + } else { + await this.evaluateChunk(chunk, inputs, results); + } + } catch (err) { + const error = err instanceof Error ? err.message : String(err); + for (const i of chunk) { + results[i] = Object.fromEntries(inputs[i]!.rules.map((r) => [r.id, { error }])); + } + } + } } + return results; + } + private async evaluateChunk( + chunk: number[], + inputs: SemanticJudgeInput[], + results: Array>, + ): Promise { + const nodes: Record = {}; + const questions: Record = {}; + chunk.forEach((inputIndex, k) => { + const input = inputs[inputIndex]!; + nodes[`n${k}`] = nodeState(input.node); + for (const rule of input.rules) { + questions[`${rule.id}__n${k}`] = choiceQuestion( + rule, + `For state.nodes.n${k}: ${rule.rubric || rule.description}`, + ); + } + }); + const data = await this.post({ nodes }, questions); + chunk.forEach((inputIndex, k) => { + for (const rule of inputs[inputIndex]!.rules) { + results[inputIndex]![rule.id] = toResult(rule, data.answers?.[`${rule.id}__n${k}`]); + } + }); + } + + private async post(state: unknown, questions: Record): Promise { const res = await fetch(this.endpoint, { method: 'POST', headers: { 'content-type': 'application/json', authorization: `Bearer ${this.apiKey}` }, - body: JSON.stringify({ - model: this.model, - state: { - title: node.title, - type: node.resolvedType, - parents: node.resolvedParents?.map((p) => p.title) ?? [], - ...node.schemaData, - }, - questions, - }), + body: JSON.stringify({ model: this.model, state, questions }), }); - if (!res.ok) { const errorText = await res.text().catch(() => ''); throw new Error(`Jev API error (${res.status}): ${errorText}`); } + return (await res.json()) as JevResponse; + } +} - const data = (await res.json()) as { answers?: Record }; - const results: Record = {}; - for (const rule of rules) { - const answer = data.answers?.[rule.id]; - if (!answer?.choice || !(answer.choice in rule.verdicts)) { - results[rule.id] = { error: `Jev returned no valid verdict for rule "${rule.id}" (choice: ${answer?.choice})` }; - continue; - } - results[rule.id] = { - verdict: answer.choice, - confidence: answer.confidence, - probabilities: answer.probabilities, - evidence: describeDistribution(answer), - }; +type JevResponse = { answers?: Record }; + +function nodeState(node: SemanticJudgeInput['node']) { + return { + title: node.title, + type: node.resolvedType, + parents: node.resolvedParents?.map((p) => p.title) ?? [], + ...node.schemaData, + }; +} + +function choiceQuestion(rule: SemanticRuleSpec, instructions: string) { + return { type: 'choice', instructions, criteria: rule.verdicts }; +} + +function toResult(rule: SemanticRuleSpec, answer: JevChoiceAnswer | undefined): SemanticJudgeResult { + if (!answer?.choice || !(answer.choice in rule.verdicts)) { + return { error: `Jev returned no valid verdict for rule "${rule.id}" (choice: ${answer?.choice})` }; + } + return { + verdict: answer.choice, + confidence: answer.confidence, + probabilities: answer.probabilities, + evidence: describeDistribution(answer), + }; +} + +const estimateTokens = (value: unknown) => Math.ceil(JSON.stringify(value).length / 4); + +/** Split same-rule-set inputs into request-sized chunks, in order, never splitting a node. */ +function chunkGroup(indices: number[], inputs: SemanticJudgeInput[]): number[][] { + const chunks: number[][] = []; + let current: number[] = []; + let tokens = 0; + let questions = 0; + for (const i of indices) { + const input = inputs[i]!; + const cost = estimateTokens(nodeState(input.node)) + estimateTokens(input.rules); + const count = input.rules.length; + if (current.length > 0 && (tokens + cost > JEV_BATCH_MAX_TOKENS || questions + count > JEV_BATCH_MAX_QUESTIONS)) { + chunks.push(current); + current = []; + tokens = 0; + questions = 0; } - return results; + current.push(i); + tokens += cost; + questions += count; } + if (current.length > 0) chunks.push(current); + return chunks; } function describeDistribution(answer: JevChoiceAnswer): string { diff --git a/src/judge/types.ts b/src/judge/types.ts index 79417d1e..fd45885b 100644 --- a/src/judge/types.ts +++ b/src/judge/types.ts @@ -34,6 +34,12 @@ export type SemanticJudgeResult = SemanticJudgeVerdict | { error: string }; export interface SemanticJudgeProvider { readonly name: string; evaluate(input: SemanticJudgeInput): Promise>; + /** + * Optional: judge many nodes in one call, so a provider can fold them into fewer requests. Returns one + * result map per input, in order. A failure that affects only some inputs is reported as `{ error }` + * results for those inputs' rules; throw only when nothing could be judged. + */ + evaluateBatch?(inputs: SemanticJudgeInput[]): Promise>>; } /** Wire APIs a provider can speak. Names the request shape, never the vendor or model. */ diff --git a/src/schema/gating.ts b/src/schema/gating.ts index b09b4e84..650ea35c 100644 --- a/src/schema/gating.ts +++ b/src/schema/gating.ts @@ -1,4 +1,5 @@ -import { judgeNode } from '../judge/judge-node'; +import type { NodeJudgement } from '../judge/judge-node'; +import { judgeNodes } from '../judge/judge-node'; import type { SemanticJudgeProvider } from '../judge/types'; import type { SpaceGraph } from '../space-graph'; import type { Rule, RuleSeverity, RuleViolation, SpaceNode } from '../types'; @@ -205,10 +206,19 @@ export async function evaluateRulesWithGating( return applicable; } - async function evaluateApplicableRules(node: SpaceNode): Promise<{ blockingRuleId?: string }> { - let blockingRuleId: string | undefined; + /** A node whose deterministic rules have run and whose semantic rules await the judge. */ + type PreparedNode = { + violations: RuleViolation[]; + semantic: Rule[]; + blockingRuleId?: string; + judged(judgements: NodeJudgement[]): void; + }; + + async function prepareNode(node: SpaceNode): Promise { + const local: RuleViolation[] = []; + const prepared: PreparedNode = { violations: local, semantic: [], judged: () => {} }; const applicable = await applicableRules(node, (rule, error) => - violations.push(expressionErrorViolation(rule, node.label, error)), + local.push(expressionErrorViolation(rule, node.label, error)), ); function report( @@ -216,7 +226,7 @@ export async function evaluateRulesWithGating( detail: { severity?: RuleSeverity; verdict?: string; guidance?: string; evidence?: string; error?: string } = {}, ): void { const severity = detail.severity ?? ruleSeverity(rule); - violations.push({ + local.push({ file: node.label, ruleId: rule.id, category: rule.category, @@ -227,13 +237,15 @@ export async function evaluateRulesWithGating( ...(detail.guidance !== undefined && { guidance: detail.guidance }), }); // A judge error says nothing about the node, so it never gates. - if (!detail.error && ruleGatesDescendants(rule, severity) && !blockingRuleId) blockingRuleId = rule.id; + if (!detail.error && ruleGatesDescendants(rule, severity) && !prepared.blockingRuleId) { + prepared.blockingRuleId = rule.id; + } } - // Cheap deterministic rules first; the judge is only consulted afterwards, once per node. + // Cheap deterministic rules first; the judge is only consulted afterwards. for (const rule of applicable.filter((r) => r.mode !== 'semantic')) { const outcome = await tryEvaluateExpression(rule.check as string, buildEvalContext(node, nodes, nodeIndex)); - if ('error' in outcome) violations.push(expressionErrorViolation(rule, node.label, outcome.error)); + if ('error' in outcome) local.push(expressionErrorViolation(rule, node.label, outcome.error)); else if (outcome.value !== true) report(rule); } @@ -244,12 +256,12 @@ export async function evaluateRulesWithGating( nodes, nodeIndex, ); - violations.push(...prechecked.violations); + local.push(...prechecked.violations); const gatingRule = prechecked.failed.find((r) => ruleGatesDescendants(r)); - if (gatingRule && !blockingRuleId) blockingRuleId = gatingRule.id; - const semantic = prechecked.judgeable; - if (judge && semantic.length > 0) { - for (const j of await judgeNode(judge, node, semantic)) { + if (gatingRule && !prepared.blockingRuleId) prepared.blockingRuleId = gatingRule.id; + if (judge) prepared.semantic = prechecked.judgeable; + prepared.judged = (judgements) => { + for (const j of judgements) { if (j.failed) { report(j.rule, { severity: j.severity, @@ -260,52 +272,76 @@ export async function evaluateRulesWithGating( }); } } + }; + return prepared; + } + + /** + * Evaluate a set of nodes whose gating is already decided. Their semantic rules go to the judge together, + * so a batching judge can share requests; violations are recorded in node order. + */ + async function evaluateNodes(batch: readonly SpaceNode[]): Promise> { + const prepared: PreparedNode[] = []; + for (const node of batch) prepared.push(await prepareNode(node)); + if (judge) { + const pending = batch.map((node, i) => ({ node, rules: prepared[i]!.semantic })); + const judgements = await judgeNodes(judge, pending); + prepared.forEach((p, i) => { + p.judged(judgements[i] ?? []); + }); } - return { blockingRuleId }; + for (const p of prepared) violations.push(...p.violations); + return prepared.map((p) => ({ blockingRuleId: p.blockingRuleId })); } // Non-hierarchy nodes sit outside the subtree cascade — always evaluated, never gated. - for (const node of graph.nonHierarchy) { - await evaluateApplicableRules(node); - } + await evaluateNodes(graph.nonHierarchy); const visited = new Set(); - const queue: Array<{ node: SpaceNode; ancestorOrigin?: string }> = []; + // Hierarchy nodes are judged one depth at a time: a node's verdicts decide whether its children are evaluated. + let wave: Array<{ node: SpaceNode; ancestorOrigin?: string }> = []; for (const node of [...graph.hierarchyRoots, ...graph.orphans]) { - queue.push({ node, ancestorOrigin: undefined }); + wave.push({ node, ancestorOrigin: undefined }); } - let qi = 0; - while (qi < queue.length) { - const { node, ancestorOrigin } = queue[qi++]!; - if (visited.has(node.title)) continue; - visited.add(node.title); - - const selfSeed = seedByTitle.get(node.title); - const isGatedByAncestor = ancestorOrigin !== undefined; - const isSelfBlocked = selfSeed !== undefined; - const isGated = isGatedByAncestor || isSelfBlocked; - const shouldEvaluate = !isGated || includeGated; + while (wave.length > 0) { + const current: typeof wave = []; + for (const entry of wave) { + if (visited.has(entry.node.title)) continue; + visited.add(entry.node.title); + current.push(entry); + } + const isGated = (e: (typeof wave)[number]) => e.ancestorOrigin !== undefined || seedByTitle.has(e.node.title); + const toEvaluate = current.filter((e) => !isGated(e) || includeGated); + const evaluated = await evaluateNodes(toEvaluate.map((e) => e.node)); + const blockingByTitle = new Map(toEvaluate.map((e, i) => [e.node.title, evaluated[i]!.blockingRuleId])); - const { blockingRuleId } = shouldEvaluate ? await evaluateApplicableRules(node) : {}; + const next: typeof wave = []; + for (const { node, ancestorOrigin } of current) { + const selfSeed = seedByTitle.get(node.title); + const isGatedByAncestor = ancestorOrigin !== undefined; + const isSelfBlocked = selfSeed !== undefined; + const blockingRuleId = blockingByTitle.get(node.title); - let childOrigin: string | undefined = ancestorOrigin; - if (isGatedByAncestor) { - gateOrigins.get(ancestorOrigin!)!.gatedFiles.push(node.label); - } else if (isSelfBlocked) { - const origin = originFor(node.title, { file: selfSeed!.file, reason: selfSeed!.reason }); - origin.gatedFiles.push(node.label); - childOrigin = node.title; - } else if (blockingRuleId) { - originFor(node.title, { file: node.label, reason: 'rule', ruleId: blockingRuleId }); - childOrigin = node.title; - } else { - childOrigin = undefined; - } + let childOrigin: string | undefined = ancestorOrigin; + if (isGatedByAncestor) { + gateOrigins.get(ancestorOrigin!)!.gatedFiles.push(node.label); + } else if (isSelfBlocked) { + const origin = originFor(node.title, { file: selfSeed!.file, reason: selfSeed!.reason }); + origin.gatedFiles.push(node.label); + childOrigin = node.title; + } else if (blockingRuleId) { + originFor(node.title, { file: node.label, reason: 'rule', ruleId: blockingRuleId }); + childOrigin = node.title; + } else { + childOrigin = undefined; + } - for (const child of graph.hierarchyChildren.get(node.title) ?? []) { - queue.push({ node: child, ancestorOrigin: childOrigin }); + for (const child of graph.hierarchyChildren.get(node.title) ?? []) { + next.push({ node: child, ancestorOrigin: childOrigin }); + } } + wave = next; } return { diff --git a/src/schema/validate-rules.ts b/src/schema/validate-rules.ts index 342b7012..0ad54406 100644 --- a/src/schema/validate-rules.ts +++ b/src/schema/validate-rules.ts @@ -1,4 +1,4 @@ -import { judgeNode } from '../judge/judge-node'; +import { judgeNodes } from '../judge/judge-node'; import type { SemanticJudgeProvider } from '../judge/types'; import type { Rule, RuleViolation, SpaceNode } from '../types'; import { @@ -117,15 +117,19 @@ export async function validateRules( const judge = options.judge; const runnableSemanticRules = judge ? semanticRules : semanticRules.filter((r) => r.check); if (runnableSemanticRules.length > 0) { + const perNode: RuleViolation[][] = []; // kept in node order + const pending: Array<{ node: SpaceNode; rules: Rule[]; found: RuleViolation[] }> = []; for (const node of nodes) { if (options.judgeTarget !== undefined && node.label !== options.judgeTarget) continue; + const found: RuleViolation[] = []; + perNode.push(found); const applicable: Rule[] = []; for (const r of runnableSemanticRules) { if (r.type && r.type !== node.resolvedType) continue; if (r.when) { const when = await tryEvaluateExpression(r.when, buildEvalContext(node, nodes, nodeIndex)); if ('error' in when) { - violations.push(expressionErrorViolation(r, node.label, when.error)); + found.push(expressionErrorViolation(r, node.label, when.error)); continue; } if (when.value !== true) continue; @@ -139,12 +143,16 @@ export async function validateRules( // A semantic rule's `check` fails fast before the judge. const prechecked = await applySemanticChecks(applicable, node, nodes, nodeIndex); - violations.push(...prechecked.violations); - if (!judge) continue; + found.push(...prechecked.violations); + if (judge) pending.push({ node, rules: prechecked.judgeable, found }); + } - for (const j of await judgeNode(judge, node, prechecked.judgeable)) { + // One judge call for every node, so a batching judge can share requests. + const judged = judge ? await judgeNodes(judge, pending) : []; + pending.forEach(({ node, found }, i) => { + for (const j of judged[i] ?? []) { if (!j.failed) continue; - violations.push({ + found.push({ file: node.label, ruleId: j.rule.id, category: j.rule.category, @@ -154,9 +162,9 @@ export async function validateRules( ...(j.verdict !== undefined && { verdict: j.verdict }), ...(j.guidance !== undefined && { guidance: j.guidance }), }); - if (!j.error) failedNodeLabels.add(node.label); } - } + }); + for (const found of perNode) violations.push(...found); } return violations; diff --git a/tests/judge/judge-node.test.ts b/tests/judge/judge-node.test.ts index 1948012c..f2743e61 100644 --- a/tests/judge/judge-node.test.ts +++ b/tests/judge/judge-node.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'bun:test'; -import { judgeNode, type NodeJudgement } from '../../src/judge/judge-node'; +import { judgeNode, judgeNodes, type NodeJudgement } from '../../src/judge/judge-node'; import type { SemanticJudgeProvider, SemanticJudgeResult } from '../../src/judge/types'; import type { Rule } from '../../src/schema/metadata-contract'; import type { SpaceNode } from '../../src/types'; @@ -36,3 +36,70 @@ describe('judgeNode', () => { }); }); }); + +describe('judgeNodes', () => { + const rule = { + id: passFailRule.id, + description: passFailRule.description, + category: passFailRule.category, + mode: 'semantic', + severity: { fail: 'error' }, + verdicts: passFailRule.verdicts, + } as unknown as Rule; + const other = { ...node, title: 'Other', label: 'other.md' } as SpaceNode; + + it('calls a judge without evaluateBatch once per node', async () => { + const seen: string[] = []; + const judge: SemanticJudgeProvider = { + name: 'single', + evaluate: (input) => { + seen.push(input.node.title); + return Promise.resolve({ [passFailRule.id]: { verdict: 'pass' } }); + }, + }; + const out = await judgeNodes(judge, [ + { node: node as SpaceNode, rules: [rule] }, + { node: other, rules: [rule] }, + ]); + expect(seen).toEqual([node.title, 'Other']); + expect(out.map((o) => o.length)).toEqual([1, 1]); + }); + + it('sends every node to evaluateBatch at once and maps results back by position', async () => { + let calls = 0; + const judge: SemanticJudgeProvider = { + name: 'batch', + evaluate: () => Promise.reject(new Error('unused')), + evaluateBatch: (inputs) => { + calls++; + return Promise.resolve( + inputs.map((i) => ({ + [passFailRule.id]: i.node.title === 'Other' ? { verdict: 'fail' } : { verdict: 'pass' }, + })), + ); + }, + }; + const out = await judgeNodes(judge, [ + { node: node as SpaceNode, rules: [rule] }, + { node: other, rules: [] }, + { node: other, rules: [rule] }, + ]); + expect(calls).toBe(1); + expect(out[0]?.[0]).toMatchObject({ failed: false, verdict: 'pass' }); + expect(out[1]).toEqual([]); + expect(out[2]?.[0]).toMatchObject({ failed: true, verdict: 'fail' }); + }); + + it('marks every rule as an error, not a pass, when the batch call throws', async () => { + const judge: SemanticJudgeProvider = { + name: 'batch', + evaluate: () => Promise.reject(new Error('unused')), + evaluateBatch: () => Promise.reject(new Error('boom')), + }; + const out = await judgeNodes(judge, [ + { node: node as SpaceNode, rules: [rule] }, + { node: other, rules: [rule] }, + ]); + expect(out.flat().every((j) => j.failed && j.error === 'boom')).toBe(true); + }); +}); diff --git a/tests/judge/provider/jev.test.ts b/tests/judge/provider/jev.test.ts index da3ae032..cd500994 100644 --- a/tests/judge/provider/jev.test.ts +++ b/tests/judge/provider/jev.test.ts @@ -1,6 +1,6 @@ import { afterEach, describe, expect, it } from 'bun:test'; import { createModelProvider } from '../../../src/judge/factory'; -import { JevJudgeProvider } from '../../../src/judge/provider/jev'; +import { JEV_BATCH_MAX_QUESTIONS, JevJudgeProvider } from '../../../src/judge/provider/jev'; import { graded, node, passFailRule } from '../helpers'; const realFetch = globalThis.fetch; @@ -80,4 +80,101 @@ describe('JevJudgeProvider', () => { 'OPENROUTER_API_KEY', ); }); + + describe('evaluateBatch', () => { + const jev = () => new JevJudgeProvider({ api: 'openrouter-decisions', apiKey: 'k' }); + const nodeNamed = (title: string, extra: Record = {}) => ({ + ...node, + title, + schemaData: { ...node.schemaData, title, ...extra }, + }); + /** Answers every question in the request with `pass`, or fails the request for which `fail` returns true. */ + function answering(fail: (body: { state: { nodes?: unknown } }) => boolean = () => false) { + const bodies: Array<{ state: { nodes?: Record }; questions: Record }> = []; + globalThis.fetch = (async (_url: string, init: RequestInit) => { + const body = JSON.parse(init.body as string); + bodies.push(body); + if (fail(body)) return new Response('upstream down', { status: 500 }); + const answers = Object.fromEntries( + Object.keys(body.questions).map((q) => [q, { choice: 'pass', probabilities: { pass: 1, fail: 0 } }]), + ); + return new Response(JSON.stringify({ answers }), { status: 200 }); + }) as unknown as typeof fetch; + return bodies; + } + + it('shares one request between nodes with the same rule set, one question per node and rule', async () => { + const bodies = answering(); + const results = await jev().evaluateBatch([ + { node: nodeNamed('A'), rules: [passFailRule, graded] }, + { node: nodeNamed('B'), rules: [passFailRule, graded] }, + ]); + expect(bodies).toHaveLength(1); + expect(Object.keys(bodies[0]!.state.nodes!)).toEqual(['n0', 'n1']); + expect(Object.keys(bodies[0]!.questions).sort()).toEqual([ + 'customer-voice__n0', + 'customer-voice__n1', + 'framing__n0', + 'framing__n1', + ]); + expect((bodies[0]!.questions.framing__n1 as { instructions: string }).instructions).toStartWith( + 'For state.nodes.n1:', + ); + expect(results.map((r) => Object.keys(r).sort())).toEqual([ + ['customer-voice', 'framing'], + ['customer-voice', 'framing'], + ]); + }); + + it('sends nodes with different rule sets in separate requests', async () => { + const bodies = answering(); + await jev().evaluateBatch([ + { node: nodeNamed('A'), rules: [passFailRule] }, + { node: nodeNamed('B'), rules: [graded] }, + ]); + expect(bodies).toHaveLength(2); + }); + + it('sends a lone node as a plain single-node request', async () => { + const bodies = answering(); + await jev().evaluateBatch([{ node: nodeNamed('A'), rules: [passFailRule] }]); + expect(bodies[0]!.state.nodes).toBeUndefined(); + expect(Object.keys(bodies[0]!.questions)).toEqual(['customer-voice']); + }); + + it('splits a group by the question cap without splitting a node', async () => { + const bodies = answering(); + const count = JEV_BATCH_MAX_QUESTIONS + 5; + const results = await jev().evaluateBatch( + Array.from({ length: count }, (_, i) => ({ node: nodeNamed(`N${i}`), rules: [passFailRule] })), + ); + expect(bodies.map((b) => Object.keys(b.questions).length)).toEqual([JEV_BATCH_MAX_QUESTIONS, 5]); + expect(results).toHaveLength(count); + expect(results.every((r) => 'verdict' in r['customer-voice']!)).toBe(true); + }); + + it('splits a group by estimated size, and sends an oversized node alone and whole', async () => { + const bodies = answering(); + const big = (title: string) => nodeNamed(title, { body: 'x'.repeat(60_000) }); // ~15k tokens each + const huge = nodeNamed('Huge', { body: 'y'.repeat(200_000) }); // ~50k tokens, over the cap alone + await jev().evaluateBatch([ + { node: big('A'), rules: [passFailRule] }, + { node: big('B'), rules: [passFailRule] }, + { node: big('C'), rules: [passFailRule] }, + { node: huge, rules: [passFailRule] }, + ]); + expect(bodies.map((b) => (b.state.nodes ? Object.keys(b.state.nodes).length : 1))).toEqual([2, 1, 1]); + expect(JSON.stringify(bodies[2]!.state)).toContain('y'.repeat(200_000)); + }); + + it('maps a failed request to errors on its own nodes only', async () => { + answering((body) => JSON.stringify(body).includes('"title":"A"')); + const results = await jev().evaluateBatch([ + { node: nodeNamed('A', { body: 'x'.repeat(100_000) }), rules: [passFailRule] }, + { node: nodeNamed('B', { body: 'x'.repeat(100_000) }), rules: [passFailRule] }, + ]); + expect(results[0]?.['customer-voice']).toMatchObject({ error: expect.stringContaining('Jev API error (500)') }); + expect(results[1]?.['customer-voice']).toMatchObject({ verdict: 'pass' }); + }); + }); }); diff --git a/tests/schema/gating.test.ts b/tests/schema/gating.test.ts index 515e1643..77338ec9 100644 --- a/tests/schema/gating.test.ts +++ b/tests/schema/gating.test.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from 'bun:test'; +import type { SemanticJudgeProvider } from '../../src/judge/types'; import { computeHierarchyDepths, evaluateRulesWithGating, @@ -185,4 +186,43 @@ describe('evaluateRulesWithGating', () => { expect(violations.find((v) => v.ruleId === 'global-fail')).toBeDefined(); }); + + describe('batched judging', () => { + const semanticRule = { + id: 'sem', + category: 'best-practice', + description: 'semantic', + mode: 'semantic', + severity: { fail: 'error' }, + verdicts: { pass: 'ok', fail: 'not ok' }, + } as unknown as Rule; + + it('judges each depth in one batch and still gates children of a failing node', async () => { + const goal = makeSpaceNode('Goal', 'goal'); + const oppA = makeSpaceNode('Opp A', 'opportunity', 'Goal'); + const oppB = makeSpaceNode('Opp B', 'opportunity', 'Goal'); + const solA = makeSpaceNode('Sol A', 'solution', 'Opp A'); + const solB = makeSpaceNode('Sol B', 'solution', 'Opp B'); + const nodes = [goal, oppA, oppB, solA, solB]; + const graph = buildSpaceGraph(nodes, hierarchy); + + const batches: string[][] = []; + const provider: SemanticJudgeProvider = { + name: 'batching', + evaluate: () => Promise.reject(new Error('expected evaluateBatch')), + evaluateBatch: async (inputs) => { + batches.push(inputs.map((i) => i.node.title)); + return inputs.map((i) => + Object.fromEntries(i.rules.map((r) => [r.id, { verdict: i.node.title === 'Opp A' ? 'fail' : 'pass' }])), + ); + }, + }; + + const { violations, gates } = await evaluateRulesWithGating(nodes, [semanticRule], graph, [], { provider }); + expect(batches).toEqual([['Goal'], ['Opp A', 'Opp B'], ['Sol B']]); + expect(violations.map((v) => v.file)).toEqual(['Opp A.md']); + expect(gates).toHaveLength(1); + expect(gates[0]?.gatedFiles).toEqual(['Sol A.md']); + }); + }); }); From 4e9658005e304ea5ca411828e104818b145b7c49 Mon Sep 17 00:00:00 2001 From: Roger Barnes Date: Fri, 16 Oct 2026 09:18:09 +1100 Subject: [PATCH 2/9] refactor(judge): name Jev caps by the unit they limit --- docs/semantic-rules.md | 2 +- src/judge/provider/jev.ts | 35 +++++++++++++++++--------------- tests/judge/provider/jev.test.ts | 10 ++++----- 3 files changed, 25 insertions(+), 22 deletions(-) diff --git a/docs/semantic-rules.md b/docs/semantic-rules.md index fc856f3f..c41618c7 100644 --- a/docs/semantic-rules.md +++ b/docs/semantic-rules.md @@ -109,7 +109,7 @@ Gating uses the severity of the verdict. A verdict with `error` severity gates t ## How judging works - **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge. -- **Batching.** A decision model judges nodes that share the same set of semantic rules in one request, with one question per node and rule. A request is limited to about 32,000 estimated input tokens and 40 questions, so a larger group is split into several requests, never within a node. A node that fills a request alone is sent whole as its own request. Chat models judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Batching can shift probabilities slightly compared with judging each node alone. +- **Batching.** A decision model judges nodes that share the same set of semantic rules in one request, with one question per node and rule. A request is limited to about 32,000 estimated input tokens and 40 questions, so a larger batch is split into several requests, never within a node. A node that fills a request alone is sent whole as its own request. Chat models judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Batching can shift probabilities slightly compared with judging each node alone. - **Gating.** structured-context does not judge a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated. - **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. If one batched request fails, only the rules on the nodes in that request report the error. - **Input.** The judge gets the node's title, type, parent titles and schema data, including frontmatter. diff --git a/src/judge/provider/jev.ts b/src/judge/provider/jev.ts index cb963605..613b1eb9 100644 --- a/src/judge/provider/jev.ts +++ b/src/judge/provider/jev.ts @@ -15,15 +15,15 @@ export const JEV_LATEST = 'jev-latest'; export const OPENROUTER_DECISIONS_ENDPOINT = 'https://openrouter.ai/api/alpha/decisions'; /** - * Estimated input-token cap (characters / 4) for one batched request. Conservative: issue #152 cites a + * Estimated input-token cap (characters / 4) for one request's `state` plus `questions`. Conservative: issue #152 cites a * "32k token limit for state plus questions" for Jev, which is unverified. measured: a live probe accepted a * 250k-token state through OpenRouter with HTTP 200, so the real limit is higher and this cap is a cost and * latency guard, not a hard limit. */ -export const JEV_BATCH_MAX_TOKENS = 32_000; +export const JEV_REQUEST_MAX_ESTIMATED_TOKENS = 32_000; -/** Cap on questions per batched request. Sourced from the jev-rank reference client (20 pairs x 2 = 40 per request). */ -export const JEV_BATCH_MAX_QUESTIONS = 40; +/** Cap on the number of questions in one request. Sourced from the jev-rank reference client (20 pairs x 2 = 40 per request). */ +export const JEV_REQUEST_MAX_QUESTIONS = 40; type JevChoiceAnswer = { type?: string; @@ -62,28 +62,28 @@ export class JevJudgeProvider implements SemanticJudgeProvider { } /** - * Judge many nodes with fewer requests. Nodes that share the same semantic rule set go into one request, - * with every node under `state.nodes` and one question per node and rule. A group is split into chunks by - * `JEV_BATCH_MAX_TOKENS` and `JEV_BATCH_MAX_QUESTIONS`, never within a node. A node alone in its chunk, even + * Judge many nodes with fewer requests. Nodes that share the same semantic rule set form a batch, sent as one request, + * with every node under `state.nodes` and one question per node and rule. A batch is split into requests by + * `JEV_REQUEST_MAX_ESTIMATED_TOKENS` and `JEV_REQUEST_MAX_QUESTIONS`, never within a node. A node alone in its request, even * one over the cap, is sent as a plain single-node request. A failed request marks only its own nodes' rules * as errors. */ async evaluateBatch(inputs: SemanticJudgeInput[]): Promise>> { const results: Array> = inputs.map(() => ({})); - const groups = new Map(); + const batches = new Map(); inputs.forEach((input, i) => { if (input.rules.length === 0) return; const key = JSON.stringify(input.rules); - groups.set(key, [...(groups.get(key) ?? []), i]); + batches.set(key, [...(batches.get(key) ?? []), i]); }); - for (const indices of groups.values()) { - for (const chunk of chunkGroup(indices, inputs)) { + for (const indices of batches.values()) { + for (const chunk of splitIntoRequests(indices, inputs)) { try { if (chunk.length === 1) { results[chunk[0]!] = await this.evaluate(inputs[chunk[0]!]!); } else { - await this.evaluateChunk(chunk, inputs, results); + await this.evaluateRequest(chunk, inputs, results); } } catch (err) { const error = err instanceof Error ? err.message : String(err); @@ -96,7 +96,7 @@ export class JevJudgeProvider implements SemanticJudgeProvider { return results; } - private async evaluateChunk( + private async evaluateRequest( chunk: number[], inputs: SemanticJudgeInput[], results: Array>, @@ -164,8 +164,8 @@ function toResult(rule: SemanticRuleSpec, answer: JevChoiceAnswer | undefined): const estimateTokens = (value: unknown) => Math.ceil(JSON.stringify(value).length / 4); -/** Split same-rule-set inputs into request-sized chunks, in order, never splitting a node. */ -function chunkGroup(indices: number[], inputs: SemanticJudgeInput[]): number[][] { +/** Split one batch (inputs sharing a rule set) into request-sized runs, in order, never splitting a node. */ +function splitIntoRequests(indices: number[], inputs: SemanticJudgeInput[]): number[][] { const chunks: number[][] = []; let current: number[] = []; let tokens = 0; @@ -174,7 +174,10 @@ function chunkGroup(indices: number[], inputs: SemanticJudgeInput[]): number[][] const input = inputs[i]!; const cost = estimateTokens(nodeState(input.node)) + estimateTokens(input.rules); const count = input.rules.length; - if (current.length > 0 && (tokens + cost > JEV_BATCH_MAX_TOKENS || questions + count > JEV_BATCH_MAX_QUESTIONS)) { + if ( + current.length > 0 && + (tokens + cost > JEV_REQUEST_MAX_ESTIMATED_TOKENS || questions + count > JEV_REQUEST_MAX_QUESTIONS) + ) { chunks.push(current); current = []; tokens = 0; diff --git a/tests/judge/provider/jev.test.ts b/tests/judge/provider/jev.test.ts index cd500994..05af4e61 100644 --- a/tests/judge/provider/jev.test.ts +++ b/tests/judge/provider/jev.test.ts @@ -1,6 +1,6 @@ import { afterEach, describe, expect, it } from 'bun:test'; import { createModelProvider } from '../../../src/judge/factory'; -import { JEV_BATCH_MAX_QUESTIONS, JevJudgeProvider } from '../../../src/judge/provider/jev'; +import { JEV_REQUEST_MAX_QUESTIONS, JevJudgeProvider } from '../../../src/judge/provider/jev'; import { graded, node, passFailRule } from '../helpers'; const realFetch = globalThis.fetch; @@ -142,18 +142,18 @@ describe('JevJudgeProvider', () => { expect(Object.keys(bodies[0]!.questions)).toEqual(['customer-voice']); }); - it('splits a group by the question cap without splitting a node', async () => { + it('splits a batch by the question cap without splitting a node', async () => { const bodies = answering(); - const count = JEV_BATCH_MAX_QUESTIONS + 5; + const count = JEV_REQUEST_MAX_QUESTIONS + 5; const results = await jev().evaluateBatch( Array.from({ length: count }, (_, i) => ({ node: nodeNamed(`N${i}`), rules: [passFailRule] })), ); - expect(bodies.map((b) => Object.keys(b.questions).length)).toEqual([JEV_BATCH_MAX_QUESTIONS, 5]); + expect(bodies.map((b) => Object.keys(b.questions).length)).toEqual([JEV_REQUEST_MAX_QUESTIONS, 5]); expect(results).toHaveLength(count); expect(results.every((r) => 'verdict' in r['customer-voice']!)).toBe(true); }); - it('splits a group by estimated size, and sends an oversized node alone and whole', async () => { + it('splits a batch by estimated size, and sends an oversized node alone and whole', async () => { const bodies = answering(); const big = (title: string) => nodeNamed(title, { body: 'x'.repeat(60_000) }); // ~15k tokens each const huge = nodeNamed('Huge', { body: 'y'.repeat(200_000) }); // ~50k tokens, over the cap alone From 5256231741594e435a1c51c1c8178b01a8747484 Mon Sep 17 00:00:00 2001 From: Roger Barnes Date: Fri, 16 Oct 2026 09:27:09 +1100 Subject: [PATCH 3/9] refactor(judge): name the Jev batch terms NodeRulesetGroup, DecisionRequest, DecisionQuestion --- AGENTS.md | 2 +- docs/concepts.md | 6 ++++ docs/semantic-rules.md | 4 +-- src/judge/judge-node.ts | 10 +++--- src/judge/provider/jev.ts | 58 +++++++++++++++++--------------- src/judge/types.ts | 4 +-- src/schema/gating.ts | 8 ++--- src/schema/validate-rules.ts | 2 +- tests/judge/judge-node.test.ts | 14 ++++---- tests/judge/provider/jev.test.ts | 24 ++++++------- tests/schema/gating.test.ts | 16 ++++----- 11 files changed, 78 insertions(+), 70 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 36016498..1a6097a0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -59,7 +59,7 @@ There are several places that need reviewing and updating with any new feature o - `src/schema/metadata-contract.ts` — Single source of truth for the `$metadata` contract - `schemas/generated/latest/_structured_context_schema_meta.json` — Generated metaschema for the newest dialect (generated on build or with `bun run generate:schema-meta`). `schemas/generated//` folders are frozen copies, created by the `version` script (`generate:schema-meta --release`) only when a release changes the dialect; never edit them. `src/schema/meta-schemas.ts` loads them at runtime. - `src/schema/gating.ts` — Cascade gating: computes hierarchy depth and evaluates rules with subtree gating (schema/hierarchy/blocking-rule failures suppress downstream rule checks). See [docs/rules.md](docs/rules.md#cascade-gating). -- `src/judge/` — Semantic rule judges: provider interface and types (`types.ts`), verdict resolution (`verdicts.ts`), per-node and batched dispatch (`judge-node.ts`), and providers (`provider/anthropic.ts`, `provider/openai.ts`, `provider/jev.ts`, `provider/fake.ts`), provider hydration and the bundled catalogue (`factory.ts`), and `sctx providers` (`src/commands/providers.ts`). See [docs/semantic-rules.md](docs/semantic-rules.md). +- `src/judge/` — Semantic rule judges: provider interface and types (`types.ts`), verdict resolution (`verdicts.ts`), per-node and grouped dispatch (`judge-node.ts`), and providers (`provider/anthropic.ts`, `provider/openai.ts`, `provider/jev.ts`, `provider/fake.ts`), provider hydration and the bundled catalogue (`factory.ts`), and `sctx providers` (`src/commands/providers.ts`). See [docs/semantic-rules.md](docs/semantic-rules.md). - `src/schema/rule-filter.ts` — `--rule-filter` parsing and rule filtering by mode, category, severity, type or id. ## Testing diff --git a/docs/concepts.md b/docs/concepts.md index f18c7639..9db8d55c 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -173,6 +173,12 @@ A **semantic rule** (`mode: semantic`) asks a model to judge a node, for example A **judge** is the model that judges semantic rules: a decision model (Jev) or a chat model (Claude, OpenAI-compatible). For each rule, the judge gives one **verdict** from the rule's `verdicts`, for example `customer-need`, `vague` or `solution`. The rule's `severity` says which verdicts report a violation. Semantic rules cost money, so they are judged only when a run asks for it (`--semantic` or `--judge`); a rule's `check` always runs. The judge is a configured **model provider**: a bundled provider or an entry in `modelProviders`. See [docs/semantic-rules.md](semantic-rules.md). +A **NodeRulesetGroup** is the set of nodes that share the same semantic rule set. A decision model judges one group in as few decision requests as it can. + +A **DecisionRequest** is one call to a decision model's API. It carries the state of one or more nodes and a list of decision questions. + +A **DecisionQuestion** is one choice question about one node under one semantic rule, inside a decision request. + ### Cascade gating **Cascade gating** is the mechanism by which a schema failure, hierarchy structure violation, or blocking rule violation on a node suppresses rule evaluation for that node's `hierarchy` subtree. It exists so downstream checks — especially costly semantic (model-judged) rules — aren't run against inputs already known to be invalid. diff --git a/docs/semantic-rules.md b/docs/semantic-rules.md index c41618c7..b20e9ed7 100644 --- a/docs/semantic-rules.md +++ b/docs/semantic-rules.md @@ -109,9 +109,9 @@ Gating uses the severity of the verdict. A verdict with `error` severity gates t ## How judging works - **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge. -- **Batching.** A decision model judges nodes that share the same set of semantic rules in one request, with one question per node and rule. A request is limited to about 32,000 estimated input tokens and 40 questions, so a larger batch is split into several requests, never within a node. A node that fills a request alone is sent whole as its own request. Chat models judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Batching can shift probabilities slightly compared with judging each node alone. +- **Grouping.** A decision model judges each NodeRulesetGroup (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request alone is sent whole as its own decision request. Chat models judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Grouping can shift probabilities slightly compared with judging each node alone. - **Gating.** structured-context does not judge a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated. -- **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. If one batched request fails, only the rules on the nodes in that request report the error. +- **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. If one decision request fails, only the rules on the nodes in that decision request report the error. - **Input.** The judge gets the node's title, type, parent titles and schema data, including frontmatter. - **Output.** A violation from a semantic rule also includes the verdict, the judge's evidence and any guidance. diff --git a/src/judge/judge-node.ts b/src/judge/judge-node.ts index 146e2210..844a0a75 100644 --- a/src/judge/judge-node.ts +++ b/src/judge/judge-node.ts @@ -76,7 +76,7 @@ export async function judgeNode( } /** - * Judge several nodes at once. A judge with `evaluateBatch` gets every node in one call and may fold them into + * Judge several nodes at once. A judge with `evaluateNodes` gets every node in one call and may fold them into * fewer requests; any other judge is called once per node. Results align with `items`. A failure marks only the * affected node's rules as errors. */ @@ -84,7 +84,7 @@ export async function judgeNodes( judge: SemanticJudgeProvider, items: Array<{ node: SpaceNode; rules: Rule[] }>, ): Promise { - if (!judge.evaluateBatch) { + if (!judge.evaluateNodes) { const out: NodeJudgement[][] = []; for (const item of items) out.push(await judgeNode(judge, item.node, item.rules)); return out; @@ -94,15 +94,15 @@ export async function judgeNodes( .filter((a) => a.rules.length > 0); const out: NodeJudgement[][] = items.map(() => []); if (active.length === 0) return out; - let batch: JudgeResults[]; + let perNode: JudgeResults[]; try { - batch = await judge.evaluateBatch(active.map((a) => ({ node: a.node, rules: a.resolved.map((r) => r.spec) }))); + perNode = await judge.evaluateNodes(active.map((a) => ({ node: a.node, rules: a.resolved.map((r) => r.spec) }))); } catch (err) { for (const a of active) out[a.index] = errorJudgements(a.rules, errorMessage(err)); return out; } active.forEach((a, k) => { - out[a.index] = interpret(a.rules, a.resolved, batch[k] ?? {}); + out[a.index] = interpret(a.rules, a.resolved, perNode[k] ?? {}); }); return out; } diff --git a/src/judge/provider/jev.ts b/src/judge/provider/jev.ts index 613b1eb9..dba7d846 100644 --- a/src/judge/provider/jev.ts +++ b/src/judge/provider/jev.ts @@ -15,15 +15,15 @@ export const JEV_LATEST = 'jev-latest'; export const OPENROUTER_DECISIONS_ENDPOINT = 'https://openrouter.ai/api/alpha/decisions'; /** - * Estimated input-token cap (characters / 4) for one request's `state` plus `questions`. Conservative: issue #152 cites a + * Estimated input-token cap (characters / 4) for one DecisionRequest's `state` plus `questions`. Conservative: issue #152 cites a * "32k token limit for state plus questions" for Jev, which is unverified. measured: a live probe accepted a * 250k-token state through OpenRouter with HTTP 200, so the real limit is higher and this cap is a cost and * latency guard, not a hard limit. */ -export const JEV_REQUEST_MAX_ESTIMATED_TOKENS = 32_000; +export const JEV_DECISION_REQUEST_MAX_ESTIMATED_TOKENS = 32_000; -/** Cap on the number of questions in one request. Sourced from the jev-rank reference client (20 pairs x 2 = 40 per request). */ -export const JEV_REQUEST_MAX_QUESTIONS = 40; +/** Cap on the number of DecisionQuestions in one DecisionRequest. Sourced from the jev-rank reference client (20 pairs x 2 = 40 per request). */ +export const JEV_DECISION_REQUEST_MAX_QUESTIONS = 40; type JevChoiceAnswer = { type?: string; @@ -36,7 +36,7 @@ type JevChoiceAnswer = { * Judge provider for TypeSafe's Jev decision model via OpenRouter's Decisions API. * * Each node is sent once as `state`, with one `choice` question per semantic rule whose criteria are - * the rule's closed verdict set. `evaluateBatch` shares one request between nodes with the same rule set. Jev returns the chosen verdict with confidence and probabilities, + * the rule's closed verdict set. `evaluateNodes` shares one DecisionRequest between nodes with the same rule set. Jev returns the chosen verdict with confidence and probabilities, * but no prose, so `evidence` is a summary of the distribution. */ export class JevJudgeProvider implements SemanticJudgeProvider { @@ -54,7 +54,7 @@ export class JevJudgeProvider implements SemanticJudgeProvider { async evaluate(input: SemanticJudgeInput): Promise> { const { node, rules } = input; const questions: Record = {}; - for (const rule of rules) questions[rule.id] = choiceQuestion(rule, rule.rubric || rule.description); + for (const rule of rules) questions[rule.id] = decisionQuestion(rule, rule.rubric || rule.description); const data = await this.post(nodeState(node), questions); const results: Record = {}; for (const rule of rules) results[rule.id] = toResult(rule, data.answers?.[rule.id]); @@ -62,32 +62,33 @@ export class JevJudgeProvider implements SemanticJudgeProvider { } /** - * Judge many nodes with fewer requests. Nodes that share the same semantic rule set form a batch, sent as one request, - * with every node under `state.nodes` and one question per node and rule. A batch is split into requests by - * `JEV_REQUEST_MAX_ESTIMATED_TOKENS` and `JEV_REQUEST_MAX_QUESTIONS`, never within a node. A node alone in its request, even - * one over the cap, is sent as a plain single-node request. A failed request marks only its own nodes' rules + * Judge many nodes with fewer DecisionRequests. Nodes that share the same semantic rule set form a + * NodeRulesetGroup, sent as one DecisionRequest, + * with every node under `state.nodes` and one DecisionQuestion per node and rule. A NodeRulesetGroup is split into DecisionRequests by + * `JEV_DECISION_REQUEST_MAX_ESTIMATED_TOKENS` and `JEV_DECISION_REQUEST_MAX_QUESTIONS`, never within a node. A node alone in its DecisionRequest, even + * one over the cap, is sent as a plain single-node DecisionRequest. A failed DecisionRequest marks only its own nodes' rules * as errors. */ - async evaluateBatch(inputs: SemanticJudgeInput[]): Promise>> { + async evaluateNodes(inputs: SemanticJudgeInput[]): Promise>> { const results: Array> = inputs.map(() => ({})); - const batches = new Map(); + const nodeRulesetGroups = new Map(); inputs.forEach((input, i) => { if (input.rules.length === 0) return; const key = JSON.stringify(input.rules); - batches.set(key, [...(batches.get(key) ?? []), i]); + nodeRulesetGroups.set(key, [...(nodeRulesetGroups.get(key) ?? []), i]); }); - for (const indices of batches.values()) { - for (const chunk of splitIntoRequests(indices, inputs)) { + for (const indices of nodeRulesetGroups.values()) { + for (const requestInputs of planDecisionRequests(indices, inputs)) { try { - if (chunk.length === 1) { - results[chunk[0]!] = await this.evaluate(inputs[chunk[0]!]!); + if (requestInputs.length === 1) { + results[requestInputs[0]!] = await this.evaluate(inputs[requestInputs[0]!]!); } else { - await this.evaluateRequest(chunk, inputs, results); + await this.evaluateDecisionRequest(requestInputs, inputs, results); } } catch (err) { const error = err instanceof Error ? err.message : String(err); - for (const i of chunk) { + for (const i of requestInputs) { results[i] = Object.fromEntries(inputs[i]!.rules.map((r) => [r.id, { error }])); } } @@ -96,25 +97,25 @@ export class JevJudgeProvider implements SemanticJudgeProvider { return results; } - private async evaluateRequest( - chunk: number[], + private async evaluateDecisionRequest( + requestInputs: number[], inputs: SemanticJudgeInput[], results: Array>, ): Promise { const nodes: Record = {}; const questions: Record = {}; - chunk.forEach((inputIndex, k) => { + requestInputs.forEach((inputIndex, k) => { const input = inputs[inputIndex]!; nodes[`n${k}`] = nodeState(input.node); for (const rule of input.rules) { - questions[`${rule.id}__n${k}`] = choiceQuestion( + questions[`${rule.id}__n${k}`] = decisionQuestion( rule, `For state.nodes.n${k}: ${rule.rubric || rule.description}`, ); } }); const data = await this.post({ nodes }, questions); - chunk.forEach((inputIndex, k) => { + requestInputs.forEach((inputIndex, k) => { for (const rule of inputs[inputIndex]!.rules) { results[inputIndex]![rule.id] = toResult(rule, data.answers?.[`${rule.id}__n${k}`]); } @@ -146,7 +147,7 @@ function nodeState(node: SemanticJudgeInput['node']) { }; } -function choiceQuestion(rule: SemanticRuleSpec, instructions: string) { +function decisionQuestion(rule: SemanticRuleSpec, instructions: string) { return { type: 'choice', instructions, criteria: rule.verdicts }; } @@ -164,8 +165,8 @@ function toResult(rule: SemanticRuleSpec, answer: JevChoiceAnswer | undefined): const estimateTokens = (value: unknown) => Math.ceil(JSON.stringify(value).length / 4); -/** Split one batch (inputs sharing a rule set) into request-sized runs, in order, never splitting a node. */ -function splitIntoRequests(indices: number[], inputs: SemanticJudgeInput[]): number[][] { +/** Plan the DecisionRequests for one NodeRulesetGroup (inputs sharing a rule set): runs of nodes, in order, never splitting a node. */ +function planDecisionRequests(indices: number[], inputs: SemanticJudgeInput[]): number[][] { const chunks: number[][] = []; let current: number[] = []; let tokens = 0; @@ -176,7 +177,8 @@ function splitIntoRequests(indices: number[], inputs: SemanticJudgeInput[]): num const count = input.rules.length; if ( current.length > 0 && - (tokens + cost > JEV_REQUEST_MAX_ESTIMATED_TOKENS || questions + count > JEV_REQUEST_MAX_QUESTIONS) + (tokens + cost > JEV_DECISION_REQUEST_MAX_ESTIMATED_TOKENS || + questions + count > JEV_DECISION_REQUEST_MAX_QUESTIONS) ) { chunks.push(current); current = []; diff --git a/src/judge/types.ts b/src/judge/types.ts index fd45885b..c11ad7be 100644 --- a/src/judge/types.ts +++ b/src/judge/types.ts @@ -35,11 +35,11 @@ export interface SemanticJudgeProvider { readonly name: string; evaluate(input: SemanticJudgeInput): Promise>; /** - * Optional: judge many nodes in one call, so a provider can fold them into fewer requests. Returns one + * Optional: judge many nodes in one call, so a provider can fold them into fewer model calls. Returns one * result map per input, in order. A failure that affects only some inputs is reported as `{ error }` * results for those inputs' rules; throw only when nothing could be judged. */ - evaluateBatch?(inputs: SemanticJudgeInput[]): Promise>>; + evaluateNodes?(inputs: SemanticJudgeInput[]): Promise>>; } /** Wire APIs a provider can speak. Names the request shape, never the vendor or model. */ diff --git a/src/schema/gating.ts b/src/schema/gating.ts index 650ea35c..18b911f1 100644 --- a/src/schema/gating.ts +++ b/src/schema/gating.ts @@ -278,13 +278,13 @@ export async function evaluateRulesWithGating( /** * Evaluate a set of nodes whose gating is already decided. Their semantic rules go to the judge together, - * so a batching judge can share requests; violations are recorded in node order. + * so a judge with `evaluateNodes` can share model calls; violations are recorded in node order. */ - async function evaluateNodes(batch: readonly SpaceNode[]): Promise> { + async function evaluateNodes(wave: readonly SpaceNode[]): Promise> { const prepared: PreparedNode[] = []; - for (const node of batch) prepared.push(await prepareNode(node)); + for (const node of wave) prepared.push(await prepareNode(node)); if (judge) { - const pending = batch.map((node, i) => ({ node, rules: prepared[i]!.semantic })); + const pending = wave.map((node, i) => ({ node, rules: prepared[i]!.semantic })); const judgements = await judgeNodes(judge, pending); prepared.forEach((p, i) => { p.judged(judgements[i] ?? []); diff --git a/src/schema/validate-rules.ts b/src/schema/validate-rules.ts index 0ad54406..232a1bc5 100644 --- a/src/schema/validate-rules.ts +++ b/src/schema/validate-rules.ts @@ -147,7 +147,7 @@ export async function validateRules( if (judge) pending.push({ node, rules: prechecked.judgeable, found }); } - // One judge call for every node, so a batching judge can share requests. + // One judge call for every node, so a judge with `evaluateNodes` can share model calls. const judged = judge ? await judgeNodes(judge, pending) : []; pending.forEach(({ node, found }, i) => { for (const j of judged[i] ?? []) { diff --git a/tests/judge/judge-node.test.ts b/tests/judge/judge-node.test.ts index f2743e61..d8ac8615 100644 --- a/tests/judge/judge-node.test.ts +++ b/tests/judge/judge-node.test.ts @@ -48,7 +48,7 @@ describe('judgeNodes', () => { } as unknown as Rule; const other = { ...node, title: 'Other', label: 'other.md' } as SpaceNode; - it('calls a judge without evaluateBatch once per node', async () => { + it('calls a judge without evaluateNodes once per node', async () => { const seen: string[] = []; const judge: SemanticJudgeProvider = { name: 'single', @@ -65,12 +65,12 @@ describe('judgeNodes', () => { expect(out.map((o) => o.length)).toEqual([1, 1]); }); - it('sends every node to evaluateBatch at once and maps results back by position', async () => { + it('sends every node to evaluateNodes at once and maps results back by position', async () => { let calls = 0; const judge: SemanticJudgeProvider = { - name: 'batch', + name: 'multi-node', evaluate: () => Promise.reject(new Error('unused')), - evaluateBatch: (inputs) => { + evaluateNodes: (inputs) => { calls++; return Promise.resolve( inputs.map((i) => ({ @@ -90,11 +90,11 @@ describe('judgeNodes', () => { expect(out[2]?.[0]).toMatchObject({ failed: true, verdict: 'fail' }); }); - it('marks every rule as an error, not a pass, when the batch call throws', async () => { + it('marks every rule as an error, not a pass, when the evaluateNodes call throws', async () => { const judge: SemanticJudgeProvider = { - name: 'batch', + name: 'multi-node', evaluate: () => Promise.reject(new Error('unused')), - evaluateBatch: () => Promise.reject(new Error('boom')), + evaluateNodes: () => Promise.reject(new Error('boom')), }; const out = await judgeNodes(judge, [ { node: node as SpaceNode, rules: [rule] }, diff --git a/tests/judge/provider/jev.test.ts b/tests/judge/provider/jev.test.ts index 05af4e61..989ca867 100644 --- a/tests/judge/provider/jev.test.ts +++ b/tests/judge/provider/jev.test.ts @@ -1,6 +1,6 @@ import { afterEach, describe, expect, it } from 'bun:test'; import { createModelProvider } from '../../../src/judge/factory'; -import { JEV_REQUEST_MAX_QUESTIONS, JevJudgeProvider } from '../../../src/judge/provider/jev'; +import { JEV_DECISION_REQUEST_MAX_QUESTIONS, JevJudgeProvider } from '../../../src/judge/provider/jev'; import { graded, node, passFailRule } from '../helpers'; const realFetch = globalThis.fetch; @@ -81,7 +81,7 @@ describe('JevJudgeProvider', () => { ); }); - describe('evaluateBatch', () => { + describe('evaluateNodes', () => { const jev = () => new JevJudgeProvider({ api: 'openrouter-decisions', apiKey: 'k' }); const nodeNamed = (title: string, extra: Record = {}) => ({ ...node, @@ -105,7 +105,7 @@ describe('JevJudgeProvider', () => { it('shares one request between nodes with the same rule set, one question per node and rule', async () => { const bodies = answering(); - const results = await jev().evaluateBatch([ + const results = await jev().evaluateNodes([ { node: nodeNamed('A'), rules: [passFailRule, graded] }, { node: nodeNamed('B'), rules: [passFailRule, graded] }, ]); @@ -128,7 +128,7 @@ describe('JevJudgeProvider', () => { it('sends nodes with different rule sets in separate requests', async () => { const bodies = answering(); - await jev().evaluateBatch([ + await jev().evaluateNodes([ { node: nodeNamed('A'), rules: [passFailRule] }, { node: nodeNamed('B'), rules: [graded] }, ]); @@ -137,27 +137,27 @@ describe('JevJudgeProvider', () => { it('sends a lone node as a plain single-node request', async () => { const bodies = answering(); - await jev().evaluateBatch([{ node: nodeNamed('A'), rules: [passFailRule] }]); + await jev().evaluateNodes([{ node: nodeNamed('A'), rules: [passFailRule] }]); expect(bodies[0]!.state.nodes).toBeUndefined(); expect(Object.keys(bodies[0]!.questions)).toEqual(['customer-voice']); }); - it('splits a batch by the question cap without splitting a node', async () => { + it('splits a NodeRulesetGroup by the question cap without splitting a node', async () => { const bodies = answering(); - const count = JEV_REQUEST_MAX_QUESTIONS + 5; - const results = await jev().evaluateBatch( + const count = JEV_DECISION_REQUEST_MAX_QUESTIONS + 5; + const results = await jev().evaluateNodes( Array.from({ length: count }, (_, i) => ({ node: nodeNamed(`N${i}`), rules: [passFailRule] })), ); - expect(bodies.map((b) => Object.keys(b.questions).length)).toEqual([JEV_REQUEST_MAX_QUESTIONS, 5]); + expect(bodies.map((b) => Object.keys(b.questions).length)).toEqual([JEV_DECISION_REQUEST_MAX_QUESTIONS, 5]); expect(results).toHaveLength(count); expect(results.every((r) => 'verdict' in r['customer-voice']!)).toBe(true); }); - it('splits a batch by estimated size, and sends an oversized node alone and whole', async () => { + it('splits a NodeRulesetGroup by estimated size, and sends an oversized node alone and whole', async () => { const bodies = answering(); const big = (title: string) => nodeNamed(title, { body: 'x'.repeat(60_000) }); // ~15k tokens each const huge = nodeNamed('Huge', { body: 'y'.repeat(200_000) }); // ~50k tokens, over the cap alone - await jev().evaluateBatch([ + await jev().evaluateNodes([ { node: big('A'), rules: [passFailRule] }, { node: big('B'), rules: [passFailRule] }, { node: big('C'), rules: [passFailRule] }, @@ -169,7 +169,7 @@ describe('JevJudgeProvider', () => { it('maps a failed request to errors on its own nodes only', async () => { answering((body) => JSON.stringify(body).includes('"title":"A"')); - const results = await jev().evaluateBatch([ + const results = await jev().evaluateNodes([ { node: nodeNamed('A', { body: 'x'.repeat(100_000) }), rules: [passFailRule] }, { node: nodeNamed('B', { body: 'x'.repeat(100_000) }), rules: [passFailRule] }, ]); diff --git a/tests/schema/gating.test.ts b/tests/schema/gating.test.ts index 77338ec9..794ce6aa 100644 --- a/tests/schema/gating.test.ts +++ b/tests/schema/gating.test.ts @@ -187,7 +187,7 @@ describe('evaluateRulesWithGating', () => { expect(violations.find((v) => v.ruleId === 'global-fail')).toBeDefined(); }); - describe('batched judging', () => { + describe('judging by depth', () => { const semanticRule = { id: 'sem', category: 'best-practice', @@ -197,7 +197,7 @@ describe('evaluateRulesWithGating', () => { verdicts: { pass: 'ok', fail: 'not ok' }, } as unknown as Rule; - it('judges each depth in one batch and still gates children of a failing node', async () => { + it('judges each depth in one evaluateNodes call and still gates children of a failing node', async () => { const goal = makeSpaceNode('Goal', 'goal'); const oppA = makeSpaceNode('Opp A', 'opportunity', 'Goal'); const oppB = makeSpaceNode('Opp B', 'opportunity', 'Goal'); @@ -206,12 +206,12 @@ describe('evaluateRulesWithGating', () => { const nodes = [goal, oppA, oppB, solA, solB]; const graph = buildSpaceGraph(nodes, hierarchy); - const batches: string[][] = []; + const calls: string[][] = []; const provider: SemanticJudgeProvider = { - name: 'batching', - evaluate: () => Promise.reject(new Error('expected evaluateBatch')), - evaluateBatch: async (inputs) => { - batches.push(inputs.map((i) => i.node.title)); + name: 'multi-node', + evaluate: () => Promise.reject(new Error('expected evaluateNodes')), + evaluateNodes: async (inputs) => { + calls.push(inputs.map((i) => i.node.title)); return inputs.map((i) => Object.fromEntries(i.rules.map((r) => [r.id, { verdict: i.node.title === 'Opp A' ? 'fail' : 'pass' }])), ); @@ -219,7 +219,7 @@ describe('evaluateRulesWithGating', () => { }; const { violations, gates } = await evaluateRulesWithGating(nodes, [semanticRule], graph, [], { provider }); - expect(batches).toEqual([['Goal'], ['Opp A', 'Opp B'], ['Sol B']]); + expect(calls).toEqual([['Goal'], ['Opp A', 'Opp B'], ['Sol B']]); expect(violations.map((v) => v.file)).toEqual(['Opp A.md']); expect(gates).toHaveLength(1); expect(gates[0]?.gatedFiles).toEqual(['Sol A.md']); From 14006dd240f03017e39445b61e91efc1e35fefef Mon Sep 17 00:00:00 2001 From: Roger Barnes Date: Fri, 16 Oct 2026 09:34:00 +1100 Subject: [PATCH 4/9] refactor(judge): rename leftover chunk variable to requests --- src/judge/provider/jev.ts | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/judge/provider/jev.ts b/src/judge/provider/jev.ts index dba7d846..81069178 100644 --- a/src/judge/provider/jev.ts +++ b/src/judge/provider/jev.ts @@ -167,7 +167,7 @@ const estimateTokens = (value: unknown) => Math.ceil(JSON.stringify(value).lengt /** Plan the DecisionRequests for one NodeRulesetGroup (inputs sharing a rule set): runs of nodes, in order, never splitting a node. */ function planDecisionRequests(indices: number[], inputs: SemanticJudgeInput[]): number[][] { - const chunks: number[][] = []; + const requests: number[][] = []; let current: number[] = []; let tokens = 0; let questions = 0; @@ -180,7 +180,7 @@ function planDecisionRequests(indices: number[], inputs: SemanticJudgeInput[]): (tokens + cost > JEV_DECISION_REQUEST_MAX_ESTIMATED_TOKENS || questions + count > JEV_DECISION_REQUEST_MAX_QUESTIONS) ) { - chunks.push(current); + requests.push(current); current = []; tokens = 0; questions = 0; @@ -189,8 +189,8 @@ function planDecisionRequests(indices: number[], inputs: SemanticJudgeInput[]): tokens += cost; questions += count; } - if (current.length > 0) chunks.push(current); - return chunks; + if (current.length > 0) requests.push(current); + return requests; } function describeDistribution(answer: JevChoiceAnswer): string { From 5d99f62c9eaa7ef2b2e60db68d135e96140ee148 Mon Sep 17 00:00:00 2001 From: Roger Barnes Date: Fri, 16 Oct 2026 11:49:46 +1100 Subject: [PATCH 5/9] fix(judge): split a single node's questions across Jev requests; nest-state reference per TypeSafe guidance A node with more rules than the question cap now has its DecisionQuestions split across DecisionRequests. Batched questions refer to the node as `nodes.nK`. Drop spec history from the token-cap comment. Add tests for a missing batched answer and rule ids containing `__n0`. --- src/judge/provider/jev.ts | 26 +++++++++---------- tests/judge/factory.test.ts | 3 ++- tests/judge/provider/jev.test.ts | 43 +++++++++++++++++++++++++++++++- 3 files changed, 57 insertions(+), 15 deletions(-) diff --git a/src/judge/provider/jev.ts b/src/judge/provider/jev.ts index 81069178..8681f4fc 100644 --- a/src/judge/provider/jev.ts +++ b/src/judge/provider/jev.ts @@ -14,15 +14,10 @@ export const JEV_LATEST = 'jev-latest'; export const OPENROUTER_DECISIONS_ENDPOINT = 'https://openrouter.ai/api/alpha/decisions'; -/** - * Estimated input-token cap (characters / 4) for one DecisionRequest's `state` plus `questions`. Conservative: issue #152 cites a - * "32k token limit for state plus questions" for Jev, which is unverified. measured: a live probe accepted a - * 250k-token state through OpenRouter with HTTP 200, so the real limit is higher and this cap is a cost and - * latency guard, not a hard limit. - */ +/** Estimated input-token cap (characters / 4) for one DecisionRequest's `state` plus `questions`: a conservative cost and latency guard. */ export const JEV_DECISION_REQUEST_MAX_ESTIMATED_TOKENS = 32_000; -/** Cap on the number of DecisionQuestions in one DecisionRequest. Sourced from the jev-rank reference client (20 pairs x 2 = 40 per request). */ +/** Cap on the number of DecisionQuestions in one DecisionRequest. */ export const JEV_DECISION_REQUEST_MAX_QUESTIONS = 40; type JevChoiceAnswer = { @@ -53,11 +48,15 @@ export class JevJudgeProvider implements SemanticJudgeProvider { async evaluate(input: SemanticJudgeInput): Promise> { const { node, rules } = input; - const questions: Record = {}; - for (const rule of rules) questions[rule.id] = decisionQuestion(rule, rule.rubric || rule.description); - const data = await this.post(nodeState(node), questions); const results: Record = {}; - for (const rule of rules) results[rule.id] = toResult(rule, data.answers?.[rule.id]); + // A node with more rules than one DecisionRequest allows is split by rule; the state is sent whole each time. + for (let start = 0; start < rules.length; start += JEV_DECISION_REQUEST_MAX_QUESTIONS) { + const part = rules.slice(start, start + JEV_DECISION_REQUEST_MAX_QUESTIONS); + const questions: Record = {}; + for (const rule of part) questions[rule.id] = decisionQuestion(rule, rule.rubric || rule.description); + const data = await this.post(nodeState(node), questions); + for (const rule of part) results[rule.id] = toResult(rule, data.answers?.[rule.id]); + } return results; } @@ -66,7 +65,8 @@ export class JevJudgeProvider implements SemanticJudgeProvider { * NodeRulesetGroup, sent as one DecisionRequest, * with every node under `state.nodes` and one DecisionQuestion per node and rule. A NodeRulesetGroup is split into DecisionRequests by * `JEV_DECISION_REQUEST_MAX_ESTIMATED_TOKENS` and `JEV_DECISION_REQUEST_MAX_QUESTIONS`, never within a node. A node alone in its DecisionRequest, even - * one over the cap, is sent as a plain single-node DecisionRequest. A failed DecisionRequest marks only its own nodes' rules + * one over the token cap, is sent whole as a plain single-node DecisionRequest; one with more rules than the question cap has its rules split + * across DecisionRequests. A failed DecisionRequest marks only its own nodes' rules * as errors. */ async evaluateNodes(inputs: SemanticJudgeInput[]): Promise>> { @@ -110,7 +110,7 @@ export class JevJudgeProvider implements SemanticJudgeProvider { for (const rule of input.rules) { questions[`${rule.id}__n${k}`] = decisionQuestion( rule, - `For state.nodes.n${k}: ${rule.rubric || rule.description}`, + `For node \`nodes.n${k}\`: ${rule.rubric || rule.description}`, ); } }); diff --git a/tests/judge/factory.test.ts b/tests/judge/factory.test.ts index 4742146f..a9993ea0 100644 --- a/tests/judge/factory.test.ts +++ b/tests/judge/factory.test.ts @@ -10,6 +10,7 @@ import { AnthropicJudgeProvider } from '../../src/judge/provider/anthropic'; import { FakeJudgeProvider } from '../../src/judge/provider/fake'; import { JevJudgeProvider } from '../../src/judge/provider/jev'; import { OpenAIJudgeProvider } from '../../src/judge/provider/openai'; +import { passFailRule } from './helpers'; describe('hydrateProvider', () => { it('prefers the explicit value over the env var and leaves unset fields undefined', () => { @@ -64,7 +65,7 @@ describe('createModelProvider', () => { schemaData: {}, resolvedParents: [], }, - rules: [], + rules: [passFailRule], }) .catch(() => {}); expect(calls[0]).toBe('https://openrouter.ai/api/alpha/decisions'); diff --git a/tests/judge/provider/jev.test.ts b/tests/judge/provider/jev.test.ts index 989ca867..9a649690 100644 --- a/tests/judge/provider/jev.test.ts +++ b/tests/judge/provider/jev.test.ts @@ -118,7 +118,7 @@ describe('JevJudgeProvider', () => { 'framing__n1', ]); expect((bodies[0]!.questions.framing__n1 as { instructions: string }).instructions).toStartWith( - 'For state.nodes.n1:', + 'For node `nodes.n1`:', ); expect(results.map((r) => Object.keys(r).sort())).toEqual([ ['customer-voice', 'framing'], @@ -167,6 +167,47 @@ describe('JevJudgeProvider', () => { expect(JSON.stringify(bodies[2]!.state)).toContain('y'.repeat(200_000)); }); + it('reports an error on the rules of a node with no answer in a batched response, leaving other nodes intact', async () => { + globalThis.fetch = (async () => + new Response( + JSON.stringify({ + answers: { 'customer-voice__n0': { choice: 'pass' }, 'customer-voice__n2': { choice: 'fail' } }, + }), + { status: 200 }, + )) as unknown as typeof fetch; + const results = await jev().evaluateNodes( + ['A', 'B', 'C'].map((title) => ({ node: nodeNamed(title), rules: [passFailRule] })), + ); + expect(results[0]?.['customer-voice']).toMatchObject({ verdict: 'pass' }); + expect(results[1]?.['customer-voice']).toHaveProperty('error'); + expect(results[2]?.['customer-voice']).toMatchObject({ verdict: 'fail' }); + }); + + it('resolves answers for a rule id that itself contains the node suffix', async () => { + const bodies = answering(); + const tricky = { ...passFailRule, id: 'tricky__n0' }; + const results = await jev().evaluateNodes([ + { node: nodeNamed('A'), rules: [tricky] }, + { node: nodeNamed('B'), rules: [tricky] }, + ]); + expect(Object.keys(bodies[0]!.questions)).toEqual(['tricky__n0__n0', 'tricky__n0__n1']); + expect(results.map((r) => r.tricky__n0)).toEqual([ + expect.objectContaining({ verdict: 'pass' }), + expect.objectContaining({ verdict: 'pass' }), + ]); + }); + + it('splits a single node with more rules than the question cap across requests, sending its state whole each time', async () => { + const bodies = answering(); + const count = JEV_DECISION_REQUEST_MAX_QUESTIONS + 3; + const rules = Array.from({ length: count }, (_, i) => ({ ...passFailRule, id: `r${i}` })); + const results = await jev().evaluateNodes([{ node: nodeNamed('A'), rules }]); + expect(bodies.map((b) => Object.keys(b.questions).length)).toEqual([JEV_DECISION_REQUEST_MAX_QUESTIONS, 3]); + expect(bodies.every((b) => b.state.nodes === undefined)).toBe(true); + expect(Object.keys(results[0]!)).toHaveLength(count); + expect(Object.values(results[0]!).every((r) => 'verdict' in r)).toBe(true); + }); + it('maps a failed request to errors on its own nodes only', async () => { answering((body) => JSON.stringify(body).includes('"title":"A"')); const results = await jev().evaluateNodes([ From 8beef90243a8d542cc7b48b367e3273d4e5e7fa0 Mon Sep 17 00:00:00 2001 From: Roger Barnes Date: Fri, 16 Oct 2026 11:50:00 +1100 Subject: [PATCH 6/9] docs: name which providers batch; define node ruleset group; lowercase batching terms in prose --- docs/concepts.md | 6 +++--- docs/config.md | 6 +++--- docs/semantic-rules.md | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/concepts.md b/docs/concepts.md index 9db8d55c..738aff07 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -173,11 +173,11 @@ A **semantic rule** (`mode: semantic`) asks a model to judge a node, for example A **judge** is the model that judges semantic rules: a decision model (Jev) or a chat model (Claude, OpenAI-compatible). For each rule, the judge gives one **verdict** from the rule's `verdicts`, for example `customer-need`, `vague` or `solution`. The rule's `severity` says which verdicts report a violation. Semantic rules cost money, so they are judged only when a run asks for it (`--semantic` or `--judge`); a rule's `check` always runs. The judge is a configured **model provider**: a bundled provider or an entry in `modelProviders`. See [docs/semantic-rules.md](semantic-rules.md). -A **NodeRulesetGroup** is the set of nodes that share the same semantic rule set. A decision model judges one group in as few decision requests as it can. +A **node ruleset group** is the set of nodes that share the same semantic rules. It forms within one evaluation step of [cascade gating](#cascade-gating): one hierarchy depth, or the set of non-hierarchy nodes. Nodes are keyed on the rules left after `when` and `check` filtering. A judge on the `openrouter-decisions` or `typesafe-systemone` API judges one group in as few decision requests as it can. -A **DecisionRequest** is one call to a decision model's API. It carries the state of one or more nodes and a list of decision questions. +A **decision request** is one call to a decisions API. It carries the state of one or more nodes and a list of decision questions. -A **DecisionQuestion** is one choice question about one node under one semantic rule, inside a decision request. +A **decision question** is one choice question about one node under one semantic rule, inside a decision request. ### Cascade gating diff --git a/docs/config.md b/docs/config.md index 6214076d..dea778e9 100644 --- a/docs/config.md +++ b/docs/config.md @@ -79,9 +79,9 @@ A provider's `api` names the **request shape**, potentially supported by multipl | API | Shape | Bundled providers | |---|---|---| -| `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities; nodes with the same rules share a request | `openrouter-jev`, `openrouter-openai-decisions` | -| `typesafe-systemone` | Same decisions shape (currently) | `typesafe-jev` | -| `openai-decisions` | OpenAI's Decisions API: same concept, its own request/response shape | `openai-decisions` | +| `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities; nodes with the same rules share a decision request | `openrouter-jev`, `openrouter-openai-decisions` | +| `typesafe-systemone` | Same decisions shape (currently), including shared decision requests | `typesafe-jev` | +| `openai-decisions` | OpenAI's Decisions API: same concept, its own request/response shape; one node per request | `openai-decisions` | | `anthropic-messages` | Chat: one request per rule, structured JSON verdict | `anthropic`, `anthropic-haiku-5-5`, `anthropic-env` | | `openai-completions` | Chat: OpenAI-compatible; works with Ollama, vLLM and OpenRouter | `openai`, `openai-env`, `openrouter-openai` | diff --git a/docs/semantic-rules.md b/docs/semantic-rules.md index b20e9ed7..ff6babf9 100644 --- a/docs/semantic-rules.md +++ b/docs/semantic-rules.md @@ -109,7 +109,7 @@ Gating uses the severity of the verdict. A verdict with `error` severity gates t ## How judging works - **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge. -- **Grouping.** A decision model judges each NodeRulesetGroup (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request alone is sent whole as its own decision request. Chat models judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Grouping can shift probabilities slightly compared with judging each node alone. +- **Grouping.** A judge on the `openrouter-decisions` or `typesafe-systemone` API judges each node ruleset group (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. This includes the bundled `openrouter-openai-decisions` provider. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request on tokens is sent whole as its own decision request. A node with more than 40 rules has its rules split across decision requests. Other providers, including `openai-decisions` and chat models, judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Grouping can shift verdicts and probabilities slightly compared with judging each node alone. - **Gating.** structured-context does not judge a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated. - **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. If one decision request fails, only the rules on the nodes in that decision request report the error. - **Input.** The judge gets the node's title, type, parent titles and schema data, including frontmatter. From 17c242bd6e0c50c93a8f77b0fce0ac3806828ab0 Mon Sep 17 00:00:00 2001 From: Roger Barnes Date: Fri, 16 Oct 2026 11:50:29 +1100 Subject: [PATCH 7/9] test(judge): live drift comparison of single versus batched Jev judging --- tests/judge/live.test.ts | 72 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 69 insertions(+), 3 deletions(-) diff --git a/tests/judge/live.test.ts b/tests/judge/live.test.ts index 315a79c3..4c82cb7e 100644 --- a/tests/judge/live.test.ts +++ b/tests/judge/live.test.ts @@ -3,9 +3,9 @@ * SCTX_LIVE_JUDGE_TESTS=1 (`bun run test:judge`), and each provider only when its API key is set. */ import { describe, expect, it } from 'bun:test'; -import { createModelProvider } from '../../src/judge/factory'; -import type { SemanticJudgeInput, SemanticRuleSpec } from '../../src/judge/types'; -import { node, passFailRule } from './helpers'; +import { BUNDLED_PROVIDERS, createModelProvider } from '../../src/judge/factory'; +import type { SemanticJudgeInput, SemanticJudgeResult, SemanticRuleSpec } from '../../src/judge/types'; +import { graded, node, passFailRule } from './helpers'; const CASES = [ { name: 'openrouter-jev', api: 'openrouter-decisions' as const, keyEnv: 'OPENROUTER_API_KEY' }, @@ -38,3 +38,69 @@ describe.skipIf(process.env.SCTX_LIVE_JUDGE_TESTS !== '1')('live judges', () => ); } }); + +/** + * Drift between judging nodes one at a time (`evaluate`) and batched (`evaluateNodes`). Prints per-node, per-rule + * verdicts and probability deltas; asserts only that both paths return a valid verdict, so the numbers are for a + * person to read. Run with `bun test tests/judge/live.test.ts -t drift` and the env below. + */ +describe.skipIf(process.env.SCTX_LIVE_JUDGE_TESTS !== '1')('live judges: batching drift', () => { + const titles = [ + 'I cannot find my receipts when tax time comes', + 'Build a receipt scanner app', + 'Increase filing revenue by 20%', + 'Tax forms are confusing and I never know which one I need', + ]; + const inputs: SemanticJudgeInput[] = titles.map((title, i) => ({ + node: { ...node, label: `n${i}.md`, title, schemaData: { ...node.schemaData, title } }, + rules: [passFailRule, graded], + })); + + for (const name of ['openrouter-jev', 'openrouter-openai-decisions']) { + it.skipIf(!process.env.OPENROUTER_API_KEY)( + `${name}: single versus batched verdicts and probabilities`, + async () => { + const provider = createModelProvider(name, BUNDLED_PROVIDERS[name]!, { + OPENROUTER_API_KEY: process.env.OPENROUTER_API_KEY, + }); + if (!provider.evaluateNodes) throw new Error(`${name} does not batch`); + const single: Array> = []; + for (const input of inputs) single.push(await provider.evaluate(input)); + const batched = await provider.evaluateNodes(inputs); + + const lines: string[] = []; + let flips = 0; + let maxDelta = 0; + inputs.forEach((input, i) => { + for (const rule of input.rules) { + const a = single[i]?.[rule.id]; + const b = batched[i]?.[rule.id]; + if (!a || !b || 'error' in a || 'error' in b) { + expect(a).not.toHaveProperty('error'); + expect(b).not.toHaveProperty('error'); + continue; + } + const deltas = Object.keys(rule.verdicts).map((v) => { + const d = (b.probabilities?.[v] ?? 0) - (a.probabilities?.[v] ?? 0); + maxDelta = Math.max(maxDelta, Math.abs(d)); + return `${v} ${d >= 0 ? '+' : ''}${d.toFixed(3)}`; + }); + if (a.verdict !== b.verdict) flips++; + lines.push( + `${input.node.label} ${rule.id}: single=${a.verdict} batched=${b.verdict}${a.verdict === b.verdict ? '' : ' (FLIP)'}; delta ${deltas.join(', ')}`, + ); + } + }); + console.log( + `[drift ${name}] verdict flips: ${flips}; max |probability delta|: ${maxDelta.toFixed(3)}\n${lines.join('\n')}`, + ); + for (const r of [...single, ...batched]) { + for (const rule of [passFailRule, graded]) { + expect(Object.keys(rule.verdicts)).toContain((r[rule.id] as { verdict: string }).verdict); + } + } + }, + 180_000, + ); + } +}); From 70816492b8f40413d8cf65bba505c15db5cd0f8c Mon Sep 17 00:00:00 2001 From: Roger Barnes Date: Fri, 16 Oct 2026 12:08:36 +1100 Subject: [PATCH 8/9] refactor(gating): rename evaluateNodes/wave to evaluateNodeSet/depthNodes --- src/schema/gating.ts | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/src/schema/gating.ts b/src/schema/gating.ts index 18b911f1..70727871 100644 --- a/src/schema/gating.ts +++ b/src/schema/gating.ts @@ -277,14 +277,15 @@ export async function evaluateRulesWithGating( } /** - * Evaluate a set of nodes whose gating is already decided. Their semantic rules go to the judge together, - * so a judge with `evaluateNodes` can share model calls; violations are recorded in node order. + * Evaluate one set of nodes together, so their semantic rules go to the judge in one pass and a judge + * with `evaluateNodes` can share model calls. Gating for the set is already decided; violations are + * recorded in node order. */ - async function evaluateNodes(wave: readonly SpaceNode[]): Promise> { + async function evaluateNodeSet(nodes: readonly SpaceNode[]): Promise> { const prepared: PreparedNode[] = []; - for (const node of wave) prepared.push(await prepareNode(node)); + for (const node of nodes) prepared.push(await prepareNode(node)); if (judge) { - const pending = wave.map((node, i) => ({ node, rules: prepared[i]!.semantic })); + const pending = nodes.map((node, i) => ({ node, rules: prepared[i]!.semantic })); const judgements = await judgeNodes(judge, pending); prepared.forEach((p, i) => { p.judged(judgements[i] ?? []); @@ -295,28 +296,28 @@ export async function evaluateRulesWithGating( } // Non-hierarchy nodes sit outside the subtree cascade — always evaluated, never gated. - await evaluateNodes(graph.nonHierarchy); + await evaluateNodeSet(graph.nonHierarchy); const visited = new Set(); // Hierarchy nodes are judged one depth at a time: a node's verdicts decide whether its children are evaluated. - let wave: Array<{ node: SpaceNode; ancestorOrigin?: string }> = []; + let depthNodes: Array<{ node: SpaceNode; ancestorOrigin?: string }> = []; for (const node of [...graph.hierarchyRoots, ...graph.orphans]) { - wave.push({ node, ancestorOrigin: undefined }); + depthNodes.push({ node, ancestorOrigin: undefined }); } - while (wave.length > 0) { - const current: typeof wave = []; - for (const entry of wave) { + while (depthNodes.length > 0) { + const current: typeof depthNodes = []; + for (const entry of depthNodes) { if (visited.has(entry.node.title)) continue; visited.add(entry.node.title); current.push(entry); } - const isGated = (e: (typeof wave)[number]) => e.ancestorOrigin !== undefined || seedByTitle.has(e.node.title); + const isGated = (e: (typeof depthNodes)[number]) => e.ancestorOrigin !== undefined || seedByTitle.has(e.node.title); const toEvaluate = current.filter((e) => !isGated(e) || includeGated); - const evaluated = await evaluateNodes(toEvaluate.map((e) => e.node)); + const evaluated = await evaluateNodeSet(toEvaluate.map((e) => e.node)); const blockingByTitle = new Map(toEvaluate.map((e, i) => [e.node.title, evaluated[i]!.blockingRuleId])); - const next: typeof wave = []; + const next: typeof depthNodes = []; for (const { node, ancestorOrigin } of current) { const selfSeed = seedByTitle.get(node.title); const isGatedByAncestor = ancestorOrigin !== undefined; @@ -341,7 +342,7 @@ export async function evaluateRulesWithGating( next.push({ node: child, ancestorOrigin: childOrigin }); } } - wave = next; + depthNodes = next; } return { From 8beadfe4f472976c77a142b989c7ddd8dc181e11 Mon Sep 17 00:00:00 2001 From: Roger Barnes Date: Fri, 16 Oct 2026 12:08:38 +1100 Subject: [PATCH 9/9] docs: note that a failed split request errors all of a node's rules --- docs/semantic-rules.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/semantic-rules.md b/docs/semantic-rules.md index ff6babf9..a963ce97 100644 --- a/docs/semantic-rules.md +++ b/docs/semantic-rules.md @@ -109,7 +109,7 @@ Gating uses the severity of the verdict. A verdict with `error` severity gates t ## How judging works - **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge. -- **Grouping.** A judge on the `openrouter-decisions` or `typesafe-systemone` API judges each node ruleset group (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. This includes the bundled `openrouter-openai-decisions` provider. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request on tokens is sent whole as its own decision request. A node with more than 40 rules has its rules split across decision requests. Other providers, including `openai-decisions` and chat models, judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Grouping can shift verdicts and probabilities slightly compared with judging each node alone. +- **Grouping.** A judge on the `openrouter-decisions` or `typesafe-systemone` API judges each node ruleset group (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. This includes the bundled `openrouter-openai-decisions` provider. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request on tokens is sent whole as its own decision request. A node with more than 40 rules has its rules split across decision requests. If a node's questions are split across several requests and one fails, all of that node's rules are reported as errors. Other providers, including `openai-decisions` and chat models, judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Grouping can shift verdicts and probabilities slightly compared with judging each node alone. - **Gating.** structured-context does not judge a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated. - **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. If one decision request fails, only the rules on the nodes in that decision request report the error. - **Input.** The judge gets the node's title, type, parent titles and schema data, including frontmatter.