Repository navigation
feat(judge): batch Jev semantic questions across nodes that share a rule set - #161
Conversation
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
5b77b7e to
6fa1b76
Compare
mindsocket
left a comment
There was a problem hiding this comment.
Fix these in the spirit of the feedback, on the top of the PR stack, since the docs have changed already in the stack.
| | `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities | `openrouter-jev`, `openrouter-openai-decisions` | | ||
| | `typesafe-systemone` | Same decisions shape (currently) | `typesafe-jev` | | ||
| | `openai-decisions` | OpenAI's Decisions API: same concept, its own request/response shape | `openai-decisions` | | ||
| | `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities; nodes with the same rules share a decision request | `openrouter-jev`, `openrouter-openai-decisions` | |
There was a problem hiding this comment.
This is not relevant to config. Don't add trivia to docs.
|
|
||
| - **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge in one call. | ||
| - **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge. | ||
| - **Grouping.** A judge on the `openrouter-decisions` or `typesafe-systemone` API judges each node ruleset group (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. This includes the bundled `openrouter-openai-decisions` provider. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request on tokens is sent whole as its own decision request. A node with more than 40 rules has its rules split across decision requests. If a node's questions are split across several requests and one fails, all of that node's rules are reported as errors. Other providers, including `openai-decisions` and chat models, judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Grouping can shift verdicts and probabilities slightly compared with judging each node alone. |
There was a problem hiding this comment.
This is way too much detail for user docs. Don't hardcode and repeat so much implementation detail and focus on "need to know" stuff that people using the project might care about. Fine to mention the high level approach in case someone's curious, but they can dig into the code themselves beyond that.
| - **Grouping.** A judge on the `openrouter-decisions` or `typesafe-systemone` API judges each node ruleset group (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. This includes the bundled `openrouter-openai-decisions` provider. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request on tokens is sent whole as its own decision request. A node with more than 40 rules has its rules split across decision requests. If a node's questions are split across several requests and one fails, all of that node's rules are reported as errors. Other providers, including `openai-decisions` and chat models, judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Grouping can shift verdicts and probabilities slightly compared with judging each node alone. | ||
| - **Gating.** structured-context does not judge a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated. | ||
| - **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. | ||
| - **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. If one decision request fails, only the rules on the nodes in that decision request report the error. |
There was a problem hiding this comment.
Again, irrelevant. The output will tell people this if it happens.
…ule set Nodes with the same semantic rule set now share one Jev request: all nodes go under state.nodes, with one question per node and rule. A group is split by an estimated-token cap and a question cap, never within a node; a lone or oversized node is sent whole as a plain single-node request. A failed request marks only its own nodes' rules as errors. Chat providers are unchanged. Gating now judges one hierarchy depth at a time so a batch never needs a verdict that is still pending; violation order per node is unchanged. measured: probe against openrouter ~typesafe/jev-latest (jev-1.14-20261014) - one request with 3 nodes in state.nodes and 3 questions returned 3 answers keyed by question id (HTTP 200). - drift, same question alone vs in the 3-node batch: mean |dp| 0.047, max 0.12 (n1 0.98 -> 1.00, n2 unchanged, n3 0.97 -> 0.85 on fail; no verdict flipped). 3 nodes, 1 rule: indicative only. - size: 45k-token state and a 250k-token state were both accepted (HTTP 200); no 32k limit was reached, so JEV_BATCH_MAX_TOKENS (32k, from issue #152, unverified) is a conservative guard, not a hard limit.
…equest, DecisionQuestion
…-state reference per TypeSafe guidance A node with more rules than the question cap now has its DecisionQuestions split across DecisionRequests. Batched questions refer to the node as `nodes.nK`. Drop spec history from the token-cap comment. Add tests for a missing batched answer and rule ids containing `__n0`.
…e batching terms in prose
Closes #155. Stacked on #160.
Summary
openrouter-decisionsandtypesafe-systemoneAPIs, including the bundledopenrouter-openai-decisions.openai-decisionsand chat providers are unchanged.`nodes.nK`, following TypeSafe's guidance.Drift
A live test (
bun run test:judge,-t drift) judges the same nodes one at a time and batched. One run, 4 nodes × 2 rules:openrouter-jev: 1 verdict flip of 8, largest probability change 0.41.openrouter-openai-decisions: 0 flips of 8, largest change 0.24.measured: one run only. The test has no single-against-single control, so run-to-run variation is not separated from the effect of batching. hypothesis: the flip is on a borderline node; the other provider judged it
weakon its own.