Skip to content

feat(judge): batch Jev semantic questions across nodes that share a rule set - #161

Merged
mindsocket merged 9 commits into
mainfrom
issue-155
Oct 16, 2026
Merged

mindsocket merged 9 commits into
mainfrom
issue-155

Conversation

@mindsocket

@mindsocket mindsocket commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Closes #155. Stacked on #160.

Summary

  • Jev-API providers batch semantic questions across nodes that share a rule set (a node ruleset group), within one evaluation step of cascade gating.
  • Batching applies to providers on the openrouter-decisions and typesafe-systemone APIs, including the bundled openrouter-openai-decisions. openai-decisions and chat providers are unchanged.
  • A decision request holds at most 32,000 estimated tokens and 40 decision questions. Both caps are conservative guesses. A node over the token cap is sent whole; a node with more than 40 rules has its questions split across requests.
  • If one of a split node's requests fails, all of that node's rules are reported as errors.
  • Nested state is referenced as `nodes.nK`, following TypeSafe's guidance.

Drift

A live test (bun run test:judge, -t drift) judges the same nodes one at a time and batched. One run, 4 nodes × 2 rules:

  • openrouter-jev: 1 verdict flip of 8, largest probability change 0.41.
  • openrouter-openai-decisions: 0 flips of 8, largest change 0.24.

measured: one run only. The test has no single-against-single control, so run-to-run variation is not separated from the effect of batching. hypothesis: the flip is on a borderline node; the other provider judged it weak on its own.

@mindsocket
mindsocket added this pull request to stack #162 October 16, 2026 01:21
@mindsocket mindsocket changed the title issue 155 feat(judge): batch Jev semantic questions across nodes that share a rule set Oct 16, 2026
@mindsocket
mindsocket removed this pull request from stack #162 October 16, 2026 03:55
@mindsocket
mindsocket added this pull request to stack #167 October 16, 2026 03:55
@mindsocket
mindsocket marked this pull request as ready for review October 16, 2026 03:56
@claude

claude Bot commented Oct 16, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

@mindsocket
mindsocket force-pushed the issue-155 branch 3 times, most recently from 5b77b7e to 6fa1b76 Compare October 16, 2026 04:12

@mindsocket mindsocket left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix these in the spirit of the feedback, on the top of the PR stack, since the docs have changed already in the stack.

Comment thread docs/config.md
| `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities | `openrouter-jev`, `openrouter-openai-decisions` |
| `typesafe-systemone` | Same decisions shape (currently) | `typesafe-jev` |
| `openai-decisions` | OpenAI's Decisions API: same concept, its own request/response shape | `openai-decisions` |
| `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities; nodes with the same rules share a decision request | `openrouter-jev`, `openrouter-openai-decisions` |

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not relevant to config. Don't add trivia to docs.

Comment thread docs/semantic-rules.md

- **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge in one call.
- **Order.** On each node, expression rules run first. Then structured-context applies each semantic rule's [`check`](#preconditions-check) and sends the rules that pass to the judge.
- **Grouping.** A judge on the `openrouter-decisions` or `typesafe-systemone` API judges each node ruleset group (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. This includes the bundled `openrouter-openai-decisions` provider. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request on tokens is sent whole as its own decision request. A node with more than 40 rules has its rules split across decision requests. If a node's questions are split across several requests and one fails, all of that node's rules are reported as errors. Other providers, including `openai-decisions` and chat models, judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Grouping can shift verdicts and probabilities slightly compared with judging each node alone.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is way too much detail for user docs. Don't hardcode and repeat so much implementation detail and focus on "need to know" stuff that people using the project might care about. Fine to mention the high level approach in case someone's curious, but they can dig into the code themselves beyond that.

Comment thread docs/semantic-rules.md
- **Grouping.** A judge on the `openrouter-decisions` or `typesafe-systemone` API judges each node ruleset group (see [concepts](concepts.md#semantic-rules)) in as few decision requests as it can, with one decision question per node and rule. This includes the bundled `openrouter-openai-decisions` provider. A decision request is limited to about 32,000 estimated input tokens and 40 decision questions, so a larger group is split into several decision requests, never within a node. A node that fills a decision request on tokens is sent whole as its own decision request. A node with more than 40 rules has its rules split across decision requests. If a node's questions are split across several requests and one fails, all of that node's rules are reported as errors. Other providers, including `openai-decisions` and chat models, judge one node per request. Within a space, the nodes at one hierarchy depth are judged together, because their verdicts decide whether the next depth is judged. Grouping can shift verdicts and probabilities slightly compared with judging each node alone.
- **Gating.** structured-context does not judge a gated node. [Coverage output](rules.md#coverage-output) shows which nodes were gated.
- **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate.
- **Judge errors.** If the judge fails (for example, a network error or output that structured-context cannot read), structured-context reports a violation. A judge error does not gate. If one decision request fails, only the rules on the nodes in that decision request report the error.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, irrelevant. The output will tell people this if it happens.

Base automatically changed from issue-156 to main October 16, 2026 04:34
…ule set

Nodes with the same semantic rule set now share one Jev request: all nodes
go under state.nodes, with one question per node and rule. A group is split
by an estimated-token cap and a question cap, never within a node; a lone or
oversized node is sent whole as a plain single-node request. A failed request
marks only its own nodes' rules as errors. Chat providers are unchanged.

Gating now judges one hierarchy depth at a time so a batch never needs a
verdict that is still pending; violation order per node is unchanged.

measured: probe against openrouter ~typesafe/jev-latest (jev-1.14-20261014)
- one request with 3 nodes in state.nodes and 3 questions returned 3 answers
  keyed by question id (HTTP 200).
- drift, same question alone vs in the 3-node batch: mean |dp| 0.047, max
  0.12 (n1 0.98 -> 1.00, n2 unchanged, n3 0.97 -> 0.85 on fail; no verdict
  flipped). 3 nodes, 1 rule: indicative only.
- size: 45k-token state and a 250k-token state were both accepted (HTTP 200);
  no 32k limit was reached, so JEV_BATCH_MAX_TOKENS (32k, from issue #152,
  unverified) is a conservative guard, not a hard limit.
…-state reference per TypeSafe guidance

A node with more rules than the question cap now has its DecisionQuestions split across
DecisionRequests. Batched questions refer to the node as `nodes.nK`. Drop spec history from the
token-cap comment. Add tests for a missing batched answer and rule ids containing `__n0`.
@mindsocket
mindsocket merged commit cc3e135 into main Oct 16, 2026
2 checks passed
@mindsocket
mindsocket deleted the issue-155 branch October 16, 2026 06:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Jev batching: one call per node; consider fewer, larger calls (per space)

1 participant