You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Jev batching: one call per node; consider fewer, larger calls (per space) #155
JevJudgeProvider.evaluate makes one request per node (state = node, questions = that node's semantic rules). All of a node's semantic rules already share one call; nodes do not.
measured: src/judge/provider/jev.ts — evaluate(input) takes a single SemanticJudgeInput (one node) and posts one fetch; src/schema/gating.ts calls judgeNode once per node inside evaluateApplicableRules.
Motivation
Roger's review comment on PR #151 (docs/semantic-rules.md):
Side note: check what we do with jev to batch rule execution, and if it can be made more efficient (fewer calls at potentially slightly more cost) then ensure an issue is tracked to implement that.
A space with N ungated nodes and semantic rules costs N sequential HTTP round-trips. Fewer, larger calls (batch several nodes' questions into one request) could cut wall-clock latency and per-call overhead, at potentially slightly higher per-token cost.
Questions to settle
Does the OpenRouter Decisions API accept multiple states per request (array of states, or a batch endpoint)? If not, this reduces to client-side concurrency (parallelise per-node calls) rather than batching.
If batching is possible: latency/throughput tradeoff, cost per call vs per token, and how per-node errors map back to nodes.
Should batching also apply to the chat providers (anthropic/openai), where fan-out/prompt-batching patterns apply?
Activity