Skip to content

Jev batching: one call per node; consider fewer, larger calls (per space) #155

Closed

Description

@mindsocket

Current behavior

JevJudgeProvider.evaluate makes one request per node (state = node, questions = that node's semantic rules). All of a node's semantic rules already share one call; nodes do not.

measured: src/judge/provider/jev.ts — evaluate(input) takes a single SemanticJudgeInput (one node) and posts one fetch; src/schema/gating.ts calls judgeNode once per node inside evaluateApplicableRules.

Motivation

Roger's review comment on PR #151 (docs/semantic-rules.md):

Side note: check what we do with jev to batch rule execution, and if it can be made more efficient (fewer calls at potentially slightly more cost) then ensure an issue is tracked to implement that.

A space with N ungated nodes and semantic rules costs N sequential HTTP round-trips. Fewer, larger calls (batch several nodes' questions into one request) could cut wall-clock latency and per-call overhead, at potentially slightly higher per-token cost.

Questions to settle

  1. Does the OpenRouter Decisions API accept multiple states per request (array of states, or a batch endpoint)? If not, this reduces to client-side concurrency (parallelise per-node calls) rather than batching.
  2. If batching is possible: latency/throughput tradeoff, cost per call vs per token, and how per-node errors map back to nodes.
  3. Should batching also apply to the chat providers (anthropic/openai), where fan-out/prompt-batching patterns apply?

Related patterns: https://docs.typesafe.ai/patterns/fan-out

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions