Skip to content

τ³-bench: offer the reranker and qwen search methods through the relay #28

Description

@aitianshi

Priority

extreme

Components

  • app/nullius (backend)
  • app/web (routes)
  • templates and static files
  • docs and the site
  • catalogue (benchmarks, agents, capabilities)
  • the standard (ports, kits, relay, runs)
  • CI, workflows and skills

Today

τ³-bench's banking_knowledge offers 11 of the upstream's search methods. Nine are recorded as not offered in upstream-settings.json, for two reasons, both checked against the upstream at v1.0.1:

  • The four reranker methods (bm25_reranker, bm25_reranker_grep, openai_embeddings_reranker, openai_embeddings_reranker_grep). The pointwise reranker asks a chat model through the OpenAI client (knowledge/postprocessors/pointwise_llm_reranker.py:59, model gpt-5.2), which reads OPENAI_BASE_URL. In a run that variable is the embeddings route, whose one model is text-embedding-3-large, or it is not set at all.
  • The five qwen methods (qwen_embeddings, ..._grep, ..._reranker, ..._reranker_grep, alltools-qwen). Their embedder calls OpenRouter at an address written in the code (knowledge/embedders/openrouter_embedder.py:118, base_url="https://openrouter.ai/api/v1") with OPENROUTER_API_KEY, past the relay, which never hands a driver a key.

Wanted

The nine methods offered, with every call still through the run's relay.

  • A relay route that sends a request by what it asks for: an embeddings request as the embedding model, a chat or responses request as the reranker's model. The driver's one OPENAI_BASE_URL then serves both, and a method without embeddings (bm25_reranker) gets the route for its reranker alone.
  • For qwen, a route to OpenRouter on the run's relay, and the embedder pointed at it. The address is written in the upstream's code, so this needs either a change upstream (reading it from the environment) or one line of ours setting it in the driver; the second is code beyond glue and is the owner's yes to give.

Acceptance

  • A relay route can pin one model for embeddings requests and another for chat requests, each recorded as the model sent.
  • The four reranker methods are values of retrieval, each with what it needs said (needs).
  • The owner has said how the qwen embedder is pointed at the relay.
  • The five qwen methods are values of retrieval, or stay recorded as not offered with the reason.
  • A run of each offered method on a few tasks makes every model call through the relay (its record shows the reranker's and the embedder's calls).

Dependencies and related

none

Activity

  1. self-assigned this
    on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions