Say three verified upstream defects in the benchmarks' upstream limitations #18

Priority

medium

Components

• app/nullius (backend)
• app/web (routes)
• templates and static files
• docs and the site
• catalogue (benchmarks, agents, capabilities)
• the standard (ports, kits, relay, runs)
• CI, workflows and skills

Today

Three defects of the upstreams, at the commits the catalogue pins, are written only in each benchmark's upstream-settings.json as the reason an option is not offered. The benchmark pages do not say them, so a reader of LoCoMo, LongMemEval or τ³-bench does not learn why those choices are missing.

Each was checked against the pinned source, by reading it (nothing of the upstreams was run):

• LoCoMo at 3eb6f2c, task_eval/rag_utils.py: init_query_model('contriever') reads context_tokenizer, a local it assigns only in its dragon branch (line 80), so the call raises UnboundLocalError. get_embeddings with openai passes the whole input list for every batch of 24 (line 134), so gpt_utils.py:99 refuses the result (Lengths of embeddings and dialogs do not match) for more than 24 texts.
• LongMemEval at 9e0b455, src/index_expansion/temp_query_search_pruning.py:88: json.load(sys.argv[1]) is given the file's name, not the file, and stops with AttributeError: 'str' object has no attribute 'read'.
• τ³-bench at v1.0.1, src/tau2/runner/helpers.py:52 calls every task loader as task_loader(task_split_name=...); the loaders registered as telecom_full and telecom_small (domains/telecom/environment.py:188, 192) take no argument, so they raise TypeError.

Wanted

Each defect is one line of its benchmark's Upstream limitations, saying what is not offered because of it. They are the upstreams' to fix and are not filed there from here.

Acceptance

• LoCoMo's upstream limitations say the Contriever and OpenAI retrievers fail as published.
• LongMemEval's upstream limitations say time-aware query expansion stops as published.
• τ³-bench's upstream limitations say the telecom_full and telecom_small task sets fail to load at v1.0.1, and how the same tasks run.
• The Supported pages under docs/benchmarks/supported/ carry the same lines.

Dependencies and related

none