LongMemEval: offer the upstream's index expansion from the authors' released caches #36

Priority

extreme

Components

• app/nullius (backend)
• app/web (routes)
• templates and static files
• docs and the site
• catalogue (benchmarks, agents, capabilities)
• the standard (ports, kits, relay, runs)
• CI, workflows and skills

Today

LongMemEval's retrieval can expand each key of its index with facts, keyphrases or a summary the authors extracted beforehand (run_retrieval.py --index_expansion_method, five methods, six ways to join). The driver always passes --index_expansion_method none, and the settings inventory records the option as not offered: "the expansions are read from caches the authors released on Google Drive without a digest ... there is no pinned copy to fetch".

The released archive downloads without an account (68 MB, five JSON files, one per method), so a digest can be taken here and the file pinned as the datasets are. What is missing is a way for a benchmark to declare such a file: a definition can name datasets and nothing else to fetch.

Wanted

• A benchmark may declare files beside its datasets (files: a URL and its sha256), and an option's value may need one (needs.files). The run downloads what the chosen values need, checked by its sha256, and tells the driver where it is.
• LongMemEval offers the upstream's index expansion for its flat retrievers: the method, by the retriever's granularity, and how the expansion joins the key, read from the authors' released caches.

Time-aware query expansion stays out: it fails as published.

Acceptance

• files and needs.files are validated: a name, an https URL, a sha256; a needed file must be declared.
• A run whose option values need a file downloads it once, refuses a file whose sha256 differs, and gives the driver its path (NULLIUS_FILE_<NAME>).
• LongMemEval's options offer the five expansion methods, each only with a retriever of its granularity, and the six join modes.
• The driver passes the method, the join mode and the method's cache file to run_retrieval.py, the cache read out of the archive by its exact name.
• A run with an expansion on a few questions ends with a retrieval log and an answer per question.
• The settings inventory and the benchmark's limitation say what is offered now.

Dependencies and related

none