Run LongMemEval on one question type #15

Priority

medium

Components

• app/nullius (backend)
• app/web (routes)
• templates and static files
• docs and the site
• catalogue (benchmarks, agents, capabilities)
• the standard (ports, kits, relay, runs)
• CI, workflows and skills

Today

The LongMemEval driver's units() (driver.py:136) returns every question of the dataset. --limit N takes the first N in dataset order and --shard splits by the unit id's hash, so one question type cannot be run alone, for example the 78 knowledge-update questions that test updated facts.

Wanted

A bench option question_type with the values all (the default), knowledge-update, temporal-reasoning, multi-session, single-session-user, single-session-assistant and single-session-preference, for anyone measuring one memory ability. It keeps only that type's questions, abstention variants included, in dataset order and before --limit. I run this in a user-catalogue copy of the benchmark as --bench-option question_type=knowledge-update.

Acceptance

• --bench-option question_type=knowledge-update on longmemeval_oracle runs exactly its 78 knowledge-update questions.
• Without the option, the run covers every question, as today.

Dependencies and related

none