Repository navigation
Description
This issue is the task brief for phase 9 of the second workstream, memory accuracy. docs/internal/plan.md holds the same plan, and the pull request keeps the two matched.
Goal
A run that changes only recall or the reader takes minutes, not 20 minutes, and a
smoke run checks a change in a few minutes. This phase changes bench/ only.
Phase 8 ran seven configurations on sample A. Each run captured the same memory again,
because the episodes value and the budget act after capture. A run took 1,133 to
2,085 seconds with 4 workers (observed, duration_s in the result files), and most of
that time was capture: about four engine round trips for each of about 2,700 sessions
(read from the code, not timed).
The diagnosis replays capture again. An internet outage stopped four runs, and each
lost every finished question, because a run writes its result file only at the end.
Tasks
- 1. Keep the memory of a run. With
--keep, the runner does not empty the namespaces, and
it keeps one database for each question. The name of the database states what decides
the memory: the sample, the extractor variant, the episode setting, the fact sources,
the ranking, the last commit that changedsrc/hotmemory, and the question id. The
runner records the name in the result file. One database holds one question, because
BM25 ranks the whole table before the namespace filter applies. In a shared table, the
memory of other questions changes the block and the known facts ofcapture. The
owner agreed to this layout on 2026-10-16. - 2. Reuse the memory. With
--reuse, the runner opens the kept databases, skips
capture, and runsrecall, the reader, and the judge. It refuses to run if a
database with the expected name does not exist. - 3. Let
bench.diagnoseread a kept database, so that it does not replaycapture. - 4. A smoke run.
--smokeruns 18 questions, 3 of each type from sample A, on
MemoryStore, in the process. A committed file lists the 18 ids. The smoke run checks
that a change runs and that the block looks right. It decides no gain: 18 questions can
be off by about 23 points, andMemoryStoreranks by vector only. - 5. Keep finished questions. The runner appends each finished row to a file as it goes,
and--resumeskips the rows that the file holds. A failed call to a model fails its
question and not the whole run. - 6. Prove the reuse. Run
searchat 64,000 characters on sample A with--keep, then
searchat 32,000 andcitedat 64,000 with--reuse. Each answer must equal the
answer of the phase 8 run of the same configuration, from the caches. Record the time
of each run. - 7. Update
bench/README.mdanddocs/benchmarks.mdwith the new options and the times.
Acceptance criteria
- A run with
--reusegives the same block, answer, and mark for each question as a run
that captures, and it takes less than 5 minutes on sample A with cached replies. - A smoke run with cached replies takes less than 3 minutes.
- A run that loses its model connection keeps its finished questions, and
--resume
completes it. - No file in
src/hotmemorychanges.
Stop and ask if
- A reused memory gives a different block than a fresh capture for any question.
- The kept databases of one sample need more than 10 GB on the local stack.
Verification
- Per commit:
make verify. Per commit that touches a driver:make local-up, thenmake integration. - Each benchmark run writes its configuration and results to one file in
bench/results/. The numbers indocs/benchmarks.mdcome from those files. - Before the pull request: the docs audit of the
pr-workflowskill, and a search of the diff for the names of private repositories and deployment details.
Activity