Skip to content

Phase 9: faster evaluation #25

Closed

Description

@rohan-hotdata

This issue is the task brief for phase 9 of the second workstream, memory accuracy. docs/internal/plan.md holds the same plan, and the pull request keeps the two matched.

Goal

A run that changes only recall or the reader takes minutes, not 20 minutes, and a
smoke run checks a change in a few minutes. This phase changes bench/ only.

Phase 8 ran seven configurations on sample A. Each run captured the same memory again,
because the episodes value and the budget act after capture. A run took 1,133 to
2,085 seconds with 4 workers (observed, duration_s in the result files), and most of
that time was capture: about four engine round trips for each of about 2,700 sessions
(read from the code, not timed).
The diagnosis replays capture again. An internet outage stopped four runs, and each
lost every finished question, because a run writes its result file only at the end.

Tasks

  • 1. Keep the memory of a run. With --keep, the runner does not empty the namespaces, and
    it keeps one database for each question. The name of the database states what decides
    the memory: the sample, the extractor variant, the episode setting, the fact sources,
    the ranking, the last commit that changed src/hotmemory, and the question id. The
    runner records the name in the result file. One database holds one question, because
    BM25 ranks the whole table before the namespace filter applies. In a shared table, the
    memory of other questions changes the block and the known facts of capture. The
    owner agreed to this layout on 2026-10-16.
  • 2. Reuse the memory. With --reuse, the runner opens the kept databases, skips
    capture, and runs recall, the reader, and the judge. It refuses to run if a
    database with the expected name does not exist.
  • 3. Let bench.diagnose read a kept database, so that it does not replay capture.
  • 4. A smoke run. --smoke runs 18 questions, 3 of each type from sample A, on
    MemoryStore, in the process. A committed file lists the 18 ids. The smoke run checks
    that a change runs and that the block looks right. It decides no gain: 18 questions can
    be off by about 23 points, and MemoryStore ranks by vector only.
  • 5. Keep finished questions. The runner appends each finished row to a file as it goes,
    and --resume skips the rows that the file holds. A failed call to a model fails its
    question and not the whole run.
  • 6. Prove the reuse. Run search at 64,000 characters on sample A with --keep, then
    search at 32,000 and cited at 64,000 with --reuse. Each answer must equal the
    answer of the phase 8 run of the same configuration, from the caches. Record the time
    of each run.
  • 7. Update bench/README.md and docs/benchmarks.md with the new options and the times.

Acceptance criteria

  • A run with --reuse gives the same block, answer, and mark for each question as a run
    that captures, and it takes less than 5 minutes on sample A with cached replies.
  • A smoke run with cached replies takes less than 3 minutes.
  • A run that loses its model connection keeps its finished questions, and --resume
    completes it.
  • No file in src/hotmemory changes.

Stop and ask if

  • A reused memory gives a different block than a fresh capture for any question.
  • The kept databases of one sample need more than 10 GB on the local stack.

Verification

  • Per commit: make verify. Per commit that touches a driver: make local-up, then make integration.
  • Each benchmark run writes its configuration and results to one file in bench/results/. The numbers in docs/benchmarks.md come from those files.
  • Before the pull request: the docs audit of the pr-workflow skill, and a search of the diff for the names of private repositories and deployment details.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions