Skip to content

Phase 10: the ranking, and episode chunks #26

Closed

Description

@rohan-hotdata

This issue is the task brief for phase 10 of the second workstream, memory accuracy. docs/internal/plan.md holds the same plan, and the pull request keeps the two matched.

Goal

The default ranking of HotdataStore is the one that scores best, and the reason
for the gap between the two rankings is known. The reader gets the accuracy of phase 8
for fewer tokens, because the block holds the chunks of a session that match, and not
the whole session.

In phase 8, search at 64,000 characters scored 45 on both samples with about 12,800
reader tokens at the median. At 32,000 characters it scored 43 on sample A with 5,715
tokens. A session is one episode, so an episode that holds one needed turn still fills
the budget with every other turn. load already cuts a document into chunks of
chunk_chars, and each chunk is an episode.

The runs use the options of phase 9. A run with --keep captures once for each ranking,
chunk size, and commit of src/hotmemory, and each other budget or episodes value is a run
with --reuse. A commit that changes src/hotmemory makes the kept memory of earlier
commits unusable: delete it by hand after the runs that need it. Check each library change
first with --smoke --workers 4. The first smoke run after a change to capture pays
for extraction.

Tasks

  • 1. Explain the gap on the 419 retrieval questions, with no reader model. Run the fused
    query with one arm left out at a time: BM25 and content, content and cue, and BM25
    alone. Episodes have no cues, so in the retrieval run the cue arm adds nothing. Find
    which arm costs the most and on which question types.
  • 2. Run the memory arm on sample A with each ranking, with search at 64,000 characters.
    The fused memory of the phase 9 proof is kept, at commit b5443ac of src/hotmemory.
    The vector ranking needs one run with --keep.
  • 3. Propose the default ranking and the surface of episode chunks in docs/contracts.md,
    and stop for the owner's agreement. The ranking: the vector ranking, or a rebuilt
    fusion that ranks BM25 inside the scope or fuses two arms with no weights. The chunks,
    as a starting point: capture with episode takes chunk_chars, cuts the text at
    blank lines as load does, and stores chunk n under <episode>-<n>. The extractor
    still reads the whole text, so its prompt and its cache do not change. The proposal
    names which chunks a fact cites, and how "cited" uses them.
  • 4. Build the agreed change for both drivers, with CHANGELOG.md, docs/contracts.md,
    docs/guarantees.md, and the skill commands. If the default ranking changes, the
    oracle test and the benchmark runner follow.
  • 5. Run search with chunks of 1,000, 2,000, and 4,000 characters, at budgets of 8,000,
    16,000, and 32,000 characters, on sample A. Each chunk size is one run with --keep,
    and each budget after the first is a run with --reuse. Report accuracy against
    reader tokens, next to the phase 8 runs.
  • 6. Run the best configuration on sample B, rerun the retrieval score on all 419 questions
    with the agreed ranking, and diagnose the misses.

Acceptance criteria

  • docs/benchmarks.md reports the ablation of task 1, the retrieval score of the agreed
    ranking, and each chunk size and budget.
  • The search inside capture uses the same ranking as recall, and a test proves it.
  • A configuration with chunks scores at least 45 on sample A and 45 on sample B with
    fewer reader tokens at the median than 12,849, or the report states why none does.
  • MemoryStore and HotdataStore return the same block for the same calls with chunks.

Stop and ask if

  • The ablation shows that the gap comes from the data and not from an arm, for example
    that BM25 favors a long session.
  • Chunks need a new Store method or a new record field.
  • No chunk size reaches 43 on sample A within 6,000 reader tokens at the median.

Verification

  • Per commit: make verify. Per commit that touches a driver: make local-up, then make integration.
  • Each benchmark run writes its configuration and results to one file in bench/results/. The numbers in docs/benchmarks.md come from those files.
  • Before the pull request: the docs audit of the pr-workflow skill, and a search of the diff for the names of private repositories and deployment details.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions