Repository navigation
Description
Context
The checked-in benchmarks/corpus/router-cache directory contains the v4
manifest, campaign, schedule, synthetic PowerPlan and observations, legacy
fixtures, public oracles, and a synthetic sealed bundle. The benchmark index
summarizes their evidence boundary, but the directory has no local guide that
maps each file to its producer, consumer, identity, and eligibility.
This makes it unnecessarily easy for a new contributor to mistake synthetic or
public material for a production holdout or provider-backed observation.
Outcome
Add benchmarks/corpus/router-cache/README.md as a concise fixture map, and link
it from benchmarks/README.md. The guide documents existing files only and does
not change the corpus protocol or any benchmark result.
Scope
- Inventory
manifest.json,campaign.json,power-plan.json,
schedule.json,cost-model.json,minimal-live-pilot.json, the synthetic
calibration/final observation JSONL files,oracles/,synthetic-sealed/,
and the legacy files. - For each group, state its schema/identity role, which current command or
evaluator consumes it (or that it is legacy/unconsumed), and whether it is
answer-free, answer-bearing, evaluator-only, synthetic, legacy, or report
input. - Add a short artifact flow from calibration observations to PowerPlan,
schedule, arm execution, sealed evaluation, FinalObservation, and final
report. Mark which boundaries are implemented, fixture-only, or still lack a
production runner/recorder; do not invent commands. - State prominently that checked-in public/synthetic artifacts are permanently
ineligible as independent provider holdouts or general economic evidence. - Explain that the synthetic sealed bundle exists for evaluator tests and must
never be exposed to an arm runner or provider prompt. - Link the directory guide from
benchmarks/README.md.
Non-goals
- Changing JSON/JSONL content, schema, digests, tasks, schedules, PowerPlan,
oracles, evaluator behavior, statistics, or accepted evidence reports. - Adding a provider runner, private holdout material, credentials, paid runs,
new economic claims, or publication authorization. - Moving or renaming benchmark files.
Acceptance criteria
- Every current top-level router-cache fixture and both subdirectories are
represented in the guide. - Schema/identity, producer/consumer, and evidence eligibility statements
match current source anddocs/BENCHMARKING.md. - Implemented, fixture-only, legacy, and missing production boundaries are
explicit; the guide does not present the open broker/recorder work as
shipped. - The guide clearly separates public answer-free launch material, public
legacy oracles, evaluator-only synthetic sealed material, and observation
inputs. - It states that neither the synthetic PowerPlan nor synthetic final
observations support a provider or savings claim. - No fixture bytes, paths, digests, or behavior change.
- The new link from
benchmarks/README.mdresolves.
Starting points
benchmarks/README.mddocs/BENCHMARKING.mdbenchmarks/corpus/router-cache/manifest.jsonbenchmarks/corpus/router-cache/campaign.jsonbenchmarks/corpus/router-cache/power-plan.jsonbenchmarks/corpus/router-cache/schedule.jsoncrates/needle-bench/src/schedule.rscrates/needle-bench/src/sealed_oracle.rscrates/needle-bench/src/final_gate.rs
Validation
Run git diff --check, verify every documented relative path exists, and check
the artifact-flow terminology against the current benchmark schemas and
docs/BENCHMARKING.md. No Cargo build, provider credential, sealed private
material, or benchmark execution is required.
Activity