Repository navigation
Description
Context
Needle already has a frozen-corpus manifest, typed BenchmarkOracle, content
digests, focused-test preflight, campaign cost readiness, and response-quality
evaluation. The current public manifest contains four tasks. Two task IDs are
labelled holdout, but their prompts, oracle files, source needles, focused
tests, and expected evidence have already been published in Git history. They
are therefore public development fixtures and cannot support an independent
holdout claim.
The current runner boundary also does not completely prevent oracle material
from reaching a model-facing process. A real holdout package must remain
outside the public checkout and must fail closed until both independently
sealed task/oracle material and a power-analysis plan are supplied.
Expected outcome
Needle has a versioned, answer-free sealed-corpus protocol that can be fully
implemented and tested with synthetic public fixtures. Real holdout task and
oracle bytes enter only through an evaluator-owned external bundle and never
through model, worker, lifecycle, launch-payload, or ordinary log surfaces.
The protocol accepts an immutable PowerPlan produced by #7, binds its digest into the
final schedule, and checks exact per-route repetition counts without deriving
or inventing them. In the public checkout, or whenever the external sealed
bundle or validated power plan is absent, preflight truthfully returns
provider_run_ready = false. Supplying real never-published holdouts and
running a powered campaign remain a separate maintainer-owned activity.
Scope
- Introduce an answer-free corpus/campaign schema revision for calibration,
synthetic fixtures, and externally sealed holdouts. Bind task identity,
repository URL/SHA, route, split/evidence class, prompt and focused-command
policy without placing evaluator answers, source needles, expected symbols,
oracle paths, or raw oracle bytes in model-facing manifests. - Treat the two currently public task IDs ending in
-holdoutas synthetic
development fixtures that are permanently ineligible for independent
holdout evidence. Preserve them only where useful for deterministic schema,
evaluator, and adversarial tests; do not relabel their results as statistical
evidence. - Define an evaluator-only external bundle contract for real holdout task and
oracle bytes. Bind every sealed entry to a task ID, repository SHA, schema,
focused-test identity, and content digest. The evaluator resolves and reads
the material only after an arm finishes; model-facing processes receive no
filesystem handle or answer-bearing metadata. - Add a two-stage schedule contract. Calibration observations are validated and
supplied to Harden the paired-analysis, PowerPlan, and BCa final benchmark gate #7 to produce an immutablePowerPlan. The frozen final schedule
then references that plan digest and records its exact per-route required
pair counts. Every real holdout task/repetition requiresFinalArm::ALL,
calibration and holdout remain disjoint, andautomatic_retries = false. - Permit synthetic public
PowerPlanand sealed-bundle fixtures for offline
protocol tests. Synthetic values exercise the complete schema but are
explicitly ineligible for provider evidence or economic claims. - Extend
preflight_frozen_corpus/corpus-preflightto validate source
cleanliness, origin/SHA, safe paths, schema and digest binding, task and arm
uniqueness, focused-test policy, campaign/cost/power-plan digests, bundle
availability, and the non-adaptive retry policy. - Create bounded model/worker/lifecycle launch payloads containing only the
task prompt and approved execution metadata. They must omit oracle paths,
oracle digests where those act as answer handles, evidence needles, expected
files/symbols/claims, evaluator answers, and external bundle locations. - Make every missing, stale, malformed, mutated, leaked, duplicate, or
incompatible input fail closed with bounded diagnostics. Absence of a real
sealed bundle or validatedPowerPlanis an expected not-ready state, not a
reason to invent data. - Keep evaluator results, oracle version/digest, model output, infrastructure
failure, and focused-test evidence separate. Editing an oracle or schedule
after freeze invalidates the package and cannot retroactively validate an
observation. - Add offline fixture tests for valid synthetic packages, answer-bearing launch
payloads, public-task misuse, missing/mutated external bundles, stale power
plans, schedule mismatch, source drift, duplicate identities, malformed
commands, and campaign/digest mismatch.
Non-goals
- Creating, publishing, or committing real holdout prompts, task identities,
oracle bytes, needles, expected answers, or evaluator secrets. - Producing the real power-plan count, running paid provider arms, selecting
models/pricing, or making a statistical savings claim. - Treating the currently published
-holdouttasks as recoverable secret
holdouts; Git history makes them permanently unsuitable for that purpose. - Hard-coding a global holdout repetition count, deriving it inside corpus
preflight, or accepting a count without a validated Harden the paired-analysis, PowerPlan, and BCa final benchmark gate #7PowerPlandigest. - Changing semantic cache/worker behavior, verifier TestPlan execution, adding
a remote corpus service, or authorizing automatic retries or publication.
Acceptance criteria
- The answer-free schema is versioned and rejects unknown fields. Public
manifests and model-facing launch payloads contain no oracle path, raw
oracle bytes, source needles, expected file/symbol/claim text, evaluator
result, or external bundle location. - The existing public task IDs
ripgrep-no-ignore-vcs-locate-holdoutand
ripgrep-null-data-trace-holdoutare explicitly marked synthetic or
otherwise prevented from contributing to independent holdout evidence. - A real holdout entry is usable only through an evaluator-owned external
bundle with matching schema, task, repository SHA, focused-test identity,
and digest; the arm process cannot resolve or read that bundle. - The final schedule references one immutable
PowerPlandigest from Harden the paired-analysis, PowerPlan, and BCa final benchmark gate #7
and exactly matches its per-route required pair counts. Missing, stale,
mismatched, or synthetic-only plans returnprovider_run_ready = false. - Every real holdout task/repetition schedules
FinalArm::ALL; calibration
entries remain separate, automatic retries are disabled, and a schedule
digest detects omissions, substitutions, reordered identities, or extra
repetitions. - A complete synthetic fixture can exercise schema, scheduling, evaluator,
and mutation checks offline, but is clearly marked ineligible for provider
evidence. Without a real sealed bundle, preflight remains not ready. - Preflight validates exact source origin/SHA and cleanliness, safe paths,
cost/campaign/power-plan digests, focused-command policy, unique task/arm
identities, and bundle immutability. Every violation produces a bounded
fail-closed reason. - Quality failure, stale hit, infrastructure failure, evaluator error,
bundle mutation, and oracle mismatch remain explicit and block promotion;
no failed record or missing arm is silently omitted. - Offline adversarial tests cover public-task misuse, oracle/needle leakage,
missing or mutated bundle, stale or mismatched power plan, schedule count
mismatch, duplicate identity, malformed command, source drift, and
campaign digest mismatch.
Validation
cargo test --locked -p needle-bench
cargo test --locked -p needle-bench corpus
cargo test --locked -p needle-bench final_gate
cargo run --locked -p needle-bench --bin corpus-preflight -- --workspace-root . --manifest benchmarks/corpus/router-cache/manifest.json --source-repository target/router-cache-source --output target/corpus-preflight/report.json
The checked-in public fixture must finish without provider calls and report
provider_run_ready = false when no external sealed bundle and validated real
PowerPlan are supplied. Focused tests may run only when explicitly requested
against a clean pinned source checkout.
Starting points
crates/needle-bench/src/corpus.rs- corpus/oracle types, source validation,
evaluator checks, cost readiness, and fail-closed preflight.crates/needle-bench/src/final_gate.rs- manifest, task/arm identities,
observations, power estimation, and the current final gate.crates/needle-app/src/minimal_live_pilot/protocol.rs- current protocol,
model-facing task selection, oracle loading, and quality-spec boundary.benchmarks/corpus/router-cache/manifest.json,campaign.json, and
oracles/- public development corpus that must not be represented as an
independent holdout.docs/BENCHMARKING.mdandPROJECT_STATUS.md- evidence and paid-run
boundaries.
Dependencies
#7 owns the deterministic,
digest-bound PowerPlan produced from validated calibration observations. This
issue owns the answer-free corpus, external evaluator bundle, and frozen
schedule that references that plan. Both implementations can use synthetic
fixtures and may proceed in parallel; neither requires nor authorizes a live
provider run.
The future maintainer-only powered campaign owns creation and custody of fresh,
never-published holdout tasks/oracles, collection of calibration observations,
freezing the real power plan and final schedule, provider budget approval, and
any public statistical interpretation.
Ownership and risk
The benchmark maintainer owns the corpus protocol and evaluator boundary.
Oracle leakage or reuse of public answers invalidates an evidence claim;
overly-strict preflight only delays a run. Keep evaluator access physically and
logically separate, bind every transition by digest, and preserve truthful
provider_run_ready semantics.
Activity
[-]Prepare a holdout-ready benchmark corpus with sealed independent oracles[/-][+]Build the sealed holdout corpus protocol and evaluator boundary[/+]