Repository navigation
knowledge: distill AutoTuneBench treatment and trial validity - #306
Conversation
Distinguish treatment exposure, arm eligibility and observed protocol checks under the existing benchmark owner, retaining source conflicts and limits. Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (author) Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
Regenerate the native skill index and coverage from canonical source material while preserving existing owner metadata and validation states. Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (author) Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
python tools/build_skill_index.py --source-revision c6e91d437308ef8d5565528206fa554a3490e806 --checkThe declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Selection/task metadata (not a relevance verdict): {"adapter":"audit","basis":"trusted_path_mapping","matched_paths":["src/inferrsi/data/skill_library/methods/inference-benchmark-design/SKILL.md","src/inferrsi/data/skill_library/methods/inference-benchmark-design/metadata.json","src/inferrsi/data/skill_library/methods/inference-benchmark-design/references/autotunebench-source.json","src/inferrsi/data/skill_library/methods/inference-benchmark-design/references/autotunebench-trial-validity.md"],"tasks":[{"evidence_scope":"http_fixed_workload","id":"audit-threshold","request":"Audit the fixed input report using its contract. Compute the median of per-trial request means, including every request in each trial, and the percentage reduction relative to the baseline aggregate.","split":"holdout"}]}Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Projection check selection (not an executed result): {"declared_in":"src/inferrsi/data/repo_packs/inferrsi/knowledge/README.md","source_revision":"c6e91d437308ef8d5565528206fa554a3490e806","status":"ready"}Provider/settings: Uncovered / unsupported paths (85)
|
Dong1017
added
the
area:knowledge
ChatGPT Codex Connector
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
LGTM for the bounded source-only AutoTuneBench amendment at 941c2699. No actionable source-fidelity or integration blocker was found in this review; goal acceptance and measured usefulness remain separate.
- The reference preserves the paper’s invalid-arm/shared-wall-null conflict, the exported loop’s missing first-keep stop, partial gate enforcement and unresolved source median disagreement. Selected ledger counts and best values independently match the retained interpretation; source-specific thresholds are not promoted to repository defaults.
- The six canonical changes retain existing owners and sources. All 77 discovery IDs remain; the other 76 cards change only their source pin/URL. The modified card’s ten source-file hashes match bytes fetched from the actual material commit.
- Current-tree Ready CI, isolated installed-wheel checks and declaration policy pass. The installed checks cover resource hashes, including the two additions.
- SkillEval remains PLANNED with zero comparisons and unsupported snapshot entries. Source delivery is not evidence of agent benefit, benchmark reproduction or completion of the wider #91 backlog. The existing owner retains any integration decision.
Inspected revisions, evidence and verification limits
Head: 941c26990d6cc1799f760d086e4f50e81f6977dc; base: 919ed42c0d01dc0aa3b613085b9041d735d2d4db; material: c6e91d437308ef8d5565528206fa554a3490e806. Full 85-path inventory: six canonical changes followed by 79 projections. Tested merge ffb04650c28bc046d995ecba90bec37bf1a70139 has those base/head parents and tree 0d54e8deda61c2610a67ade3017c5751373da722, identical to head. Python 3.10/3.11 each report 3,381 passed, 13 skipped and 284 subtests passed; both fresh isolated wheel smokes pass. Declaration binds this head to policy base with no errors.
Checked the pinned upstream loop/CLI, ingestion/store, G3/G5/schema, adapter, README/tabulation and selected ledger; all 23 recorded Git blob identities and sizes match the complete pinned tree. Read paper HTML sections, relevant tables/captions and references, and checked separately stated repository/paper licensing. PDF figures were not independently rendered and PDF/source-wide SHA-256 digests were not all recomputed; the author’s complete visual readset is not attributed to this review. The ten generated owner source-file SHA-256 values were independently recomputed.
No earlier submitted reviews or inline findings were present; the existing bot review completed without findings. No repository tests/generators, model/device runs, installations or paid experiments were executed by this review. Hosted evidence and static source checks are distinguished above. This COMMENT does not submit formal approval, merge, or grant campaign acceptance.
Governing issue and delivery slice
Refs #91. Refs #177.
This source-only amendment extends the existing
inference-benchmark-designowner with AutoTuneBench trial-validity guidance. It distinguishes treatment production from delivery to the next eligible consumer, arm eligibility from benefit/null conclusions, and protocol claims from reached checks and persisted evidence.Sources and knowledge scope
The repository is pinned to
64cc8d7ca5d0ee40c4e2189b2c2c8dfb1a2a7c2f, paired with paper v1. One reference and an ordinary provenance JSON retain 23 source/rights resources and the complete 15-page paper readset, including six figures and seven tables.The reference preserves the conflict between the paper's invalid-arm precedence and its shared-wall-null interpretation. It also records that the exported first-keep path does not implement the stated stop, without asserting that unavailable external checkpoints never ran. Feedback summaries remain distinct from actual prompt receipts. Weak comparators, quarantined anomalies and the unresolved source median conflict remain explicit. Source-specific numerical bars do not become InferRSI defaults.
Repository MIT and paper CC BY 4.0 attribution are recorded separately, with third-party exclusions. Source code, raw ledger, PDF and figures are not redistributed. Results remain author-reported; knowledge remains candidate/not_run and efficacy remains not_evaluated.
The six canonical paths add the two resources, an owner pointer, incremental source metadata and both matrix mappings. Actual main
919ed42c0d01dc0aa3b613085b9041d735d2d4dbis preserved, including NeMo/ASR and JustGoal knowledge and the unrelated H3 campaign evidence. No skill ID, Case, controller, runner, schema or backend is added.Actual canonical source M
c6e91d437308ef8d5565528206fa554a3490e806precedes the unmodified native projections in H941c26990d6cc1799f760d086e4f50e81f6977dc(tree0d54e8deda61c2610a67ade3017c5751373da722). The GitHub App source tree matched the author's staged tree exactly; generated publication bytes likewise match the native output tree. All 77 discovery cards cite the reachable actual M. The other 76 cards change only source commit/URL; Accelerator Verify methodology adds no measured implementation or backend coverage.Validation and limits
f8cf0200ae7936dbb4791eac59686244919a3219plus its generated working tree. This is historical evidence; it was not rerun locally at the new API H or its current-main combination.ffb04650c28bc046d995ecba90bec37bf1a70139has parents main/H and exactly the H tree. Authorship declarations and the stable knowledge-model-policy check passed.python tools/build_skill_index.py --source-revision c6e91d437308ef8d5565528206fa554a3490e806 --check: exit 0, 77 metadata records, 291.1 seconds. H, all 85 input hashes and the tool were unchanged before/after.Another maintenance workflow merged this PR as
f5a21bf0bd684fab70bfae02562076eb478bd627on 2026-11-02. Its actual tree equals the reviewed H tree. The post-merge CPU smoke passed eight tests, and post-merge declarations passed.Knowledge authorship declaration