Repository navigation
knowledge: distill JustGoal controlled evaluation contracts - #302
Conversation
Preserve controlled harness comparisons, independent acceptance, repair accounting and workspace/stream/cache evidence under the existing benchmark owner. Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (material author) Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
Regenerate discovery records and coverage from the committed source material, preserving existing owners and validation states. Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (material author) Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
python tools/build_skill_index.py --source-revision ce1d889fc06e8dfe48327fac13cd3a871ebdcda8 --checkThe declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Selection/task metadata (not a relevance verdict): {"adapter":"audit","basis":"trusted_path_mapping","matched_paths":["src/inferrsi/data/skill_library/methods/inference-benchmark-design/SKILL.md","src/inferrsi/data/skill_library/methods/inference-benchmark-design/metadata.json","src/inferrsi/data/skill_library/methods/inference-benchmark-design/references/justgoal-evaluation-contracts.md","src/inferrsi/data/skill_library/methods/inference-benchmark-design/references/justgoal-source.json"],"tasks":[{"evidence_scope":"http_fixed_workload","id":"audit-threshold","request":"Audit the fixed input report using its contract. Compute the median of per-trial request means, including every request in each trial, and the percentage reduction relative to the baseline aggregate.","split":"holdout"}]}Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Projection check selection (not an executed result): {"declared_in":"src/inferrsi/data/repo_packs/inferrsi/knowledge/README.md","source_revision":"ce1d889fc06e8dfe48327fac13cd3a871ebdcda8","status":"ready"}Provider/settings: Uncovered / unsupported paths (85)
|
Dong1017
added
the
area:knowledge
ChatGPT Codex Connector
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Scoped integration review — P2 risk (new upstream dependency, not a new JustGoal-source finding). PR #301 has now merged into main. It and this PR touch 83 common paths, including the canonical inference-benchmark-design/SKILL.md, metadata.json, both knowledge matrices, and generated catalog records.
- Observed: current
maincontains the newly added cache-aware ASR reference/source; this PR head contains the JustGoal reference/source but predates that ASR addition. The available PR #302 test-merge ref still uses the pre-#301 main parent. Thus the existing clean-merge/CI evidence does not yet demonstrate that a merge against today's main retains both sources and consistent generated records. This is an integration/provenance regression risk, not a confirmed current merge conflict. - Suggested minimal reconciliation: integrate the latest main into this PR; preserve both ASR and JustGoal links/source inventories in the canonical owner and matrices; regenerate projections from a reachable combined source revision; then attach the combined-tree source/index check and relevant Ready CI evidence. No model, hardware, or paid evaluation is requested.
Revision and file evidence
- Latest main:
6c9f90daae2d404915378b910f581b4dcff7e7da(PR #301); reviewed PR #302 head:19ab07ebb7c5fedcde81741a3b6df4e4be56a738; its recorded base:eabad4ad57c0af9407d33da83bb117201d5e5cce. - Exposed
refs/pull/302/merge:fce777f51bb77172fe94872c31c24ff55f97efee, parentseabad4ad...and19ab07eb...(before PR #301 merged). - Representative overlap:
src/inferrsi/data/skill_library/methods/inference-benchmark-design/{SKILL.md,metadata.json},src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/inference-benchmark-design.json,docs/{reference-matrix,searchspace-matrix,skill-coverage}.md. - No local tests were run for this review.
Retain ASR and JustGoal canonical resources and current applicability before regenerating native projections from the combined source revision. Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (material author) Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
Regenerate discovery records and coverage from the combined ASR and JustGoal source revision, preserving existing owners and validation states. Agent-assisted-by: Codex / gpt-6.1-sol [effort=max] (material author) Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com>
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
LGTM for the source-documentation and generated-index scope at 08a389bc.
- The paper and selected DSA source support the reference's stated boundaries: equal-scope comparisons, independent acceptance, released failures versus repaired results, and timing/workspace limits. I found no new blocking issue in this slice.
- The earlier integration concern is cleared for this head: ASR and JustGoal resources are both retained, the material pin is reachable, and the tested merge has the same complete tree as the reviewed head.
- Current software checks and per-head policy pass. Skill Eval remains planned, zero comparisons, with unsupported snapshot paths; this review does not establish kernel correctness, performance or agent efficacy.
- The existing PR owner retains the merge decision. This conclusion does not accept the wider governing Issues.
Evidence and review limits
Reviewed head: 08a389bc00e19933ec58cc788b42647024258f5b; API base and assessed main: 6c9f90daae2d404915378b910f581b4dcff7e7da. Combined material: ce1d889fc06e8dfe48327fac13cd3a871ebdcda8. Tested merge: 4c1c75636261fec8026597418da67f65edd6183c; both trees are eeb8ac6b135dae77a445074f041b6ef76873a5ee.
Reviewed all 85 changed paths, including the canonical route/metadata, matrices and generated deltas; the other 76 cards change only source URL/commit. Read all 21 pages of the pinned paper, visually checked its principal figures/tables, and checked the recorded primary-source regions and identities. Independently matched nine upstream file hashes and five owner-resource hashes against the current head, material pin and generated owner card.
Ready CI: inspected Python 3.10 full-check log (3,381 tests and 284 subtests passed; 13 skips, one warning); Python 3.11 and installed-package checks report success. Per-head policy succeeds; independent pull_request_target Skill Eval planned/published but did not execute comparisons. Local verification was source/byte inspection, not a local test-suite or upstream kernel run. No model/device execution, numerical/performance reproduction, service qualification or paid evaluation was performed.
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
The earlier P2 integration concern is cleared at 08a389bc. Both ASR and JustGoal sources are preserved, with consistent regenerated records and passing checks on an identical test-merge tree. This follow-up covers integration/projection consistency; complete JustGoal primary-source review remains outside this clearance, so this is not whole-PR LGTM.
- The previous finding is addressed: current-main ASR content is retained, JustGoal references are unchanged from the earlier head, and the combined source revision is reachable.
- All eight generated owner-resource hashes match fetched source bytes; the other 76 cards change only source URLs/pins. Candidate/
not_runboundaries remain intact. - Ready checks passed. Software/package evidence does not establish primary-source fidelity, model/hardware performance or agent efficacy; SkillEval remains planned with zero comparisons.
Inspected revisions and verification limits
Previous head: 19ab07ebb7c5fedcde81741a3b6df4e4be56a738; current head: 08a389bc00e19933ec58cc788b42647024258f5b; base: 6c9f90daae2d404915378b910f581b4dcff7e7da.
Combined material ce1d889fc06e8dfe48327fac13cd3a871ebdcda8 has the previous head and base as parents. Head directly descends from it; the following delta contains exactly 79 generated paths. Projected metadata/applicability matches canonical inputs.
Test merge 4c1c75636261fec8026597418da67f65edd6183c and head share complete tree eeb8ac6b135dae77a445074f041b6ef76873a5ee. Full-check 3.11 checked out that merge: 3381 passed, 284 subtests passed, 13 skipped, 1 warning. Installed-package 3.11 built that revision and reported 125 source-grounded records.
Source bytes, hashes, commit ancestry, projection deltas and hosted logs were inspected. No local repository tests, generator execution, complete upstream paper/code review, model or hardware experiments were performed. The prior review was integration-scoped and did not establish full JustGoal source fidelity. No reviewer thread is resolved or formal approval issued by this COMMENT.
Governing Issue and delivery slice
Refs #91. Refs #177.
Delivers the bounded JustGoal repository + paired-paper knowledge slice under the existing
inference-benchmark-designowner, preserving the cache-aware ASR procedure, current applicability and source inventory already merged into main. It retains same-model/budget/scope comparisons, independent complete-output acceptance, released failures versus repaired timings, and workspace/stream/cache contracts.Changed artifacts and evidence
Primary sources: JustGoal repository and its paired paper, both fixed at
f052fc0b28fc7f10b2ef1ae7aa4682aabbbe868d.The already-merged cache-aware ASR reference remains unchanged.
The JustGoal reference and ordinary supporting source JSON retain nine pinned primary files and the complete 21-page paper readset. The existing ASR reference and both JustGoal resources are byte-identical to their source parents, and all current-main ASR metadata/applicability and both matrix entries are retained. The existing skill route/metadata and both maintained matrices link this source-only methodology. No new skill ID, case, runner, runtime, schema or backend implementation is added. Paper benchmarks remain author-reported; candidate/not_run and unknown paper licensing are preserved. Source kernels, paper/figures and the omitted proprietary helper are not redistributed.
flowchart LR A["Pinned JustGoal repo + paper"] --> B["Existing benchmark owner"] F["Merged ASR reference"] --> B B --> C["ASR + JustGoal source resources"] C --> D["Native catalog and material reads"] C --> E["Two maintained matrices"]The combined source merge
ce1d889fc06e8dfe48327fac13cd3a871ebdcda8retains current main6c9f90daae2d404915378b910f581b4dcff7e7daand the prior contribution as parents. Unmodified native generators then produced08a389bc00e19933ec58cc788b42647024258f5bfrom that committed material. All 77 discovery cards now cite that reachable material commit; the other 76 cards change only their source commit/URL. The source-specific Accelerator row is benchmark verification guidance, with no new measured resource coverage or compatibility claim.Checks and dependencies
4c1c75636261fec8026597418da67f65edd6183chas exactly this head's complete tree.knowledge-model-policy, reference/qualification and plan/publication checks passed; Codex bot review completed without findings. Paid execution was deliberately skipped. SkillEval has zero comparisons/task calls and 85 unsupported/uncovered paths under the inherited snapshot limitation, not a result about agent efficacy.The paper's single-shape paired timing and whole-definition official-evaluator timing remain distinct. Released-version retest counts are not first-generated-attempt rates. The source kernels are not a complete campaign runner. The wider #91 backlog and empirical qualification remain open to their owners; no Issue-completion or independent-approval claim is made.
Knowledge authorship declaration