Repository navigation
Conversation
Skill evaluationPLANNED: not executed; there are no task outcomes yet.
Author verification steps (nonpaid; not yet run)
python tools/build_skill_index.py --source-revision 32623f7c062227d038bf48c59ed85792872d46b0 --checkThe declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy. Evidence and logs · Retention: 30 days. Plan bindings, budget, frozen exam and advisory detailsAdvisory: obtain missing evidenceRetained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Model/settings:
Automatic execution / cumulative PR budget: {"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}Projection check selection (not an executed result): {"declared_in":"src/inferrsi/data/repo_packs/inferrsi/knowledge/_provenance.json","source_revision":"32623f7c062227d038bf48c59ed85792872d46b0","status":"ready"}Provider/settings: Uncovered / unsupported paths (8)
|
|
Review of the full PR (new PR, no earlier review of mine): base Not approving yet; the content itself is careful (both gates, the weight-only scope limit and the "does not establish" list read well, and I checked that the 60+ generated records outside Blocking / needs action
Provenance questions (please confirm or downgrade the marker)
Minor
Not verified by me: the HuggingArch file hashes/blobs, and the PR's local |
b5b7dda to
18c9a16
Compare
|
Incremental review What changed: I compared the per-file patches of New at this head
Still open from last round (no change in this delta)
Static review of the delta plus the hosted CI logs; I did not run anything locally. |
|
Incremental review Fixed in this delta
Still open
Static review of the delta plus hosted CI status. I didn't run anything locally. |
|
Coverage review at head Fixed earlier (unchanged here)
Still open
Reviewed commit range: |
|
[inferrsi-oncall smoke] @Dong1017 please ignore — testing on-call wake 2026-11-07T05:42:13Z |
|
On-call proxy for Dong1017's queue (inferrsi-oncall): smoke wake received. I am not Dong1017. Acknowledged the test mention on #351 (comment) ( Verified: |
1694a0b to
19f12f4
Compare
|
Thanks for the four rounds. Replies to the coverage review at 1. 2. Source pin on squash merge. I'd propose no change, because this matches how
The pin is now content commit 3.
The full re-audit, which included an independent read-only pass, found more problems than the four you named. Fixed:
4. Merge state. Rebased; the branch is no longer 5. Minor.
On your earlier open question: Local verification at the new head is in the PR body. Same scope as before: documentation and reference material only, |
|
@Dong1017 Before requesting the knowledge-policy approval on #351, could you decide the merge order between this PR and #359? Whichever lands second has to rebase and regenerate the skill index:
Either works for me. Since #359's |
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 3f560642a44a1496aa4984ab936517693be2c788 and head 19f12f4ed00997a2a79cfd07299411e67410c7b2 (full diff; prior-reviews.md has no earlier findings). Dong-bot will re-read after the author pushes.
Blocker
Scope of the checkpoint anchor disagrees with itself on per-rank weight bytes.
The procedure defines the anchor as two gates. Gate 1 is the bidirectional checkpoint join. Gate 2 is the per-rank weight ledger, and the reference states the consequence directly: passing the model-level byte anchor certifies nothing about per-rank weight bytes, so any L1 figure quoted per device or under a parallelism degree needs gate 2 (references/huggingarch-checkpoint-anchor.md, "Gate 2"). SKILL.md says the same thing before a per-device or per-degree figure is quoted. metadata.json verification.note (hunk at line 98) matches that wording: the prerequisite establishes nothing about per-rank bytes without its second gate.
The limitations string contradicts that pair. It leaves per-rank placement unanchored with no gate-2 condition:
The checkpoint anchor covers analytical weight demand only; KV traffic, per-rank placement and end-to-end serving performance remain unanchored by it.
Locations:
src/inferrsi/data/skill_library/methods/performance-ceiling-analysis/metadata.jsonlimitations, line 122- the same sentence copied into
src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/performance-ceiling-analysis.jsonat lines 214 and 257
A metadata consumer will treat declared per-rank layout as out of scope. The skill text tells them to run the ledger before quoting a per-device figure. Those cannot both be the contract.
Expected correction: make the limitation state the same scope as the verification note and the reference. KV traffic and end-to-end serving performance stay unanchored. Per-rank weight bytes stay unanchored until gate 2 passes. Gate 2 checks declared layout, format, and the non-dividing-degree fit policy; it does not prove runtime placement or KV counts. Regenerate the knowledge pack so both limitation copies match metadata.json.
Consistent in this head
SKILL.md, the anchor reference,docs/reference-matrix.md, andverification.noteagree that the weight term goes through the bidirectional join plus the per-rank ledger, that KV growth is a separate justification, and that this source has no end-to-end predicted-versus-measured serving result.[code]/[doc]markers in the anchor reference match the inspection notes inhuggingarch-source-identity.jsonfor the git files read here: join residuals and name summing in_matching.py;run_weight_ledger_checkmapping in_weight_ledger.py;UTILIZATION_CEILING/above_solinkernel_bench/emit.py;ResultIdentity.keyinscope.py;_IMPLIESin_ablation.py. HTTP 422, the fit policy, andvalidate()stay[doc], with the uninspected files named.- The two-threshold rule is marked as this reference's reading where the source does not say why. The guard-ablation study is carried with the source's own non-reproducibility limits.
docs/skill-coverage.mdmovesperformance-ceiling-analysisfrom 18 readsets to 20 and adds the HuggingArch row. The knowledge snapshot pointer moves fromd648f25732dee1c974da62a94c46a642ad52cef8to1536aeb6cf6b6d3345388bac7f4f9537273efa40across the pack.
Not verified here
- Did not run
python tools/build_skill_index.py --source-revision 1536aeb6cf6b6d3345388bac7f4f9537273efa40 --checkorpython tools/build_skill_coverage.py --check. Did not recompute content hashes. - Did not confirm
1536aeb6is an ancestor of19f12f4whoseskill_librarybytes match the pack. Did not re-fetchshenh10/HuggingArchatfacbbbc96856e558b9084a94a5070b2e2c86cf0b. - The seeded diff cuts off inside the blog document source of
huggingarch-source-identity.jsonat"snapshot": "unk, and the next line is too long to read. The snapshot value, the "two figures" record, and JSON closure were not checked. CI was not inspected.
Optional
SKILL.mdgate 1 says both "agree in physical element count" and "compare bytes rather than elements wherever precision is mixed." The reference keeps element agreement on the per-tensor join and switches the checksum total to bytes when precision is mixed. One sentence inSKILL.mdpointing at that split would keep the summary from being read as flipping the primary gate.- The anchor doc's "Why L1 needs this" states "one header read per shard, plus the shard index" with no
[code]or[doc]marker.
This review does not approve the change.
|
#359 first. Hold #351, and request
I have not opened #359's |
|
Incremental review Resolved
Still open
Optional
|
L1 divides a count by a rated peak and never looks at the model again, so a wrong weight count reaches the floor unchallenged. Add a prerequisite with two gates, distilled from HuggingArch at its root commit: - Gate 1 joins the safetensors headers with the analysis's stored tensors in both directions, compares summed rows per name in physical elements, and treats byte totals as a checksum on that join. - Gate 2 requires a declared per-axis placement, a format and a non-divisibility policy before any per-rank weight figure. Gate 1 needs no layout, so it certifies nothing per rank. State what the anchor leaves open: KV size and growth (windowed and global attention have byte-identical weights), cache precision, activations, collectives and achieved rates. Add the two-threshold rule for measurements: a row is rejected only when its own work over its own time exceeds the peak it recorded; a row faster than a separately computed floor is flagged and both sides are re-derived. The source's numeric constants stay provenance only. Every claim about the source carries a [code], [doc], [tree] or [blog] marker with a locator. The source identity record gives the 12 files' blobs, sha256, sizes and read ranges, all matched against the commit's tree. Companion-post figures absent from the repository are recorded as unverified. Add the metadata sources, a limitation, two readsets and a reference-matrix row. locally_verified stays false and empirical_validation stays not_run. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>
Regenerate with `build_skill_index.py --source-revision 32623f7` and `build_skill_coverage.py`. Only the performance-ceiling-analysis record, the shared _provenance.json pin and the coverage view change. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>
19f12f4 to
b3f1009
Compare
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base 4b3afa55f278ca314e07d5f8acbd0a18f3fa72bf and head b3f10097ffb7211f76c6bfb1b5659ff8094d99b0.
Incremental range 19f12f4ed00997a2a79cfd07299411e67410c7b2..b3f10097ffb7211f76c6bfb1b5659ff8094d99b0 (prior review base 3f560642a44a1496aa4984ab936517693be2c788). Dong-bot will re-read after the author pushes.
Blocker
Prior finding still open: the checkpoint anchor still disagrees with itself on per-rank weight bytes.
The review at head 19f12f4e blocked because the limitations string leaves per-rank placement unanchored with no gate-2 condition (The checkpoint anchor covers analytical weight demand only; KV traffic, per-rank placement and end-to…). Gate 2 in references/huggingarch-checkpoint-anchor.md, SKILL.md, and metadata.json verification.note say the opposite: a passing model-level byte anchor certifies nothing about per-rank weight bytes, and any L1 figure quoted per device or under a parallelism degree needs gate 2.
The readable incremental diff has no hunk for:
src/inferrsi/data/skill_library/methods/performance-ceiling-analysis/references/huggingarch-checkpoint-anchor.mdsrc/inferrsi/data/skill_library/methods/performance-ceiling-analysis/SKILL.mdsrc/inferrsi/data/skill_library/methods/performance-ceiling-analysis/metadata.json
Those three texts are what this re-review can still see. Attach the gate-2 condition to every per-rank, per-device, and per-degree weight-byte claim, in the same words as Gate 2, SKILL.md, and verification.note.
docs/reference-matrix.md does add the HuggingArch row and calls the reference a two-gate weight anchor. Its boundary sentence is “Anchors stored weights only; KV, collectives and E2E performance unanchored”. That row does not replace the limitations string above.
Readable range
docs/skill-coverage.md: performance-ceiling-analysis readsets 18 → 20, HuggingArch listed; input digestsha256:ee418cd9ed09c7e5c745808637387ad7a02a9ede459563ba2e56d4229d299e67.- Base advanced
3f560642→4b3afa55, so previous..head also carries other tree changes (knowledge-contribution workflow, historical plan banners, faster-cosyvoice rows,evidence/h3-ref2va-910c-4logical-20261101-297/**). This review judges the anchor-doc blocker. Those other paths are not new blockers against4b3afa55.
Not verified
pr/diff.patchline 2746 is one line the reader would not return. Lines 1–2744 already exceed 100000 characters; the file is 150155 bytes. That line follows a mid-token line inevidence/h3-ref2va-910c-4logical-20261101-297/patch/c12-candidate.patch(assert model._alignment_ca), the 252nd line of a+1,252test hunk inside a patch declared+1,633. Bytes after that point are unread. If the limitations edit lives only there, it is still unverified.- Tests not run. CI not checked. PR body and any knowledge-authorship declaration were not in the seed.
This comment does not approve the PR and does not change its scope.
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 4b3afa55f278ca314e07d5f8acbd0a18f3fa72bf and head b3f10097ffb7211f76c6bfb1b5659ff8094d99b0. The branch was rebased and its commits rewritten, so I compared the PR diff at my last reviewed head 19f12f4ed00997a2a79cfd07299411e67410c7b2 (base 3f560642a44a1496aa4984ab936517693be2c788) with the PR diff at this head, file by file.
Both items from my previous review are resolved:
- Per-rank scope wording. The
metadata.jsonlimitation now says per-rank weight bytes stay unanchored until the second gate passes, and that Gate 2 checks the declared layout and format, not runtime placement. This matchesSKILL.md(Gate 2),verification.noteand the anchor reference. Both copies in the regeneratedknowledge/common/skills/performance-ceiling-analysis.jsoncarry the new text, and the old "per-rank placement … remain unanchored" wording is gone. declarationsCI. Green on this head, together withfull-check(3.10/3.11),installed-package(3.10/3.11),pr-references,planandpublish.- The optional Gate 1 clarification is also in: the per-name comparison stays in elements, and only the post-join byte totals switch to bytes (
SKILL.mditem 4; anchor reference, Procedure steps 4–5).
I also re-read the rewritten references/huggingarch-checkpoint-anchor.md and spot-checked its [code] locators against HuggingArch at facbbbc96856e558b9084a94a5070b2e2c86cf0b. These match the pinned source: emit.py L30–32 (UTILIZATION_CEILING), op_calibration.py L59 and L924–939 (ANOMALY_THRESHOLD, measured_anomalous), _matching.py L157–171 and L324–342 (summed physical elements, empty-header refusal, two residuals), _weight_ledger.py L37–51, and _ablation.py L40. The blog material is kept as [blog] and unverified, and the adoption boundary keeps the numeric constants as provenance only.
Not verified: I did not run the tests locally. I did not check every [doc] locator or the per-file digests in huggingarch-source-identity.json.
|
@Dong1017 Pls review it |
|
PR review verdict — DO NOT PASS Reviewed base Addressed in the PR body (file text at this head still unread)
Blocking
Disposition
Not verified here
I will re-read this head once |
Refs #348 (the issue's acceptance owner decides closure; a merged PR does not satisfy the goal by itself).
Rebuilt on
mainafter #359This head replaces the earlier branch. The old branch had three commits; the first,
0a2e2335, was written byclaude-opus-5, which the knowledge-model policy does not admit. Rebasing would have carried that text forward. So the content was rewritten from the primary sources in one session (claude-opus-5-5, effortxhigh). The new text was not built from the old wording.facbbbc96856e558b9084a94a5070b2e2c86cf0b. Every blob was recomputed and matched against the commit's tree.19f12f4eare addressed (see Changes since the last review).What this changes
methods/performance-ceiling-analysis/SKILL.md.../references/huggingarch-checkpoint-anchor.md.../references/huggingarch-source-identity.json.../metadata.jsondocs/reference-matrix.mddocs/skill-coverage.md,knowledge/_provenance.json,knowledge/common/skills/performance-ceiling-analysis.jsonThe two gates
The anchor covers stored weights only. Windowed and global attention with equal head dimensions have byte-identical weights, so KV size and growth stay unanchored, as do cache precision, activations, collectives and achieved rates.
The two-threshold rule
kernel_bench/emit.pyabove_sol,UTILIZATION_CEILING = 1.0).0.95 ×the roofline. That roofline is recomputed from caller-supplied FLOPs and bytes at the row's own test-time peaks, and the flag is computed only when both numbers are supplied (backend/inference/op_calibration.py).Provenance
Markers:
[code]is executable code in an inspected range.[doc]covers prose, docstrings, comments and message strings, plus behaviour in uninspected files.[tree]is a fact from the commit's tree listing.[blog]is the companion post.Every marker has a locator.
The pin is HuggingArch's root commit (
init repo, authored 2026-11-04, no parents). GitHub reports the repository as created on 2026-11-03.The companion post is dated 2026-10-08 and was retrieved once, on 2026-11-10. Its sections 2.5, 2.6, 3.6 and 3.7 were read in full. It predates the pin, so none of it is attributed to the pin.
canonical_module_path()name join that the pinned code does not use. The reference follows the pin.What the source does not establish:
A separate read-only pass checked every marker, locator, read range and digest against the downloaded sources. It found no factual errors and no
[code]misuse; its nine locator and wording refinements are folded in.locally_verifiedstaysfalseandempirical_validationstaysnot_run. HuggingArch was not run, no checkpoint was joined, and nothing was measured.Changes since the last review (
19f12f4e)SKILL.md, the verification note and the reference say the same.SKILL.mdand the procedure now say the per-name comparison stays in elements and only the totals switch to bytes. The source's own split is unsettled within the inspected scope: its gate table lists the per-name mismatch as a gate, whilematch_checkpoint_to_hfcalls it a diagnostic. The reference records both, and the InferRSI procedure fails on a mismatch either way.latency < 0.95 × rooflineat the row's test-time peaks, computed only with both numerators supplied, and the time is returned unclamped. That code lives inbackend/inference/op_calibration.py, which is now in the readset; the facbbbc9 bloba4074fd5matches the tree.[doc].op_calibration.pyandLICENSEare added.backend/measure/scope.pyis recorded as read but not cited.Source pin after squash merge
The shared pin in
knowledge/_provenance.jsonis the content commit,32623f7c. After a squash merge it is a branch-only commit. #359's workflow document lists that as a known limit of the current process, and--checkstill works wherever the commit is reachable, as it is through this PR's ref.Out of scope
As #348 specifies:
Class A/Class Bvocabulary. The source's classes A–E are a different taxonomy and are not imported either.skill-index.jsonchange.Verification at this head
Head
b3f10097ffb7211f76c6bfb1b5659ff8094d99b0, onmain4b3afa55: content commit32623f7c, thenb3f10097regenerating the record, the shared pin and the coverage view.Local results:
--checkruns pass.ruff check src tests toolspasses.tests/check_installed.pysmoke passes from an installed wheel built at this head (128 records).maincheckout at4b3afa55in the same run environment. They are in skill-eval andyibuapi_runtimetests: this workstation has git 2.34 and nopythonon PATH. No new failures.The generator ran through a local wrapper that drops only
--no-lazy-fetch, which git 2.34 lacks. This clone has no promisor remote, so the hashed bytes are unaffected. CI reruns the same checks with its own git.knowledge_declaration_paths.pyreports 7 protected paths. No exception is requested.Knowledge author declaration
🤖 Generated with Claude Code