Skip to content

docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis - #351

Open
fake0fan wants to merge 2 commits into
InferMatrixLab:mainfrom
fake0fan:docs/ceiling-checkpoint-anchor
Open

fake0fan wants to merge 2 commits into
InferMatrixLab:mainfrom
fake0fan:docs/ceiling-checkpoint-anchor

Conversation

@fake0fan

@fake0fan fake0fan commented Nov 4, 2026 •

Copy link
Copy Markdown

Refs #348 (the issue's acceptance owner decides closure; a merged PR does not satisfy the goal by itself).

Rebuilt on main after #359

This head replaces the earlier branch. The old branch had three commits; the first, 0a2e2335, was written by claude-opus-5, which the knowledge-model policy does not admit. Rebasing would have carried that text forward. So the content was rewritten from the primary sources in one session (claude-opus-5-5, effort xhigh). The new text was not built from the old wording.

  • The HuggingArch files were re-read at the same pin, facbbbc96856e558b9084a94a5070b2e2c86cf0b. Every blob was recomputed and matched against the commit's tree.
  • The open review items on 19f12f4e are addressed (see Changes since the last review).
  • Under prototype: reduce knowledge projection and declaration churn #359's source snapshot, the regeneration now touches 3 generated files instead of 81.
  • The declaration lists one author with an admitted model and effort, so no exception is requested.

What this changes

File Change
methods/performance-ceiling-analysis/SKILL.md Item 4 under Evidence to inspect first (the two gates); a third known L1 limitation; Measurements against L1; a short Reviewed implementation source section
.../references/huggingarch-checkpoint-anchor.md New: what the source does, the procedure for each gate, what the anchor leaves open, the two-threshold rule, the archived ablation study, the companion post, and what the source does not establish
.../references/huggingarch-source-identity.json New: pin and commit metadata, a tree summary, 12 files with git blob, sha256, size and read ranges, and the companion post's retrieval identity
.../metadata.json A git source and a document source, one limitation, an extended verification note, and two readsets inserted in the existing order
docs/reference-matrix.md One §2 row
docs/skill-coverage.md, knowledge/_provenance.json, knowledge/common/skills/performance-ceiling-analysis.json Regenerated at the content commit

The two gates

  1. Per-tensor join. Join the safetensors headers with the analysis's stored tensors in both directions and report both residuals. For each name, sum the stored rows that share it and compare physical elements with the header. Only the byte totals that follow switch to bytes, and they are a checksum: equal totals do not pass a join with a residual or a mismatch.
  2. Per-rank weight ledger. Gate 1 needs no layout, so it certifies nothing per rank. Before any per-rank weight figure, every stored weight needs a declared placement per stored axis and a format. A degree that does not divide its part count follows a declared policy (refuse, co-own or pad), never a substitute divisor. Gate 2 checks the declared layout and format, not the placement a framework uses at run time.

The anchor covers stored weights only. Windowed and global attention with equal head dimensions have byte-identical weights, so KV size and growth stay unanchored, as do cache precision, activations, collectives and achieved rates.

The two-threshold rule

  • At upload, HuggingArch refuses a row whose achieved compute or bandwidth exceeds the peak the row itself recorded (kernel_bench/emit.py above_sol, UTILIZATION_CEILING = 1.0).
  • At estimate time, it flags, and never refuses, a row whose latency is below 0.95 × the roofline. That roofline is recomputed from caller-supplied FLOPs and bytes at the row's own test-time peaks, and the flag is computed only when both numbers are supplied (backend/inference/op_calibration.py).
  • The labelled reading: rejection is reserved for a row that contradicts its own record; a floor computed from other counts may only flag. The source's constants stay provenance only, and any flag tolerance belongs to the acceptance issue.

Provenance

  • Markers:

    • [code] is executable code in an inspected range.
    • [doc] covers prose, docstrings, comments and message strings, plus behaviour in uninspected files.
    • [tree] is a fact from the commit's tree listing.
    • [blog] is the companion post.
    • Interpretations are labelled Reading.

    Every marker has a locator.

  • The pin is HuggingArch's root commit (init repo, authored 2026-11-04, no parents). GitHub reports the repository as created on 2026-11-03.

  • The companion post is dated 2026-10-08 and was retrieved once, on 2026-11-10. Its sections 2.5, 2.6, 3.6 and 3.7 were read in full. It predates the pin, so none of it is attributed to the pin.

    • Three figures in the post appear in no inspected repository file and are listed as not to be cited:
      • an agent self-approval rate of 94 % → 6 %;
      • per-derivation turns, minutes and cost;
      • a corpus of more than 280,000 rows.
    • The post also describes a canonical_module_path() name join that the pinned code does not use. The reference follows the pin.
  • What the source does not establish:

    • No inspected file compares predictions with measured end-to-end serving results.
    • Collectives are unmeasured.
    • The measured corpus is not in the repository.
  • A separate read-only pass checked every marker, locator, read range and digest against the downloaded sources. It found no factual errors and no [code] misuse; its nine locator and wording refinements are folded in.

  • locally_verified stays false and empirical_validation stays not_run. HuggingArch was not run, no checkpoint was joined, and nothing was measured.

Changes since the last review (19f12f4e)

  • Per-rank wording (amy-why-3459, cursor[bot]). The metadata limitation now reads: KV size and growth, activation traffic, collectives and end-to-end serving performance stay unanchored; per-rank weight bytes stay unanchored until the second gate passes; and that gate checks declared layout and format, not run-time placement. SKILL.md, the verification note and the reference say the same.
  • Gate 1 elements versus bytes (optional point). SKILL.md and the procedure now say the per-name comparison stays in elements and only the totals switch to bytes. The source's own split is unsettled within the inspected scope: its gate table lists the per-name mismatch as a gate, while match_checkpoint_to_hf calls it a diagnostic. The reference records both, and the InferRSI procedure fails on a mismatch either way.
  • Anomaly badge, tightened as promised. It is no longer "faster than the roofline". It now reads latency < 0.95 × roofline at the row's test-time peaks, computed only with both numerators supplied, and the time is returned unclamped. That code lives in backend/inference/op_calibration.py, which is now in the readset; the facbbbc9 blob a4074fd5 matches the tree.
  • Marker set. Tree-listing facts and blog content get their own markers instead of being folded into [doc].
  • Readset. 12 files (previously 10): op_calibration.py and LICENSE are added. backend/measure/scope.py is recorded as read but not cited.

Source pin after squash merge

The shared pin in knowledge/_provenance.json is the content commit, 32623f7c. After a squash merge it is a branch-only commit. #359's workflow document lists that as a known limit of the current process, and --check still works wherever the commit is reachable, as it is through this PR's ref.

Out of scope

As #348 specifies:

Verification at this head

Head b3f10097ffb7211f76c6bfb1b5659ff8094d99b0, on main 4b3afa55: content commit 32623f7c, then b3f10097 regenerating the record, the shared pin and the coverage view.

python tools/build_skill_index.py --source-revision 32623f7c062227d038bf48c59ed85792872d46b0
python tools/build_skill_coverage.py
python tools/build_skill_index.py --source-revision 32623f7c062227d038bf48c59ed85792872d46b0 --check
python tools/build_skill_coverage.py --check
python tools/knowledge_declaration_paths.py --base origin/main --head HEAD

Local results:

  • Both --check runs pass.
  • ruff check src tests tools passes.
  • The knowledge and skill-library test files pass.
  • The clean-wheel tests/check_installed.py smoke passes from an installed wheel built at this head (128 records).
  • Full suite: 153 failures, all also failing on an untouched main checkout at 4b3afa55 in the same run environment. They are in skill-eval and yibuapi_runtime tests: this workstation has git 2.34 and no python on PATH. No new failures.

The generator ran through a local wrapper that drops only --no-lazy-fetch, which git 2.34 lacks. This clone has no promisor remote, so the hashed bytes are unaffected. CI reruns the same checks with its own git.

knowledge_declaration_paths.py reports 7 protected paths. No exception is requested.

Knowledge author declaration

{
  "schema_version": 1,
  "head_sha": "b3f10097ffb7211f76c6bfb1b5659ff8094d99b0",
  "human_only": false,
  "material_ai_authors": [
    {
      "tool": "Claude Code",
      "model_id": "claude-opus-5-5",
      "effort": "xhigh",
      "role": "author"
    }
  ]
}

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Skill evaluation

PLANNED: not executed; there are no task outcomes yet.
No approval, model efficacy or current-head test result is implied.

  • Next: run the pinned source/index --check below at head b3f10097ffb7211f76c6bfb1b5659ff8094d99b0.
  • Task selection: no supported automatic comparison; choose a relevant source/content check and disposition uncovered claims.
  • Selected comparisons: 0 · Estimated task calls: 0. Uncovered/unsupported changes: 8; Canonical library paths: 4; Generated projection paths: 2; Other paths: 2. These are paths, not independent claims.
  • Return: command, input/source pin, exit status, outputs/failures and untested claims for independent verified/refuted/unverified disposition.
Author verification steps (nonpaid; not yet run)
  1. Bind the checkout. Run git rev-parse HEAD; it must equal b3f10097ffb7211f76c6bfb1b5659ff8094d99b0. If it differs, obtain a current plan; do not reuse old results.
  2. Check source/index consistency. At that PR checkout, using the reviewed repository tool and its documented development dependencies, run:
python tools/build_skill_index.py --source-revision 32623f7c062227d038bf48c59ed85792872d46b0 --check

The declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
No supported automatic comparison is selected. Review the canonical changes and explain which claims need evidence, are unaffected, or remain unverified; do not treat generated records as separate behavioral claims.
Uncovered scope by location: Canonical library paths: 4; Generated projection paths: 2; Other paths: 2. Path counts are not claim counts; no metadata-only conclusion is inferred.
3. Return evidence for independent review: head, check/task/source pin, command, environment, exit status, outputs/failures and untested claims. Label scripted controls. Request each claim's disposition: verified, refuted or unverified; this plan proves neither model provenance nor benefit.
Check recipes and evidence limits

Evidence and logs · Retention: 30 days.
Retained artifacts are verifiable only before Unix time 1796976875 while the linked artifact is available. Expired evidence is unverified.

Plan bindings, budget, frozen exam and advisory details

Advisory: obtain missing evidence

Retained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Applicable only while these revisions and inputs match the current PR.
Paid execution requires trusted label authorization and cumulative PR reservation. Unknown accounting blocks new paid calls; planning is not execution.
Pipeline evidence only; offline fixed outputs do not establish skill efficacy.
Unchanged-correct is valid neutral evidence, not incremental benefit.
Content/source review and authorship admission remain separate.
Trusted regrading checks retained answers, not model/hardware provenance or a hostile executor; this is not an efficacy certification.

  • Retained evidence is planned / unexecuted.
  • No supported comparison cases were selected.
  • Uncovered/unsupported scope needs a case or explicit reviewer disposition.
    Plan SHA-256: 55fb22348030a4df7d821941946927ad04eb64150d76f39b9d014fbef135e393
    Base: 4b3afa55f278ca314e07d5f8acbd0a18f3fa72bf · Head: b3f10097ffb7211f76c6bfb1b5659ff8094d99b0
    Trusted implementation: 4b3afa55f278ca314e07d5f8acbd0a18f3fa72bf

Model/settings: {"effort":"not_applicable","name":"claude-opus-5-5"}
Budget: {"comparisons":0,"estimated_calls":0,"limits_per_comparison":{"campaign_seconds":600,"max_task_calls":2,"repeats":1,"timeout_seconds":180,"token_stop_new_trials":200000},"provider_config_sha256":"b28d80f8194fbe0f5a9f01515efb24b218979fd45f702776560da276f4b30d04","reservation":null}

Skill Comparison Cases Estimated calls

Automatic execution / cumulative PR budget:

{"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}

Projection check selection (not an executed result):

{"declared_in":"src/inferrsi/data/repo_packs/inferrsi/knowledge/_provenance.json","source_revision":"32623f7c062227d038bf48c59ed85792872d46b0","status":"ready"}

Provider/settings: {"client":{"api_version":"2023-06-01","endpoint":"https://yibuapi.com/v1/messages","kind":"anthropic-messages-full-context-v1","max_output_tokens":4096,"max_read_bytes":32768,"max_request_bytes":262144,"max_requests":1,"max_tool_calls":12,"provider":"yibuapi"},"limits":{"token_stop_new_trials":200000},"model":{"effort":"not_applicable","name":"claude-opus-5-5"},"pricing":{"currency":"USD","expires_at":1793271228,"group":"default","max_input_tokens":1000000,"model":"claude-opus-5-5","rate_evidence":"https://github.com/InferMatrixLab/InferRSI/issues/215#issuecomment-6467531341","rates":{"cache_read":"0.4","cache_write":"10","input":"8","output":"40"},"request_fee":"0","unit":"per_million_tokens"},"spend_limit":{"amount":"200","currency":"USD"}}

Uncovered / unsupported paths (8)
  • uncovered: "docs/reference-matrix.md"
  • uncovered: "docs/skill-coverage.md"
  • uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/_provenance.json"
  • uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/performance-ceiling-analysis.json"
  • uncovered: "src/inferrsi/data/skill_library/methods/performance-ceiling-analysis/SKILL.md"
  • uncovered: "src/inferrsi/data/skill_library/methods/performance-ceiling-analysis/metadata.json"
  • uncovered: "src/inferrsi/data/skill_library/methods/performance-ceiling-analysis/references/huggingarch-checkpoint-anchor.md"
  • uncovered: "src/inferrsi/data/skill_library/methods/performance-ceiling-analysis/references/huggingarch-source-identity.json"

Copy link
Copy Markdown
Collaborator

Review of the full PR (new PR, no earlier review of mine): base 663f3b327998526a13aa31e72ad729025c18983c … head b5b7dda6959c92e730fe78fad66e6948d5951cb6 (commits 741c2ec2 content + b5b7dda6 generated refresh).

Not approving yet; the content itself is careful (both gates, the weight-only scope limit and the "does not establish" list read well, and I checked that the 60+ generated records outside performance-ceiling-analysis change only the ca201c99… → 741c2ec2… pin). The items below need to be resolved or answered first.

Blocking / needs action

  1. declarations check is failing at this head (job). The PR declaration names model_id: "claude-opus-5" with effort: "unknown", while .github/knowledge-model-policy.json on main only admits claude-opus-5-5 (and the GPT-6 models) with effort xhigh/max. So every protected path lands in exception and needs approval from the policy approval owner. Please either correct the declaration if the actual model/effort was an admitted one, or request the exception approval explicitly — as is, the PR cannot satisfy the policy.
  2. Source pin will be orphaned on squash merge (src/inferrsi/data/repo_packs/inferrsi/knowledge/README.md L18 and L33, plus every regenerated record). 741c2ec2… is a branch-only commit; after a squash merge, python tools/build_skill_index.py --source-revision 741c2ec2… --check and all record URLs point at a commit that is not on main. You raised this yourself in the PR body; I think it needs a decision before merge (merge commit / rebase-merge, or a post-merge regeneration step that a maintainer owns), rather than being left open.

Provenance questions (please confirm or downgrade the marker)

  1. references/huggingarch-checkpoint-anchor.md — several claims are tagged [code] but the identity file's inspection scope seems narrower than the claim:
    • Gate 1 "join on the full tensor name … the analysis side composes its names from declared root, stack and module segments [code]" (around L28–38), while _matching.py is recorded as "Matching internals not traced".
    • "Row refused at ingest (HTTP 422) [code]" in the two-threshold table (around L97): the inspected file is kernel_bench/emit.py (gate function + constant, "row construction not traced"). Which inspected file shows the 422 ingest refusal? If it is in the upload/backend path that was not inspected, this should be [doc].
    • "The repository carries 16 provider environment files … [code]" (around L180): which file/listing supports this? If it comes from the tree listing, please say so in the identity method.
  2. The blog-sourced "2× long-context prefill baseline error" figure: the identity file says sections from 3.7 onward were only reached via a generated summary. Please state which section the figure comes from; if it is ≥3.7 it is second-hand and I would drop it rather than carry it even as "unverified".

Minor

  1. metadata.json note says "Single-commit repository created 2026-11-03", while the identity file records commit_date: 2026-11-04T02:24:14Z (repo created 2026-11-03T16:33Z). Both are true, but please phrase it so the date in metadata isn't read as the commit date.
  2. The provenance_readsets list was also re-sorted (case-sensitive order); fine, just mention it in the PR body so the diff isn't mistaken for removals.

Not verified by me: the HuggingArch file hashes/blobs, and the PR's local ruff/pytest results (53 pre-existing failures). I'll re-review on the next push.

Copy link
Copy Markdown
Collaborator

Incremental review b5b7dda6..18c9a16c, checked against my previous review. The branch was rebased: content commit 741c2ec2 (on 0b19c5c3) became 75afe47c (on 663f3b32), and the generated commit is now 18c9a16c (chore: refresh skill coverage view). Not approving yet.

What changed: I compared the per-file patches of 741c2ec2 and 75afe47c for the 5 substantive files; apart from hunk offsets and rebase context in metadata.json (the FlexGen/Mooncake entries from main), the content is identical. The generated commit now only touches docs/skill-coverage.md (input digest plus one row); the PR is down to 6 files, from the 69 regenerated records described in the PR body.

New at this head

  1. full-check (3.10, 3.11) and installed-package (3.10, 3.11) now fail. I read the full-check 3.11 log (6 failed, 3538 passed); I did not inspect the installed-package logs. tests/test_collect_selection.py::test_frozen_offline_pilot (4 cases) reports package:inferrsi.common.skills.performance_ceiling_analysis as index_unavailable / "Stale packaged skill projection; rebuild with build_skill_index.py", and tests/test_skill_index.py::test_complete_index_retains_metadata_states_and_unknown_license and ::test_generated_index_matches_current_sources_without_historical_git_objects fail too. My read is that the rebase dropped the skill-index regeneration and only the coverage view was refreshed, but please verify: rerun python tools/build_skill_index.py --source-revision <rev> and commit the result (or explain the intended state), and update the PR body's "Generated — 69 files" line to match.

Still open from last round (no change in this delta)

  1. declarations still fails (job): "Missing, invalid or stale knowledge author declaration". The declaration still needs to name an admitted model/effort, or get an explicit exception approval.
  2. Source pin on squash merge: whatever revision the regenerated index is pinned to (previously 741c2ec2, now presumably 75afe47c) is branch-only and will be orphaned by a squash merge. This still needs a merge-strategy or post-merge regeneration decision.
  3. [code] vs [doc] markers in references/huggingarch-checkpoint-anchor.md (Gate 1 name composition vs _matching.py "not traced"; the HTTP 422 ingest refusal; the "16 provider environment files"), and the "2× long-context prefill baseline error" figure's section: unchanged, questions as before.
  4. Minor: metadata.json "created 2026-11-03" vs commit_date wording, and mentioning the provenance_readsets re-sort in the PR body.

Static review of the delta plus the hosted CI logs; I did not run anything locally.

@Dong1017 Dong1017 added the area:knowledge Knowledge line: packs, skill library, knowledge admission label Nov 4, 2026 — with ChatGPT Codex Connector

Copy link
Copy Markdown
Collaborator

Incremental review 18c9a16c..1694a0b5 (one new commit, 1694a0b5 "chore: regenerate skill index at the content commit"), checked against my previous review and the updated PR body. Not approving yet.

Fixed in this delta

  1. Stale skill index (my item 1 last round). I checked all 80 changed files under src/inferrsi/data/repo_packs/inferrsi/knowledge/: 79 change only the source pin 7aa3344b… → 75afe47c… (README L18 and L32 included), and common/skills/performance-ceiling-analysis.json additionally gains the two new references/huggingarch-* paths and hashes, the HuggingArch source entry, the extended verification note and the new limitation, all consistent with the canonical metadata.json. installed-package (3.10/3.11) is now green. full-check (3.10/3.11) was still running when I looked, so I can't yet confirm the six test_collect_selection / test_skill_index failures are gone; please confirm once it finishes.

Still open

  1. declarations still fails at this head (job). The declaration now truthfully lists claude-opus-5 / claude-opus-5-5 with effort unknown and requests an exception with the 80 regenerated records in scope, which is the right way to handle it. It still needs a knowledge-policy-approval comment from an authorized independent owner, and that approval must be renewed after any further push, so it's best requested after the last commit.
  2. Source pin on squash merge. The regenerated records and knowledge/README.md (L18, L32) are now pinned to 75afe47c, which is still a branch-only commit and would be orphaned by a squash merge. A merge-strategy decision (merge or rebase-merge) or a maintainer-owned post-merge regeneration step is still needed.
  3. [code] vs [doc] markers in references/huggingarch-checkpoint-anchor.md (Gate 1 name composition vs _matching.py "not traced", the HTTP 422 ingest refusal, the "16 provider environment files") and the section for the "2× long-context prefill baseline error" figure. This delta doesn't touch that file, so the questions from last round still stand.
  4. Minor: the metadata.json "created 2026-11-03" vs commit_date wording. Also, the regenerated performance-ceiling-analysis.json re-sorts provenance_readsets (uppercase INTEGRATION_NOTES.md/README.md/STATIC_CHECKS.md before cases/…), and the PR body's "only … changes beyond the source pin" description doesn't mention that.

Static review of the delta plus hosted CI status. I didn't run anything locally.

Copy link
Copy Markdown
Collaborator

Coverage review at head 1694a0b558c78f3920dc4f8667fee3b7c3b11c61 (base 663f3b327998526a13aa31e72ad729025c18983c). Incremental from my prior short-SHA note; no further commits. Not approving yet.

Fixed earlier (unchanged here)

  • Skill-index regeneration at content commit 75afe47c… addressed the stale packaged projection / test_collect_selection / test_skill_index failures from the rebase.

Still open

  1. declarations / knowledge-model policy. The declaration correctly lists claude-opus-5 / claude-opus-5-5 with effort unknown and requests an exception over the protected paths. It still needs a top-level knowledge-policy-approval comment from an authorized independent owner (and renewal after any further push).
  2. Source pin on squash merge. Regenerated records and knowledge/README.md pin to branch-only 75afe47c07e337e9592010cb67b079777515fd08. A squash merge would orphan that pin. Please get a merge-strategy decision (merge / rebase-merge) or a maintainer-owned post-merge regeneration step before merge.
  3. [code] vs [doc] markers in references/huggingarch-checkpoint-anchor.md (unchanged in this head): Gate 1 name composition vs _matching.py "not traced"; the HTTP 422 ingest refusal claim vs inspected kernel_bench/emit.py scope; the "16 provider environment files" claim; and the section for the "2× long-context prefill baseline error" figure. Please confirm, downgrade the marker, or point at the inspected file that supports each [code] tag.
  4. Merge state: mergeable_state=dirty against current main; rebase/regenerate as needed after the policy approval path is clear.
  5. Minor: metadata.json "created 2026-11-03" vs commit_date wording; regenerated performance-ceiling-analysis.json re-sorts provenance_readsets.

Reviewed commit range: 663f3b327998526a13aa31e72ad729025c18983c … 1694a0b558c78f3920dc4f8667fee3b7c3b11c61.

@Dong1017

Dong1017 commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

[inferrsi-oncall smoke] @Dong1017 please ignore — testing on-call wake 2026-11-07T05:42:13Z

@cursor

cursor Bot commented Nov 7, 2026

Copy link
Copy Markdown

On-call proxy for Dong1017's queue (inferrsi-oncall): smoke wake received. I am not Dong1017.

Acknowledged the test mention on #351 (comment) (2026-11-07T05:42:13Z). Per the request to ignore, no review and no verdict on this PR.

Verified: pr/meta.json (InferMatrixLab/InferRSI #351, triggering comment 6702938114). Not verified: diff, CI, or any code change — none were in scope for this wake test.

@Dong1017 Dong1017 added the kind:docs Documentation and template delivery label Nov 10, 2026
@fake0fan
fake0fan force-pushed the docs/ceiling-checkpoint-anchor branch from 1694a0b to 19f12f4 Compare November 10, 2026 07:41
@fake0fan

Copy link
Copy Markdown
Author

Thanks for the four rounds. Replies to the coverage review at 1694a0b5. New head: 19f12f4ed00997a2a79cfd07299411e67410c7b2, rebased onto main 3f560642.

1. declarations / knowledge-model policy. No change in substance. The declaration still names claude-opus-5 and claude-opus-5-5 with effort unknown. head_sha now points at the new head. The exception paths were recomputed from protected_paths: 86 paths, one more than before because main added vllm-ascend. It still needs a fresh knowledge-policy-approval from the approval owner for this head.

2. Source pin on squash merge. I'd propose no change, because this matches how main already works:

  • docs/agent-collaboration.md makes squash merge the required integration method, so merge or rebase-merge is not available.
  • The last 14 knowledge/README.md refreshes on main all pin branch-only commits. main currently pins d648f257 from knowledge: validate NInfer state and completion contracts #325.
  • Such pins still resolve on GitHub through the PR refs. I checked d648f257, 630bb57a and 7aa3344b through the API.
  • build_skill_index.py --check renders the projection without historical git objects.

The pin is now content commit 1536aeb6. A main-reachable pin would be a maintainer-owned post-merge regeneration; this PR doesn't change the convention.

3. [code] vs [doc]. I re-retrieved every cited file at facbbbc9, and all blobs and sha256 match the record. I then re-audited every marker, not only the four you named. The markers now have explicit definitions: [code] means executable code in an inspected file; docstrings, comments and behaviour in uninspected files count as [doc]. Every marker also has a locator. On your items:

  • Gate 1 name composition. Now [doc], citing docs/validation.md §4.2–4.3 and the _matching.py docstrings. The lookup (SafetensorsHeader.select) and composition (spec_tree) are in uninspected files. I've now read _matching.py in full. What stays [code]: CheckpointJoin.of builds both residuals, and shared names are summed in match_checkpoint_to_hf and bind_leaves_to_hf.
  • HTTP 422. You were right that it is not in emit.py. It is in docs/measured_calibration.md §4.3, an inspected section, so the refusal is now [doc]. The gate it applies (above_sol, utilization, UTILIZATION_CEILING) stays [code].
  • 16 environment files. These come from the GitHub tree listing: 16 *.txt files directly under kernel_bench/envs/. The document now labels this a listing fact, and the identity method says so. I dropped "one of which is the reference framework": the only support is §3.1, which is outside the recorded scope.
  • 2× prefill figure. It is in blog §2.6, "Guard:错了,有没有东西接得住", before §3.7, so it falls in the verbatim-retrieved part. I kept it, labelled unverified, and corrected the description: the post attributes the error to an agent-written deployment analysis that the tool caught, not to itself. cited_for records the section.

The full re-audit, which included an independent read-only pass, found more problems than the four you named. Fixed:

  • Gate 1 said each header tensor is claimed by "exactly one" leaf. The source sums the leaves that share a name, so "exactly one" is wrong.
  • The Gate 2 failure conditions, the swa_window gating and the two-channel ablation drop are all docstring-level. The behaviour lives in capacity and validate(), neither inspected, so all three are now [doc].
  • The rule that a degree which does not divide must be refused, co-owned or padded came from docs/parallelism.md, which was not in the readset. It is now added with blob, sha256, bytes and the one section read.
  • The two-threshold rationale ("a legitimate implementation can change the counts") did not fit the source. Both thresholds use the engine's plan numerators, and the source treats a row above peak as a wrong numerator. The reading is now that rejection is reserved for a row that contradicts its own record, and it is labelled as this document's reading.
  • Wording overstated the source in several places: "two or three range reads" (now one header read per shard), "dominates decode", "independent corroboration", "squashed", "computed once" and "analytical throughout". Each is corrected.

4. Merge state. Rebased; the branch is no longer dirty.

5. Minor.

  • metadata.json now reads "created on 2026-11-03 and has a single commit, dated 2026-11-04".
  • The provenance_readsets re-sort is reverted. Against main, the sidecar and the generated record now differ by two insertions only.

On your earlier open question: full-check (3.10/3.11) finished green at 1694a0b5.

Local verification at the new head is in the PR body. Same scope as before: documentation and reference material only, locally_verified: false, empirical_validation: not_run.

@fake0fan

Copy link
Copy Markdown
Author

@Dong1017 Before requesting the knowledge-policy approval on #351, could you decide the merge order between this PR and #359?

Whichever lands second has to rebase and regenerate the skill index:

Either works for me. Since #359's full-check is still red, I'd lean towards #359 first if it is close, so that #351 doesn't add another 81-file regeneration for it to absorb. Which do you prefer?

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base 3f560642a44a1496aa4984ab936517693be2c788 and head 19f12f4ed00997a2a79cfd07299411e67410c7b2 (full diff; prior-reviews.md has no earlier findings). Dong-bot will re-read after the author pushes.

Blocker

Scope of the checkpoint anchor disagrees with itself on per-rank weight bytes.

The procedure defines the anchor as two gates. Gate 1 is the bidirectional checkpoint join. Gate 2 is the per-rank weight ledger, and the reference states the consequence directly: passing the model-level byte anchor certifies nothing about per-rank weight bytes, so any L1 figure quoted per device or under a parallelism degree needs gate 2 (references/huggingarch-checkpoint-anchor.md, "Gate 2"). SKILL.md says the same thing before a per-device or per-degree figure is quoted. metadata.json verification.note (hunk at line 98) matches that wording: the prerequisite establishes nothing about per-rank bytes without its second gate.

The limitations string contradicts that pair. It leaves per-rank placement unanchored with no gate-2 condition:

The checkpoint anchor covers analytical weight demand only; KV traffic, per-rank placement and end-to-end serving performance remain unanchored by it.

Locations:

  • src/inferrsi/data/skill_library/methods/performance-ceiling-analysis/metadata.json limitations, line 122
  • the same sentence copied into src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/performance-ceiling-analysis.json at lines 214 and 257

A metadata consumer will treat declared per-rank layout as out of scope. The skill text tells them to run the ledger before quoting a per-device figure. Those cannot both be the contract.

Expected correction: make the limitation state the same scope as the verification note and the reference. KV traffic and end-to-end serving performance stay unanchored. Per-rank weight bytes stay unanchored until gate 2 passes. Gate 2 checks declared layout, format, and the non-dividing-degree fit policy; it does not prove runtime placement or KV counts. Regenerate the knowledge pack so both limitation copies match metadata.json.

Consistent in this head

  • SKILL.md, the anchor reference, docs/reference-matrix.md, and verification.note agree that the weight term goes through the bidirectional join plus the per-rank ledger, that KV growth is a separate justification, and that this source has no end-to-end predicted-versus-measured serving result.
  • [code] / [doc] markers in the anchor reference match the inspection notes in huggingarch-source-identity.json for the git files read here: join residuals and name summing in _matching.py; run_weight_ledger_check mapping in _weight_ledger.py; UTILIZATION_CEILING / above_sol in kernel_bench/emit.py; ResultIdentity.key in scope.py; _IMPLIES in _ablation.py. HTTP 422, the fit policy, and validate() stay [doc], with the uninspected files named.
  • The two-threshold rule is marked as this reference's reading where the source does not say why. The guard-ablation study is carried with the source's own non-reproducibility limits.
  • docs/skill-coverage.md moves performance-ceiling-analysis from 18 readsets to 20 and adds the HuggingArch row. The knowledge snapshot pointer moves from d648f25732dee1c974da62a94c46a642ad52cef8 to 1536aeb6cf6b6d3345388bac7f4f9537273efa40 across the pack.

Not verified here

  • Did not run python tools/build_skill_index.py --source-revision 1536aeb6cf6b6d3345388bac7f4f9537273efa40 --check or python tools/build_skill_coverage.py --check. Did not recompute content hashes.
  • Did not confirm 1536aeb6 is an ancestor of 19f12f4 whose skill_library bytes match the pack. Did not re-fetch shenh10/HuggingArch at facbbbc96856e558b9084a94a5070b2e2c86cf0b.
  • The seeded diff cuts off inside the blog document source of huggingarch-source-identity.json at "snapshot": "unk, and the next line is too long to read. The snapshot value, the "two figures" record, and JSON closure were not checked. CI was not inspected.

Optional

  • SKILL.md gate 1 says both "agree in physical element count" and "compare bytes rather than elements wherever precision is mixed." The reference keeps element agreement on the per-tensor join and switches the checksum total to bytes when precision is mixed. One sentence in SKILL.md pointing at that split would keep the summary from being read as flipping the primary gate.
  • The anchor doc's "Why L1 needs this" states "one header read per shard, plus the shard index" with no [code] or [doc] marker.

This review does not approve the change.

@cursor

cursor Bot commented Nov 10, 2026

Copy link
Copy Markdown

#359 first. Hold #351, and request knowledge-policy-approval only on the head you rebase after #359 is on main.

I have not opened #359's full-check log or re-measured that single-owner result; this follows your report on this thread. If that red check becomes a long blocker, say so here and the order switches to #351 at 19f12f4 with the 86-path exception approved as written.

Copy link
Copy Markdown
Collaborator

Incremental review 1694a0b558c78f3920dc4f8667fee3b7c3b11c61..19f12f4ed00997a2a79cfd07299411e67410c7b2 (rebased onto main 3f560642a44a1496aa4984ab936517693be2c788, so I re-read the canonical diff in full: SKILL.md, metadata.json, references/huggingarch-checkpoint-anchor.md, plus the generated performance-ceiling-analysis.json), checked against my coverage review and your reply. Not approving yet.

Resolved

  • [code] / [doc] markers. The definitions at the top of the anchor reference are explicit, and the four items I named are now handled as you describe: Gate 1 name composition and the HTTP 422 refusal are [doc] with locators, the 16 env files are labelled a tree-listing fact, and the 2× prefill example is kept as blog-sourced and unverified. The extra corrections ("exactly one" leaf replaced by summing shared names, Gate 2 conditions and ablation channel drop downgraded to [doc], docs/parallelism.md added to the readset) read correctly.
  • Merge state. Rebased; no longer dirty.
  • Minor. The metadata.json date wording is fixed and the provenance_readsets re-sort is reverted.
  • Source pin. Thanks for the evidence that squash merge is required and that main already pins branch-only commits. I accept that this PR follows the existing convention; a main-reachable pin is a separate maintainer decision.
  • CI. full-check (3.10/3.11), installed-package (3.10/3.11) and pr-references are green on this head.

Still open

  1. declarations is failing on this head: both CI — knowledge declaration policy runs on 19f12f4e concluded failure, since there is no knowledge-policy-approval yet. Per the merge-order discussion, if prototype: reduce knowledge projection and declaration churn #359 lands first this head will be rebased and regenerated anyway, so I'll re-review on whichever head carries the approval.
  2. Scope wording is inconsistent on per-rank bytes. src/inferrsi/data/skill_library/methods/performance-ceiling-analysis/metadata.json line 122 says "KV traffic, per-rank placement and end-to-end serving performance remain unanchored by it", with no Gate 2 condition. SKILL.md (Gate 2) and verification.note say per-rank weight bytes are unanchored only without the second gate. Please align the limitation, for example: KV traffic and end-to-end serving performance remain unanchored; per-rank weight bytes stay unanchored until Gate 2 passes, and Gate 2 checks declared layout and format, not runtime placement. Then regenerate so both copies in knowledge/common/skills/performance-ceiling-analysis.json match. (cursor[bot] flagged the same point.)

Optional

  • SKILL.md lines 43–51: Gate 1 says tensors must "agree in physical element count" and later "compare bytes rather than elements wherever precision is mixed". A reader can take that as switching the primary gate to bytes. One clause saying the per-tensor join stays on elements and only the checksum total switches to bytes would remove the ambiguity, matching the anchor reference.

fake0fan and others added 2 commits November 11, 2026 07:59
L1 divides a count by a rated peak and never looks at the model again,
so a wrong weight count reaches the floor unchallenged. Add a
prerequisite with two gates, distilled from HuggingArch at its root
commit:

- Gate 1 joins the safetensors headers with the analysis's stored
  tensors in both directions, compares summed rows per name in physical
  elements, and treats byte totals as a checksum on that join.
- Gate 2 requires a declared per-axis placement, a format and a
  non-divisibility policy before any per-rank weight figure. Gate 1
  needs no layout, so it certifies nothing per rank.

State what the anchor leaves open: KV size and growth (windowed and
global attention have byte-identical weights), cache precision,
activations, collectives and achieved rates.

Add the two-threshold rule for measurements: a row is rejected only when
its own work over its own time exceeds the peak it recorded; a row
faster than a separately computed floor is flagged and both sides are
re-derived. The source's numeric constants stay provenance only.

Every claim about the source carries a [code], [doc], [tree] or [blog]
marker with a locator. The source identity record gives the 12 files'
blobs, sha256, sizes and read ranges, all matched against the commit's
tree. Companion-post figures absent from the repository are recorded
as unverified. Add the metadata sources, a limitation, two readsets and
a reference-matrix row. locally_verified stays false and
empirical_validation stays not_run.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>
Regenerate with `build_skill_index.py --source-revision 32623f7` and
`build_skill_coverage.py`. Only the performance-ceiling-analysis
record, the shared _provenance.json pin and the coverage view change.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>
@fake0fan
fake0fan force-pushed the docs/ceiling-checkpoint-anchor branch from 19f12f4 to b3f1009 Compare November 11, 2026 08:13

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS

Reviewed base 4b3afa55f278ca314e07d5f8acbd0a18f3fa72bf and head b3f10097ffb7211f76c6bfb1b5659ff8094d99b0.
Incremental range 19f12f4ed00997a2a79cfd07299411e67410c7b2..b3f10097ffb7211f76c6bfb1b5659ff8094d99b0 (prior review base 3f560642a44a1496aa4984ab936517693be2c788). Dong-bot will re-read after the author pushes.

Blocker

Prior finding still open: the checkpoint anchor still disagrees with itself on per-rank weight bytes.

The review at head 19f12f4e blocked because the limitations string leaves per-rank placement unanchored with no gate-2 condition (The checkpoint anchor covers analytical weight demand only; KV traffic, per-rank placement and end-to…). Gate 2 in references/huggingarch-checkpoint-anchor.md, SKILL.md, and metadata.json verification.note say the opposite: a passing model-level byte anchor certifies nothing about per-rank weight bytes, and any L1 figure quoted per device or under a parallelism degree needs gate 2.

The readable incremental diff has no hunk for:

  • src/inferrsi/data/skill_library/methods/performance-ceiling-analysis/references/huggingarch-checkpoint-anchor.md
  • src/inferrsi/data/skill_library/methods/performance-ceiling-analysis/SKILL.md
  • src/inferrsi/data/skill_library/methods/performance-ceiling-analysis/metadata.json

Those three texts are what this re-review can still see. Attach the gate-2 condition to every per-rank, per-device, and per-degree weight-byte claim, in the same words as Gate 2, SKILL.md, and verification.note.

docs/reference-matrix.md does add the HuggingArch row and calls the reference a two-gate weight anchor. Its boundary sentence is “Anchors stored weights only; KV, collectives and E2E performance unanchored”. That row does not replace the limitations string above.

Readable range

  • docs/skill-coverage.md: performance-ceiling-analysis readsets 18 → 20, HuggingArch listed; input digest sha256:ee418cd9ed09c7e5c745808637387ad7a02a9ede459563ba2e56d4229d299e67.
  • Base advanced 3f560642 → 4b3afa55, so previous..head also carries other tree changes (knowledge-contribution workflow, historical plan banners, faster-cosyvoice rows, evidence/h3-ref2va-910c-4logical-20261101-297/**). This review judges the anchor-doc blocker. Those other paths are not new blockers against 4b3afa55.

Not verified

  • pr/diff.patch line 2746 is one line the reader would not return. Lines 1–2744 already exceed 100000 characters; the file is 150155 bytes. That line follows a mid-token line in evidence/h3-ref2va-910c-4logical-20261101-297/patch/c12-candidate.patch (assert model._alignment_ca), the 252nd line of a +1,252 test hunk inside a patch declared +1,633. Bytes after that point are unread. If the limitations edit lives only there, it is still unverified.
  • Tests not run. CI not checked. PR body and any knowledge-authorship declaration were not in the seed.

This comment does not approve the PR and does not change its scope.

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 4b3afa55f278ca314e07d5f8acbd0a18f3fa72bf and head b3f10097ffb7211f76c6bfb1b5659ff8094d99b0. The branch was rebased and its commits rewritten, so I compared the PR diff at my last reviewed head 19f12f4ed00997a2a79cfd07299411e67410c7b2 (base 3f560642a44a1496aa4984ab936517693be2c788) with the PR diff at this head, file by file.

Both items from my previous review are resolved:

  • Per-rank scope wording. The metadata.json limitation now says per-rank weight bytes stay unanchored until the second gate passes, and that Gate 2 checks the declared layout and format, not runtime placement. This matches SKILL.md (Gate 2), verification.note and the anchor reference. Both copies in the regenerated knowledge/common/skills/performance-ceiling-analysis.json carry the new text, and the old "per-rank placement … remain unanchored" wording is gone.
  • declarations CI. Green on this head, together with full-check (3.10/3.11), installed-package (3.10/3.11), pr-references, plan and publish.
  • The optional Gate 1 clarification is also in: the per-name comparison stays in elements, and only the post-join byte totals switch to bytes (SKILL.md item 4; anchor reference, Procedure steps 4–5).

I also re-read the rewritten references/huggingarch-checkpoint-anchor.md and spot-checked its [code] locators against HuggingArch at facbbbc96856e558b9084a94a5070b2e2c86cf0b. These match the pinned source: emit.py L30–32 (UTILIZATION_CEILING), op_calibration.py L59 and L924–939 (ANOMALY_THRESHOLD, measured_anomalous), _matching.py L157–171 and L324–342 (summed physical elements, empty-header refusal, two residuals), _weight_ledger.py L37–51, and _ablation.py L40. The blog material is kept as [blog] and unverified, and the adoption boundary keeps the numeric constants as provenance only.

Not verified: I did not run the tests locally. I did not check every [doc] locator or the per-file digests in huggingarch-source-identity.json.

@fake0fan

Copy link
Copy Markdown
Author

@Dong1017 Pls review it

@cursor

cursor Bot commented Nov 11, 2026

Copy link
Copy Markdown

PR review verdict — DO NOT PASS

Reviewed base 4b3afa55f278ca314e07d5f8acbd0a18f3fa72bf and head b3f10097ffb7211f76c6bfb1b5659ff8094d99b0 (content commit 32623f7c062227d038bf48c59ed85792872d46b0) from the PR body and the thread. This head is the post-#359 rewrite, so it replaces the tree last reviewed at 19f12f4ed00997a2a79cfd07299411e67410c7b2.

Addressed in the PR body (file text at this head still unread)

  • Per-rank scope matches the open item on 19f12f4e: KV size and growth, activation traffic, collectives, and end-to-end serving stay unanchored; per-rank weight bytes stay unanchored until gate 2 passes; gate 2 checks declared layout and format, not run-time placement. The body says metadata.json, SKILL.md, the verification note, and the anchor reference share that wording.
  • Gate 1 keeps the per-name comparison in elements; only the checksum totals move to bytes. A per-name mismatch fails the InferRSI procedure either way, including where match_checkpoint_to_hf calls it a diagnostic.
  • The declaration is one admitted author: Claude Code, claude-opus-5-5, effort xhigh, head_sha b3f10097ffb7211f76c6bfb1b5659ff8094d99b0. No exception. knowledge_declaration_paths.py is reported at 7 protected paths. That attestation (rewrite from the facbbbc9 pin, with 0a2e2335 left behind) is what drops the old claude-opus-5 / unknown exception.
  • Generated refresh is the three files the prototype: reduce knowledge projection and declaration churn #359-first order called for (performance-ceiling-analysis.json, _provenance.json, docs/skill-coverage.md) plus the canonical edits and docs/reference-matrix.md. Those eight paths match the skill-eval uncovered list.
  • Branch-only pin 32623f7c after squash stays the convention already accepted on the previous head.

Blocking

  1. No successful declarations result for b3f10097 is in the thread. That check failed on 19f12f4e and the heads before it. This declaration is a new attestation, so PASS needs the job green at this head_sha. I have not compared this text with 0a2e2335.
  2. Skill-eval at this head is still planned, not executed (comment 6627156289). It asks for python tools/build_skill_index.py --source-revision 32623f7c062227d038bf48c59ed85792872d46b0 --check with exit status and output. The body reports a local pass for both --check runs, ruff, the knowledge and skill-library tests, and the installed-wheel smoke (128 records), and says the 153 failures match an untouched main at 4b3afa55 (git 2.34, skill-eval and yibuapi_runtime). I did not run those commands. The local wrapper drops --no-lazy-fetch; the hosted run is what confirms the same bytes.

Disposition

  • The eight uncovered paths are this anchor's docs and generated projection. Scope stays documentation. locally_verified remains false and empirical_validation remains not_run. HuggingArch constants, including 0.95, stay provenance; any flag tolerance stays with the acceptance issue.
  • Still out of scope: [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303 Class A/B, vendoring a kernel corpus, a new skill id, and any measurement.

Not verified here

  • Canonical wording, locators, and the facbbbc9 blob/sha256 record (including op_calibration.py blob a4074fd5).
  • Hosted CI on b3f10097.

I will re-read this head once declarations and the source-revision --check are green on b3f10097. This review does not change scope.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:knowledge Knowledge line: packs, skill library, knowledge admission kind:docs Documentation and template delivery

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants