skill: add model-level speed-of-light analysis (model-sol-analysis) by fake0fan · Pull Request #386 · InferMatrixLab/InferRSI

fake0fan · 2026-11-10T08:55:13Z

Refs #385 (the issue's acceptance owner decides closure; a merged PR does not satisfy the goal by itself). Builds on #351.

Draft, stacked on #351. The first three commits are #351's. Review only the last three:

64adb54a: the skill.
817f57e3: a provisional index regeneration, so that CI can run.
dbe0cf04: one clean-wheel count.

After #351 lands, this rebases onto main. The index is regenerated for real only after #359 lands, which avoids another repository-wide pin-only rewrite. Until #351 merges, the diff against main includes #351's content, so the knowledge-policy declaration below lists its author too. No approval is requested at this head.

What this adds

methods/performance-ceiling-analysis prices one device, one operation class and one traffic tier. Its roofline contract states that it does not walk a model, sum devices or compose stages. So "what is the decode ceiling of model M under layout L on card H?" had no procedure.

methods/model-sol-analysis is a new candidate owner, distilled from HuggingArch (MIT) at e69d20a028e5b17524aa48b53bddf8b9aad5d15b. It works in eight steps:

Anchor the counts before pricing them. The weight join and per-rank ledger come from docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351. Shapes must compose. KV type contracts and KV growth laws are checked by letting the context grow without bound. A gate that could not be judged is a failure.
Bind each phase: prefill and decode lengths, with attention counted in causal pairs, not s².
Price the fused graph the framework runs at its pinned release, with a source anchor for every fusion. Traffic comes from fused-graph boundaries.
Evaluate per rank: declared non-divisibility policies (refuse, co-own, pad; never a silent divisor), the MoE busiest-rank factor from a measured skew, and storage width separated from compute width.
Make communication explicit: bytes and hops per collective algorithm (ring, one-shot, two-shot, in-switch reduction), intra- versus inter-node links, hierarchical all-reduce.
Compose the phase and check capacity first.
Report headline figures with their labels.
Calibrate beside the model, never over it.

The one design decision to review

HuggingArch's "SOL" is Σ_op max(t_compute, t_memory, t_comm, t_launch), a serial sum over fused ops ([code]). The source rejects the model-level max(Σ flops/peak, Σ bytes/bw) as one that "under-counts the true time". That holds only under its own stated assumption that "no compute hides a collective or another kernel's launch", and its own code comment calls the whole-step maximum "a bound and not an estimate". A stack that overlaps kernels can finish below the per-op sum, but never below the resource-total maximum.

So this skill reports both, in separate columns:

Figure	Definition	May reject a result?
Conditional bound	`max over resources (irreducible per-rank demand / peak rate)`	Yes, if counts are irreducible (for example, the cheapest admissible collective) and every peak is at the clock the device actually runs that work at
Serial estimate	`Σ_op max(compute, memory, summed link phases + latency, launch floor)` from the counts actually run, no overlap	No; it may only flag a faster result

Link latencies and launch floors enter the estimate, never the bound. A spec-sheet peak computed below the clock the device actually runs, which the source shows real rows exceeding, is not a bound input. This is the same split as #351's two-threshold rule.

Files

File	Change
`methods/model-sol-analysis/SKILL.md`	Procedure, inputs, output, stop conditions, known gaps
`.../references/huggingarch-model-sol.md`	Source reading; every claim about the source is `[code]` or `[doc]` with a locator, and interpretations are labelled
`.../references/huggingarch-source-identity.json`	44 retrieved files: blob, sha256, bytes and per-file inspection scope (files read but not cited, and files hashed only, are listed as such)
`.../scripts/model_sol.py` + `references/model-sol-contract.md` + `references/model-sol-example.json`	Pure helper: `join_checkpoint`, `kv_cache_bytes`, `phase_floors`. JSON in, JSON out, no I/O, `INSUFFICIENT_EVIDENCE` instead of defaults
`tests/test_model_sol_analysis.py`	43 synthetic tests, covering hand-calculated cases, residuals, the checksum-passes-where-join-fails case, evidence boundaries, a bound ≤ estimate property, no I/O, and reading at the checkout revision
`tests/check_installed.py`	Clean-wheel smoke for the helper
`skill-index.json`, library README, `docs/knowledge-base.md`, count tests	Register one skill. The documented counts already lagged the registry by one (79 versus 80); they now read 81 and 144
`docs/searchspace-matrix.md`, `docs/reference-matrix.md`	Accelerator and Network rows; one source row next to #351's
`methods/performance-ceiling-analysis/SKILL.md`	One cross-link to the new owner
`docs/skill-coverage.md`, `repo_packs/inferrsi/knowledge/**`	Provisional regeneration (see above)

Provenance

Three read-only passes read the source in parallel: cost model, parallelism/KV, and validation/calibration/limits. Every retrieved file's git blob and size matched the tree listing at the pin.
The load-bearing [code] claims were rechecked against the source text before writing. A separate independent read-only audit then checked every marker and locator; its findings are folded in.
docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 keeps its own pin facbbbc9…, the repository's root commit. At this pin, every file docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 relied on is byte-identical except docs/measured_calibration.md and docs/parallelism.md, and docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351's claims still hold.
locally_verified: false, empirical_validation: not_run. No accelerator was used, HuggingArch was not run, and no predicted figure has been compared with a measurement.

Found along the way, not changed here

performance-ceiling-analysis gives attention FLOPs as a full seq² for DiT, whose attention is bidirectional, where that is right. Applied to causal LLM prefill, it would overstate attention work by about 2×, since causal pairs count about half of seq² (inference, recorded in the source reading).
docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351's description of the anomaly badge is loose. In code it fires only below a fixed fraction (0.95) of the roofline at the row's own test-time peaks. This will be tightened when docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 is reworked after prototype: reduce knowledge projection and declaration churn #359.

Out of scope

The SOL ruler, Class A/B vocabulary and gap metric of [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303. This skill produces model-level floors and estimates; it neither freezes a ruler nor ranks agent flows.
Vendoring HuggingArch code, specs or its measured corpus. The helper is written from the documented method.
Any measurement, numeric gate, policy change or index-format change.

Verification at this head

Head dbe0cf04. The skill commits are 64adb54a (content), 817f57e3 (provisional regeneration) and dbe0cf04 (clean-wheel count).

python tools/build_skill_index.py --source-revision 64adb54a2f96307d647791d1204a5123ebffe3e0 --check
python tools/build_skill_coverage.py --check

Local, Python 3.11:

ruff check src tests tools passes.
tests/test_model_sol_analysis.py: 43 tests pass.
test_skill_index, test_collect_selection, test_knowledge_contract, test_skill_coverage_view, test_skill_library and test_performance_ceiling pass.
The clean-wheel tests/check_installed.py smoke passes, with 129 records and the new helper exercised.
Full suite: 154 failures, identical test-for-test to an untouched main checkout at 3f560642 run in the same environment, with no new failures. All are in skill-eval and yibuapi_runtime tests, which this PR does not touch; this workstation has git 2.34 and no python on PATH.

The generator ran through a local git wrapper that drops only --no-lazy-fetch, which git 2.34 lacks. This clone has no promisor remote, so the hashed blob bytes are unaffected.

Knowledge author declaration

{
  "schema_version": 1,
  "head_sha": "dbe0cf048a1a4c534ed4bba81bb0ca5b85efa070",
  "human_only": false,
  "material_ai_authors": [
    {
      "tool": "Claude Code",
      "model_id": "claude-opus-5",
      "effort": "unknown",
      "role": "author"
    },
    {
      "tool": "Claude Code",
      "model_id": "claude-opus-5-5",
      "effort": "xhigh",
      "role": "author"
    }
  ]
}

claude-opus-5 appears only because this branch carries #351's content commit. Once #351 has merged and this PR is rebased, the declaration will list claude-opus-5-5 / xhigh alone, and no exception should be needed. Until then the policy check fails, and no approval is requested.

🤖 Generated with Claude Code

`methods/performance-ceiling-analysis` L1 divides a count by a rated peak and never revisits the model, so an error in the count reaches the floor unchallenged and no later roofline step re-examines it. The skill had no procedure for checking the demand side; `safetensors` appeared nowhere under this owner while the loading mechanics already lived next door. Add a two-gate prerequisite: the bidirectional per-tensor checkpoint join, and the separate per-rank weight ledger. The first needs no parallelism layout, so passing it certifies nothing about per-device bytes, and the distinction is stated rather than implied. State the blind spot too: the anchor covers the weight traffic term only, since a windowed and a global attention with equal head dimensions have byte-identical weights. Record the reviewed external source it derives from at a pinned commit, with per-file git blob and sha256 digests and per-file inspection scope. Documentation claims and source-verified claims are marked separately. Carry the source's two-threshold rule, which grants rejection authority only to a rated-peak bound and not to a per-operation roofline, and its archived guard-ablation study with the source's own limitations intact. What the source does not establish is recorded as explicitly as what it does: no end-to-end validation of predicted against measured serving performance within the inspected scope, unmeasured collectives upstream, and companion-blog corpus figures that the repository documents do not state and that must not be cited. `locally_verified` stays false; `empirical_validation` stays not_run. Agent-assisted-by: Claude Code / claude-opus-5 (author) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>

Define the two markers precisely. [code] means executable code in an inspected file. [doc] means the project's prose, docstrings and comments included, and also covers any behaviour that lives in a file that was not inspected. Re-audit every marker against the pinned commit and give each one a locator. Downgrade the claims that rested on docstrings or documentation: - the name lookup and composition behind the join - the layout-free join - which conditions fail the weight ledger - what the swa_window lever gates - the two-channel ablation drop - the HTTP 422 upload refusal The speed-of-light gate function, the lever vocabulary, the join's two residuals and the summing of shared names stay [code]. Record the environment-file count as a tree-listing fact. Correct statements against the source: - A header tensor is claimed by a leaf, not by exactly one; leaves sharing a name are summed. - The rule for a degree that does not divide comes from docs/parallelism.md, which now joins the readset with its digests and inspected section. - Both thresholds of the two-threshold rule use the engine's own counts, so the old rationale (rejection tracks invariance, because a legitimate implementation can change the counts) does not fit a source that refuses that case as a wrong numerator. The labelled reading is now that rejection is reserved for a row that contradicts its own record; a separately computed floor may only flag. - A sharded checkpoint needs one header read per shard, not two or three reads. - KV traffic can dominate decode only at long context or large batch. - The ablation study is archived evidence, not independent corroboration. - A single commit is not evidence of squashing. Locate the blog's 2x prefill example in section 2.6, inside the part retrieved verbatim, and describe it as an agent-written analysis that the tool caught. Reword the repository creation date so it is not read as the commit date, and keep the existing readset order. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>

Regenerate with `build_skill_index.py --source-revision 1536aeb` and `build_skill_coverage.py`. Apart from the performance-ceiling-analysis record and its coverage row, every generated file changes only its source pin. Agent-assisted-by: Claude Code / claude-opus-5-5 (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>

performance-ceiling-analysis prices one device, one operation class and one traffic tier. Its contract says it does not walk a model, sum devices or compose stages, so a question such as "what is the decode ceiling of this model under this layout on this card" had no procedure. Add methods/model-sol-analysis, distilled from HuggingArch at a pinned commit. The procedure: - anchors weights, shapes and KV growth before pricing anything; - binds each phase with causal pair counts; - prices the fused graph the serving framework runs, with a source anchor for every fusion; - evaluates per rank under declared non-divisibility policies and MoE skew; - makes communication explicit; - keeps measured kernels beside the analytical figure, never over it. It reports a resource-total bound apart from a serial per-op estimate. The source publishes its per-op sum as "SOL", while its own code comment calls the whole-step maximum the bound. The per-op sum assumes no overlap between kernels, so here it is an estimate that may only flag a faster result; only the bound may reject one. The bound uses irreducible counts, such as the cheapest admissible collective, and peaks at the clock the device actually runs. Link latencies and launch floors enter only the estimate. A pure helper composes supplied counts: the checkpoint join, KV bytes by growth law, and per-phase bound and estimate. It does no I/O and is covered by synthetic tests. The source identity records 44 retrieved files with blob, sha256, bytes and inspection scope, and the source reading marks each claim [code] or [doc] with a locator. Register the skill, add the matrix rows, link it from performance-ceiling-analysis, and correct the library counts, which already lagged the registry by one. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>

Provisional regeneration so that CI can run on the Draft. Regenerate with `build_skill_index.py --source-revision 64adb54` and `build_skill_coverage.py`. Besides the new model-sol-analysis record, two things change: - the performance-ceiling-analysis record, for its new cross-link; - the coverage rows. Every other record changes only its source pin and the skill-index.json hash that the new registry entry moves. The final regeneration happens after InferMatrixLab#351 and InferMatrixLab#359 land. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>

The installed package now ships 129 source-grounded records, one more for model-sol-analysis. Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Chenguang Zheng <zhengcg0505@gmail.com>

github-actions · 2026-11-10T08:56:01Z

Skill evaluation

PLANNED: not executed; there are no task outcomes yet.
No approval, model efficacy or current-head test result is implied.

Next: run the pinned source/index --check below at head dbe0cf048a1a4c534ed4bba81bb0ca5b85efa070.
Task selection: no supported automatic comparison; choose a relevant source/content check and disposition uncovered claims.
Selected comparisons: 0 · Estimated task calls: 0. Uncovered/unsupported changes: 112; Canonical library paths: 13; Generated projection paths: 82; Other paths: 17. These are paths, not independent claims.
Return: command, input/source pin, exit status, outputs/failures and untested claims for independent verified/refuted/unverified disposition.

Author verification steps (nonpaid; not yet run)

Bind the checkout. Run git rev-parse HEAD; it must equal dbe0cf048a1a4c534ed4bba81bb0ca5b85efa070. If it differs, obtain a current plan; do not reuse old results.
Check source/index consistency. At that PR checkout, using the reviewed repository tool and its documented development dependencies, run:

python tools/build_skill_index.py --source-revision 64adb54a2f96307d647791d1204a5123ebffe3e0 --check

The declared source pin matches this head's canonical library bytes. The command has not run: retain its exit code and full drift/success output. It checks the generated projection, not claim truth or skill efficacy.
No supported automatic comparison is selected. Review the canonical changes and explain which claims need evidence, are unaffected, or remain unverified; do not treat generated records as separate behavioral claims.
Uncovered scope by location: Canonical library paths: 13; Generated projection paths: 82; Other paths: 17. Path counts are not claim counts; no metadata-only conclusion is inferred.
3. Return evidence for independent review: head, check/task/source pin, command, environment, exit status, outputs/failures and untested claims. Label scripted controls. Request each claim's disposition: verified, refuted or unverified; this plan proves neither model provenance nor benefit.
Check recipes and evidence limits

Evidence and logs · Retention: 30 days.
Retained artifacts are verifiable only before Unix time 1796894387 while the linked artifact is available. Expired evidence is unverified.

Plan bindings, budget, frozen exam and advisory details

Advisory: obtain missing evidence

Retained evidence availability: planned / unexecuted. No approval or merge authorization is issued.
Applicable only while these revisions and inputs match the current PR.
Paid execution requires trusted label authorization and cumulative PR reservation. Unknown accounting blocks new paid calls; planning is not execution.
Pipeline evidence only; offline fixed outputs do not establish skill efficacy.
Unchanged-correct is valid neutral evidence, not incremental benefit.
Content/source review and authorship admission remain separate.
Trusted regrading checks retained answers, not model/hardware provenance or a hostile executor; this is not an efficacy certification.

Retained evidence is planned / unexecuted.
No supported comparison cases were selected.
Uncovered/unsupported scope needs a case or explicit reviewer disposition.
Plan SHA-256: 8ea8c9099e6d3aacfeb2622cc60679107e8d170850476ce89470169805a6c3f2
Base: d699e570025d768b38528bd620d60b17553996ef · Head: dbe0cf048a1a4c534ed4bba81bb0ca5b85efa070
Trusted implementation: d699e570025d768b38528bd620d60b17553996ef

Model/settings: {"effort":"not_applicable","name":"claude-opus-5-5"}
Budget: {"comparisons":0,"estimated_calls":0,"limits_per_comparison":{"campaign_seconds":600,"max_task_calls":2,"repeats":1,"timeout_seconds":180,"token_stop_new_trials":200000},"provider_config_sha256":"b28d80f8194fbe0f5a9f01515efb24b218979fd45f702776560da276f4b30d04","reservation":null}

Skill	Comparison	Cases	Estimated calls

Automatic execution / cumulative PR budget:

{"reason":"Trusted configuration or pricing missing, invalid or expired","remaining_usd":"unknown","reserved_or_unresolved_usd":"unknown","spent_or_legacy_usd":"unknown","state":"blocked"}

Projection check selection (not an executed result):

{"declared_in":"src/inferrsi/data/repo_packs/inferrsi/knowledge/README.md","source_revision":"64adb54a2f96307d647791d1204a5123ebffe3e0","status":"ready"}

Provider/settings: {"client":{"api_version":"2023-06-01","endpoint":"https://yibuapi.com/v1/messages","kind":"anthropic-messages-full-context-v1","max_output_tokens":4096,"max_read_bytes":32768,"max_request_bytes":262144,"max_requests":1,"max_tool_calls":12,"provider":"yibuapi"},"limits":{"token_stop_new_trials":200000},"model":{"effort":"not_applicable","name":"claude-opus-5-5"},"pricing":{"currency":"USD","expires_at":1793271228,"group":"default","max_input_tokens":1000000,"model":"claude-opus-5-5","rate_evidence":"https://github.com/InferMatrixLab/InferRSI/issues/215#issuecomment-6467531341","rates":{"cache_read":"0.4","cache_write":"10","input":"8","output":"40"},"request_fee":"0","unit":"per_million_tokens"},"spend_limit":{"amount":"200","currency":"USD"}}

Uncovered / unsupported paths (112)

uncovered: "PHASE1_REPORT.md"
uncovered: "SUBMISSION_NOTES.md"
uncovered: "docs/knowledge-base.md"
uncovered: "docs/plans/2026-10-14-phase1-report.md"
uncovered: "docs/plans/2026-10-18-inferrsi-rename.md"
uncovered: "docs/plans/2026-10-18-negative-diagnostics.md"
uncovered: "docs/plans/2026-10-19-campaign-plan-rfc.md"
uncovered: "docs/plans/2026-10-28-submission-notes.md"
uncovered: "docs/reference-matrix.md"
uncovered: "docs/reviews/source-audit-ac70272.md"
uncovered: "docs/searchspace-matrix.md"
uncovered: "docs/skill-coverage.md"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/README.md"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-aclgraph-capture-replay.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-device-health.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-distributed-parallel.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-graph-mode.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-model-adaptation.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-operator-profiling.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-operator-tuning.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-optimization-casebook.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-performance-analysis.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/ascend-video-diffusion-optimization.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/ascendc-tiling-design.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/cann-attribution-overlap.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/torch-npu.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/torchnpu-device-memory-lifecycle.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/torchnpu-pinned-host-memory-lifecycle.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/ascend/skills/vllm-ascend.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/assess-serving-preemption.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/assess-split-backward-schedule.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/audit-fixed-workload-claim.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/audit-h3-metal-fasth3-semantics.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/audit-h3-runtime-residency.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/audit-llama-runtime-state.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/audit-recurrent-kv-finalization.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/audit-sd-cpp-residency.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/audit-sglang-diffusion-lifetimes.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/audit-vllm-kv-lifetimes.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/bound-windowed-attention-replay.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/budget-speculative-verification.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/calibrate-serving-simulation.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/cgroup-resource-controls.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/checkpoint-recovery-consistency.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/choose-parallel-training-plan.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/cite-check-downstream-pr.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/claim-contract.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/comfy-ui.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/competitor-scan.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/dcp-async-save-performance.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/dense-performance-evidence.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/diagnose-vllm-admission-pressure.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/diffusers-invocation-state.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/draft-acceptance-issue.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/dwarfstar.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/evaluate-elastic-serving-placement.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/generation-cache-lifecycle.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/host-memory-management.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/host-overhead-optimization.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/husky.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/inference-benchmark-design.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/inference-optimization-casebook.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/kernel-writing-guide.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/kv-tier-management.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/lazy-evaluation-stream-lifetimes.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/lily.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/lithos-metal.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/local-dream.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/mistral-rs.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/mlc-llm.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/mlx-swift-lm.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/model-sol-analysis.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/npu-skill-factory.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/optimization-search-policy.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/os-process-placement.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/performance-ceiling-analysis.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/plan-activation-residency.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/project-maya.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/pytorch.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/repository-evolution-analysis.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/rsi-evidence.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/skill-evolution-playbook.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/splash.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/storage-io-measurement.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/training-input-pipeline.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/training-resource-budget.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/video-diffusion-optimization.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/vllm-metal.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/common/skills/weight-loading-optimization.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/cuda/skills/audit-afd-p2p-exchanges.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/cuda/skills/audit-flare-session-lifecycle.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/cuda/skills/diagnose-attention-bottlenecks.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/cuda/skills/host-overhead-profiling.json"
uncovered: "src/inferrsi/data/repo_packs/inferrsi/knowledge/cuda/skills/reason-hopper-kernels.json"
uncovered: "src/inferrsi/data/skill_library/README.md"
uncovered: "src/inferrsi/data/skill_library/methods/model-sol-analysis/SKILL.md"
uncovered: "src/inferrsi/data/skill_library/methods/model-sol-analysis/metadata.json"
uncovered: "src/inferrsi/data/skill_library/methods/model-sol-analysis/references/huggingarch-model-sol.md"
uncovered: "src/inferrsi/data/skill_library/methods/model-sol-analysis/references/huggingarch-source-identity.json"
uncovered: "src/inferrsi/data/skill_library/methods/model-sol-analysis/references/model-sol-contract.md"
Additional paths omitted; see retained evidence.

amy-why-3459 · 2026-11-10T08:58:35Z

Review (COMMENT only, not an approval)

Reviewed range: the three commits you flagged as this PR's own, 19f12f4ed00997a2a79cfd07299411e67410c7b2..dbe0cf048a1a4c534ed4bba81bb0ca5b85efa070 (64adb54a, 817f57e3, dbe0cf04). The #351 commits are covered on #351. I focused on scripts/model_sol.py, references/model-sol-contract.md and tests/test_model_sol_analysis.py. Thanks for following up #385 so quickly. The split between bound and estimate is implemented the way the issue describes.

Verified

phase_floors(): the bound is max_r Σ_op bound_demand_r × repeat / rate_r, with no latencies or launch floors. The estimate is Σ_op repeat × max(per-rate times, summed link time + latency, launch). Each op's max term is at least its share of every per-rate total, and bound_demand ≤ demand is enforced, so the contract's claim that the estimate "is never below the bound" holds by construction.
A peak rate requires clock_basis, and mixed peak/synthetic/measured rates go to INSUFFICIENT_EVIDENCE. This matches the stated rule that only a bound may reject a result.
Unknown dtypes, missing sources, unit mismatches and an empty header all produce unknowns instead of defaults.

Issues

kv_cache_bytes([], n) reports COMPUTED with total_bytes == 0. In scripts/model_sol.py, an empty layers list leaves unknowns empty, so the result is COMPUTED with a total of 0 and bounded_as_context_grows: null. join_checkpoint explicitly flags an empty header ("an empty header passes every join"), and phase_floors flags phase.ops: empty, so this looks like an inconsistency. A KV total of 0 can make a capacity check pass silently. Suggestion: append "layers: empty" as an unknown, and add a test next to the kv_cache_bytes([], -1) case, which only covers the negative-context error.
Overlapping stored names in join_checkpoint are not detected. claimed.update(selection) only records that a header tensor was selected, not how many times. If the inventory has both model.layers.X.mlp.down_proj.weight and model.layers.0.mlp.down_proj.weight, header tensor model.layers.0… is counted in both matched rows. Neither residual reports it. The per-row diffs and the byte checksum will then disagree in a way that is hard to read. Since the join is the gate, please report header tensors selected by more than one stored name, for example in a multiply_claimed_header_tensors list that makes join_clean false, and add a test.
CI and stacking. On dbe0cf04, declarations failed. You note that the diff includes docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351's content, and docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 itself still lacks a knowledge-policy approval. full-check (3.10/3.11) is still in progress, and installed-package passes on both. Like you said, no approval at this head. I'll re-review after the rebase onto main once docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 lands and the index is regenerated after prototype: reduce knowledge projection and declaration churn #359.

Questions

phase_floors(): within one op, compute and memory take a max (full overlap inside the op), while link phases are summed and the latency added. Please state in the contract that the estimate assumes full compute/memory overlap within an op. Today it only says "no overlap between ops". A reader might otherwise take the serial estimate as pessimistic in every respect.
kv_cache_bytes(): the scope of the first slice in task: add a model-level speed-of-light analysis skill distilled from HuggingArch #385 is still open. Which attention families does SKILL.md commit to covering, mapped to the four laws (global, window, compressed, constant)? For example, is MLA's latent cache expected to be expressed as global with a smaller bytes_per_position?

Not verified here

I did not check the [code]/[doc] locators in references/huggingarch-model-sol.md, or the 44 entries in huggingarch-source-identity.json, against HuggingArch at e69d20a0….
Tests were not run locally. The provisional regeneration (817f57e3, 81 records) was not checked beyond the file list.

No approval implied.

fake0fan and others added 6 commits November 10, 2026 06:53

fake0fan mentioned this pull request Nov 10, 2026

task: add a model-level speed-of-light analysis skill distilled from HuggingArch #385

Open

Dong1017 added area:knowledge Knowledge line: packs, skill library, knowledge admission kind:task Bounded delivery task with a delegated slice labels Nov 10, 2026

Navigation Menu

Repository navigation

skill: add model-level speed-of-light analysis (model-sol-analysis) - #386

skill: add model-level speed-of-light analysis (model-sol-analysis)#386
fake0fan wants to merge 6 commits into
InferMatrixLab:mainfrom
fake0fan:feat/model-sol-analysis

fake0fan commented Nov 10, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 10, 2026 •

edited

Loading

Advisory: obtain missing evidence

Uh oh!

amy-why-3459 commented Nov 10, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

fake0fan commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this adds

The one design decision to review

Files

Provenance

Found along the way, not changed here

Out of scope

Verification at this head

Knowledge author declaration

Uh oh!

github-actions Bot commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Skill evaluation

Advisory: obtain missing evidence

Uh oh!

amy-why-3459 commented Nov 10, 2026

Review (COMMENT only, not an approval)

Verified

Issues

Questions

Not verified here

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fake0fan commented Nov 10, 2026 •

edited

Loading

github-actions Bot commented Nov 10, 2026 •

edited

Loading