task: add a model-level speed-of-light analysis skill distilled from HuggingArch #385

Current state: Proposed on 2026-11-10 by the issue author; no implementation is accepted. Draft PR #386 (head dbe0cf04) holds a candidate implementation stacked on #351, which it builds on. CI and the knowledge-policy check are expected to report there; no approval is requested while it is stacked.
Decision needed: Accept, revise or reject a new methods/ owner for model-level speed-of-light analysis, from the maintainers. This also bears on open question 1 of #303 (new methods/ owner or a second slice of performance-ceiling-analysis); see the boundary below.
Next action: The issue author keeps the Draft PR in Draft until three things happen: this scope is accepted, #351 lands, and the skill index is regenerated after #359. The Draft carries a provisional regeneration so that CI can run.

Goal and acceptance

• Problem / desired outcome: methods/performance-ceiling-analysis computes a conditional L1 floor for one device, one operation class and one traffic tier. Its roofline contract states that it does not walk a model, sum devices or compose stages. So an agent asked "what is the prefill / decode ceiling of model M under layout L on hardware H?" has no procedure for the model-level question. It assembles one by hand from per-op formulas, without:

  • ground-truth anchors on the counts,
  • a rule for which graph to price (fused or unfused),
  • per-rank geometry or injected communication,
  • a separation between what is a bound and what is a conditional estimate.

  HuggingArch (MIT) implements this question end to end. task: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #348 / docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 already carry one prerequisite from it, the checkpoint weight anchor. This issue distills the rest of the method into a portable candidate procedure.

• Completion criteria:

  • A new registered candidate skill methods/model-sol-analysis (SKILL.md, metadata.json, references/), with one skill-index.json entry and the count updates that follow.
  • The procedure walks a model to per-phase (prefill, decode) floors: anchored counts, the fused graph with provenance for every fusion, per-rank geometry under a declared layout, communication as explicit terms, and KV traffic per attention family with its growth law stated.
  • Every figure is labelled as either an invariant bound (per-resource totals over rated peaks) or a conditional estimate (per-op serial composition, calibrated rates). Only a bound may reject a result, consistent with the bounds, predictions and decisions reference.
  • A pure offline helper with tests. It takes saved inputs (a checkpoint header, an analytical tensor inventory, per-op counts, rated hardware facts) and returns join residuals, a byte checksum, and both the bound and the estimate per phase. It does no I/O, no network access and no device execution, and it returns INSUFFICIENT_EVIDENCE rather than guess.
  • A typed source identity at one pinned commit, with per-file blob, sha256, bytes and inspection scope. Claims in the reference documents carry [code] / [doc] markers with locators, following the convention reviewed on docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351.
  • One docs/reference-matrix.md row and the matching docs/searchspace-matrix.md row(s); regenerated docs/skill-coverage.md and skill index.
  • locally_verified: false, empirical_validation: not_run. No measurement, benchmark or performance claim.

Scope and constraints

• In scope: the new methods/model-sol-analysis owner, its helper and tests, the registry entry, matrix rows, count updates, and generated views. A one-line cross-link from performance-ceiling-analysis to the new owner.
• Out of scope:
  • The per-task SOL ruler, Class A/B vocabulary and gap metric proposed in [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303. This skill produces model-level floors and estimates; it does not define a frozen scoring ruler or rank agent flows. Choosing this owner for model-level analysis does not decide [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303's question for fused-operator ruler work.
  • Importing or vendoring HuggingArch code, specs or its measured kernel corpus; the helper is written from the documented method.
  • Any measurement, profiler run, device execution or numeric gate in the tree.
  • Changes to the knowledge-model policy or to prototype: reduce knowledge projection and declaration churn #359's index representation.
• Relevant dependencies, shared interfaces or limits:
  • Builds on docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 (weight-demand anchor in performance-ceiling-analysis): the new skill references it rather than duplicating it. The PR is stacked on docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 and rebases once docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 lands.
  • prototype: reduce knowledge projection and declaration churn #359 changes the generated index representation. The Draft PR carries a provisional schema-v1 regeneration so that CI runs. The final regeneration happens after prototype: reduce knowledge projection and declaration churn #359 lands, which avoids another repository-wide pin-only rewrite on main.
  • weight-loading-optimization keeps loading mechanics; calibrate-serving-simulation keeps serving-level (queueing, SLO) prediction.

Ownership and authorization

• Proposed by / actual requester: repository owner (via authorized agent)
• Status: Proposed
• Accountable parent (Issue-level owner, if assigned): TBD
• Slice implementer(s) / acting contributor (if assigned): issue author, if accepted
• Goal-acceptance owner: TBD
• Accepted decision or scoped delegation: Not yet granted

Inputs and deliverables

• Relevant sources, revisions or linked issues:
  • github.com/shenh10/HuggingArch @ e69d20a028e5b17524aa48b53bddf8b9aad5d15b (MIT), observed 2026-11-10. Readset: docs/inference_cost_model.md, docs/evaluation_model.md, docs/symbolic_evaluation.md, docs/design.md, docs/parallelism.md, docs/kv_cache_storage.md, docs/attention_families.md, docs/moe_expert_axis.md, docs/quantization.md, docs/validation.md, docs/measured_calibration.md, docs/known_limitations.md, plus the implementing code files needed to verify load-bearing claims. The identity file records exact per-file scope.
  • docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351 retains its own pin facbbbc9… for the weight anchor; the two owners keep separate source revisions.
  • Related: task: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #348, docs: anchor L1 weight demand against the checkpoint in performance-ceiling-analysis #351, [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303, trim & improve: reduce knowledge contribution cost and generated-file churn #355 / prototype: reduce knowledge projection and declaration churn #359.
• Expected artifacts: one PR as described under completion criteria.

Completion and closure

• Who accepts completion and may close: the named acceptance owner, not the PR merge.
• Remaining uncertainty / next decision: whether model-level analysis belongs in a new owner (proposed here) or as further slices of performance-ceiling-analysis; whether the helper's bound/estimate split should be reused by [RFC] Per-task SOL modeling and a KDA-flow gap metric for fused operators #303 if that RFC is accepted.