Skip to content

Comparison freshness gate, construct-only ratchet legs, async comparison refresh - #73

Merged
tap merged 2 commits into
mainfrom
claude/sample-rate-solutions-comparison-mqc190
Oct 9, 2026
Merged

tap merged 2 commits into
mainfrom
claude/sample-rate-solutions-comparison-mqc190

Conversation

@tap

@tap tap commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

What this changes

PR 1 of the comparison plan (the rest: competitor set, bridge, rational).

  • Freshness gate for the comparison. compare.yml now harvests its counts with provenance (tree SHA, DspTap pin, run ID, toolchains) into async/bench/compare_counts.json (scripts/harvest_compare.py) and uploads it. scripts/update_compare_docs.py derives COMPARISON.md's embedded tables and key figures from that record, and a new compare-docs CI job fails on drift, the same shape as icount-docs.
  • Async refresh. Seeded from today's compare.yml run 37920542901 on fc2949e (DspTap ef6fdc4). Every cell reproduces the 2026-09-25 tables except the one the Helium Q15 dot moved: M55 Q15 steady state, 1,163 → 861. Footnote ² and the construction paragraph are reworded around the generated figures; a doubled parenthetical is gone.
  • Construct-only ratchet legs, after rational's: async construct_q15, bridge construct_{up,down}_q15_eco. Each is the Q15 workload with its stream removed, the converter built and the same input fixture synthesized, then one block processed, so stream − construct is the loop alone. Baselines recorded on all three targets; README tables regenerated.

Why

The SampleRateTap rows in COMPARISON.md were hand-typed under a date line, so a DspTap bump that moved a count failed nothing: the Helium dot landed on 2026-09-30 and the doc stayed at the September numbers until this morning's run exposed it.

The construct legs are the mitigation PERFORMANCE.md has queued since July and bridge's M7e asked for. On the M33 the Q15 totals are mostly constructor (async 89 %, bridge 78 %), so a hot-path regression there was diluted several times over before the ±3 % gate saw it.

Verification

  • Comparison numbers: CI-measured (run 21 on main), not local. The derivation matches the hand tables cell for cell at the 2026-09-25 pin; update_compare_docs.py is idempotent.
  • Ratchet: built with CI's configure line and the tree's plugin on M55, M33 and Hexagon (pinned toolchains, plugin-enabled qemu-hexagon 8.2.2). Every existing baseline is identical to the instruction before and after; only the new rows are added. The PR's ratchet jobs passed on all three targets, so the new rows match CI's runners.
  • Cross-check worth having: pipeline_q15 − construct_q15 on M33 is 1,188/frame against the 2 s/4 s split's 1,138. The 50 is the servo's one-time lock-in, which the split counts as setup; both are measured and PERFORMANCE.md says which answers which question.
  • Why the fixture is in the construct leg: the pipeline synthesizes 0.25 s of input with libm sin(), 52 M soft-double instructions on M33. Without it in the subtrahend, "stream − construct" was 1,729/frame, i.e. not the stream. Rational's construct_q15_eco has that gap; a follow-up could align it.
  • After The 2026-10 adversarial audit and its response: S0 to S5, family version 0.6.0 #72 merged: main is merged in. On the merged tree: no conflicts, the new retired-words gate reports nothing, both freshness gates regenerate with no diff, gcc -Werror with every engine's WERROR and the C ABIs on, 305/305 tests, book clean.
  • Host: clang-format and clang-tidy clean on the changed units (bridge's icount TU is under the tidy gate; async's is excluded per style.yml). Both workflows parse.
  • Not run: the new compare.yml harvest step end to end on CI (it runs on dispatch only). The harvest script was exercised on run 21's log, which is the same input the step writes.

Notes for the reviewer

  • New file async/bench/compare_counts.json is the comparison's equivalent of baselines.json: a measured record with provenance. Re-measure by dispatching compare.yml, downloading the artifact, committing it and the regenerated block.
  • Provenance pin vs. tree pin. The record names DspTap ef6fdc4; after The 2026-10 adversarial audit and its response: S0 to S5, family version 0.6.0 #72 the tree pins d7ebe4b (the chain.h fix, rational only). The counts stand — nothing async or bridge compiles changed — and the gate deliberately does not require pin equality, so a bump that moves no kernel does not force a 10-minute dispatch. The provenance line says what was measured; the reader decides whether the pin since matters.
  • The 12-channel pipeline has no construct leg (different FIFO and fixture sizes); its gate keeps the old dilution. PERFORMANCE.md says so.
  • Docs that still cite the 2 s/4 s steady state (1,138 / 3,326 in the README, book and Pico READMEs) remain correct; they describe the split method, which stays.
  • Host throughput table untouched, labelled indicative, as decided.

🤖 Generated with Claude Code

https://claude.ai/code/session_01G3HxzEGiZp7jStoMuYNuBK

claude added 2 commits October 9, 2026 15:49
The comparison's SampleRateTap rows were hand-typed under a date line, so
the DspTap bump to the Helium Q15 dot (ef6fdc4) moved one of them — the
M55 Q15 steady state, 1,163 -> 861 instructions/frame — and nothing
failed. Now compare.yml harvests its CMP_COUNT lines with their
provenance (tree SHA, DspTap pin, run, toolchains) into
async/bench/compare_counts.json (scripts/harvest_compare.py) and uploads
it; scripts/update_compare_docs.py derives COMPARISON.md's embedded
tables and key figures from that record; and a compare-docs CI job fails
on drift, as icount-docs does for the README tables. Seeded from today's
run 37920542901 on fc2949e: every cell reproduces the 2026-09-25 tables
except the one that moved. Footnote 2 and the construction paragraph are
reworded around the generated figures (float is now cheaper than Q15 on
the M55 by 5 %, not 42 %), and a doubled parenthetical is gone.

Construct-only ratchet legs, after rational's: async construct_q15 and
bridge construct_{up,down}_q15_eco are the Q15 workloads with their
stream removed — the converter built and the same input fixture
synthesized, then one block processed — gated beside the streaming
workloads so stream minus construct is the loop alone. The fixture is in
the construct leg deliberately (24,000 soft-double sin() calls on M33).
Measured on all three targets; every existing baseline identical to the
instruction. Shares of the Q15 totals: async M33 88.8 %, Hexagon 70.2 %,
M55 17.9 %; bridge M33 78 %, Hexagon 55 %, M55 19 %. Cross-check: async
M33 pipeline minus construct is 1,188/frame against the 2 s/4 s split's
1,138 — the 50 is the servo's one-time lock-in, which the split counts as
setup. PERFORMANCE.md and bridge/PLAN.md (M7e) record the numbers; the
README tables regenerate; the book's scenario counts follow.

Verified: gcc -Werror build with the ratchet workloads, 279/279 tests;
clang-format and clang-tidy clean on the changed units; three-target
ratchet green locally against the committed files; update_compare_docs
idempotent; the book builds with zero warnings; both workflows parse.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G3HxzEGiZp7jStoMuYNuBK
@tap
tap merged commit 43f3e29 into main Oct 9, 2026
30 checks passed
@tap
tap deleted the claude/sample-rate-solutions-comparison-mqc190 branch October 9, 2026 17:58
tap pushed a commit that referenced this pull request Oct 10, 2026
The comparison plan (#73, #75, #76, #77) and its follow-up (#78) closed;
this is the plan for what the family's plans and ledgers still carry
open, sequenced into twelve weeks in three blocks, with "production-ready"
defined as a thirteen-row checklist per engine so the verdict can be
checked rather than believed.

Drafted and then audited by two independent adversarial reviews (one
against the trees' facts and settled decisions, one against engineering
judgement). The audit changed the draft materially: the book's bridge
chapters are done (#68), so they are off the list; rational has no
cross-engine validation by design (scipy and decimate.h are its legs);
TSan runs async only; four host compilers and three QEMU targets, not the
draft's counts; the +1.9 % M55 ratio23_q15_eco drift is ab8fea3's stated
move, not a DspTap bump, and the chain rows sit +926/+1,102; the hardware
block never ran bridge or rational on silicon; nothing builds with
-fno-exceptions; TSan has never seen the converter with two threads; no
footprint, stack or worst-single-call figure exists; Setup 1's "< 1 ppm"
criterion tests the counting, not the servo, and cheap USB dongles may not
own a crystal; the timestamp servo is a four-to-five-week extension, not an
overload, and must not be frozen in the release that adds it; "C ABIs
frozen" needs a reconciled NULL model, an extension rule, SOVERSION and a
compat test. Section 7 holds the six decisions the maintainer takes before
week 1, each with a recommendation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G3HxzEGiZp7jStoMuYNuBK
tap pushed a commit that referenced this pull request Oct 10, 2026
The comparison plan (#73, #75, #76, #77) and its follow-up (#78) closed;
this is the plan for what the family's plans and ledgers still carry
open, sequenced into twelve weeks in three blocks, with "production-ready"
defined as a thirteen-row checklist per engine so the verdict can be
checked rather than believed.

Drafted and then audited by two independent adversarial reviews (one
against the trees' facts and settled decisions, one against engineering
judgement). The audit changed the draft materially: the book's bridge
chapters are done (#68), so they are off the list; rational has no
cross-engine validation by design (scipy and decimate.h are its legs);
TSan runs async only; four host compilers and three QEMU targets, not the
draft's counts; the +1.9 % M55 ratio23_q15_eco drift is ab8fea3's stated
move, not a DspTap bump, and the chain rows sit +926/+1,102; the hardware
block never ran bridge or rational on silicon; nothing builds with
-fno-exceptions; TSan has never seen the converter with two threads; no
footprint, stack or worst-single-call figure exists; Setup 1's "< 1 ppm"
criterion tests the counting, not the servo, and cheap USB dongles may not
own a crystal; the timestamp servo is a four-to-five-week extension, not an
overload, and must not be frozen in the release that adds it; "C ABIs
frozen" needs a reconciled NULL model, an extension rule, SOVERSION and a
compat test. Section 7 holds the six decisions the maintainer takes before
week 1, each with a recommendation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G3HxzEGiZp7jStoMuYNuBK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants