Repository navigation
Conversation
fold_packs and spare_slices look F192 subset sums up a byte at a time. LLVM vectorized those lookups into vpgatherqq, which perf showed as the hot instructions of fold_packs (9.6% of the hash_batch samples). Hiding the tables' length, as primitives' frobenius and the verifier's Scaled do, keeps bounds checks that stop the vectorizer, and the loads stay scalar. The proof is unchanged (digests of whole batch proofs, 8 shapes, equal). hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads, the Openings span: 66 -> 24 ms. Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
Every layer builds the power basis of its challenge s and moves weights in and out of it on one thread, between the parallel passes. That serial time was about 1.7 ms of a 28 ms layer: - PowerBasis::new eliminated rows with a branch per row and bit, a coin flip, and transposed the matrix in and out bit by bit. It now eliminates columns, the powers as they come and the inverse's columns as they are needed, masked rather than branched. - PowerField::new's Barrett division shifted the modulus in bit by bit; it shifts words. - PowerBasis::to_poly and PolyEval::eval lookups were vectorized into AVX-512 gathers (Round::new, the tail's move back to the standard basis); hiding the tables' length keeps them scalar. The proof is unchanged (digests of whole batch proofs, 8 shapes, equal). hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads, per layer: PowerField::new 334 -> 92 us, Round::new 556 -> 187 us, the standard basis row 300 -> 90 us. Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
keccak::linear indexed every position (the theta columns, the z - 1 shift, the rho rotation modulo 64), and on the witness's eight-word positions LLVM vectorized those loops across positions with vpgatherqq: perf put linear at 5.4% of hash_batch's samples. It now works a lane at a time on slices, the rotation as two runs, so each position is whole-register loads. The witness is unchanged (the round test, the witness tests, and digests of whole batch proofs, 8 shapes, equal). Witness::from_committed, the VM's rebuild of the reduction's witness, runs the same code. hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads, the Witness span: 43 -> 24 ms. Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
…not 40 perf put the Skip pass at 27.6% of hash_batch's samples, its hot loop the carry-less products: per slot and eight slices, five products e[x+1] e[x+2] and their 192-bit weights, 40 VPCLMULQDQ, at this machine's VPCLMULQDQ throughput. A row's sum is now e[2] (w[0] e[1] + w[1] e[3]) + e[4] (w[2] e[3] + w[3] e[0]) + w[4] (e[0] e[1]). The two stars are linear in the values, so they are multiplied on the group's 16 packs and extended to the 65 mixed points by XORs like the values are; a slot then takes 24 products. The extension is walked once for the values and once per star word rather than stored, which keeps the factors in registers. Even and odd slices share an accumulator, and SkipBlock::get returns the slot's polynomial (four words) on every arm. Only the AVX-512 (AVX-512BW, VPCLMULQDQ, GFNI) arm changes; AVX2, NEON and the portable kernel are as they were. The sums are the same polynomials, so the proof is unchanged (digests of whole batch proofs, 8 shapes, equal; the kernel test against the portable sums). hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads, 4 runs alternated, the Skip spans summed over the layers: 149.8 -> 119.8 ms. One thread, KECCAK_N_LOG=14: 138.6 -> 113.1 ms. Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
parithosh
force-pushed
the
pr-keccak-gkr
branch
from
November 9, 2026 18:45
c661d9d to
ea774a5
Compare
The product pass's staging (products, then their bit transposes into column bytes) was 52 of a block's 92 us, bound by the shuffle port: per lane byte, about 350 qword and byte shuffles for the 8x8 byte transposes, and three zero-extensions of every pack in every product it enters. - With AVX-512 VBMI, each pair of staged registers is two two-source byte permutes of four positions' products, a blend and one 512-bit bit transpose, in the order the AVX2 transposes leave, so the matrices are unchanged. The AVX2 transposes stay for GFNI machines without VBMI. - A lane byte's packs are widened to a word once, as they are embedded, and the products read the words as they are. The staged registers are the same bytes, so the proof is unchanged (the GFNI against digit buckets test; digests of whole batch proofs, 8 shapes, equal). One block, cache-hot: staging 52 -> 38 us. hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads, 5 runs alternated, the Product spans summed over the layers: 91.4 -> 84.0 ms. Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
…copies Round::add_pair, 8% of hash_batch's samples, copied both rows shifted one and two columns along (with kappa added), wrote B = w (kappa + T[j + 1]) + kappa w[j + 2] to scratch by mul_row and read it back in dot_pair: per pair 22.5 us, of which about 5.5 us the copies. PowerField::pair_sums now takes the rows and where each run of 64 entries reads its shifted entries, and on x86 builds both rows' B in registers and dots them with T[j + 2] at once. kappa moves into the constant, kappa (w[j] + w[j + 2]), so B = w T[j + 1] + constant. The portable form, and so NEON, keeps the copies, mul_row and dot_pair. B is the same reduced value, so the round sums, and the proof, are unchanged (digests of whole batch proofs, 8 shapes, equal); the power basis test checks the x86 kernel against the portable one. One pair, cache-hot: 22.5 -> 17.1 us. hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads, 4 runs alternated, spans summed over the layers: Tables 46.0 -> 41.4 ms, Tail 46.7 -> 41.8 ms. Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
Between its parallel passes a layer works on one thread: per layer, about 0.9 ms of the layer's 25 ms on this machine. - The claim's weights move into the power basis once, for the Skip pass and the rounds alike; each moved them itself. - The local rounds' tables hold the 1600 positions, not the 2048 of the cube: the entries past them are zero, so a round's last pair takes a zero where it needs one and the sums and folds are the same. The four tables are one, entry by entry, as the rounds read them. Spreading these loops over the pool was tried too and lost: on this machine the dispatch costs more than the loops. The proof is unchanged (digests of whole batch proofs, 8 shapes, equal). hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads, 4 runs alternated, a layer's time outside its four passes, summed over the layers: 11.5 -> 7.7 ms. Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
fold_packs gave a task to each position of a committed lane, and the task walked the position through every block: one cache line in each 100 KB block (three for a state after chi), a page apart, where the next position's task read the line beside it. A task now takes sixteen side by side positions, which it reads together in each block. The sums are the same, so the proof is unchanged (digests of whole batch proofs, 8 shapes, equal). hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads: fold_packs 21.5 -> 11.5 ms (one span, 2 runs each of 4, 8, 16 and 32 positions a task; 16 was best); 4 runs alternated, the Openings span 22.3 -> 15.8 ms. Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
parithosh
force-pushed
the
pr-keccak-gkr
branch
from
November 9, 2026 19:22
ea774a5 to
cde0b65
Compare
TomWambsgans
added a commit
that referenced
this pull request
Nov 15, 2026
…basis setup per layer Three prover-only changes to the experimental Keccak GKR: - `PowerField::pair_sums` builds a pair of rows' factor `B = w * T[j+1] + constant`, kappa folded into the constant `kappa * (w[j] + w[j+2])`, in registers where its dot with `T[j+2]` reads it (x86, NEON and portable arms), the shifted entries read through precomputed run offsets. `Round::factors`, `mul_row`, `dot_pair`, the shifted copies and the per-worker scratch buffers are gone. - The claim's weights are converted to the power basis once per layer and shared by the Skip pass and `Round::new`; the local rounds keep the 1600 live positions, interleaved as `[w, t0, t1, t2]`, instead of the 2048-entry cube. - `PowerBasis::new` eliminates columns with masks rather than a branch per row, and `PowerField::new`'s Barrett division shifts the modulus by words. Proofs unchanged: `identical_proofs` passes on the Mac, h1, h2 and h4, and differential tests against the previous kernels and against the stable tree's Keccak GKR (whole proofs of 8 batch shapes, on NEON, AVX-512, 128-bit PCLMUL and portable builds) matched byte for byte. On the Keccak GKR alone (`hash_batch`, `KECCAK_N_LOG=17`) the pair sums take the Tables span down by 8% to 12% on h1 and h2. From leanEthereum/leanVM PR #730 (commits 5ce1cec, 0624447, 4d31de0, authored by Amp); the branchless power basis is also in gattaca-com/leanVM (5d93998 / 2ba2730). Of #730, the VBMI staging (9d2e514) measured nothing on AVX2 and an uncertain gain on AVX-512 for 143 lines and was left out, as were the parallel local rounds of gattaca's 360ce3c (1600 positions do not pay for the pool). This branch before -> after, `--prover experimental`, 3 rounds x 2 passes, interleaved, medians: - h1 (Zen 4): aggregate --xmss 900 --log-inv-rate 1 0.800 -> 0.795 s (-0.6%), at 4 threads -0.3%, --sphincs 410 -0.3%, recursion --n 2 --xmss-per-leaf 900 --log-inv-rate 2 -0.8%. - h2 (Arrow Lake): -0.7%, -0.7%, -0.7%, -0.5%. - h4 (Zen 5): xmss 900 within noise (+0.4%), at 4 threads 0.777 -> 0.762 s (-1.9%), sphincs 410 -1.9%, recursion 0.090 -> 0.087 s (-3.3%). - Mac (M4 Max): xmss 900 0.504 -> 0.487 s (-3.3%), at 4 threads -0.3%, sphincs 410 -0.5%, recursion 0.185 -> 0.178 s (-4.0%). - Peak memory unchanged. Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai> Co-authored-by: alexX512 <alexey@gattaca.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TomWambsgans
added a commit
that referenced
this pull request
Nov 15, 2026
…basis setup per layer Three prover-only changes to the experimental Keccak GKR: - `PowerField::pair_sums` builds a pair of rows' factor `B = w * T[j+1] + constant`, kappa folded into the constant `kappa * (w[j] + w[j+2])`, in registers where its dot with `T[j+2]` reads it (x86, NEON and portable arms), the shifted entries read through precomputed run offsets. `Round::factors`, `mul_row`, `dot_pair`, the shifted copies and the per-worker scratch buffers are gone. - The claim's weights are converted to the power basis once per layer and shared by the Skip pass and `Round::new`; the local rounds keep the 1600 live positions, interleaved as `[w, t0, t1, t2]`, instead of the 2048-entry cube. - `PowerBasis::new` eliminates columns with masks rather than a branch per row, and `PowerField::new`'s Barrett division shifts the modulus by words. Proofs unchanged: `identical_proofs` passes on the Mac, h1, h2 and h4, and differential tests against the previous kernels and against the stable tree's Keccak GKR (whole proofs of 8 batch shapes, on NEON, AVX-512, 128-bit PCLMUL and portable builds) matched byte for byte. On the Keccak GKR alone (`hash_batch`, `KECCAK_N_LOG=17`) the pair sums take the Tables span down by 8% to 12% on h1 and h2. From leanEthereum/leanVM PR #730 (commits 5ce1cec, 0624447, 4d31de0, authored by Amp); the branchless power basis is also in gattaca-com/leanVM (5d93998 / 2ba2730). Of #730, the VBMI staging (9d2e514) measured nothing on AVX2 and an uncertain gain on AVX-512 for 143 lines and was left out, as were the parallel local rounds of gattaca's 360ce3c (1600 positions do not pay for the pool). This branch before -> after, `--prover experimental`, 3 rounds x 2 passes, interleaved, medians: - h1 (Zen 4): aggregate --xmss 900 --log-inv-rate 1 0.800 -> 0.795 s (-0.6%), at 4 threads -0.3%, --sphincs 410 -0.3%, recursion --n 2 --xmss-per-leaf 900 --log-inv-rate 2 -0.8%. - h2 (Arrow Lake): -0.7%, -0.7%, -0.7%, -0.5%. - h4 (Zen 5): xmss 900 within noise (+0.4%), at 4 threads 0.777 -> 0.762 s (-1.9%), sphincs 410 -1.9%, recursion 0.090 -> 0.087 s (-3.3%). - Mac (M4 Max): xmss 900 0.504 -> 0.487 s (-3.3%), at 4 threads -0.3%, sphincs 410 -0.5%, recursion 0.185 -> 0.178 s (-4.0%). - Peak memory unchanged. Co-authored-by: Parithosh Jayanthi <17509050+parithosh@users.noreply.github.com> Co-authored-by: Alexey Perevyshin <61140353+alexX512@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
|
took 3a31169 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prover-only,
crates/keccak_gkr: no protocol or verifier change, and the proof is byte-identical (digests of whole batch proofs for 8 shapes equal base). 8 commits, each reviewable on its own and each with its own measurement in its message.Profiled with perf and the span tree on base, the time went to:
fold_packs(9.6%),keccak::linear(5.4%),PowerBasis::to_polyandPolyEval::eval;add_pair(7%): two Barrett-reducedmul_rows plus shifted-row copies;fold_packsandspare_slices(Openings 66 -> 24 ms)to_polyandevalWitness::from_committedvpermt2b, packs widened once (Product 91 -> 84 ms)fold_packstasks take 16 side-by-side positions (21.5 -> 11.5 ms)Measured.
cargo bench -p keccak_gkr --bench hash_batch,KECCAK_N_LOG=17, depth 12, Xeon (Ice Lake, AVX-512) Amp orb, default 8 threads, 6 base/head runs alternated, eachBENCH_REPEAT=3.LEANVM_NUM_THREADS=16, 3 runsNEON speed is unmeasured: its hot kernels are unchanged, and its tests pass (aarch64
neoverse-n2build under qemu, plus aarch64 clippy-D warnings).Tried and dropped: an unreduced fused pair kernel (fewer clmuls but ALU-bound, 24.7 us against 22.5); storing the Skip extension (50 KB a group, slower); a single-pass Skip walker (spilled); a precomputed high-point array (2x slower); unchecked indexing (no change); widening packs without VBMI (no gain alone); spreading the per-layer serial loops over the pool (the dispatch costs more than the loops).
Checks:
cargo testall,cargo clippyall,cargo docall,cargo fmt --check, the x86 cross-check from AGENTS.md; keccak_gkr tests and proof digests on haswell, alderlake and skylake-avx512+gfni+vpclmulqdq builds; keccak_gkr tests on aarch64 (NEON) under qemu. Rebased onto currentmain(#3f2ff324):cargo test --release -p keccak_gkr,aggregate_two_to_one, clippy and fmt rerun there.