Skip to content

keccak_gkr: 26% faster prover, proof unchanged - #730

Open
parithosh wants to merge 8 commits into
leanEthereum:mainfrom
parithosh:pr-keccak-gkr
Open

parithosh wants to merge 8 commits into
leanEthereum:mainfrom
parithosh:pr-keccak-gkr

Conversation

@parithosh

Copy link
Copy Markdown
Contributor

Prover-only, crates/keccak_gkr: no protocol or verifier change, and the proof is byte-identical (digests of whole batch proofs for 8 shapes equal base). 8 commits, each reviewable on its own and each with its own measurement in its message.

Profiled with perf and the span tree on base, the time went to:

  • the Skip pass (27.6% of samples), bound by VPCLMULQDQ throughput: 40 per slot per 8 slices;
  • the Product pass (16.6%): staging bound by the shuffle port (52 us per block), the GFNI accumulate by port 0;
  • AVX-512 gathers LLVM made from table lookups and per-position indexing: fold_packs (9.6%), keccak::linear (5.4%), PowerBasis::to_poly and PolyEval::eval;
  • add_pair (7%): two Barrett-reduced mul_rows plus shifted-row copies;
  • about 1.7 ms of serial work per layer.
commit change arms
1 no gathers in fold_packs and spare_slices (Openings 66 -> 24 ms) all
2 power basis by masked column elimination, Barrett division by word shifts, no gathers in to_poly and eval all
3 the witness linear map a lane at a time, no gathers (Witness 43 -> 24 ms); also Witness::from_committed all
4 the AVX-512 Skip kernel as two "stars", 24 VPCLMULQDQ a slot instead of 40 (Skip 150 -> 120 ms) AVX-512 only
5 Product staging by VBMI vpermt2b, packs widened once (Product 91 -> 84 ms) GFNI with VBMI; AVX2 transposes kept otherwise
6 a table pair's round sums in one pass, B built in registers, no shifted copies (one pair 22.5 -> 17.1 us) x86; NEON keeps the old path
7 less serial work per layer: weights moved to the power basis once, local tables trimmed to 1600 entries (11.5 -> 7.7 ms) all
8 fold_packs tasks take 16 side-by-side positions (21.5 -> 11.5 ms) all

Measured. cargo bench -p keccak_gkr --bench hash_batch, KECCAK_N_LOG=17, depth 12, Xeon (Ice Lake, AVX-512) Amp orb, default 8 threads, 6 base/head runs alternated, each BENCH_REPEAT=3.

base this PR
prove TOTAL 537.5 ms (513.1 to 565.3, sd 19.6) 395.3 ms (387.1 to 403.7, sd 6.7), -26.5%
layers and openings 423.1 ms (sd 14.5) 303.9 ms (sd 4.8), -28.2%
witness 43.1 ms 22.8 ms
prove TOTAL, LEANVM_NUM_THREADS=16, 3 runs 336.7 ms 263.6 ms, -21.7%

NEON speed is unmeasured: its hot kernels are unchanged, and its tests pass (aarch64 neoverse-n2 build under qemu, plus aarch64 clippy -D warnings).

Tried and dropped: an unreduced fused pair kernel (fewer clmuls but ALU-bound, 24.7 us against 22.5); storing the Skip extension (50 KB a group, slower); a single-pass Skip walker (spilled); a precomputed high-point array (2x slower); unchecked indexing (no change); widening packs without VBMI (no gain alone); spreading the per-layer serial loops over the pool (the dispatch costs more than the loops).

Checks: cargo testall, cargo clippyall, cargo docall, cargo fmt --check, the x86 cross-check from AGENTS.md; keccak_gkr tests and proof digests on haswell, alderlake and skylake-avx512+gfni+vpclmulqdq builds; keccak_gkr tests on aarch64 (NEON) under qemu. Rebased onto current main (#3f2ff324): cargo test --release -p keccak_gkr, aggregate_two_to_one, clippy and fmt rerun there.

ampagent and others added 4 commits November 9, 2026 18:26
fold_packs and spare_slices look F192 subset sums up a byte at a time. LLVM
vectorized those lookups into vpgatherqq, which perf showed as the hot
instructions of fold_packs (9.6% of the hash_batch samples). Hiding the
tables' length, as primitives' frobenius and the verifier's Scaled do, keeps
bounds checks that stop the vectorizer, and the loads stay scalar.

The proof is unchanged (digests of whole batch proofs, 8 shapes, equal).

hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads, the
Openings span: 66 -> 24 ms.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd
Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
Every layer builds the power basis of its challenge s and moves weights in
and out of it on one thread, between the parallel passes. That serial time
was about 1.7 ms of a 28 ms layer:

- PowerBasis::new eliminated rows with a branch per row and bit, a coin flip,
  and transposed the matrix in and out bit by bit. It now eliminates columns,
  the powers as they come and the inverse's columns as they are needed,
  masked rather than branched.
- PowerField::new's Barrett division shifted the modulus in bit by bit; it
  shifts words.
- PowerBasis::to_poly and PolyEval::eval lookups were vectorized into AVX-512
  gathers (Round::new, the tail's move back to the standard basis); hiding the
  tables' length keeps them scalar.

The proof is unchanged (digests of whole batch proofs, 8 shapes, equal).

hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads, per layer:
PowerField::new 334 -> 92 us, Round::new 556 -> 187 us, the standard basis
row 300 -> 90 us.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd
Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
keccak::linear indexed every position (the theta columns, the z - 1 shift, the
rho rotation modulo 64), and on the witness's eight-word positions LLVM
vectorized those loops across positions with vpgatherqq: perf put linear at
5.4% of hash_batch's samples. It now works a lane at a time on slices, the
rotation as two runs, so each position is whole-register loads.

The witness is unchanged (the round test, the witness tests, and digests of
whole batch proofs, 8 shapes, equal). Witness::from_committed, the VM's
rebuild of the reduction's witness, runs the same code.

hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads, the Witness
span: 43 -> 24 ms.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd
Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
…not 40

perf put the Skip pass at 27.6% of hash_batch's samples, its hot loop the
carry-less products: per slot and eight slices, five products e[x+1] e[x+2]
and their 192-bit weights, 40 VPCLMULQDQ, at this machine's VPCLMULQDQ
throughput.

A row's sum is now e[2] (w[0] e[1] + w[1] e[3]) + e[4] (w[2] e[3] + w[3] e[0])
+ w[4] (e[0] e[1]). The two stars are linear in the values, so they are
multiplied on the group's 16 packs and extended to the 65 mixed points by
XORs like the values are; a slot then takes 24 products. The extension is
walked once for the values and once per star word rather than stored, which
keeps the factors in registers. Even and odd slices share an accumulator, and
SkipBlock::get returns the slot's polynomial (four words) on every arm.

Only the AVX-512 (AVX-512BW, VPCLMULQDQ, GFNI) arm changes; AVX2, NEON and the
portable kernel are as they were. The sums are the same polynomials, so the
proof is unchanged (digests of whole batch proofs, 8 shapes, equal; the kernel
test against the portable sums).

hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads, 4 runs
alternated, the Skip spans summed over the layers: 149.8 -> 119.8 ms.
One thread, KECCAK_N_LOG=14: 138.6 -> 113.1 ms.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd
Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
ampagent and others added 4 commits November 9, 2026 19:18
The product pass's staging (products, then their bit transposes into column
bytes) was 52 of a block's 92 us, bound by the shuffle port: per lane byte,
about 350 qword and byte shuffles for the 8x8 byte transposes, and three
zero-extensions of every pack in every product it enters.

- With AVX-512 VBMI, each pair of staged registers is two two-source byte
  permutes of four positions' products, a blend and one 512-bit bit
  transpose, in the order the AVX2 transposes leave, so the matrices are
  unchanged. The AVX2 transposes stay for GFNI machines without VBMI.
- A lane byte's packs are widened to a word once, as they are embedded, and
  the products read the words as they are.

The staged registers are the same bytes, so the proof is unchanged (the GFNI
against digit buckets test; digests of whole batch proofs, 8 shapes, equal).

One block, cache-hot: staging 52 -> 38 us. hash_batch, KECCAK_N_LOG=17, Xeon
(Ice Lake, AVX-512), 8 threads, 5 runs alternated, the Product spans summed
over the layers: 91.4 -> 84.0 ms.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd
Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
…copies

Round::add_pair, 8% of hash_batch's samples, copied both rows shifted one and
two columns along (with kappa added), wrote B = w (kappa + T[j + 1]) +
kappa w[j + 2] to scratch by mul_row and read it back in dot_pair: per pair
22.5 us, of which about 5.5 us the copies.

PowerField::pair_sums now takes the rows and where each run of 64 entries
reads its shifted entries, and on x86 builds both rows' B in registers and
dots them with T[j + 2] at once. kappa moves into the constant,
kappa (w[j] + w[j + 2]), so B = w T[j + 1] + constant. The portable form, and
so NEON, keeps the copies, mul_row and dot_pair.

B is the same reduced value, so the round sums, and the proof, are unchanged
(digests of whole batch proofs, 8 shapes, equal); the power basis test checks
the x86 kernel against the portable one.

One pair, cache-hot: 22.5 -> 17.1 us. hash_batch, KECCAK_N_LOG=17, Xeon (Ice
Lake, AVX-512), 8 threads, 4 runs alternated, spans summed over the layers:
Tables 46.0 -> 41.4 ms, Tail 46.7 -> 41.8 ms.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd
Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
Between its parallel passes a layer works on one thread: per layer, about
0.9 ms of the layer's 25 ms on this machine.

- The claim's weights move into the power basis once, for the Skip pass and
  the rounds alike; each moved them itself.
- The local rounds' tables hold the 1600 positions, not the 2048 of the
  cube: the entries past them are zero, so a round's last pair takes a zero
  where it needs one and the sums and folds are the same. The four tables
  are one, entry by entry, as the rounds read them.

Spreading these loops over the pool was tried too and lost: on this machine
the dispatch costs more than the loops.

The proof is unchanged (digests of whole batch proofs, 8 shapes, equal).

hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads, 4 runs
alternated, a layer's time outside its four passes, summed over the layers:
11.5 -> 7.7 ms.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd
Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
fold_packs gave a task to each position of a committed lane, and the task
walked the position through every block: one cache line in each 100 KB
block (three for a state after chi), a page apart, where the next position's
task read the line beside it. A task now takes sixteen side by side
positions, which it reads together in each block.

The sums are the same, so the proof is unchanged (digests of whole batch
proofs, 8 shapes, equal).

hash_batch, KECCAK_N_LOG=17, Xeon (Ice Lake, AVX-512), 8 threads:
fold_packs 21.5 -> 11.5 ms (one span, 2 runs each of 4, 8, 16 and 32
positions a task; 16 was best); 4 runs alternated, the Openings span
22.3 -> 15.8 ms.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1b1ea-c4c6-70fb-a5f5-5e6e88ce2fbd
Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
TomWambsgans added a commit that referenced this pull request Nov 15, 2026
…basis setup per layer

Three prover-only changes to the experimental Keccak GKR:

- `PowerField::pair_sums` builds a pair of rows' factor `B = w * T[j+1] + constant`, kappa folded into the constant `kappa * (w[j] + w[j+2])`, in registers where its dot with `T[j+2]` reads it (x86, NEON and portable arms), the shifted entries read through precomputed run offsets. `Round::factors`, `mul_row`, `dot_pair`, the shifted copies and the per-worker scratch buffers are gone.
- The claim's weights are converted to the power basis once per layer and shared by the Skip pass and `Round::new`; the local rounds keep the 1600 live positions, interleaved as `[w, t0, t1, t2]`, instead of the 2048-entry cube.
- `PowerBasis::new` eliminates columns with masks rather than a branch per row, and `PowerField::new`'s Barrett division shifts the modulus by words.

Proofs unchanged: `identical_proofs` passes on the Mac, h1, h2 and h4, and differential tests against the previous kernels and against the stable tree's Keccak GKR (whole proofs of 8 batch shapes, on NEON, AVX-512, 128-bit PCLMUL and portable builds) matched byte for byte. On the Keccak GKR alone (`hash_batch`, `KECCAK_N_LOG=17`) the pair sums take the Tables span down by 8% to 12% on h1 and h2.

From leanEthereum/leanVM PR #730 (commits 5ce1cec, 0624447, 4d31de0, authored by Amp); the branchless power basis is also in gattaca-com/leanVM (5d93998 / 2ba2730). Of #730, the VBMI staging (9d2e514) measured nothing on AVX2 and an uncertain gain on AVX-512 for 143 lines and was left out, as were the parallel local rounds of gattaca's 360ce3c (1600 positions do not pay for the pool).

This branch before -> after, `--prover experimental`, 3 rounds x 2 passes, interleaved, medians:
- h1 (Zen 4): aggregate --xmss 900 --log-inv-rate 1 0.800 -> 0.795 s (-0.6%), at 4 threads -0.3%, --sphincs 410 -0.3%, recursion --n 2 --xmss-per-leaf 900 --log-inv-rate 2 -0.8%.
- h2 (Arrow Lake): -0.7%, -0.7%, -0.7%, -0.5%.
- h4 (Zen 5): xmss 900 within noise (+0.4%), at 4 threads 0.777 -> 0.762 s (-1.9%), sphincs 410 -1.9%, recursion 0.090 -> 0.087 s (-3.3%).
- Mac (M4 Max): xmss 900 0.504 -> 0.487 s (-3.3%), at 4 threads -0.3%, sphincs 410 -0.5%, recursion 0.185 -> 0.178 s (-4.0%).
- Peak memory unchanged.

Co-authored-by: Parithosh Jayanthi <parithosh@forkchoice.ai>
Co-authored-by: alexX512 <alexey@gattaca.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TomWambsgans added a commit that referenced this pull request Nov 15, 2026
…basis setup per layer

Three prover-only changes to the experimental Keccak GKR:

- `PowerField::pair_sums` builds a pair of rows' factor `B = w * T[j+1] + constant`, kappa folded into the constant `kappa * (w[j] + w[j+2])`, in registers where its dot with `T[j+2]` reads it (x86, NEON and portable arms), the shifted entries read through precomputed run offsets. `Round::factors`, `mul_row`, `dot_pair`, the shifted copies and the per-worker scratch buffers are gone.
- The claim's weights are converted to the power basis once per layer and shared by the Skip pass and `Round::new`; the local rounds keep the 1600 live positions, interleaved as `[w, t0, t1, t2]`, instead of the 2048-entry cube.
- `PowerBasis::new` eliminates columns with masks rather than a branch per row, and `PowerField::new`'s Barrett division shifts the modulus by words.

Proofs unchanged: `identical_proofs` passes on the Mac, h1, h2 and h4, and differential tests against the previous kernels and against the stable tree's Keccak GKR (whole proofs of 8 batch shapes, on NEON, AVX-512, 128-bit PCLMUL and portable builds) matched byte for byte. On the Keccak GKR alone (`hash_batch`, `KECCAK_N_LOG=17`) the pair sums take the Tables span down by 8% to 12% on h1 and h2.

From leanEthereum/leanVM PR #730 (commits 5ce1cec, 0624447, 4d31de0, authored by Amp); the branchless power basis is also in gattaca-com/leanVM (5d93998 / 2ba2730). Of #730, the VBMI staging (9d2e514) measured nothing on AVX2 and an uncertain gain on AVX-512 for 143 lines and was left out, as were the parallel local rounds of gattaca's 360ce3c (1600 positions do not pay for the pool).

This branch before -> after, `--prover experimental`, 3 rounds x 2 passes, interleaved, medians:
- h1 (Zen 4): aggregate --xmss 900 --log-inv-rate 1 0.800 -> 0.795 s (-0.6%), at 4 threads -0.3%, --sphincs 410 -0.3%, recursion --n 2 --xmss-per-leaf 900 --log-inv-rate 2 -0.8%.
- h2 (Arrow Lake): -0.7%, -0.7%, -0.7%, -0.5%.
- h4 (Zen 5): xmss 900 within noise (+0.4%), at 4 threads 0.777 -> 0.762 s (-1.9%), sphincs 410 -1.9%, recursion 0.090 -> 0.087 s (-3.3%).
- Mac (M4 Max): xmss 900 0.504 -> 0.487 s (-3.3%), at 4 threads -0.3%, sphincs 410 -0.5%, recursion 0.185 -> 0.178 s (-4.0%).
- Peak memory unchanged.

Co-authored-by: Parithosh Jayanthi <17509050+parithosh@users.noreply.github.com>
Co-authored-by: Alexey Perevyshin <61140353+alexX512@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TomWambsgans

Copy link
Copy Markdown
Collaborator

took 3a31169
grazie mille

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants