Skip to content

perf(cuda): speed up bit-unpacking in dyn dispatch - #11629

Merged
0ax1 merged 1 commit into
developfrom
ad/dyn-dispatch-lane-slices
Nov 19, 2026
Merged

0ax1 merged 1 commit into
developfrom
ad/dyn-dispatch-lane-slices

Conversation

@0ax1

@0ax1 0ax1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

The dynamic dispatch kernel unpacks each 1024-element FastLanes chunk with one thread per lane, but 32- and 64-bit chunks have only 32 and 16 lanes, so half or three quarters of each 64-thread block waited at the barrier during the unpack. This splits those lanes into 16-row slices so every thread decodes one, and spreads the work over all chunks of a stage. The generator emits the per-row decode code once, as slices, and builds the whole-lane decoders used by the standalone kernels from them.

Fused u32 and u64 kernels in dynamic_dispatch_cuda are 11–19% faster, e.g. alp_for_bp_6bw_f64 at 100M elements goes from 369 µs to 299 µs.

The dynamic dispatch kernel runs 64-thread blocks, but it assigned one thread
per FastLanes lane when unpacking a 1024-element chunk. 32- and 64-bit chunks
have only 32 and 16 lanes, so half or three quarters of the block waited at
the barrier while the rest unpacked.

Split 32- and 64-bit lanes into 16-row slices so that every thread decodes one
slice, and distribute (chunk, slice, lane) work across the block, which also
parallelizes multi-chunk input stages such as dictionary values. The generator
now emits the per-row decode code once, as slices, and builds the whole-lane
decoders used by the standalone kernels from them.

Fused u32 and u64 kernels in the dynamic_dispatch_cuda benchmarks are 11-19%
faster, e.g. ALP(FoR(BitPacked)) f64 369 -> 299 us at 100M elements.

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
@0ax1 0ax1 added the changelog/performance A performance improvement label Nov 19, 2026
@0ax1 0ax1 changed the title Speed up bit-unpacking in the dynamic dispatch kernel perf(cuda): speed up bit-unpacking in the dynamic dispatch kernel Nov 19, 2026
@0ax1 0ax1 changed the title perf(cuda): speed up bit-unpacking in the dynamic dispatch kernel perf(cuda): speed up bit-unpacking in the dyn dispatch kernel Nov 19, 2026
@0ax1 0ax1 changed the title perf(cuda): speed up bit-unpacking in the dyn dispatch kernel perf(cuda): speed up bit-unpacking in dyn dispatch Nov 19, 2026
@0ax1
0ax1 requested a review from robert3005 November 19, 2026 12:47
@0ax1
0ax1 enabled auto-merge (squash) November 19, 2026 12:48
@0ax1
0ax1 merged commit 42b2a8c into develop Nov 19, 2026
119 of 120 checks passed
@0ax1
0ax1 deleted the ad/dyn-dispatch-lane-slices branch November 19, 2026 12:55
@robert3005

Copy link
Copy Markdown
Contributor

I trust that the math checks out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants