Skip to content

cuda(perf): decode dyn dispatch input stages once per resident block - #11631

Merged
0ax1 merged 3 commits into
developfrom
ad/dyn-dispatch-persistent-blocks
Nov 19, 2026
Merged

0ax1 merged 3 commits into
developfrom
ad/dyn-dispatch-persistent-blocks

Conversation

@0ax1

@0ax1 0ax1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

The fused kernel runs one block per 2048 output elements, and every block first decodes the plan's input stages (dictionary values, run ends) into its own shared memory. With large dictionaries or many runs, most of the kernel's time went into decoding the same input stages over and over.

This adds a loop_ranges kernel that launches only as many blocks as fit on the GPU at once and loops each over several 2048-element ranges. Each block decodes the input stages once and reuses them for all its ranges, cutting redundant decodes by roughly 12× at 100M elements. Plans with fewer than 64 input elements keep the original kernel, where the decode is cheap and looping doesn't pay off.

dynamic_dispatch_cuda at 100M elements:

Benchmark Before After
dict_4096vals_bp12bw_codes/dispatch_u32 1142.8 µs 673.9 µs
dict_4096vals_bp12bw_codes/dispatch_u64 2348.7 µs 1377.9 µs
runend_1024runs/dispatch_u32 370.7 µs 206.6 µs
dict_1024vals_bp10bw_codes/dispatch_u32 341.9 µs 312.2 µs

0ax1 added 2 commits November 19, 2026 13:29
Every block of the fused kernel decodes the whole dictionary or all runs into shared memory, so the existing 64- and 256-value dictionaries and 100 runs hide how that cost grows. Add 1024- and 4096-value dictionaries, including u64 values, and 1024 runs.

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Every block of the fused kernel decodes the plan's input stages, such as
dictionary values or run ends, into its own shared memory before producing
its 2048 output elements. With one block per 2048 elements, a 4096-value
dictionary is decoded twice for every block's worth of output.

Add a loop_ranges kernel that launches only as many blocks as are resident at
once, from the occupancy API, and loops each block over several output ranges
at a grid stride, so the input stages are decoded once per resident block.
Plans whose input stages decode fewer than 64 elements keep the original
kernel, where looping costs more than it saves. The loop lives in a separate
kernel because it changes the register allocation of the original kernel and
slowed down plans without input stages by 3-6%; the original kernel's code is
unchanged.

In dynamic_dispatch_cuda at 100M elements, a 4096-value dictionary takes
674 us instead of 1143 us (u32) and 1378 us instead of 2349 us (u64), 1024
runs take 207 us instead of 371 us, and a 1024-value dictionary 312 us instead
of 342 us. Plans without input stages are unchanged.

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
@0ax1 0ax1 added the changelog/performance A performance improvement label Nov 19, 2026
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
@0ax1 0ax1 changed the title Decode dynamic dispatch input stages once per resident block cuda(perf): decode dyn dispatch input stages once per resident block Nov 19, 2026
@0ax1
0ax1 requested a review from robert3005 November 19, 2026 13:56
@0ax1
0ax1 enabled auto-merge (squash) November 19, 2026 13:57
@codspeed

codspeed Bot commented Nov 19, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 2029 untouched benchmarks
🆕 4 new benchmarks
⏩ 6 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 Walltime cuda/dict_1024vals_bp10bw_codes/dispatch_u32[100M] N/A 1.2 ms N/A
🆕 Walltime cuda/dict_4096vals_bp12bw_codes/dispatch_u32[100M] N/A 1.5 ms N/A
🆕 Walltime cuda/dict_4096vals_bp12bw_codes/dispatch_u64[100M] N/A 3.6 ms N/A
🆕 Walltime cuda/runend_1024runs/dispatch_u32[100M] N/A 878.6 µs N/A

Comparing ad/dyn-dispatch-persistent-blocks (ac400d4) with develop (42b2a8c)

Open in CodSpeed

Footnotes

  1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@0ax1
0ax1 merged commit 3da99e9 into develop Nov 19, 2026
107 of 108 checks passed
@0ax1
0ax1 deleted the ad/dyn-dispatch-persistent-blocks branch November 19, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants