Repository navigation
cuda(perf): decode dyn dispatch input stages once per resident block - #11631
Merged
Merged
Conversation
Every block of the fused kernel decodes the whole dictionary or all runs into shared memory, so the existing 64- and 256-value dictionaries and 100 runs hide how that cost grows. Add 1024- and 4096-value dictionaries, including u64 values, and 1024 runs. Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Every block of the fused kernel decodes the plan's input stages, such as dictionary values or run ends, into its own shared memory before producing its 2048 output elements. With one block per 2048 elements, a 4096-value dictionary is decoded twice for every block's worth of output. Add a loop_ranges kernel that launches only as many blocks as are resident at once, from the occupancy API, and loops each block over several output ranges at a grid stride, so the input stages are decoded once per resident block. Plans whose input stages decode fewer than 64 elements keep the original kernel, where looping costs more than it saves. The loop lives in a separate kernel because it changes the register allocation of the original kernel and slowed down plans without input stages by 3-6%; the original kernel's code is unchanged. In dynamic_dispatch_cuda at 100M elements, a 4096-value dictionary takes 674 us instead of 1143 us (u32) and 1378 us instead of 2349 us (u64), 1024 runs take 207 us instead of 371 us, and a 1024-value dictionary 312 us instead of 342 us. Plans without input stages are unchanged. Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
0ax1
enabled auto-merge (squash)
November 19, 2026 13:57
Merging this PR will not alter performance1 warning
Performance Changes
Comparing Footnotes
|
robert3005
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The fused kernel runs one block per 2048 output elements, and every block first decodes the plan's input stages (dictionary values, run ends) into its own shared memory. With large dictionaries or many runs, most of the kernel's time went into decoding the same input stages over and over.
This adds a
loop_rangeskernel that launches only as many blocks as fit on the GPU at once and loops each over several 2048-element ranges. Each block decodes the input stages once and reuses them for all its ranges, cutting redundant decodes by roughly 12× at 100M elements. Plans with fewer than 64 input elements keep the original kernel, where the decode is cheap and looping doesn't pay off.dynamic_dispatch_cudaat 100M elements:dict_4096vals_bp12bw_codes/dispatch_u32dict_4096vals_bp12bw_codes/dispatch_u64runend_1024runs/dispatch_u32dict_1024vals_bp10bw_codes/dispatch_u32