Repository navigation
cuda(perf): pass the dyn dispatch plan as kernel parameter - #11630
Merged
Merged
Conversation
0ax1
enabled auto-merge (squash)
November 19, 2026 13:24
robert3005
approved these changes
Nov 19, 2026
Contributor
|
you need to resolve conflicts with your other prs |
Every fused dynamic dispatch launch allocated a device buffer for its packed plan and copied the plan into it from pageable host memory, then tracked the buffer until the kernel finished. Plans are 100-200 bytes, so this was pure per-launch overhead. Pass the plan by value as a 1 KB __grid_constant__ kernel parameter instead. Plans that do not fit fall back to unfused dispatch, like plans that exceed the shared memory limit. The kernel now reads the plan, including the scalar ops it re-reads on every tile, from the constant cache rather than global memory. A fused dictionary decode takes 16.4 us of host time instead of 20.3 us at 16K elements, and 21.2 us instead of 24.6 us at 4M elements. Kernel time in the dynamic_dispatch_cuda benchmarks drops by 15-24% for u32 outputs (e.g. FoR(BitPacked) u32 at 100M elements from 211 us to 160 us) and by 2-6% for u64 outputs. Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
0ax1
force-pushed
the
ad/dyn-dispatch-plan-param
branch
from
November 20, 2026 14:42
b468cca to
f8487f5
Compare
Merging this PR will improve performance by 30.89%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every fused dynamic dispatch launch allocated a device buffer for its 100–200 byte plan, copied the plan into it from pageable host memory, and tracked the buffer until the kernel finished.
This change now passes the plan by value as a 1 KB
__grid_constant__kernel parameter instead; plans that don't fit fall back to unfused dispatch, like plans over the shared memory limit. The kernel now also reads the plan, including the scalar ops it re-reads on every tile, from the constant cache instead of global memory.Total time per fused dictionary decode (host wall time including the kernel):
Kernel time in
dynamic_dispatch_cuda(100M elements, plan uploaded outside the timed region):for_bitpacked_6bw/dispatch_u32dict_256vals_bp8bw_codes/dispatch_u32dict_nowiden_u32_to_u32/dispatch_u32alp_for_bp_6bw_f32/dispatch_f32alp_for_bp_6bw_f64/dispatch_f64alp_for_bp_6bw_f64_1pct_patches/dispatch_f64