Skip to content

cuda(perf): pass the dyn dispatch plan as kernel parameter - #11630

Merged
0ax1 merged 1 commit into
developfrom
ad/dyn-dispatch-plan-param
Nov 20, 2026
Merged

0ax1 merged 1 commit into
developfrom
ad/dyn-dispatch-plan-param

Conversation

@0ax1

@0ax1 0ax1 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Every fused dynamic dispatch launch allocated a device buffer for its 100–200 byte plan, copied the plan into it from pageable host memory, and tracked the buffer until the kernel finished.

This change now passes the plan by value as a 1 KB __grid_constant__ kernel parameter instead; plans that don't fit fall back to unfused dispatch, like plans over the shared memory limit. The kernel now also reads the plan, including the scalar ops it re-reads on every tile, from the constant cache instead of global memory.

Total time per fused dictionary decode (host wall time including the kernel):

Elements Before After
16K 20.3 µs 16.4 µs
256K 20.7 µs 16.7 µs
4M 24.6 µs 21.2 µs

Kernel time in dynamic_dispatch_cuda (100M elements, plan uploaded outside the timed region):

Benchmark Before After
for_bitpacked_6bw/dispatch_u32 210.9 µs 159.5 µs
dict_256vals_bp8bw_codes/dispatch_u32 240.4 µs 184.0 µs
dict_nowiden_u32_to_u32/dispatch_u32 183.0 µs 146.9 µs
alp_for_bp_6bw_f32/dispatch_f32 228.4 µs 180.8 µs
alp_for_bp_6bw_f64/dispatch_f64 369.3 µs 360.8 µs
alp_for_bp_6bw_f64_1pct_patches/dispatch_f64 452.2 µs 424.1 µs

@0ax1 0ax1 added the changelog/performance A performance improvement label Nov 19, 2026
@0ax1 0ax1 changed the title Pass the dynamic dispatch plan as a kernel parameter cuda(perf): pass the dyn dispatch plan as kernel parameter Nov 19, 2026
@0ax1
0ax1 requested a review from robert3005 November 19, 2026 13:23
@0ax1
0ax1 enabled auto-merge (squash) November 19, 2026 13:24
@robert3005

Copy link
Copy Markdown
Contributor

you need to resolve conflicts with your other prs

Every fused dynamic dispatch launch allocated a device buffer for its packed
plan and copied the plan into it from pageable host memory, then tracked the
buffer until the kernel finished. Plans are 100-200 bytes, so this was pure
per-launch overhead.

Pass the plan by value as a 1 KB __grid_constant__ kernel parameter instead.
Plans that do not fit fall back to unfused dispatch, like plans that exceed
the shared memory limit. The kernel now reads the plan, including the scalar
ops it re-reads on every tile, from the constant cache rather than global
memory.

A fused dictionary decode takes 16.4 us of host time instead of 20.3 us at
16K elements, and 21.2 us instead of 24.6 us at 4M elements. Kernel time in
the dynamic_dispatch_cuda benchmarks drops by 15-24% for u32 outputs (e.g.
FoR(BitPacked) u32 at 100M elements from 211 us to 160 us) and by 2-6% for
u64 outputs.

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
@0ax1
0ax1 force-pushed the ad/dyn-dispatch-plan-param branch from b468cca to f8487f5 Compare November 20, 2026 14:42
@codspeed

codspeed Bot commented Nov 20, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 30.89%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 3 improved benchmarks
✅ 2030 untouched benchmarks
⏩ 6 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime cuda/dict_4096vals_bp12bw_codes/dispatch_u64[100M] 3.6 ms 2.3 ms +60.63%
⚡ Walltime cuda/dict_4096vals_bp12bw_codes/dispatch_u32[100M] 1.5 ms 1.2 ms +26.01%
⚡ Walltime cuda/bitpacked_u16/unpack/5bw[100M] 618.1 µs 557.9 µs +10.78%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ad/dyn-dispatch-plan-param (f8487f5) with develop (cb0ebee)

Open in CodSpeed

Footnotes

  1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@0ax1
0ax1 merged commit 33eab03 into develop Nov 20, 2026
76 checks passed
@0ax1
0ax1 deleted the ad/dyn-dispatch-plan-param branch November 20, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants