Skip to content

cuda(perf): don't drain the stream after host-to-device copies - #11634

Merged
0ax1 merged 1 commit into
developfrom
ad/cuda-copy-no-drain
Nov 19, 2026
Merged

0ax1 merged 1 commit into
developfrom
ad/cuda-copy-no-drain

Conversation

@0ax1

@0ax1 0ax1 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

CudaExecutionCtx::copy_to_device and ensure_on_device waited for a stream callback after enqueueing each copy. That callback fires only once all earlier work on the stream has finished, so every small host-to-device copy during decode and Arrow export (e.g. FSST symbol tables, binary-export pointer tables) stalled the host until the GPU drained, leaving the GPU idle while the host queued the next kernels.

The wait isn't needed: later work on the same stream is already ordered after the copy, and readers on other streams wait for the buffer's recorded write. Both methods now return the buffer as soon as the copy is enqueued. DeviceReadAt, which hands buffers across streams, keeps the waiting copy.

Measured end to end with BenchPress (TPC-H SF10 lineitem, GPU read through the CUDA FFI, hot page cache, 5 iterations, two runs each):

Before After
Read time 41.85 ms 40.25 ms
Throughput 196.0 GB/s 203.9 GB/s

CudaExecutionCtx::copy_to_device and ensure_on_device awaited a stream callback after enqueueing the copy, which waits for all earlier work on the stream, not just the copy. Later work on the same stream is already ordered after the copy, and other streams wait for it through the buffer's access tracking, so return the buffer right away.
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
@0ax1 0ax1 added the changelog/performance A performance improvement label Nov 19, 2026
@0ax1
0ax1 requested a review from robert3005 November 19, 2026 14:57
@0ax1
0ax1 enabled auto-merge (squash) November 19, 2026 14:57
@0ax1
0ax1 merged commit 0682576 into develop Nov 19, 2026
103 of 105 checks passed
@0ax1
0ax1 deleted the ad/cuda-copy-no-drain branch November 19, 2026 15:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants