Skip to content

refactor(cuda): avoid access tracking for address-only lookups - #10884

Merged
0ax1 merged 3 commits into
developfrom
ad/cuda-device-address
Nov 4, 2026
Merged

0ax1 merged 3 commits into
developfrom
ad/cuda-device-address

Conversation

@0ax1

@0ax1 0ax1 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds CudaSlice::device_address for callers that only need an allocation's
address, without ordering or recording a device access.

Use it when building output-pointer metadata in FSST, OnPair, and Zstd,
and for address checks in dynamic-dispatch tests. Actual GPU reads and
writes remain tracked through launch arguments or device-pointer guards.

Changes

  • Add and document the untracked device_address accessor.
  • Replace address-only device_ptr calls that immediately discarded
    their access guards.
  • Keep the tracked read in CudaDeviceBuffer::new: it orders the allocation
    stream after pending writes on other streams. Document why its guard
    can be dropped immediately.
  • Simplify the Zstd comments identifying where output writes are tracked.

Each replaced lookup avoids two CUDA context checks, two access-state
mutex acquisitions, and unnecessary read-tracking updates. These sites
did not require cross-stream waits, so this removes bookkeeping rather
than synchronization needed by the workload.

No builds, tests, or benchmarks were run for this change.

@0ax1 0ax1 added the changelog/performance A performance improvement label Oct 27, 2026
@0ax1

0ax1 commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

@claude review this pr in depth

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @0ax1's task in 1m 17s —— View job


Review of PR #10884: CudaSlice::device_address

  • Read the diff and the Access / SyncOnDrop ordering model in driver/src/memory.rs
  • Check each call site: does dropping the tracked read change any ordering?
  • Look for other sites with the same pattern
  • Write up the review

Verdict: LGTM. The change is correct and doesn't change stream ordering at any of the sites. I have a couple of nits on comments and on how the PR describes the performance gain. I didn't find any bugs.

Correctness

In the old code, let (p, _) = x.device_ptr(s) ran Access::before(.., Read) and then dropped the guard straight away, which ran Access::after(.., Read) before any work used the pointer. So the "read" it recorded never covered the kernel that later used the address. The only real effect was in before(): if the last write was on a different stream, the calling stream waited for it.

I checked whether that wait could matter at each site:

Site Allocation Stream used Cross-stream write possible?
fsst.rs:244, fsst.rs:338 fresh ctx.device_alloc ctx.stream() (= owner) No, no writes yet
onpair.rs:498 fresh ctx.device_alloc ctx.stream() (= owner) No
zstd.rs:131, zstd_buffers.rs:88 fresh ctx.device_alloc ctx.stream() (= owner) No
dynamic_dispatch tests (643, 2313, 2351) stream.clone_htod same stream No, the write is on the owner

None of these sites could wait, so dropping the tracked read keeps ordering identical. The real writes are still tracked where they happen:

  • FSST and OnPair pass the buffer through .arg(&mut output) / .arg(&mut bytes) on the launch builder.
  • Zstd keeps record_device_output = device_output.device_ptr_mut(stream) alive across decompress_async (zstd.rs:269, zstd_buffers.rs:121).

The change also removes a small side effect. On the owner stream, after(Read) sets allocation_stream_pending = true. In the dispatch tests that flag had just been cleared by the clone_htod write. Leaving it set would make a later cross-stream access record a snapshot event that isn't needed. The new code avoids that, which is more accurate.

The new method is a reasonable addition to the API. CudaDeviceBuffer::offset_ptr / cuda_device_ptr() already hand out untracked addresses (e.g. zstd.rs:124, zstd.rs:155-158), so this adds the same thing for a bare CudaSlice. The rustdoc is clear about the caller's responsibility, and the intra-doc link to DevicePtr::device_ptr resolves because the trait is in the same module.

Suggestions (nits)

  1. Explain why device_buffer.rs:118 keeps the tracked read. The PR body explains why CudaDeviceBuffer::new still calls device_ptr: it makes the allocation stream wait for writes on other streams. But there's no comment in the code, and let (device_ptr, _) = cuda_slice.device_ptr(cuda_slice.stream()) is now the only (x, _) discard left in vortex-cuda. Someone tidying up later will probably switch it to device_address() and silently lose that ordering. A one-line comment would prevent that. Fix this →

  2. The performance description is slightly off. At every changed site the stream is the owner and nothing has been written from another stream, so "may make the stream wait" never actually happened here. The saving per call is two bind_to_thread calls (cuCtxGetCurrent, plus cuCtxSetCurrent if the context isn't current) and two parking_lot lock acquisitions, one each in before and after, not just "a lock". Each call happens once per decode, so the gain is real but small. changelog/performance is fine, but the PR reads more naturally as a cleanup that also saves a little work.

  3. Optional: the comments in zstd.rs:129-130 and zstd_buffers.rs:86-87 could be shorter. "Only the allocation address builds the pointer metadata" reads a little oddly. Since the method name now says what's going on, something like // The write is tracked by record_device_output around decompress_async. would be enough. This is purely cosmetic.

  4. Optional: CudaView has no equivalent. That's fine since no caller needs one yet. If one comes up, an offset-aware device_address on the views would mirror this method.

Checks

I didn't compile or run anything; this review comes from reading the code. CUDA tests need a GPU runner, so CI will cover them.

@codspeed

codspeed Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 1765 untouched benchmarks
⏩ 130 skipped benchmarks1
🗄️ 47 archived benchmarks run2


Comparing ad/cuda-device-address (19829c2) with develop (0151998)

Open in CodSpeed

Footnotes

  1. 130 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. 47 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

0ax1 added 3 commits November 4, 2026 13:12
Several kernels and tests only needed an allocation's address, to check
its alignment or to build pointer metadata, and got it from
`device_ptr`, discarding the guard right away. Since #10722, every such
call is a tracked read that locks the access state, binds the context
and may make the stream wait, all for an access that never happens.

Add `CudaSlice::device_address`, which only returns the address, and use
it at those sites. `CudaDeviceBuffer::new` keeps its tracked read, whose
wait orders the allocation stream after writes on other streams.

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
@0ax1
0ax1 force-pushed the ad/cuda-device-address branch from 4cde242 to 19829c2 Compare November 4, 2026 13:23
@0ax1
0ax1 marked this pull request as ready for review November 4, 2026 13:23
@0ax1 0ax1 changed the title refactor(cuda): read device addresses without tracking an access refactor(cuda): avoid access tracking for address-only lookups Nov 4, 2026
@0ax1
0ax1 enabled auto-merge (squash) November 4, 2026 13:24
@0ax1
0ax1 requested review from mprammer and robert3005 November 4, 2026 13:25
@0ax1
0ax1 merged commit 531d126 into develop Nov 4, 2026
132 checks passed
@0ax1
0ax1 deleted the ad/cuda-device-address branch November 4, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants