[Andy-Jost]

Summary

A pickled Device named its device by ordinal, so a process with a different CUDA_VISIBLE_DEVICES unpickled it as a different GPU. Device.__reduce__ now sends the UUID, and the receiver maps it back with _device_from_uuid() from #3126. Unpickling raises RuntimeError, naming the UUID and the visible devices, when the device is not visible.

Changes

• _device.pyx: Device.__reduce__ returns (_device_from_uuid, (uuid,)). The class docstring describes the pickled form.
• 1.3.0 release-note entry.
• Tests: an in-process round trip returns the same thread-local singleton; a child with reversed visibility receives the device with the parent's UUID; a child that cannot see the device gets the error.

Related Work

Closes #3196. Stacked on #3126, which adds _device_from_uuid().

🤖 Generated with Claude Code

[coderabbitai]

📝 Summary

Summary by CodeRabbit

• New Features

  • Pickled devices and device memory resources now match devices by UUID, so they can be restored when device ordinals differ between processes.
  • Virtual memory resource peer settings are stored as a tuple, including when supplied as a one-time iterable.

• Bug Fixes

  • Restoring an object that references a device not visible to the receiving process now raises a RuntimeError identifying the missing device.
  • Clarified that plain pickle cannot transfer file descriptors required for virtual memory resources.

Device and memory resource pickling now identifies devices by UUID. The receiving process maps each UUID to a locally visible device and raises RuntimeError when no match exists. Virtual memory resource options also store peer IDs as a tuple.

Changes: UUID-based pickling

Layer / File(s)	Summary
Device UUID serialization cuda_core/cuda/core/_device.pyx, cuda_core/cuda/core/_device.pyi, cuda_core/tests/test_device.py	Device.__reduce__ serializes the device UUID. _device_from_uuid resolves it to a visible device or raises RuntimeError. Tests cover same-process pickle identity.
Memory resource UUID reconstruction cuda_core/cuda/core/_memory/_device_memory_resource.pyx, cuda_core/cuda/core/_memory/_device_memory_resource.pyi, cuda_core/cuda/core/_memory/_virtual_memory_resource.pyx, cuda_core/cuda/core/_memory/_virtual_memory_resource.pyi, cuda_core/tests/memory_ipc/test_device_uuid.py, cuda_core/tests/test_memory.py, cuda_core/docs/source/release/1.3.0-notes.rst	Memory resource pickles identify devices and peers by UUID and resolve them to receiving-process ordinals. Virtual memory resource options store peers as a tuple. IPC tests cover reordered and unavailable devices, buffer data, and peer access.

Assessment against linked issues

Objective	Addressed	Explanation
Pickle Device by UUID, resolve it in the receiving process regardless of ordinal, and raise RuntimeError when it is not visible [3196]	✅

Out-of-scope changes

Code Change	Explanation
Change device memory resource and virtual memory resource serialization, and materialize virtual memory peer options as tuples (cuda_core/cuda/core/_memory/_device_memory_resource.pyx:344-354; cuda_core/cuda/core/_memory/_virtual_memory_resource.pyx:169-173,471-476,1086-1094)	The linked objective concerns Device pickling; it does not specify changes to resource serialization or peer option storage.

Possibly related issues

• cuda.core: IPC pickling rebuilds memory resources with the sender's device indices #3097 — The PR changes DeviceMemoryResource and VirtualMemoryResource pickling to serialize device identities by UUID.

Suggested reviewers: mdboom, juenglin

Priority: ➖ Normal

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: 🔵 Low · up to ed9de

The change makes Device and memory resource pickling resolve GPUs by UUID, and no production defect was found. One new IPC test can leave its memory pool open when it fails, which can cause misleading follow-on failures in later tests. The fix is small and can be done before or shortly after merge.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

• Source mode: Auto
• Repository: NVIDIA/cuda-python
• Branch: main
• File: .coderabbit.yaml
• Commit: afa6249
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 6c0f850a-0fcc-4f2e-abda-c0c16ee5a016

📥 Commits

Reviewing files that changed from the base of the PR and between afa6249 and ed9de37.

📒 Files selected for processing: 10

• cuda_core/cuda/core/_device.pyi
• cuda_core/cuda/core/_device.pyx
• cuda_core/cuda/core/_memory/_device_memory_resource.pyi
• cuda_core/cuda/core/_memory/_device_memory_resource.pyx
• cuda_core/cuda/core/_memory/_virtual_memory_resource.pyi
• cuda_core/cuda/core/_memory/_virtual_memory_resource.pyx
• cuda_core/docs/source/release/1.3.0-notes.rst
• cuda_core/tests/memory_ipc/test_device_uuid.py
• cuda_core/tests/test_device.py
• cuda_core/tests/test_memory.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

[coderabbitai]

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

sed -n 140,165p cuda_core/tests/memory_ipc/test_device_uuid.py
sed -n 1,60p cuda_core/tests/memory_ipc/conftest.py

Repository: NVIDIA/cuda-python

Length of output: 2879

---

Close the owned pool and buffer in finally.

test_device_memory_resource_buffer closes buf and mr only after _send_to_child and _check_report return. If either call raises, the owned pool can stay open. The next init_cuda test can then fail with UnreleasedMemoryResourceWarning, which pytest.ini escalates to an error. Release the buffer and pool on every path. Initialize buf before the protected region so a failed allocation does not raise NameError in the cleanup block.

🐛 Suggested fix

--- "a/cuda_core/tests/memory_ipc/test_device_uuid.py"
+++ "b/cuda_core/tests/memory_ipc/test_device_uuid.py"
@@ -148,16 +148,19 @@
 def test_device_memory_resource_buffer(ipc_mempool_device_x2):
     """A pool-backed buffer comes back on the device that owns it, not on the device with the same ordinal."""
     dev0, dev1 = ipc_mempool_device_x2
     mr = DeviceMemoryResource(dev0, options=DeviceMemoryResourceOptions(max_size=POOL_SIZE, ipc_enabled=True))
-    buf = mr.allocate(NBYTES, stream=dev0.default_stream)
-    _fill(dev0, buf)
-
-    report = _send_to_child(buf, visible=(dev1, dev0))
-    _check_report(report, dev0)
-
-    buf.close()
-    dev0.sync()
-    mr.close()
+    buf = None
+    try:
+        buf = mr.allocate(NBYTES, stream=dev0.default_stream)
+        _fill(dev0, buf)
+
+        report = _send_to_child(buf, visible=(dev1, dev0))
+        _check_report(report, dev0)
+    finally:
+        if buf is not None:
+            buf.close()
+        dev0.sync()
+        mr.close()
 
 
 @pytest.mark.agent_authored(model="claude-opus-5-5")

[github-actions]

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-3198/
https://nvidia.github.io/cuda-python/pr-preview/pr-3198/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-3198/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-3198/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.