Repository navigation
cuda.core: a pickled Device names its device by ordinal #3196
Description
Device.__reduce__ pickles the device ordinal (return Device, (self.device_id,)). Ordinals are per process, so a Device sent to a process with a different CUDA_VISIBLE_DEVICES unpickles to a different GPU, with no error.
#3126 sends memory resources by UUID (#3097). A Device sent next to them still travels by ordinal. The IPC tests do this: they pass device to the child, and the child calls device.set_current(). With reordered visibility, the buffer arrives on the right GPU and the child makes the wrong GPU current.
Fix: pickle a Device by UUID. Unpickling returns the device with that UUID in the receiving process, whatever its ordinal is there, and raises RuntimeError when that device is not visible. Processes share a Device to work with shared memory, so the physical device is the only useful meaning. A different device would fail later, when the buffer is mapped.
Raised by @juenglin in #3126 (comment).
Activity
jpascucci-nv commentedon Oct 23, 2026
Today, if you pickle and move to a different system, doesn't that succeed? But with this, it would fail. (Maybe this use-case isn't supported for some other reason)
Merge remote-tracking branch 'origin/main' into ajost/device-pickle-uuid
Andy-Jost commentedon Oct 23, 2026
@jpascucci-nv Today it does not raise, but it does not do the right thing either. The pickle holds an ordinal. On the other machine,
Device(n)returns whichever GPU holds ordinalnthere. If that machine has fewer thann + 1GPUs, unpickling already raisesValueError. The current form never promised a device across machines. It carried a number.A
Devicecrosses a process boundary for one purpose in cuda.core: to work with IPC memory on one physical GPU. That works within one machine. The memory objects cannot travel further. The file descriptors that backDeviceMemoryResourceandVirtualMemoryResourcebuffers move only throughmultiprocessing. Plainpickleof aVirtualMemoryResourcebuffer raises, and plainpickleof a pool-backed buffer loads only in a process that already received the pool throughmultiprocessing. Fabric handles, the one mechanism that crosses nodes, are not supported yet.With #3198, a
Deviceunpickled on a machine that lacks that GPU raisesRuntimeError. The message names the missing UUID and the visible devices. That is the correct hard error. The ordinal form picks a different GPU in silence, and that is the bug behind #3196 and #3097.Cross-machine
Devicepickling is not a supported use. A program that means "GPUnon the other machine" should send the integer and callDevice(n)there. A program that means "this physical GPU" can senddevice.uuid, which is what the new pickle does.I will add a sentence to the docstring and the release note: the pickled form is for processes on the same machine.
fix(cuda.core): say the Device pickle is for one machine; close test …
fix(cuda.core): pickle Device by UUID (#3198)