Repository navigation
feat: Use CDI for GPU injection instead of nvidia-container-cli #398
Description
Problem Statement
GPU access currently relies on the legacy nvidia-container-runtime +
nvidia-container-cli stack at two layers: once when Docker injects GPUs
into the k3s cluster container, and again when the nvidia-device-plugin +
nvidia-container-runtime inject them into individual sandbox pods.
Proposed Design
Both layers should be migrated to CDI instead. The general idea:
- Generate a CDI spec on the host before starting the cluster:
nvidia-ctk cdi generate - Use Docker's native CDI support (available since Docker 25) to pass GPUs
into the k3s container:--device nvidia.com/gpu=all - Mount
/etc/cdiinto the k3s container, enableenable_cdi_devices = true
in the containerd config, and configure the nvidia-device-plugin to use CDI
device IDs so containerd handles injection natively
CDI is the canonical way NVIDIA supports GPU access in containerized
environments going forward. Some platforms require CDI and are incompatible
with the legacy runtime stack, so this would also broaden the set of platforms
OpenShell can run on. It also makes what gets injected explicit and
auditable via the CDI spec rather than delegating to a CLI with broad host
access.
Alternatives Considered
None
Agent Investigation
No response
Checklist
- I've reviewed existing issues and the architecture docs
- This is a design proposal, not a "please build this" request
Activity
tyeth commentedon Mar 18, 2026
Thanks for this, looks like it might solve my related issue in #404 and the PR in #411
elezar commentedon Mar 18, 2026
Thanks @klueska. This is definitely the direction we would want to move this so as to not depend on a specific runtime / k8s implementation. Using CDI as the mechanism for defining what
As of the NVIDIA Container Toolkit v1.18.0 the CDI specs for available devices SHOULD be generated automatically when installing the NVIDIA Container Toolkit. This should remove the need for explicit CDI spec generation. With that said, we may want to consider a situation where a sandbox is only given access to a subset of the GPUs so as to allow at least minimal isolation between multiple agents. Here the important thing is to separate CDI spec generation from consumption to allow for flexibility when defining new use cases. For example, if we are starting a REMOTE sandbox.
Although CDI support is enabled in docker as of Docker 25, it was an opt-in feature up to 28.2.0 -- where it was enabled by default. The logic for injection was not changed significantly between these versions.
The containerd config option that needs to be enabled is the
enable_cdiplugin option. The exact config path depends on the containerd version / CRI plugin used. Furthermore, it is not available in v1.6 of container, is disabled by default in v1.7, and enabled by default as of v2.0. The option is also marked as deprecated in the current HEAD.The default paths for CDI specs are
/etc/cdiand/var/run/cdi, with specifications in/var/run/cditaking precedence. As such we would also have to mount/var/run/cdi` to get access to the specs there. Also note that the device plugin (when CDI is activated) generates CDI specs itself -- assuming a containerized node these would have to be scoped to the container and not the host.klueska commentedon Mar 18, 2026
Thanks @elezar for the clarifications.
Regarding your first bullet -- This sounds like a host-level dependency that OpenShell would need to clarify in its README. From my understanding of how OpenShell works, I think the k3s container would still want access to all GPUs, and then each of the agent sandboxes it orchestrates would (potentially) want limited access to a subset of the GPUs (which would be made possible via the device plugin).
Regarding your fourth bullet -- I guess this means we don't actually need to inject these folders at all. Putting the device plugin into CDI mode would generate the specs on the fly so that these folders (and any generated specs) would be created on the fly.
klueska commentedon Mar 18, 2026
Depending on the version of k8s in use, it might also make sense to use the DRA driver for GPUs instead of the device plugin so that controlled GPU sharing could be introduced more easily in the future. Also, it uses CDI natively, so no need to "configure" it in CDI mode.
elezar commentedon Mar 18, 2026
Yes, I think we should separate giving the containerized node access to the driver and GPUs from ensuring that individual containers running on this containerized node (which I assume is each as sandbox).
I was also thinking of DRA in this context -- especially considering that the DRA driver doesn't have a dependency on the NVIDIA Container Toolkit. This would mean that it SHOULD be possible to simplify the node image and have the DRA driver generate CDI specs and provide the CDI hook as it does today.
elezar commentedon Jun 4, 2026
Closing this as superseded.
This issue was originally filed against the older k3s-based GPU architecture. The GPU path has since moved to specific compute driver implementations, with CDI support handled in the relevant driver/runtime paths rather than as one broad k3s migration item.
The remaining CDI-related work is now tracked more specifically in:
Those issues better capture the current architecture and remaining work.