task: assess Kubernetes GPU support preflight validation #1807

Description

Assess how the Kubernetes compute driver should determine whether a cluster exposes GPU resources before accepting GPU sandbox requests.

This follows from the --gpu-count work and should not change behavior in the current small CLI PR. The goal is to decide what signal the driver should use for fail-fast feedback when a cluster has no GPU support, without incorrectly rejecting requests that Kubernetes could schedule later.

Context

Related roadmap: #1444

The current Kubernetes driver preflight checks node status.allocatable["nvidia.com/gpu"] and treats any non-zero value as GPU support. During review of the GPU count CLI work, we identified that this check is really about whether the cluster exposes GPU resources, not whether a particular sandbox can schedule immediately.

Important distinctions:

• capacity is a more stable signal that a node advertises the GPU extended resource.
• allocatable is closer to whether pods can request that resource.
• Neither field alone answers whether a specific pod can schedule right now.
• Current resource pressure should generally be left to the Kubernetes scheduler, because a sandbox may become schedulable after other pods release GPUs.
• Autoscaled GPU node pools may complicate any check based only on currently present nodes.

Definition of Done

• Decide whether the Kubernetes preflight should use node capacity, node allocatable, both, or another signal.
• Decide whether GPU support detection should remain boolean or account for requested gpu_count.
• Decide how zero-scaled or autoscaled GPU node pools should be handled.
• Document the intended semantics in the Kubernetes driver README or architecture docs.
• Create implementation follow-up issues if behavior should change.