feat(cluster): curated vLLM model deploy for DGX Spark/Station

@pimlock

Summary

Consolidated tracking issue for local model deployment in Navigator clusters, updated with the DGX Spark and DGX Station runtime plan.

This issue consolidates #41 and #42.

Target environment

Primary V1 target hardware: NVIDIA DGX Spark and NVIDIA DGX Station (single-node k3s)
Must degrade safely on non-GPU or unsupported hardware

Runtime direction (updated decision)

Maintain a curated list of models validated for DGX Spark and DGX Station
Use vLLM as the default runtime for curated models, with config profiles tuned for this hardware
For non-curated or unsupported model requests, use one of:
- a small baseline model deployed with Ollama
- a lightweight mock server that echoes the request
Persist model weights on a PVC so weights survive restart, similar to gateway SQLite persistence
Research disk offload strategies as a follow-up; this is not in V1 scope

Phase 1 (V1): CLI-first model deploy

Implement navigator cluster model commands, with deploy as the happy path:

navigator cluster model deploy [--profile auto] [--backend auto]
navigator cluster model list
navigator cluster model status
navigator cluster model remove

Behavior for deploy:

Detect cluster and node hardware (arch, allocatable memory, GPU resources)
Select from curated DGX Spark or DGX Station model profiles when hardware matches
Deploy vLLM with hardware-tuned preset for curated profiles
If request is not curated or hardware is unsupported, fall back to Ollama small model or mock echo server
Provision and mount persistent storage for model weights
Wait for readiness and surface actionable status and errors
Auto-create an inference route for the model endpoint (route management remains under navigator inference route commands)

Phase 2: bootstrap plus managed system routes

After CLI-first works:

Integrate optional model deployment into cluster bootstrap flow
Add route ownership metadata (for example managed_by or source or system flag)
Protect model-managed routes from unsafe delete and update operations
Keep protection semantics runtime-agnostic

Auto-detect guardrails

Inputs: node architecture, node allocatable memory, GPU resource keys such as nvidia.com/gpu, plus runtime probes where available
Memory safety: require headroom before profile selection; downshift on uncertainty
Start conservative: single model, single replica, bounded context and concurrency defaults

Acceptance criteria

Phase 1:
- navigator cluster model deploy performs end-to-end deploy on an existing cluster with no manual Kubernetes steps
- Curated profile selection picks a safe default on DGX Spark and DGX Station, with safe fallback when unsupported
- vLLM preset is applied for curated models and hardware profile
- Fallback path deploys either a small Ollama model or mock echo server
- Model endpoint is reachable and a route is created automatically
- Model weights persist across restart via PVC-backed storage
- Automated tests cover deploy plus route wiring plus persistence checks
- Docs cover curated model behavior, fallback behavior, profile override behavior, and troubleshooting
Phase 2:
- Bootstrap can optionally deploy a model after cluster readiness
- Model-managed routes are clearly represented and protected
- Automated coverage exists for protected route behavior

Out of scope (initial)

Multi-model scheduling and autoscaling policies
Full operator or control-plane adoption in V1
Disk offload implementation and advanced offload tuning (research follow-up)

Originally by @pimlock on 2026-02-17T16:39:43.763-08:00

🏗️ build-plan

Implementation Plan

Issue type: feat
Complexity: High
Confidence: Medium - clear Phase 1 slice, but runtime and bootstrap validation still need iteration on real Spark/Station hardware

Summary

Add a new gateway-managed model deployment flow with nemoclaw model catalog, deploy, list, status, watch, and remove. Phase 1 should feel like sandbox creation: the CLI requests model deployment from the gateway, the gateway detects cluster hardware, validates a chosen config, performs preflight checks, orchestrates the runtime, streams status/log/event updates, and programs managed cluster inference through the existing inference.local flow.

Scope

crates/navigator-cli/src/main.rs: add a top-level model command group with catalog/deploy/list/status/watch/remove
crates/navigator-cli/src/run.rs: add model deploy/list/status/watch/remove handlers and stream rendering
crates/navigator-cli/src/model_deploy.rs: add catalog display, optional interactive selection, request construction, and watch rendering helpers
proto/navigator.proto: add model catalog plus model deployment create/get/list/delete/watch RPCs
proto/datamodel.proto: add ModelDeployment, ModelDeploymentSpec, ModelDeploymentStatus, ModelDeploymentPhase, and SupportedModelConfig
crates/navigator-server/src/grpc.rs: add model deployment CRUD/watch handlers
crates/navigator-server/src/model_deploy.rs: add supported-config filtering, platform selection, preflight checks, manifest rendering, orchestration, generated provider management, and managed cluster inference programming
crates/navigator-server/src/model_deploy_watch.rs: add watch bus and event-correlation helpers modeled after sandbox watch flows
crates/navigator-models/: shared model catalog, capability checks, deployment state helpers, and preflight primitives
README.md: document nemoclaw model usage and fallback behavior
architecture/cluster-single-node.md: document model deployment alongside gateway lifecycle commands
architecture/inference-routing.md: document how model deployment programs managed inference.local via generated providers
crates/navigator-bootstrap/ and cluster startup paths: update bootstrap/runtime wiring so Spark/Station clusters can expose GPUs into k3s workloads

Implementation Steps

Add minimal model catalog and model deployment proto/datamodel messages plus create/get/list/delete/watch RPCs.
Add server-side storage, watch bus, and a small single-model deployment orchestrator.
Add a checked-in curated config catalog plus platform-driven selection logic in the gateway.
Add preflight checks for storage class presence, free disk space, and best-effort local cache reuse before manifest application.
Add manifest rendering for vLLM StatefulSet, Service, PVC-backed storage, and mock fallback behavior.
Map Kubernetes conditions/events/logs into user-visible phases, including best-effort DownloadingWeights progress.
On success, create/update a generated provider record and program managed cluster inference via SetClusterInference / GetClusterInference.
Extend the CLI with model catalog/deploy/list/status/watch/remove, optional interactive selection, and stream rendering.
Add automated tests for selection logic, preflight checks, orchestration, watch streams, and inference integration.
Update architecture and user-facing docs.
Add a fast iteration loop for Spark/Station detection: validate k3s-in-docker GPU visibility first, then fold the required bootstrap flags/runtime setup into navigator-bootstrap.

Detection Strategy

Use an explicit platform profile as the primary selector: spark, station, or unknown.
Use live Kubernetes facts as validation gates, not as the primary chooser.
Supported config visibility should be platform-driven first, then filtered by cluster sanity checks such as architecture, allocatable memory, NVIDIA GPU resource presence, and storage.
For V1, platform selection can come from explicit cluster metadata/config with heuristic fallback only when metadata is absent.

Fast Iteration Loop

Before wiring full model deployment into the cluster bootstrap path, validate GPU visibility with a tight loop: run k3s inside Docker on Spark/Station, verify that the Kubernetes node reports the expected NVIDIA resources, and only then encode the required startup/runtime flags into bootstrap.
Treat Docker --gpus all, NVIDIA runtime/toolkit configuration, device-plugin availability, and k3s/containerd GPU exposure as prerequisites for real vLLM deployment.
Use a tiny vLLM model first for runtime validation before attempting larger curated models.

Proposed Phase 1 UX

nemoclaw model catalog
nemoclaw model deploy <name> --config <id>
nemoclaw model list
nemoclaw model status <name>
nemoclaw model watch <name>
nemoclaw model remove <name>

Progress Phases

Pending
PreflightChecks
SelectingProfile
SelectedConfig
ApplyingResources
DownloadingWeights
StartingRuntime
Ready
Error
Deleting

Test Plan

Unit tests: profile selection, supported-config filtering, preflight disk/cache rules, manifest rendering, generated provider naming/cleanup, phase mapping
Integration tests: server model deployment create/watch flows with mocked Kubernetes; CLI deploy/status/watch/remove flows with mocked gateway model APIs; managed inference programming through generated providers
E2E tests: list supported configs on a live cluster; deploy a model and verify inference.local is reachable; verify PVC-backed cache persistence; verify unsupported hardware falls back without breaking managed inference programming

Risks & Open Questions

The repo has no existing model deployment controller, so the first gateway-side reconciler should stay deliberately small and single-model focused.
The current inference API is optimized for one managed cluster inference target; cleanup may need a small explicit unset/clear path.
Spark/Station runtime success depends on GPU visibility inside k3s, which likely requires bootstrap/runtime changes in navigator-bootstrap and related cluster startup code.
Filesystem/cache detection will be environment-specific, so cache reuse should be best-effort and visible in status output.
Exact curated model IDs and vLLM startup flags still need validation during implementation.
Free-space checks for dynamically provisioned PVCs may need conservative approximations.
If a cluster lacks a usable default storage class, deploy should fail fast with a precise storage diagnostic.

Documentation Impact

Update README.md, architecture/cluster-single-node.md, and architecture/inference-routing.md
If cluster bootstrap/runtime behavior changes, update .agents/skills/debug-navigator-cluster/SKILL.md

Revision 4 - adopt platform-first selection, add Spark/Station GPU visibility loop, and note bootstrap/runtime changes for k3s GPU exposure
Revision 3 - switch to gateway-managed model deployments, align with simplified inference API, add catalog selection and preflight checks, and rename CLI surface to nemoclaw model ...
Revision 2 - rename planned CLI surface to nemoclaw inference ... and align it with the existing inference command group
Revision 1 - initial plan

Correction / finalized decision update:

We are going to bake the NVIDIA container toolkit/runtime into the cluster image for now, and keep a single cluster image rather than splitting GPU/non-GPU variants yet.

Rationale:

the current bind-mount approach proves k3s can detect the nvidia runtime, but actual pod startup still fails because the runtime binary depends on userspace loader/libs that are not reliably present inside the cluster container
baking the toolkit into the image gives us a reproducible k3s/containerd runtime environment and avoids depending on the target host having matching NVIDIA userspace packages installed in the right locations
we still rely on the host for the kernel driver, /dev/nvidia*, and Docker GPU passthrough, but we remove host userspace/runtime drift as a blocker

Expected tradeoff:

image grows modestly (roughly a few 10s of MB, likely on the order of ~35-50 MB once packaged), which is acceptable for now
if this becomes a distribution/maintenance problem later, we can revisit a split between default and GPU cluster images

Next step:

update deploy/docker/Dockerfile.cluster so the cluster image includes the NVIDIA container toolkit/runtime
redeploy Spark
verify runtimeClassName: nvidia
install/check the NVIDIA device plugin
confirm nvidia.com/gpu
then move to a tiny vLLM pod on k3s

Research summary for the DGX Spark / k3s-in-Docker GPU prerequisite for this issue:

The topology is viable. The stable path is: DGX OS owns the driver, Docker passes the GPU into the k3s node container, the k3s node image already contains the NVIDIA runtime/toolkit, and Kubernetes adds either the NVIDIA device plugin or GPU Operator on top.
K3s-specific requirement: nvidia-container-runtime must be present when k3s starts so the nvidia RuntimeClass is registered. That only solves container launch; the cluster still needs the device plugin or GPU Operator before the node advertises nvidia.com/gpu.
DGX Spark-specific constraints: ARM64 images only, and unified memory means some older tools and operator versions fail on memory reporting. Recent NVIDIA docs list DGX Spark as a supported ARM platform in the GPU Operator matrix.

Recommendation for NemoClaw:

Keep the current single cluster image approach with the NVIDIA toolkit baked into the image.
Keep Docker GPU passthrough into the cluster container with the --gpus all equivalent.
For our current k3s v1.35.2, use GPU Operator v25.10.1+ if we want the operator-managed stack, and install it with driver.enabled=false and toolkit.enabled=false. That matches the host-driver plus preinstalled-toolkit model.
If we want the smallest V1 surface area, a standalone NVIDIA device plugin is enough to prove the prerequisite for this issue. GPU Operator is better if we expect time-slicing or additional GPU lifecycle features soon.

Public signal from current docs and reports:

Official docs from k3s, k3d, and NVIDIA GPU Operator all line up on this nested-container pattern.
Public success reports exist for local k3s and minikube on DGX Spark, and there are forum reports of recent GPU Operator releases working on Spark.
Public failures mostly involve older operator versions on GB10 unified-memory behavior and an older k3s-specific toolkit container bug that NVIDIA later fixed.

Implication for this issue:

Most of the hard bootstrap work already exists in-repo: Spark and Station GPU passthrough detection and the NVIDIA toolkit in the cluster image are already present.
The remaining prerequisite is cluster manifest or bootstrap support for the NVIDIA device layer plus a smoke test:
1. kubectl get runtimeclass shows nvidia
2. node allocatable includes nvidia.com/gpu
3. a tiny ARM64 GPU test pod starts successfully
4. only then move to a tiny vLLM pod on Spark

Useful sources:

Progress log so far, organized as issue -> solution:

Issue 1: platform detection was too guessy
- Problem: we were leaning on raw GPU / memory heuristics
- Solution: switched to platform-first detection (spark, station, unknown) and use host model strings plus cluster facts as validation
Issue 2: no clean model deployment surface
- Problem: real model deploy flow had no proper CLI / domain structure
- Solution: added nemoclaw model ..., introduced crates/navigator-models, and built the gateway-managed deployment foundation
Issue 3: needed an end-to-end proof before real GPU work
- Problem: we needed to prove the deployment path itself worked
- Solution: built the mock deployment path and verified inference.local end-to-end with a sandbox request
Issue 4: Spark cluster container did not get host GPUs
- Problem: the k3s container was starting without Docker GPU passthrough
- Solution: updated bootstrap to request GPU passthrough on Spark / Station hosts
Issue 5: k3s still did not expose a usable NVIDIA runtime
- Problem: even with /dev/nvidia* in the cluster container, k3s / containerd did not have a working nvidia runtime path
- Solution: first tried bind-mounting host toolkit files, then moved to the better long-term fix: baking the NVIDIA container toolkit / runtime into deploy/docker/Dockerfile.cluster
Issue 6: remote Spark deploys were probing the wrong machine for toolkit files
- Problem: the original bind-mount approach checked local paths instead of the remote Spark host
- Solution: fixed remote probing over SSH; this was later superseded by the baked-image approach
Issue 7: NEMOCLAW_MODEL_PLATFORM was muddying the platform story
- Problem: we had both NEMOCLAW_MODEL_PLATFORM and NEMOCLAW_HOST_MODEL
- Solution: removed NEMOCLAW_MODEL_PLATFORM from runtime detection and kept NEMOCLAW_HOST_MODEL as the canonical bootstrap input
Issue 8: baked NVIDIA runtime existed but could not execute
- Problem: the runtime binary was present but failed because loader / shared libs were missing
- Solution: copied the required glibc loader / runtime libs into the cluster image
Issue 9: nvidia-container-cli still failed inside the cluster
- Problem: more runtime dependencies were missing because the dependency-copy step in the Dockerfile was incomplete
- Solution: fixed the Dockerfile dependency-copy logic and explicitly added common runtime libs like libm.so.6, librt.so.1, libgcc_s.so.1, and libstdc++.so.6
Issue 10: NVIDIA hook failed because ldconfig was missing
- Problem: device-plugin pod startup failed with stat /sbin/ldconfig: no such file or directory
- Solution: baked /sbin/ldconfig and /usr/sbin/ldconfig into the cluster image
Issue 11: needed to prove the NVIDIA runtime path actually worked
- Problem: we did not know whether the cluster image / runtime stack was truly healthy
- Solution: verified nvidia-container-cli info inside the running cluster container; it now sees the Spark GPU correctly (NVIDIA GB10, CUDA 13.0)
Issue 12: device plugin still does not register nvidia.com/gpu
- Problem: after all runtime fixes, k8s-device-plugin now fails with error getting device memory: Not Supported
- Solution: no local fix yet; this now matches the upstream DGX Spark / GB10 issue in NVIDIA/k8s-device-plugin#1482 and appears to be the current real blocker

Current state:

solved: platform detection, model deployment foundation, mock end-to-end routing, GPU passthrough into the cluster container, baked NVIDIA runtime / toolkit in the cluster image, runtime / library / ldconfig issues, and cluster-level GPU visibility via nvidia-container-cli
unresolved: upstream GB10 device-plugin memory discovery failure, so the node still does not advertise nvidia.com/gpu

Proposed next steps:

investigate whether a newer k8s-device-plugin release or config can avoid the GB10 memory-query failure
investigate a pragmatic workaround path for running a tiny vLLM pod on Spark without waiting for nvidia.com/gpu resource registration

Breakthrough update on Spark GPU bring-up:

the baked NVIDIA toolkit/runtime approach is now working end-to-end inside the cluster image
nvidia-container-cli info inside the running cluster container successfully sees the Spark GPU (NVIDIA GB10, CUDA 13.0)
the remaining blocker turned out to be the k8s-device-plugin version, not the cluster runtime wiring

What changed:

v0.17.1 of nvidia/k8s-device-plugin crashed on DGX Spark / GB10 with:
- error getting device memory: Not Supported
upgrading to v0.17.4 fixes this enough for Spark by downgrading that GB10 memory query failure from fatal to warning:
- Ignoring error getting device memory: Not Supported

What we verified:

device plugin pod is healthy on v0.17.4
plugin logs show it successfully registered nvidia.com/gpu
the node now reports nvidia.com/gpu=1
a GPU pod requesting limits.nvidia.com/gpu: 1 and using runtimeClassName: nvidia runs successfully on Spark
pod logs showed:
- GPU 0: NVIDIA GB10 (UUID: GPU-a8e27b8a-5770-ffbc-8cac-827f3a671fab)

So we have now crossed the main infrastructure milestone:

Spark + k3s + baked NVIDIA runtime/toolkit + working Kubernetes GPU resource registration

Next step:

rebuild the full cluster on Spark with the latest image/state
then deploy a tiny real vLLM pod on k3s using vllm/vllm-openai:cu130-nightly and a small model (likely Qwen/Qwen2.5-0.5B-Instruct) to validate real model serving on Spark

Major validation milestone reached on DGX Spark:

We now have a tiny real vLLM model running successfully on Spark inside k3s.

What was validated:

cluster rebuilt with the baked NVIDIA toolkit/runtime image
Spark GPU runtime inside the cluster container is healthy
k8s-device-plugin v0.17.4 works on GB10 when combined with runtimeClassName: nvidia
node advertises nvidia.com/gpu=1
a real vLLM deployment on k3s starts successfully using:
- image: vllm/vllm-openai:cu130-nightly
- model: Qwen/Qwen2.5-0.5B-Instruct
- served model name: qwen2.5-0.5b
/health returns 200
OpenAI-compatible inference works from inside the cluster

Concrete inference validation:

called /v1/chat/completions
prompt requested: Reply with exactly: spark-ok
model response was exactly: spark-ok

Important product note discovered during validation:

if another host-level vLLM process is already consuming GPU memory, the k3s deployment can fail even though the cluster/runtime wiring is correct
so nemoclaw model deploy should include preflight/admission checks for:
- existing GPU consumers
- available GPU memory / effective free capacity
- model fit heuristics
- a clear deployment failure state when the model does not fit because of current GPU usage

This means we have now validated the intended intermediate milestone:

tiny real model
real vLLM
real Spark GPU
real k3s deployment
real OpenAI-compatible inference response

Next step:

wire this deployment shape into the real nemoclaw model deploy flow so Spark/Station curated model configs create and manage the corresponding vLLM workload instead of the current mock-only path

Latest implementation progress:

nemoclaw model deploy now provisions a real curated vLLM backend for Spark configs instead of falling through to BackendNotImplemented
the gateway creates:
- a generated provider record
- a namespaced Kubernetes Deployment
- a namespaced Kubernetes Service
- the managed inference.local route pointing at that generated provider
model status / model list now refresh vLLM deployment state from Kubernetes so a successful backend transitions to Ready
delete now tears the generated Kubernetes backend back down
added namespace RBAC so the gateway service account can manage those Deployment / Service resources

Live validation on Spark:

./scripts/bin/nemoclaw model deploy spark-real --config spark-vllm-qwen25-05b
backend deployed in namespace navigator
nemoclaw model status spark-real now reports Ready
generated provider is wired into cluster inference
second deploy attempts are now rejected early with a clearer single-GPU preflight error while spark-real is active

Current limitation:

the new GPU contention preflight is intentionally simple: one active curated vLLM deployment at a time on the current single-GPU Spark path
future work should replace this with free-memory / fit-aware preflight and richer status reporting

refactor: rename NemoClaw to OpenShell (NVIDIA#17)

Merge pull request NVIDIA#17 from maxamillion/docs-podman-linux-setup

Navigation Menu

Repository navigation

feat(cluster): curated vLLM model deploy for DGX Spark/Station #17

Summary

Target environment

Runtime direction (updated decision)

Phase 1 (V1): CLI-first model deploy

Phase 2: bootstrap plus managed system routes

Auto-detect guardrails

Acceptance criteria

Out of scope (initial)

Implementation Plan

Summary

Scope

Implementation Steps

Detection Strategy

Fast Iteration Loop

Proposed Phase 1 UX

Progress Phases

Test Plan

Risks & Open Questions

Documentation Impact

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

feat(cluster): curated vLLM model deploy for DGX Spark/Station #17

Description

Summary

Target environment

Runtime direction (updated decision)

Phase 1 (V1): CLI-first model deploy

Phase 2: bootstrap plus managed system routes

Auto-detect guardrails

Acceptance criteria

Out of scope (initial)

Activity

pimlock commented on Mar 6, 2026

Implementation Plan

Summary

Scope

Implementation Steps

Detection Strategy

Fast Iteration Loop

Proposed Phase 1 UX

Progress Phases

Test Plan

Risks & Open Questions

Documentation Impact

pimlock commented on Mar 8, 2026

pimlock commented on Mar 8, 2026

pimlock commented on Mar 8, 2026

pimlock commented on Mar 8, 2026

pimlock commented on Mar 9, 2026

pimlock commented on Mar 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions