[DGX Spark] gateway never rewrites openshell:resolve:env: credential placeholder on sandbox egress — literal placeholder sent on the wire (works on DGX Station, same versions) #2161

Agent Diagnostic

Investigated from the NemoClaw side (downstream consumer of the provider/placeholder contract), full triage in NVIDIA/NemoClaw#6379:

• Walked the NemoClaw MCP bridge path: mcp add --env <NAME> creates a generic OpenShell provider holding the real secret (openshell provider create --name <n> --type generic --credential <NAME>), attaches it to the sandbox, and writes only the placeholder header Authorization: Bearer openshell:resolve:env:<NAME> into the agent adapter config. The real token never enters the sandbox by design; substitution on egress is owned by the OpenShell gateway (L7 rewrite).
• Verified provider state on the affected host via openshell-backed inspection: provider exists, credential key matches, provider attached to the sandbox, resource version stable — all metadata healthy.
• Isolation probes from inside the sandbox (container ID pinned across all steps):
  • unauthenticated GET https://api.githubcopilot.com/mcp/ → 401 (egress + route fine)
  • MCP initialize with the literal placeholder as bearer → 400 (what the wire actually carries)
  • MCP initialize with the real token as bearer → 200, full handshake (from sandbox and from host)
• Conclusion: the L7 placeholder-token rewrite never fires on this host; the literal openshell:resolve:env:... string is sent to the remote. The identical NemoClaw/OpenShell versions on a DGX Station resolve the placeholder correctly end-to-end, so this is host-/build-specific, not the downstream bridge code.

Description

On a DGX Spark host (aarch64, OpenShell CLI 0.0.72), provider credential placeholders (openshell:resolve:env:<NAME>) in sandbox egress HTTP headers are never substituted with the stored credential — the placeholder string goes out on the wire literally, so the upstream service rejects every request (400/401/403).

This breaks all placeholder-based auth from sandboxes on the affected host. Reproduced on two different agent stacks over two different egress shapes:

• openclaw sandbox, mcporter proxy path → proxy CONNECT 403
• Deep Agents sandbox, direct egress → upstream 403, MCP server skipped at pre-flight

Expected: the gateway rewrites the placeholder to the real secret on egress (as it does on a DGX Station running the same versions).

Additionally, all provider health metadata (provider get/inspection: exists, attached, credential key present) stays green while the rewrite is dead — there appears to be no diagnostic that exercises the rewrite path itself.

Reproduction Steps

1. On the affected DGX Spark host, create a sandbox with any agent.
2. Create a generic provider from a host env var and attach it to the sandbox:

   export GITHUB_PERSONAL_ACCESS_TOKEN=<real PAT>
   openshell provider create --name ghmcp-cred --type generic --credential GITHUB_PERSONAL_ACCESS_TOKEN
   # attach to sandbox

3. From inside the sandbox, send an HTTPS request carrying Authorization: Bearer openshell:resolve:env:GITHUB_PERSONAL_ACCESS_TOKEN to an authenticated endpoint (e.g. MCP initialize against https://api.githubcopilot.com/mcp/).
4. Observe the upstream reject with 400/403 — capture shows the literal placeholder string as the bearer value; the rewrite never happened.
5. Same request with the real token pasted in returns 200 from the same sandbox — egress, DNS, and auth are all fine.

Environment

• Device: DGX Spark — fails; DGX Station with identical stack — works
• OS: Ubuntu 24.04, aarch64
• Docker: 27.x
• OpenShell CLI: 0.0.72
• Downstream: NemoClaw v0.0.74 (managed MCP bridge), OpenClaw 2026.5.27 / Deep Agents dcode v0.1.30

Logs

# Deep Agents direct egress path
MCP server 'ghmcp' skipped: pre-flight failed: <...> 403 Forbidden
(no "Loaded N MCP tools" line; zero MCP tools in session)

# openclaw / mcporter proxy path
proxy CONNECT -> 403

# Isolation (inside sandbox, container ID stable):
GET https://api.githubcopilot.com/mcp/  (unauthenticated)      -> 401
MCP initialize, bearer = literal "openshell:resolve:env:..."   -> 400
MCP initialize, bearer = real token                            -> 200 full handshake

# Contrast — DGX Station, same versions, same provider setup:
Loaded 44 MCP tools ; ghmcp_get_me RC=0   (placeholder resolved on egress)

Cross-links:

• Fix PR (this repo): fix(network): fail closed when credential placeholders cannot be rewritten #2162 — fail closed when credential placeholders cannot be rewritten
• Downstream bug report: [DGX Spark][Sandbox] managed MCP unusable at runtime — OpenShell never resolves the openshell:resolve:env: credential placeholder (sends it literally); mcp status stays all-green NemoClaw#6379
• Downstream diagnostic PR: fix(mcp): probe wire-level credential resolution in mcp status and add (#6379) NemoClaw#6382 — wire-level credential-resolution probe in mcp status / mcp add (approved)

Original cross-refs: NVIDIA/NemoClaw#6379 (downstream report; a NemoClaw-side PR is planned to add a wire-level resolution probe to its mcp status so this failure mode is no longer silent there). Possibly related: #913 (L7 placeholder rewrite gaps for WSS payloads — different transport, same rewrite layer).