Repository navigation
bug: k3s gateway fails on Jetson Orin Nano: iptables nf_tables / legacy conflict makes OpenShell unusable #407
Description
Agent Diagnostic
Investigated using Claude as coding agent. No OpenShell skills were available locally (fresh install, no repo clone for agent investigation).
What was tried:
Loaded iptables-legacy on host via update-alternatives --set iptables /usr/sbin/iptables-legacy
Loaded all required kernel modules: ip_tables, iptable_filter, iptable_nat, iptable_mangle, xt_comment, xt_conntrack, xt_mark, br_netfilter
Blacklisted nf_tables kernel module and rebooted — confirmed lsmod | grep nf_tables returns empty and host iptables --version reports v1.8.7 (legacy)
Configured cgroupns=host in Docker daemon via nemoclaw setup-spark
Ran openshell gateway stop/destroy and cleaned all containers between attempts
What was found:
With nf_tables loaded: k3s kube-router panics at network_policy_controller.go:412 — RULE_INSERT failed (No such file or directory) because nf_tables extensions inside the container conflict with host iptables-legacy tables
With nf_tables blacklisted: k3s kube-proxy exits with iptables is not available on this host: error listing chain "POSTROUTING" in table "nat": exit status 4: Warning: iptables-legacy tables present, use iptables-legacy to see them — iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Invalid argument
The container bundles iptables v1.8.10 (nf_tables) internally, which requires the nf_tables kernel module. But when nf_tables is loaded on host, kube-router crashes. This creates an unresolvable conflict on Jetson Orin.
Agent could not resolve this because the iptables binary inside the gateway container (v1.8.10 nf_tables) is incompatible with the Jetson kernel in both states (nf_tables loaded and unloaded). This requires a fix in the gateway container image itself (e.g., bundling iptables-legacy or using --prefer-bundled-bin with legacy mode)
Description
What happened:
openshell gateway start --name nemoclaw (triggered via nemoclaw onboard) fails every time on Jetson Orin Nano. The k3s container exits with either:
nf_tables loaded: panic: network_policy_controller.go:412 — Failed to run iptables command — RULE_INSERT failed (No such file or directory)
nf_tables blacklisted: kube-proxy exited: iptables is not available on this host — iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Invalid argument
The gateway container bundles iptables v1.8.10 (nf_tables) which is incompatible with the Jetson Orin kernel (5.15.x-tegra) in either configuration.
What I expected:
nemoclaw onboard should complete successfully and start the OpenShell gateway on Jetson Orin Nano, as Jetson is an NVIDIA platform and NemoClaw targets NVIDIA hardware.
Reproduction Steps
REPRODUCTION STEPS
Fresh Jetson Orin Nano with JetPack 6.x (Ubuntu 22.04, kernel 5.15.x-tegra)
Install Docker (default JetPack install)
Install OpenShell:
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sudo sh
Install NemoClaw:
git clone https://github.com/NVIDIA/NemoClaw.git
cd NemoClaw
sudo ./install.sh
Run sudo nemoclaw setup-spark (configures cgroupns=host)
Run sudo nemoclaw onboard
Step [2/7] "Starting OpenShell gateway" fails at "Initializing environment"
Attempted workarounds (all failed):
update-alternatives --set iptables /usr/sbin/iptables-legacy
Blacklisting nf_tables + reboot
Loading all legacy iptable_* and xt_* modules manually
Combinations of the above
Environment
Device: NVIDIA Jetson Orin Nano
OS: Ubuntu 22.04 (JetPack 6.x)
Kernel: 5.15.148-tegra (aarch64)
Docker: Docker Engine 28.x with cgroupns=host configured
OpenShell: v0.0.7 (installed via official install.sh)
NemoClaw: installed from source (git clone, main branch, March 2026)
Host iptables: v1.8.7 (legacy) after update-alternatives
Container iptables: v1.8.10 (nf_tables) — bundled inside gateway image ghcr.io/nvidia/openshell/cluster:0.0.8
Node.js: v22.22.1
Logs
Attempt 1: nf_tables loaded (default)
F0317 18:20:46.858772 86 network_policy_controller.go:412] Failed to run iptables
command to insert in INPUT chain running [/usr/sbin/iptables -t filter -I INPUT 1 -m comment
--comment kube-router netpol - 4IA2OSFRMVNDXBVV -j KUBE-ROUTER-INPUT --wait]: exit status 4:
Warning: Extension comment revision 0 not supported, missing kernel module?
iptables v1.8.10 (nf_tables): RULE_INSERT failed (No such file or directory): rule in chain INPUT
panic: network_policy_controller.go:412
Attempt 2: nf_tables blacklisted + iptables-legacy on host
Host verification:
$ lsmod | grep nf_tables
(empty)
$ iptables --version
iptables v1.8.7 (legacy)
Gateway error:
time="2026-03-17T18:35:04Z" level=error msg="Shutdown request received: kube-proxy exited:
iptables is not available on this host : error listing chain \"POSTROUTING\" in table
\"nat\": exit status 4: # Warning: iptables-legacy tables present, use iptables-legacy to see
them\niptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Invalid argument\n"Agent-First Checklist
- I pointed my agent at the repo and had it investigate this issue
- I loaded relevant skills (e.g.,
debug-openshell-cluster,debug-inference,openshell-cli) - My agent could not resolve this — the diagnostic above explains why
Activity
github-actions commentedon Mar 17, 2026
This issue appears to have been opened without an agent investigation.
OpenShell is an agent-first project - please point your coding agent at the repo and have it diagnose this before we triage. Your agent can load skills like
debug-openshell-cluster,debug-inference,openshell-cli, andgenerate-sandbox-policy.See CONTRIBUTING.md for the full workflow.
johntmyers commentedon Mar 17, 2026
Thanks for your submission. This is not an environment we have tested on. I will add this to our list of environments for tracking purposes.
muhshan2 commentedon Mar 21, 2026
I have the same issue
johnnynunez commentedon Mar 22, 2026
Hello,
Here from NVIDIA.
muhshan2 commentedon Mar 22, 2026
Running into the same issue. Looks like the container is still using nf_tables.
Error: × K8s namespace not ready
╰─▶ dial tcp: lookup registry: Try again
DNS resolution is failing inside the gateway container. The cluster cannot pull images or create the 'openshell' namespace until DNS is fixed.
container logs:
I0322 05:42:06.864855 85 pod_startup_latency_tracker.go:108] "Observed pod startup duration" pod="kube-system/local-path-provisioner-6bc6568469-l5w5b" podStartSLOduration=15.721438976
podStartE2EDuration="22.864824344s" podCreationTimestamp="2026-03-22 05:41:44 +0000 UTC" firstStartedPulling="2026-03-22 05:41:58.906350687 +0000 UTC m=+29.548582406"
lastFinishedPulling="2026-03-22 05:42:06.049735959 +0000 UTC m=+36.691967774" observedRunningTime="2026-03-22 05:42:06.86151864 +0000 UTC m=+37.503750487" watchObservedRunningTime="2026-03-22
05:42:06.864824344 +0000 UTC m=+37.507056095"
I0322 05:42:06.865222 85 pod_startup_latency_tracker.go:108] "Observed pod startup duration" pod="agent-sandbox-system/agent-sandbox-controller-0" podStartSLOduration=22.131698794
podStartE2EDuration="23.865210386s" podCreationTimestamp="2026-03-22 05:41:43 +0000 UTC" firstStartedPulling="2026-03-22 05:41:58.780687888 +0000 UTC m=+29.422919639"
lastFinishedPulling="2026-03-22 05:42:00.514199416 +0000 UTC m=+31.156431231" observedRunningTime="2026-03-22 05:42:00.859444821 +0000 UTC m=+31.501676636" watchObservedRunningTime="2026-03-22
05:42:06.865210386 +0000 UTC m=+37.507442137"
I0322 05:42:09.160115 85 pod_startup_latency_tracker.go:108] "Observed pod startup duration" pod="kube-system/coredns-7566b5ff58-4lwlx" podStartSLOduration=17.452586326
podStartE2EDuration="25.160088851s" podCreationTimestamp="2026-03-22 05:41:44 +0000 UTC" firstStartedPulling="2026-03-22 05:42:00.128241961 +0000 UTC m=+30.770473712"
lastFinishedPulling="2026-03-22 05:42:07.835744486 +0000 UTC m=+38.477976237" observedRunningTime="2026-03-22 05:42:09.159812228 +0000 UTC m=+39.802043947" watchObservedRunningTime="2026-03-22
05:42:09.160088851 +0000 UTC m=+39.802320570"
E0322 05:42:09.294514 85 proxier.go:805] "Failed to ensure chain jumps" err=<
error appending rule: exit status 4: Warning: Extension conntrack revision 0 not supported, missing kernel module?
Warning: Extension comment revision 0 not supported, missing kernel module?
iptables v1.8.10 (nf_tables): RULE_INSERT failed (No such file or directory): rule in chain INPUT
> table="filter" srcChain="INPUT" dstChain="KUBE-EXTERNAL-SERVICES"
I0322 05:42:09.294588 85 proxier.go:768] "Sync failed" retryingTime="30s"
E0322 05:42:10.402693 85 proxier.go:805] "Failed to ensure chain jumps" err=<
error appending rule: exit status 4: Warning: Extension conntrack revision 0 not supported, missing kernel module?
Warning: Extension comment revision 0 not supported, missing kernel module?
iptables v1.8.10 (nf_tables): RULE_INSERT failed (No such file or directory): rule in chain INPUT
> table="filter" srcChain="INPUT" dstChain="KUBE-EXTERNAL-SERVICES"
I0322 05:42:10.402815 85 proxier.go:768] "Sync failed" retryingTime="30s"
time="2026-03-22T05:42:10Z" level=info msg="Starting network policy controller version v2.6.3-k3s1, built on 2026-03-04T18:39:36Z, go1.25.7"
I0322 05:42:10.920368 85 network_policy_controller.go:164] Starting network policy controller
F0322 05:42:11.026401 85 network_policy_controller.go:412] Failed to run iptables command to insert in INPUT chain running [/usr/sbin/iptables -t filter -I INPUT 1 -m comment --comment
kube-router netpol - 4IA2OSFRMVNDXBVV -j KUBE-ROUTER-INPUT --wait]: exit status 4: Warning: Extension comment revision 0 not supported, missing kernel module?
iptables v1.8.10 (nf_tables): RULE_INSERT failed (No such file or directory): rule in chain INPUT
panic: F0322 05:42:11.026401 85 network_policy_controller.go:412] Failed to run iptables command to insert in INPUT chain running [/usr/sbin/iptables -t filter -I INPUT 1 -m comment
--comment kube-router netpol - 4IA2OSFRMVNDXBVV -j KUBE-ROUTER-INPUT --wait]: exit status 4: Warning: Extension comment revision 0 not supported, missing kernel module?
iptables v1.8.10 (nf_tables): RULE_INSERT failed (No such file or directory): rule in chain INPUT
goroutine 30174 [running]:
k8s.io/klog/v2.(*loggingT).output(0xbb287a0, 0x3, 0x4000eed8a0, 0x4004b9b5e0, 0x1, {0x9293e8d?, 0x2?}, 0x4?, 0x0)
/go/pkg/mod/github.com/k3s-io/klog/v2@v2.120.1-k3s1/klog.go:965 +0x6b0
k8s.io/klog/v2.(*loggingT).printfDepth(0xbb287a0, 0x3, 0x4000eed8a0, {0x0, 0x0}, 0x1, {0x681ea2a, 0x37}, {0x400848d9c0, 0x2, ...})
/go/pkg/mod/github.com/k3s-io/klog/v2@v2.120.1-k3s1/klog.go:767 +0x198
k8s.io/klog/v2.(*loggingT).printf(...)
/go/pkg/mod/github.com/k3s-io/klog/v2@v2.120.1-k3s1/klog.go:744
k8s.io/klog/v2.Fatalf(...)
/go/pkg/mod/github.com/k3s-io/klog/v2@v2.120.1-k3s1/klog.go:1655
github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).ensureTopLevelChains.func2({0x78916d8, 0x400bb50140}, {0x6634a6b, 0x5}, {0x400a147e60, 0x6, 0x6},
{0x4003156600, 0x10}, 0x1)
/go/pkg/mod/github.com/k3s-io/kube-router/v2@v2.6.3-k3s1/pkg/controllers/netpol/network_policy_controller.go:412 +0x498
github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).ensureTopLevelChains(0x40058ca5a0)
/go/pkg/mod/github.com/k3s-io/kube-router/v2@v2.6.3-k3s1/pkg/controllers/netpol/network_policy_controller.go:472 +0x154c
github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).Run(0x40058ca5a0, 0x400a4961c0, 0x4000e02ee0, 0x400c7ef170)
/go/pkg/mod/github.com/k3s-io/kube-router/v2@v2.6.3-k3s1/pkg/controllers/netpol/network_policy_controller.go:168 +0x100
created by github.com/k3s-io/k3s/pkg/agent/netpol.Run in goroutine 497
/go/src/github.com/k3s-io/k3s/pkg/agent/netpol/netpol.go:191 +0xd40
johnnynunez commentedon Mar 22, 2026
you have to pull docker, fix it, commit with the same name and then install openshell again
johnnynunez commentedon Mar 22, 2026
https://www.linkedin.com/posts/johnnycano_nemoclaw-openshell-jetson-agx-orin-and-activity-7441364777718661120-HbwP