Repository navigation
Description
Bug Description
PR #269 (closing #146) builds the NVAs' trusted-side return routes from every tenant subnet of every environment. The proxy-only subnet that branch-net-envs.tf creates in each environment spoke (subnets_proxy_only, CIDR from var.proxy_subnets[<env>], name proxy-<regions.primary>) is still not among them: the landing entry of local.routing_config in nva.tf reads var.subnets only (if s.tenant != null), and nva.tf never references var.proxy_subnets.
The spokes have their default routes deleted (delete_default_routes_on_create = true) and reach the internet through the default-route-nva route (0.0.0.0/0 → the NVA internal load balancer) and Cloud NAT on the DMZ VPC. A regional managed load balancer in a spoke — for example a regional internal Application Load Balancer whose backend is a regional internet NEG, which is how blueprints/gemini-enterprise reaches the Google-hosted Gemini Enterprise endpoint (gemini-stage-0/network.tf: an INTERNET_FQDN_PORT NEG for vertexaisearch.cloud.google.com) — originates its backend connections from the Envoy proxies in that spoke's proxy-only subnet. Those packets leave through the NVA and are masqueraded on the DMZ interface. The reply comes back to the NVA, is un-NATed to the proxy-only address, and — because the guest has no route for that range on the landing interface (simple-nva's cloud-config adds ip route add <cidr> via <eth1 gateway> only for the listed routes) — follows the VM's default route back out the untrusted interface. The connection hangs and times out; nothing errors at apply time, and DNS keeps resolving, so it reads as a firewall or DNS problem.
Environment and Deployment Context
- Stellar Engine Version/Commit:
v5.2.0(82153050); unchanged onmain@e2396049(2026-11-01).fast/stages-aw/2-networking-a-fedramp/nva.tflines 28-37 (thelandingroutes),branch-net-envs.tflines 114-119 (subnets_proxy_only;delete_default_routes_on_createat line 107),variables.tflines 257-268 (proxy_subnets, one entry required perenvs_folderskey),modules/cloud-config-container/simple-nva/cloud-config.yamllines 57-59 (guest routes only for the listed CIDRs). - Deployment Type:
- US Region Restricted (e.g., Access Policy constraint)
- FedRAMP Moderate
- FedRAMP High
- DoD IL4
- DoD IL5
- Stand-alone / Custom
- FAST Stage (if applicable):
2-networking-a-fedramp
Steps to Reproduce
- Deploy Stages 0-2 with
2-networking-a-fedrampatv5.2.0, withproxy_subnetsset for each environment (required by the variable's validation). - In a spoke, deploy a regional managed load balancer whose backend is a regional internet NEG (for example
gemini-stage-0andgemini-stage-1with the internal deployment type). - Send a request to the load balancer from a client in the spoke.
- On an NVA:
ip route get <an address in the proxy-only CIDR>shows the route viaeth0(the DMZ side), noteth1.
Expected Behavior
Every subnet whose egress is routed through the NVAs — tenant subnets and proxy-only subnets — gets a trusted-side return route, so replies to the load balancer's proxies are delivered.
Actual Behavior
Replies to the proxy-only range leave the NVA on the untrusted side; the load balancer's backend connections time out.
Relevant Logs and Errors
None at deploy time: the plan and apply succeed. The symptom is the request hanging at the load balancer.
Additional Context
- Suggested fix — add each environment's proxy-only CIDR to the
landingroutes:(routes = flatten([ for k, v in var.envs_folders : concat( [for s in try(var.subnets[lower(k)], []) : module.env-spoke-vpc[k].subnets["${s.region}/${s.name}"].ip_cidr_range if s.tenant != null], [var.proxy_subnets[k]] ) ])
var.proxy_subnetsis keyed by theenvs_folderskey itself, not its lower-cased form, and its validation already guarantees an entry for every environment.) - [Bug] 2-networking-a-fedramp-high: NVA return-route generation only routes the FIRST tenant subnet per environment ([0] index) — additional tenant subnets get no NVA return route and lose ALL internet egress #146 reported the tenant-subnet half; it did not mention the proxy-only subnets, which is why fix(2-networking): fix NVA routing, Spot default, firewall policy prefix, and peering stagger #269 did not cover them.
Activity
aghassemlouei commentedon Nov 17, 2026
Closing as resolved on
main(v6.0.0, #394). #404 (d87fdc39) removedfast/stages-aw/2-networking-a-fedramp/nva.tfand thesimple-nvarouting appliances in favor of aSTARNetwork Connectivity Center hub (net-hub.tf,factory-ncc.tf) paired with per-spoke Cloud NAT (var.enable_cloud_nat) or centralized Secure Web Proxy next-hop egress (egress.tf), so spoke proxy-only subnets no longer rely on static guest return routes onsimple-nvainstances.