Skip to content

[Bug] 2-networking-a-fedramp: the NVA return routes still omit the proxy-only subnets, so traffic from a regional managed load balancer's proxies cannot get replies from the internet #396

Closed

Description

@JohnHales

Bug Description

PR #269 (closing #146) builds the NVAs' trusted-side return routes from every tenant subnet of every environment. The proxy-only subnet that branch-net-envs.tf creates in each environment spoke (subnets_proxy_only, CIDR from var.proxy_subnets[<env>], name proxy-<regions.primary>) is still not among them: the landing entry of local.routing_config in nva.tf reads var.subnets only (if s.tenant != null), and nva.tf never references var.proxy_subnets.

The spokes have their default routes deleted (delete_default_routes_on_create = true) and reach the internet through the default-route-nva route (0.0.0.0/0 → the NVA internal load balancer) and Cloud NAT on the DMZ VPC. A regional managed load balancer in a spoke — for example a regional internal Application Load Balancer whose backend is a regional internet NEG, which is how blueprints/gemini-enterprise reaches the Google-hosted Gemini Enterprise endpoint (gemini-stage-0/network.tf: an INTERNET_FQDN_PORT NEG for vertexaisearch.cloud.google.com) — originates its backend connections from the Envoy proxies in that spoke's proxy-only subnet. Those packets leave through the NVA and are masqueraded on the DMZ interface. The reply comes back to the NVA, is un-NATed to the proxy-only address, and — because the guest has no route for that range on the landing interface (simple-nva's cloud-config adds ip route add <cidr> via <eth1 gateway> only for the listed routes) — follows the VM's default route back out the untrusted interface. The connection hangs and times out; nothing errors at apply time, and DNS keeps resolving, so it reads as a firewall or DNS problem.

Environment and Deployment Context

  • Stellar Engine Version/Commit: v5.2.0 (82153050); unchanged on main @ e2396049 (2026-11-01). fast/stages-aw/2-networking-a-fedramp/nva.tf lines 28-37 (the landing routes), branch-net-envs.tf lines 114-119 (subnets_proxy_only; delete_default_routes_on_create at line 107), variables.tf lines 257-268 (proxy_subnets, one entry required per envs_folders key), modules/cloud-config-container/simple-nva/cloud-config.yaml lines 57-59 (guest routes only for the listed CIDRs).
  • Deployment Type:
    • US Region Restricted (e.g., Access Policy constraint)
    • FedRAMP Moderate
    • FedRAMP High
    • DoD IL4
    • DoD IL5
    • Stand-alone / Custom
  • FAST Stage (if applicable): 2-networking-a-fedramp

Steps to Reproduce

  1. Deploy Stages 0-2 with 2-networking-a-fedramp at v5.2.0, with proxy_subnets set for each environment (required by the variable's validation).
  2. In a spoke, deploy a regional managed load balancer whose backend is a regional internet NEG (for example gemini-stage-0 and gemini-stage-1 with the internal deployment type).
  3. Send a request to the load balancer from a client in the spoke.
  4. On an NVA: ip route get <an address in the proxy-only CIDR> shows the route via eth0 (the DMZ side), not eth1.

Expected Behavior

Every subnet whose egress is routed through the NVAs — tenant subnets and proxy-only subnets — gets a trusted-side return route, so replies to the load balancer's proxies are delivered.

Actual Behavior

Replies to the proxy-only range leave the NVA on the untrusted side; the load balancer's backend connections time out.

Relevant Logs and Errors

None at deploy time: the plan and apply succeed. The symptom is the request hanging at the load balancer.

Additional Context

Activity

  1. added theissue type on Nov 3, 2026
  2. added
    BugSomething isn't working
    Priority - MediumStandard features and non-blocking bugs; important for the current milestone but not urgent
    Level of Effort - LowQuick, well-defined tasks with no unknowns; takes a few hours up to one day to complete
    on Nov 3, 2026
  3. added
    Scope - BlueprintRelated to solution blueprints in the blueprints directory
    Scope - FastRelated to Fabric FAST deployment stages in the fast directory
    Scope - ModuleRelated to reusable Terraform modules in the modules directory
    on Nov 3, 2026
  4. added
    Priority - HighCritical issues blocking development or users; urgent bugs or core features for release
    and removed
    Priority - MediumStandard features and non-blocking bugs; important for the current milestone but not urgent
    on Nov 5, 2026
  5. aghassemlouei commented on Nov 17, 2026

    @aghassemlouei
    Member

    Closing as resolved on main (v6.0.0, #394). #404 (d87fdc39) removed fast/stages-aw/2-networking-a-fedramp/nva.tf and the simple-nva routing appliances in favor of a STAR Network Connectivity Center hub (net-hub.tf, factory-ncc.tf) paired with per-spoke Cloud NAT (var.enable_cloud_nat) or centralized Secure Web Proxy next-hop egress (egress.tf), so spoke proxy-only subnets no longer rely on static guest return routes on simple-nva instances.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugSomething isn't workingFramework - FedRAMP HighFedRAMP High compliance regime and controlsFramework - FedRAMP ModerateFedRAMP Moderate compliance regime and controlsGemini - EnterpriseGemini Enterprise (GE) relatedLevel of Effort - LowQuick, well-defined tasks with no unknowns; takes a few hours up to one day to completePriority - HighCritical issues blocking development or users; urgent bugs or core features for releaseScope - BlueprintRelated to solution blueprints in the blueprints directoryScope - FastRelated to Fabric FAST deployment stages in the fast directoryScope - ModuleRelated to reusable Terraform modules in the modules directory

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions