Skip to content

[Bug] 2-networking-a-fedramp: after #269, upgraded deployments keep their Spot NVAs (the group has no update policy), and on-demand NVAs still stop for host maintenance where the IL5 appliances live-migrate #409

Closed

Description

@JohnHales

Bug Description

This is a follow-up to #199, which #269 closed in v4.1.0 by adding nva_spot_vms (default false, variables.tf:223) to 2-networking-a-fedramp, so a new deployment now gets on-demand NVAs. (The closing comment on #199 names the variable enable_spot_vms; it is nva_spot_vms.) Three differences between this stage's NVA tier and the IL5 stage's appliance tier remain on main, and the first means the fix never reaches a deployment that already exists.

1. A deployment built before v4.1.0 keeps its Spot NVAs after it upgrades. Its appliances run with provisioningModel: SPOT and automaticRestart: false. Applying v4.1.0 or later replaces the instance template with a STANDARD one and points nva-mig at it, but nva-mig (nva.tf:116-133) sets no update_policy, so the group runs on the API's default. In the FedRAMP High organizations we have built from this stage, one of them deployed from v4.0.0 on 2026-11-09, the group reads:

updatePolicy:
  maxSurge: {calculated: 3, fixed: 3}
  maxUnavailable: {calculated: 3, fixed: 3}
  minimalAction: REPLACE
  replacementMethod: SUBSTITUTE
  type: OPPORTUNISTIC

The Compute Engine documentation is explicit about that type: "Compute Engine does not actively initiate requests to apply opportunistic updates on existing instances." The running appliances stay on Spot until something recreates them. Terraform tracks the group, not its instances, so no later plan shows it, and the only way to apply the fix is a manual gcloud compute instance-groups managed rolling-action replace. Every later change to the template has the same gap.

We have seen it in our own organization. On 2026-07-30 a Stage 2 apply created a new NVA template at 15:34:08 UTC and patched the group to it at 15:34:21. No appliance was recreated until we set a PROACTIVE update policy on the group by hand at 15:39:13, and the group began replacing both appliances seconds later.

The IL5 stage does not have this gap. ngfw-mig (ngfw.tf:315-325) rolls a new template out proactively without dropping below its two appliances:

update_policy = {
  minimal_action = "REPLACE"
  type           = "PROACTIVE"
  min_ready_sec  = 300
  max_surge = {
    fixed = length(local.nva_zones[each.key])
  }
  max_unavailable = {
    fixed = 0
  }
}

2. On-demand NVAs are still stopped for every host maintenance event; the IL5 appliances live-migrate. nva-template turns on Confidential VM (confidential_compute = "SEV", nva.tf:107) and sets no on_host_maintenance (nva.tf:91-94), and modules/compute-vm/main.tf:41-48 resolves any Confidential VM to TERMINATE. So with nva_spot_vms = false each appliance is still stopped and restarted for host maintenance, while the IL5 template (ngfw.tf:230-233) sets on_host_maintenance = "MIGRATE".

Compute Engine supports live migration for this configuration. The Confidential VM supported configurations page says "Live migration is supported on N2D and C3D machine types running AMD SEV," and the creation guide says "For N2D and C3D machine types that use SEV, set this to MIGRATE for live migration support." The compute-vm module synced in #311 now takes the setting from the caller (coalesce(var.scheduling_config.on_host_maintenance, ...)), which the previous module's options did not, so the stage can pass on_host_maintenance = var.nva_spot_vms ? "TERMINATE" : "MIGRATE".

We tested it. An n2d-standard-2 Confidential VM with the template's machine type, Confidential VM type (SEV), Shielded VM settings, and image (cos-stable), created with onHostMaintenance: MIGRATE, was accepted and placed on AMD Milan. gcloud compute instances simulate-maintenance-event then live-migrated it: a compute.instances.migrateOnHostMaintenance operation, lastStartTimestamp unchanged, and the instance RUNNING throughout. The host events page words the exception as "N2D machine types with AMD EPYC Milan CPU platforms running AMD SEV." Every appliance we have read landed on AMD Milan without the template asking for it, and min_cpu_platform = "AMD Milan" would guarantee it.

3. The choice is not documented where a deployment's values are chosen. nva_spot_vms appears only in the stage README's generated variables table. docs/ddg.md does not list it, and neither stage README states what Spot costs in availability or that the IL5 stage offers no Spot option. #199 asked for that trade-off to be stated so an operator can choose knowingly.

Environment and Deployment Context

Steps to Reproduce

  1. In an organization whose Stage 2 was applied before v4.1.0, read the appliances' scheduling:
    gcloud compute instances list --project <prefix>-net-vdss-host --filter='name~nva' --format='table(name,status,scheduling.provisioningModel,scheduling.automaticRestart,scheduling.onHostMaintenance)'
    They report SPOT / False / TERMINATE.
  2. Read the group's update policy:
    gcloud compute instance-groups managed describe nva-primary --region <region> --project <prefix>-net-vdss-host --format='yaml(updatePolicy)'
    It reports type: OPPORTUNISTIC.
  3. Apply the stage from v4.1.0 or later, then repeat step 1. The group's template is now STANDARD, and the running appliances still report SPOT / False.
  4. For item 2, run step 1 against a deployment built from main: STANDARD / True / TERMINATE. A 2-networking-b-il5-ngfw deployment reports MIGRATE.

Expected Behavior

  • nva-mig carries an update policy like ngfw-mig's, so an apply that changes the template replaces the appliances one surge at a time with no loss of capacity. If an automatic roll on apply is unwanted, the v4.1.0 upgrade notes tell existing deployments to run the rolling replace.
  • On-demand appliances live-migrate for host maintenance, as the IL5 appliances do.
  • docs/ddg.md lists nva_spot_vms, and the stage README states the availability trade-off.

Actual Behavior

The #269 default reaches new deployments only, and nothing in a plan or in the changelog entry for #269 tells an existing deployment that its egress appliances are still on Spot. On-demand appliances are still stopped for every host maintenance event. The choice appears only in a generated variables table.

Relevant Logs and Errors

None.

Additional Context

We have not applied v4.1.0 or later to a Spot deployment (step 3). Item 1 rests on the code at 162c7feb, on the update policy of the groups this stage built in our organizations, and on the 2026-07-30 sequence above. Item 2 rests on the code, on the live templates and appliances (SPOT, automaticRestart: false, TERMINATE, Confidential VM on), and on the live-migration test above.

Draft #404 would replace this stage's NVA tier with a Network Connectivity Center hub. Until a release ships it, these items apply to every FedRAMP deployment. If the NVA layout stays available beside the hub, they apply to that option too.

Activity

  1. added theissue type on Nov 9, 2026
  2. added
    BugSomething isn't working
    Priority - MediumStandard features and non-blocking bugs; important for the current milestone but not urgent
    Level of Effort - LowQuick, well-defined tasks with no unknowns; takes a few hours up to one day to complete
    Scope - FastRelated to Fabric FAST deployment stages in the fast directory
    on Nov 9, 2026
  3. added
    Framework - IL5DoD Impact Level 5 (IL5) compliance regime and controls
    Scope - ModuleRelated to reusable Terraform modules in the modules directory
    on Nov 9, 2026
  4. aghassemlouei commented on Nov 17, 2026

    @aghassemlouei
    Member

    Closing as resolved on main (v6.0.0, #394). #404 (d87fdc39) replaced the simple-nva appliance tier in fast/stages-aw/2-networking-a-fedramp with a STAR Network Connectivity Center hub and Cloud NAT or Secure Web Proxy next-hop egress, removing nva.tf, nva-mig, nva-template, and var.nva_spot_vms.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugSomething isn't workingFramework - FedRAMP HighFedRAMP High compliance regime and controlsFramework - FedRAMP ModerateFedRAMP Moderate compliance regime and controlsFramework - IL5DoD Impact Level 5 (IL5) compliance regime and controlsLevel of Effort - LowQuick, well-defined tasks with no unknowns; takes a few hours up to one day to completePriority - MediumStandard features and non-blocking bugs; important for the current milestone but not urgentScope - FastRelated to Fabric FAST deployment stages in the fast directoryScope - ModuleRelated to reusable Terraform modules in the modules directory

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions