Repository navigation
Description
Bug Description
This is a follow-up to #199, which #269 closed in v4.1.0 by adding nva_spot_vms (default false, variables.tf:223) to 2-networking-a-fedramp, so a new deployment now gets on-demand NVAs. (The closing comment on #199 names the variable enable_spot_vms; it is nva_spot_vms.) Three differences between this stage's NVA tier and the IL5 stage's appliance tier remain on main, and the first means the fix never reaches a deployment that already exists.
1. A deployment built before v4.1.0 keeps its Spot NVAs after it upgrades. Its appliances run with provisioningModel: SPOT and automaticRestart: false. Applying v4.1.0 or later replaces the instance template with a STANDARD one and points nva-mig at it, but nva-mig (nva.tf:116-133) sets no update_policy, so the group runs on the API's default. In the FedRAMP High organizations we have built from this stage, one of them deployed from v4.0.0 on 2026-11-09, the group reads:
updatePolicy:
maxSurge: {calculated: 3, fixed: 3}
maxUnavailable: {calculated: 3, fixed: 3}
minimalAction: REPLACE
replacementMethod: SUBSTITUTE
type: OPPORTUNISTIC
The Compute Engine documentation is explicit about that type: "Compute Engine does not actively initiate requests to apply opportunistic updates on existing instances." The running appliances stay on Spot until something recreates them. Terraform tracks the group, not its instances, so no later plan shows it, and the only way to apply the fix is a manual gcloud compute instance-groups managed rolling-action replace. Every later change to the template has the same gap.
We have seen it in our own organization. On 2026-07-30 a Stage 2 apply created a new NVA template at 15:34:08 UTC and patched the group to it at 15:34:21. No appliance was recreated until we set a PROACTIVE update policy on the group by hand at 15:39:13, and the group began replacing both appliances seconds later.
The IL5 stage does not have this gap. ngfw-mig (ngfw.tf:315-325) rolls a new template out proactively without dropping below its two appliances:
update_policy = {
minimal_action = "REPLACE"
type = "PROACTIVE"
min_ready_sec = 300
max_surge = {
fixed = length(local.nva_zones[each.key])
}
max_unavailable = {
fixed = 0
}
}
2. On-demand NVAs are still stopped for every host maintenance event; the IL5 appliances live-migrate. nva-template turns on Confidential VM (confidential_compute = "SEV", nva.tf:107) and sets no on_host_maintenance (nva.tf:91-94), and modules/compute-vm/main.tf:41-48 resolves any Confidential VM to TERMINATE. So with nva_spot_vms = false each appliance is still stopped and restarted for host maintenance, while the IL5 template (ngfw.tf:230-233) sets on_host_maintenance = "MIGRATE".
Compute Engine supports live migration for this configuration. The Confidential VM supported configurations page says "Live migration is supported on N2D and C3D machine types running AMD SEV," and the creation guide says "For N2D and C3D machine types that use SEV, set this to MIGRATE for live migration support." The compute-vm module synced in #311 now takes the setting from the caller (coalesce(var.scheduling_config.on_host_maintenance, ...)), which the previous module's options did not, so the stage can pass on_host_maintenance = var.nva_spot_vms ? "TERMINATE" : "MIGRATE".
We tested it. An n2d-standard-2 Confidential VM with the template's machine type, Confidential VM type (SEV), Shielded VM settings, and image (cos-stable), created with onHostMaintenance: MIGRATE, was accepted and placed on AMD Milan. gcloud compute instances simulate-maintenance-event then live-migrated it: a compute.instances.migrateOnHostMaintenance operation, lastStartTimestamp unchanged, and the instance RUNNING throughout. The host events page words the exception as "N2D machine types with AMD EPYC Milan CPU platforms running AMD SEV." Every appliance we have read landed on AMD Milan without the template asking for it, and min_cpu_platform = "AMD Milan" would guarantee it.
3. The choice is not documented where a deployment's values are chosen. nva_spot_vms appears only in the stage README's generated variables table. docs/ddg.md does not list it, and neither stage README states what Spot costs in availability or that the IL5 stage offers no Spot option. #199 asked for that trade-off to be stated so an operator can choose knowingly.
Environment and Deployment Context
- Stellar Engine Version/Commit:
main@162c7feb(after feat(modules)!: sync Cloud Foundation Fabric modules to v60.0.0 and add sync tooling #311). Items 1 and 3 are the same atv4.1.0andv5.2.0(82153050). Item 2 is the same there too, except that the module before feat(modules)!: sync Cloud Foundation Fabric modules to v60.0.0 and add sync tooling #311 offered no way to overrideTERMINATE. - Deployment Type:
- US Region Restricted (e.g., Access Policy constraint)
- FedRAMP Medium
- FedRAMP High
- FedRAMP Moderate
- DoD IL4
- DoD IL5
- Stand-alone / Custom
- FAST Stage (if applicable): Stage 2 (Networking), the
a-fedrampoption- Stage 0 (Bootstrap)
- Stage 1 (Resource Management)
- Stage 2 (Networking)
- Stage 3 (Security)
Steps to Reproduce
- In an organization whose Stage 2 was applied before v4.1.0, read the appliances' scheduling:
gcloud compute instances list --project <prefix>-net-vdss-host --filter='name~nva' --format='table(name,status,scheduling.provisioningModel,scheduling.automaticRestart,scheduling.onHostMaintenance)'
They reportSPOT/False/TERMINATE. - Read the group's update policy:
gcloud compute instance-groups managed describe nva-primary --region <region> --project <prefix>-net-vdss-host --format='yaml(updatePolicy)'
It reportstype: OPPORTUNISTIC. - Apply the stage from v4.1.0 or later, then repeat step 1. The group's template is now
STANDARD, and the running appliances still reportSPOT/False. - For item 2, run step 1 against a deployment built from
main:STANDARD/True/TERMINATE. A2-networking-b-il5-ngfwdeployment reportsMIGRATE.
Expected Behavior
nva-migcarries an update policy likengfw-mig's, so an apply that changes the template replaces the appliances one surge at a time with no loss of capacity. If an automatic roll on apply is unwanted, the v4.1.0 upgrade notes tell existing deployments to run the rolling replace.- On-demand appliances live-migrate for host maintenance, as the IL5 appliances do.
docs/ddg.mdlistsnva_spot_vms, and the stage README states the availability trade-off.
Actual Behavior
The #269 default reaches new deployments only, and nothing in a plan or in the changelog entry for #269 tells an existing deployment that its egress appliances are still on Spot. On-demand appliances are still stopped for every host maintenance event. The choice appears only in a generated variables table.
Relevant Logs and Errors
None.
Additional Context
We have not applied v4.1.0 or later to a Spot deployment (step 3). Item 1 rests on the code at 162c7feb, on the update policy of the groups this stage built in our organizations, and on the 2026-07-30 sequence above. Item 2 rests on the code, on the live templates and appliances (SPOT, automaticRestart: false, TERMINATE, Confidential VM on), and on the live-migration test above.
Draft #404 would replace this stage's NVA tier with a Network Connectivity Center hub. Until a release ships it, these items apply to every FedRAMP deployment. If the NVA layout stays available beside the hub, they apply to that option too.
Activity
aghassemlouei commentedon Nov 17, 2026
Closing as resolved on
main(v6.0.0, #394). #404 (d87fdc39) replaced thesimple-nvaappliance tier infast/stages-aw/2-networking-a-fedrampwith aSTARNetwork Connectivity Center hub and Cloud NAT or Secure Web Proxy next-hop egress, removingnva.tf,nva-mig,nva-template, andvar.nva_spot_vms.