Skip to content

feat(cli): guided deploy - #386

Draft
digitalresistor wants to merge 2 commits into
google:mainfrom
digitalresistor:feat/cli-guided-deploy
Draft

digitalresistor wants to merge 2 commits into
google:mainfrom
digitalresistor:feat/cli-guided-deploy

Conversation

@digitalresistor

@digitalresistor digitalresistor commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

The stellar deploy command turns the DDG prerequisites, Stage 0, and Stage 1 into reviewed, resumable steps, so operators stop running Terraform, the state migration, and the helper scripts by hand.

Part of #306. This pull request closes no issue on its own.

The branch has two commits on main, and each passes the full gate on its own. The first adds the Terraform runner, the plan guards, and the organization IAM diff, and the second adds the deploy engine, the step catalog, the commands, and the docs.

  • The stellar guide command explains each of the 24 deploy steps and the manual commands that do the same, without reading Google Cloud.
  • The stellar status command runs the check of every step, writes nothing, and exits with status 4 while a step is not done.
  • The stellar deploy command skips the steps that are done and resumes at the first open step.
  • The --stage, --only, and --from-step flags select the steps of one run.
  • The --yes flag answers only the routine questions, and manual steps, plan findings, and the organization IAM review always ask.
  • Every apply plans to a file under the plans/ folder of the environment and applies only that saved plan.
  • The plan guards in the cli/src/stellar_engine/terraform/guards.py module refuse a bootstrap-phase plan after a final apply, an import that replaces its object, and the loss of a protected Stage 0 resource.
  • A plan guard asks before an apply reverts organization policies that changed outside Terraform.
  • The organization IAM review saves a snapshot of the live policy, lists the grants that the apply removes, and needs the organization ID typed before the apply.
  • The IAM review covers the authoritative google_organization_iam_binding resources and the additive google_organization_iam_member resources of Stage 0, and compares members without case, as the provider does.
  • Keeping a removed additive grant writes it to the overrides.0-bootstrap.iam_bindings_additive answer under the key that Stage 0 uses, so the other members of that role stay intact.
  • The stellar iam snapshot, stellar iam diff, and stellar iam restore commands save, compare, and restore the organization IAM policy outside a deploy.
  • The Terraform runner removes the TF_VAR_, TF_CLI_ARGS, impersonation, and credential override variables from the child environment, so the apply matches the reviewed plan.
  • The runner writes one private log per Terraform command, whose header names the removed variables without their values.
  • The s0.migrate_state step compares the lineage and serial of the local and remote Stage 0 states, and refuses to copy when it cannot tell which state is newer.
  • Known failures, such as a missing service agent, a BigQuery service restriction, or the log bucket import bug of the Google provider, run a bounded remedy and retry the step.
  • Remedy patterns match only the Terraform error block, so a progress line that names a resource never triggers a remedy.
  • Remedy attempts count per step and cause within one run, and a new run starts the count over.
  • The disabled API remedy asks before it enables a service, and the --yes flag answers only for the bootstrap project and the projects of the deployment prefix.
  • The ADC quota project remedy recognizes the public project number 764086051850 of the gcloud OAuth client, which appears in errors when user credentials lack a quota project. The number belongs to the gcloud client, not to any deployment.
  • The run-state.json file of each environment records every step outcome, operator answer, and engine commit.
  • The deploy.lock file of each environment keeps two deploys from running at once.
  • An interrupted deploy records the end of the run and the state of an interrupted apply, then exits with status 4 so that a rerun resumes.
  • A workspace link or Terraform environment problem stops the deploy with exit status 4, because a rerun fixes it once the cause is gone.
  • The preflight shared.iam check now counts the grants that the overrides.0-bootstrap.iam answer keeps, and reports audit config overwrites.
  • The stellar init command refuses to move stage0_phase from final back to bootstrap once run-state records a final Stage 0 apply.
  • The stellar render command now points at stellar deploy instead of printing Terraform commands.
  • The cli/README.md file documents the deploy steps, the IAM commands, the exit codes, and the manual fallback.
  • The docs/ddg.md, docs/prerequisites.md, and scripts/README.md files name the deploy step that replaces each helper script, and each script stays as the manual fallback for one release.
  • The docs/prerequisites.md file now names the setIAM.sh script with its real file name instead of setIam.sh.
  • The Stage 0 org policy import block in the cli/src/stellar_engine/render/imports.py module now also requires deploy_org_resources.org_policies, so an environment that leaves org policies unmanaged plans no import.
  • Stages 2 and 3 keep following the DDG until a later pull request adds their steps.

Changes to existing commands

No Terraform variable, input or state key changes, so no upgrade note applies. Three existing commands behave differently.

  • The stellar render command now ends with Next: stellar deploy --stage <n> instead of the Terraform commands, and the stellar guide command lists those commands.
  • The stellar render command writes the Stage 0 org policy import block only when deploy_org_resources.org_policies is true.
  • The stellar init command exits with status 2 when it would move stage0_phase from final back to bootstrap after run-state records a final Stage 0 apply.
  • The preflight shared.iam check reads the full organization IAM policy, so the CloudApi.org_iam_bindings method and the api.IamBinding class give way to get_org_iam_policy, which keeps conditions and audit configs.

The new exit statuses 4 and 5 come only from the new stellar deploy, stellar status and stellar iam commands.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

Deployment & Compliance Impact

  • Applicable Regimes:
    • US Region Restricted (e.g., Access Policy constraint)
    • FedRAMP Moderate
    • FedRAMP High
    • DoD IL4
    • DoD IL5
    • General / All
  • NIST 800-53r5 Controls:
    • The change modifies no Terraform control implementation.
    • The reviewed saved plans support CM-3, and the organization IAM review and the Billing Account User grant support AC-2 and AC-6.

Checklist

Code Quality & Reusability

  • My code adheres to the Maximize Reusability principle. I have not redefined common elements and have reused existing base configurations and modules where possible.
  • I have checked that no existing module or configuration in modules/ or fast/stages-aw/ can be leveraged for this change.
  • My code follows the established naming conventions outlined in docs/naming-convention.md.

Documentation

  • I have updated the README.md of the modified module or blueprint.
  • I have added/updated documentation for inputs (variables) and outputs.

Security

  • My change adheres to GCP security best practices and the principle of least privilege.
  • I have ensured compliance with the targeted regime (FedRAMP Moderate, FedRAMP High, IL4, IL5, etc.).

Testing

  • I have tested my changes locally.
  • I have included details of my testing in this PR.

Testing Performed

  • uv run pytest from the cli/ directory passes with 1940 tests and 16 syrupy snapshots, and the first commit alone passes with 1364 tests.
  • uv run pytest -m terraform from the cli/ directory passes against the real Terraform binary with 50 tests.
  • The new Terraform contract tests in the cli/tests/test_terraform_runner.py file cover the runner argv order, the saved plan flow, the state migration, and the environment scrub:
    • test_target_skips_an_import_outside_the_target
    • test_saved_plan_flow_and_show_json_shape
    • test_migrate_state_copies_once_then_does_nothing
    • test_force_copy_overwrites_a_newer_destination
    • test_parent_variables_and_cli_args_do_not_leak
    • test_state_import_and_unlock_argv
  • The test_fix_pulls_rewrites_and_pushes_with_terraform test in the cli/tests/test_deploy_state_fix.py file pulls, rewrites, and pushes a real state for the log bucket fix.
  • The test_status_leaves_the_environment_byte_identical test checks that stellar status writes nothing.
  • Unit tests replace Google Cloud with the FakeCloud class and Terraform with the FakeTerraform class, so no unit test touches the network.
  • uv run ruff format --check, uv run ruff check, and uv run pyright from the cli/ directory report no issues.
  • python3 tools/check_boilerplate.py cli and the root pytest suite pass from the repository root.
  • The lab acceptance ran on a deployed IL5 environment with the placeholder prefix abc in organization 123456789012, where Stage 0 is applied in the final phase and an outside automation reverted 11 organization policies. Every Google Cloud call was a read, and a scripted prompter declined every apply and every remedy that writes.
    • stellar render --diff reports only the new org policy import condition and an input that main added since the last render.
    • stellar status shows every Stage 0 step done or not applicable except s0.apply_final, which asks for a decision because run-state records no final apply. It exits with status 4 and writes nothing.
    • stellar deploy --stage 0 initializes the remote backend, plans to a saved file and writes one IAM review snapshot. The IAM review finds 0 removed grants and 1 added grant and passes without a question.
    • The plan summary reports 1 add, 13 changes and 2 destroys: 11 organization policy drift reverts, 1 organization policy delete, and the replacement of a generated local tfvars file. The run stops with status 4 at the apply question and records that exit in run-state.
    • stellar iam diff --plan reports 0 removed, 1 added and 4 kept grants, and stellar iam diff against the snapshot reports 0 removed and 171 kept grants.
    • An earlier lab run found that s0.allow_bq stayed pending on a deployed environment, because the workload folder policy still blocks BigQuery for new enablements. The step now requires the BigQuery services of each Stage 0 project in the Stage 0 state, and a test covers it.

@github-actions github-actions Bot added Documentation Improvements or additions to documentation Scope - Fast Related to Fabric FAST deployment stages in the fast directory labels Oct 30, 2026
… diff

The library layer of the guided deploy: a Terraform runner that scrubs
the environment, logs each command and applies only saved plans, a
parser and guards for terraform show -json plans, an organization IAM
diff between a saved plan and a live policy snapshot, the Google Cloud
writes the deploy needs, and the deploy run-state.
  * Preflight now reads the full organization IAM policy and counts the
    grants that the Stage 0 iam overrides keep, so it no longer warns
    about kept grants; no other command changes behavior.
…ommands

stellar deploy runs the prerequisite, Stage 0 and Stage 1 steps of the
DDG with resume from run-state, a plan summary and guards before every
apply, an organization IAM review gate, and bounded remedies for known
errors; stellar guide explains the steps, stellar status checks them
without writing, and stellar iam snapshots, diffs and restores the
organization IAM policy.
  * Operators no longer run Terraform by hand for Stage 0 and Stage 1;
    the new exit status 4 means the run stopped for the operator and 5
    means a step failed.
@github-actions github-actions Bot added Enhancement New feature or request Level of Effort - High Complex task taking a week or more; consider breaking this down into smaller issues. Priority - Medium Standard features and non-blocking bugs; important for the current milestone but not urgent Security Something is insecure or can be secured labels Nov 3, 2026

@pulsingSignal pulsingSignal left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewing feat(cli): guided deploy at 93db8b7e70.

The saved-plan pipeline, environment scrubbing across Stage 0 and Stage 1 subprocesses, etag-guarded organization IAM review gate, lineage-aware state tracking, and protocol conformance tests for FakeCloud and FakeTerraform are well structured and thoroughly tested.

Requesting changes for one blocking regression on fresh deployments and four high-severity edge cases before taking this out of draft:

  • Blocking (F-01) s0.apply_workload aborts with targeted_scope (exit 5) on every fresh deployment because PR #381 added count to module "organization-logging", changing its planned resource address to module.organization-logging[0].google_logging_organization_settings.default[0].
  • High (F-02) Unrendered answers.yaml edits still report PASS on apply steps in stellar status, and missing direct or transitive requires edges allow stellar deploy --only and --from-step to skip rendering or apply stale workspace files.
  • High (F-03) pre.org_policies deadlocks with validate._org_policy_problems when org_policies_config.import_defaults is true, and pre, preflight, and iam ignore deploy_org_resources toggles (org_policies, iam, audit_config).
  • High (F-04) iam_diff._audit_overwrites skips services absent from the live policy, causing stellar iam restore to report no changes and exit 0 when organization auditConfigs were deleted.
  • High (F-05) Keeping a deleted additive google_organization_iam_member re-keys the override with the live lowercase member string instead of the planned for_each key, creating a concurrent Terraform create and delete race on replan.

Note on the pull request description under Changes to existing commands: render/imports.py still writes the Stage 0 org policy import block whenever existing_org_policies is non-empty and guards its for_each expression with var.deploy_org_resources.org_policies, rather than omitting the block at render time (F-14).

Comment on lines +86 to +89
ORG_LOGGING_SETTINGS = (
"module.organization-logging."
"google_logging_organization_settings.default[0]"
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Critical / Blocking] PR #381 (3f9d8a5) added count = var.deploy_org_resources.logging_settings ? 1 : 0 to module "organization-logging" in fast/stages-aw/0-bootstrap/organization.tf. On any fresh deployment where logging_settings is enabled, s0.apply_workload plans to create module.organization-logging[0].google_logging_organization_settings.default[0], which fails change.address == ORG_LOGGING_SETTINGS on line 424 and aborts every fresh deployment with Level.BLOCK (targeted_scope, exit code 5).

Accepting both the [0]-indexed address and the unindexed legacy address (with change.address in ORG_LOGGING_SETTINGS on line 424) fixes main while remaining compatible with pre-#381 engine checkouts. Add a sync check in cli/tests/test_repo.py against 0-bootstrap/organization.tf so future module address changes fail unit tests immediately.

Suggested change
ORG_LOGGING_SETTINGS = (
"module.organization-logging."
"google_logging_organization_settings.default[0]"
)
ORG_LOGGING_SETTINGS = frozenset(
{
"module.organization-logging[0]."
"google_logging_organization_settings.default[0]",
"module.organization-logging."
"google_logging_organization_settings.default[0]",
}
)

Comment on lines +424 to +425
if change.address == ORG_LOGGING_SETTINGS:
found.append(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pair with the frozenset update on ORG_LOGGING_SETTINGS above so _targeted_scope accepts module.organization-logging[0].

Suggested change
if change.address == ORG_LOGGING_SETTINGS:
found.append(
if change.address in ORG_LOGGING_SETTINGS:
found.append(

Comment on lines +17 to +18
"address": "module.organization-logging.google_logging_organization_settings.default[0]",
"module_address": "module.organization-logging",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update both targeted-workload.json and targeted-no-compliance.json to match the post-#381 module.organization-logging[0] address produced by fast/stages-aw/0-bootstrap/organization.tf.

Suggested change
"address": "module.organization-logging.google_logging_organization_settings.default[0]",
"module_address": "module.organization-logging",
"address": "module.organization-logging[0].google_logging_organization_settings.default[0]",
"module_address": "module.organization-logging[0]",

Comment on lines +152 to +155
if record.tfvars_sha256 != apply.inputs_sha256(ctx, stage):
return model.Verdict(
Status.PENDING, "The stage inputs changed since the last apply."
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[High] apply.inputs_sha256(ctx, stage) hashes only the rendered files already on disk in the stage workspace (terraform.tfvars, *.auto.tfvars, *-imports.tf). When an operator updates answers.yaml (for example via stellar init --set or stellar iam restore --emit-overrides) without re-rendering, stellar status reports s0.render and s1.render as FAIL (pending) while still reporting s0.apply_final and s1.apply as PASS.

Check render_verdict(ctx, stage) before comparing record.tfvars_sha256 so unrendered answer changes mark downstream apply steps PENDING.

Suggested change
if record.tfvars_sha256 != apply.inputs_sha256(ctx, stage):
return model.Verdict(
Status.PENDING, "The stage inputs changed since the last apply."
)
rendered = render_verdict(ctx, stage)
if rendered.status != Status.DONE:
return model.Verdict(
Status.PENDING, "The workspace has unrendered answer changes."
)
if record.tfvars_sha256 != apply.inputs_sha256(ctx, stage):
return model.Verdict(
Status.PENDING, "The stage inputs changed since the last apply."
)

Comment on lines +764 to +770
applies_when=_bootstrap_phase,
when=(
"stage0_phase is bootstrap and the state has not moved to the "
"bootstrap bucket"
),
remedies=APPLY_REMEDIES,
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[High] APPLY_BOOTSTRAP omits requires, so stellar deploy --only s0.apply_bootstrap bypasses s0.render, s0.ready, s0.apply_workload, and s0.allow_bq (engine._done already treats non-applicable conditional steps such as s0.allow_bq on non-AW regimes as satisfied).

Suggested change
applies_when=_bootstrap_phase,
when=(
"stage0_phase is bootstrap and the state has not moved to the "
"bootstrap bucket"
),
remedies=APPLY_REMEDIES,
)
applies_when=_bootstrap_phase,
when=(
"stage0_phase is bootstrap and the state has not moved to the "
"bootstrap bucket"
),
requires=(RENDER.id, READY.id, APPLY_WORKLOAD.id, ALLOW_BQ.id),
remedies=APPLY_REMEDIES,
)

Comment on lines +435 to +462
self.path = env.deploy_lock
self._content = json.dumps(
{"command": command, "started": now.isoformat()}
)

def acquire(self) -> None:
"""Takes the lock.

Raises:
LockHeldError: If the lock file exists.
"""
paths.make_private_dir(self.path.parent)
try:
descriptor = os.open(
self.path,
os.O_WRONLY | os.O_CREAT | os.O_EXCL,
fsutil.PRIVATE_FILE_MODE,
)
except FileExistsError:
holder = describe_holder(lock_holder(self.env) or {})
raise LockHeldError(
f"Another deploy holds {self.path}: {holder}. Delete "
f"{self.path} if no deploy is running.",
self.path,
) from None
with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
handle.write(f"{self._content}\n")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] Including os.getpid() and socket.gethostname() in Lock._content (and translating SIGTERM / SIGHUP into clean teardown alongside KeyboardInterrupt in commands/deploy.py) helps operators tell whether deploy.lock is held by a live process after a dropped SSH or Cloud Shell session.

Comment on lines +375 to +386
if (
previous_phase != models.STAGE0_FINAL
or updated_phase != models.STAGE0_BOOTSTRAP
):
return []
return [
f"stage0_phase cannot return from {models.STAGE0_FINAL} to "
f"{models.STAGE0_BOOTSTRAP}: the final Stage 0 apply already ran, "
"and a bootstrap apply would delete the organization policies and "
f"essential contacts it manages; keep stage0_phase "
f"{models.STAGE0_FINAL}."
]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] Checking previous_phase != models.STAGE0_FINAL allows an operator who manually edited answers.yaml back to stage0_phase: bootstrap after s0.apply_final to run stellar render (emitting only a warning while writing 0-bootstrap-phase.auto.tfvars) and subsequent answers.update calls without rejection. Since callers invoke phase_change_problems only when final_applied is true, rejecting updated_phase == models.STAGE0_BOOTSTRAP regardless of previous_phase closes that gap.

Comment on lines +326 to +334
return (
self.uses_remote
and (self.local is None or self.local.empty)
and self.remote.readable
and not self.remote.empty
and self.backup is not None
and self.backup.lineage is not None
and self.backup.lineage != self.remote.lineage
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] Add and not self.backup.empty here so an empty terraform.tfstate.backup carrying a fresh lineage UUID does not block s0.migrate_state as a foreign backup when GCS already holds the migrated state.

Comment on lines +183 to +193
def _restriction(policy):
"""Merges the rules of an effective orgpolicy v2 Policy."""
rules = list(policy.spec.rules) if policy.spec else []
allowed = {v for rule in rules for v in rule.values.allowed_values}
denied = {v for rule in rules for v in rule.values.denied_values}
return api.ServiceRestriction(
allow_all=any(rule.allow_all for rule in rules),
deny_all=any(rule.deny_all for rule in rules),
allowed=tuple(sorted(v.removeprefix("is:") for v in allowed)),
denied=tuple(sorted(v.removeprefix("is:") for v in denied)),
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] _restriction merges tag-conditioned policy.spec.rules (rule.condition.expression) into unconditional allow_all / deny_all / denied sets, and deploy/cloud_ops.py:_check_compliant attributes ancestor gcp.restrictServiceUsage denials to regime non-compliance when the folder policy already allows BigQuery. Filter out conditioned rules (or track them separately) and check whether the workload folder policy already permits the service before reporting regime non-compliance.

Comment on lines +631 to +635
Raises:
UnknownMembersError: If a binding's members, role or condition, an
additive member, or an audit config is unknown until apply,
because then the removed grants cannot be listed.
ValueError: If a change has a malformed role or condition.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] _plan_change (lines 553-566) skips unknown created additive google_organization_iam_member values rather than raising UnknownMembersError (only unknown deletes raise, because a created additive member never removes existing grants).

Suggested change
Raises:
UnknownMembersError: If a binding's members, role or condition, an
additive member, or an audit config is unknown until apply,
because then the removed grants cannot be listed.
ValueError: If a change has a malformed role or condition.
Raises:
UnknownMembersError: If a binding's members, role or condition, a
deleted additive member, or an audit config is unknown until
apply, because then the removed grants cannot be listed.
ValueError: If a change has a malformed role or condition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Documentation Improvements or additions to documentation Enhancement New feature or request Level of Effort - High Complex task taking a week or more; consider breaking this down into smaller issues. Priority - Medium Standard features and non-blocking bugs; important for the current milestone but not urgent Scope - Fast Related to Fabric FAST deployment stages in the fast directory Security Something is insecure or can be secured

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants