Repository navigation
feat(cli): guided deploy - #386
digitalresistor wants to merge 2 commits into
Conversation
eecf5c2 to
f43a285
Compare
… diff
The library layer of the guided deploy: a Terraform runner that scrubs
the environment, logs each command and applies only saved plans, a
parser and guards for terraform show -json plans, an organization IAM
diff between a saved plan and a live policy snapshot, the Google Cloud
writes the deploy needs, and the deploy run-state.
* Preflight now reads the full organization IAM policy and counts the
grants that the Stage 0 iam overrides keep, so it no longer warns
about kept grants; no other command changes behavior.
…ommands
stellar deploy runs the prerequisite, Stage 0 and Stage 1 steps of the
DDG with resume from run-state, a plan summary and guards before every
apply, an organization IAM review gate, and bounded remedies for known
errors; stellar guide explains the steps, stellar status checks them
without writing, and stellar iam snapshots, diffs and restores the
organization IAM policy.
* Operators no longer run Terraform by hand for Stage 0 and Stage 1;
the new exit status 4 means the run stopped for the operator and 5
means a step failed.
f43a285 to
93db8b7
Compare
pulsingSignal
left a comment
There was a problem hiding this comment.
Reviewing feat(cli): guided deploy at 93db8b7e70.
The saved-plan pipeline, environment scrubbing across Stage 0 and Stage 1 subprocesses, etag-guarded organization IAM review gate, lineage-aware state tracking, and protocol conformance tests for FakeCloud and FakeTerraform are well structured and thoroughly tested.
Requesting changes for one blocking regression on fresh deployments and four high-severity edge cases before taking this out of draft:
- Blocking (
F-01)s0.apply_workloadaborts withtargeted_scope(exit 5) on every fresh deployment because PR #381 addedcounttomodule "organization-logging", changing its planned resource address tomodule.organization-logging[0].google_logging_organization_settings.default[0]. - High (
F-02) Unrenderedanswers.yamledits still reportPASSon apply steps instellar status, and missing direct or transitiverequiresedges allowstellar deploy --onlyand--from-stepto skip rendering or apply stale workspace files. - High (
F-03)pre.org_policiesdeadlocks withvalidate._org_policy_problemswhenorg_policies_config.import_defaultsis true, andpre,preflight, andiamignoredeploy_org_resourcestoggles (org_policies,iam,audit_config). - High (
F-04)iam_diff._audit_overwritesskips services absent from the live policy, causingstellar iam restoreto report no changes and exit 0 when organizationauditConfigswere deleted. - High (
F-05) Keeping a deleted additivegoogle_organization_iam_memberre-keys the override with the live lowercase member string instead of the plannedfor_eachkey, creating a concurrent Terraformcreateanddeleterace on replan.
Note on the pull request description under Changes to existing commands: render/imports.py still writes the Stage 0 org policy import block whenever existing_org_policies is non-empty and guards its for_each expression with var.deploy_org_resources.org_policies, rather than omitting the block at render time (F-14).
| ORG_LOGGING_SETTINGS = ( | ||
| "module.organization-logging." | ||
| "google_logging_organization_settings.default[0]" | ||
| ) |
There was a problem hiding this comment.
[Critical / Blocking] PR #381 (3f9d8a5) added count = var.deploy_org_resources.logging_settings ? 1 : 0 to module "organization-logging" in fast/stages-aw/0-bootstrap/organization.tf. On any fresh deployment where logging_settings is enabled, s0.apply_workload plans to create module.organization-logging[0].google_logging_organization_settings.default[0], which fails change.address == ORG_LOGGING_SETTINGS on line 424 and aborts every fresh deployment with Level.BLOCK (targeted_scope, exit code 5).
Accepting both the [0]-indexed address and the unindexed legacy address (with change.address in ORG_LOGGING_SETTINGS on line 424) fixes main while remaining compatible with pre-#381 engine checkouts. Add a sync check in cli/tests/test_repo.py against 0-bootstrap/organization.tf so future module address changes fail unit tests immediately.
| ORG_LOGGING_SETTINGS = ( | |
| "module.organization-logging." | |
| "google_logging_organization_settings.default[0]" | |
| ) | |
| ORG_LOGGING_SETTINGS = frozenset( | |
| { | |
| "module.organization-logging[0]." | |
| "google_logging_organization_settings.default[0]", | |
| "module.organization-logging." | |
| "google_logging_organization_settings.default[0]", | |
| } | |
| ) |
| if change.address == ORG_LOGGING_SETTINGS: | ||
| found.append( |
There was a problem hiding this comment.
Pair with the frozenset update on ORG_LOGGING_SETTINGS above so _targeted_scope accepts module.organization-logging[0].
| if change.address == ORG_LOGGING_SETTINGS: | |
| found.append( | |
| if change.address in ORG_LOGGING_SETTINGS: | |
| found.append( |
| "address": "module.organization-logging.google_logging_organization_settings.default[0]", | ||
| "module_address": "module.organization-logging", |
There was a problem hiding this comment.
Update both targeted-workload.json and targeted-no-compliance.json to match the post-#381 module.organization-logging[0] address produced by fast/stages-aw/0-bootstrap/organization.tf.
| "address": "module.organization-logging.google_logging_organization_settings.default[0]", | |
| "module_address": "module.organization-logging", | |
| "address": "module.organization-logging[0].google_logging_organization_settings.default[0]", | |
| "module_address": "module.organization-logging[0]", |
| if record.tfvars_sha256 != apply.inputs_sha256(ctx, stage): | ||
| return model.Verdict( | ||
| Status.PENDING, "The stage inputs changed since the last apply." | ||
| ) |
There was a problem hiding this comment.
[High] apply.inputs_sha256(ctx, stage) hashes only the rendered files already on disk in the stage workspace (terraform.tfvars, *.auto.tfvars, *-imports.tf). When an operator updates answers.yaml (for example via stellar init --set or stellar iam restore --emit-overrides) without re-rendering, stellar status reports s0.render and s1.render as FAIL (pending) while still reporting s0.apply_final and s1.apply as PASS.
Check render_verdict(ctx, stage) before comparing record.tfvars_sha256 so unrendered answer changes mark downstream apply steps PENDING.
| if record.tfvars_sha256 != apply.inputs_sha256(ctx, stage): | |
| return model.Verdict( | |
| Status.PENDING, "The stage inputs changed since the last apply." | |
| ) | |
| rendered = render_verdict(ctx, stage) | |
| if rendered.status != Status.DONE: | |
| return model.Verdict( | |
| Status.PENDING, "The workspace has unrendered answer changes." | |
| ) | |
| if record.tfvars_sha256 != apply.inputs_sha256(ctx, stage): | |
| return model.Verdict( | |
| Status.PENDING, "The stage inputs changed since the last apply." | |
| ) |
| applies_when=_bootstrap_phase, | ||
| when=( | ||
| "stage0_phase is bootstrap and the state has not moved to the " | ||
| "bootstrap bucket" | ||
| ), | ||
| remedies=APPLY_REMEDIES, | ||
| ) |
There was a problem hiding this comment.
[High] APPLY_BOOTSTRAP omits requires, so stellar deploy --only s0.apply_bootstrap bypasses s0.render, s0.ready, s0.apply_workload, and s0.allow_bq (engine._done already treats non-applicable conditional steps such as s0.allow_bq on non-AW regimes as satisfied).
| applies_when=_bootstrap_phase, | |
| when=( | |
| "stage0_phase is bootstrap and the state has not moved to the " | |
| "bootstrap bucket" | |
| ), | |
| remedies=APPLY_REMEDIES, | |
| ) | |
| applies_when=_bootstrap_phase, | |
| when=( | |
| "stage0_phase is bootstrap and the state has not moved to the " | |
| "bootstrap bucket" | |
| ), | |
| requires=(RENDER.id, READY.id, APPLY_WORKLOAD.id, ALLOW_BQ.id), | |
| remedies=APPLY_REMEDIES, | |
| ) |
| self.path = env.deploy_lock | ||
| self._content = json.dumps( | ||
| {"command": command, "started": now.isoformat()} | ||
| ) | ||
|
|
||
| def acquire(self) -> None: | ||
| """Takes the lock. | ||
|
|
||
| Raises: | ||
| LockHeldError: If the lock file exists. | ||
| """ | ||
| paths.make_private_dir(self.path.parent) | ||
| try: | ||
| descriptor = os.open( | ||
| self.path, | ||
| os.O_WRONLY | os.O_CREAT | os.O_EXCL, | ||
| fsutil.PRIVATE_FILE_MODE, | ||
| ) | ||
| except FileExistsError: | ||
| holder = describe_holder(lock_holder(self.env) or {}) | ||
| raise LockHeldError( | ||
| f"Another deploy holds {self.path}: {holder}. Delete " | ||
| f"{self.path} if no deploy is running.", | ||
| self.path, | ||
| ) from None | ||
| with os.fdopen(descriptor, "w", encoding="utf-8") as handle: | ||
| handle.write(f"{self._content}\n") | ||
|
|
There was a problem hiding this comment.
[Low] Including os.getpid() and socket.gethostname() in Lock._content (and translating SIGTERM / SIGHUP into clean teardown alongside KeyboardInterrupt in commands/deploy.py) helps operators tell whether deploy.lock is held by a live process after a dropped SSH or Cloud Shell session.
| if ( | ||
| previous_phase != models.STAGE0_FINAL | ||
| or updated_phase != models.STAGE0_BOOTSTRAP | ||
| ): | ||
| return [] | ||
| return [ | ||
| f"stage0_phase cannot return from {models.STAGE0_FINAL} to " | ||
| f"{models.STAGE0_BOOTSTRAP}: the final Stage 0 apply already ran, " | ||
| "and a bootstrap apply would delete the organization policies and " | ||
| f"essential contacts it manages; keep stage0_phase " | ||
| f"{models.STAGE0_FINAL}." | ||
| ] |
There was a problem hiding this comment.
[Low] Checking previous_phase != models.STAGE0_FINAL allows an operator who manually edited answers.yaml back to stage0_phase: bootstrap after s0.apply_final to run stellar render (emitting only a warning while writing 0-bootstrap-phase.auto.tfvars) and subsequent answers.update calls without rejection. Since callers invoke phase_change_problems only when final_applied is true, rejecting updated_phase == models.STAGE0_BOOTSTRAP regardless of previous_phase closes that gap.
| return ( | ||
| self.uses_remote | ||
| and (self.local is None or self.local.empty) | ||
| and self.remote.readable | ||
| and not self.remote.empty | ||
| and self.backup is not None | ||
| and self.backup.lineage is not None | ||
| and self.backup.lineage != self.remote.lineage | ||
| ) |
There was a problem hiding this comment.
[Low] Add and not self.backup.empty here so an empty terraform.tfstate.backup carrying a fresh lineage UUID does not block s0.migrate_state as a foreign backup when GCS already holds the migrated state.
| def _restriction(policy): | ||
| """Merges the rules of an effective orgpolicy v2 Policy.""" | ||
| rules = list(policy.spec.rules) if policy.spec else [] | ||
| allowed = {v for rule in rules for v in rule.values.allowed_values} | ||
| denied = {v for rule in rules for v in rule.values.denied_values} | ||
| return api.ServiceRestriction( | ||
| allow_all=any(rule.allow_all for rule in rules), | ||
| deny_all=any(rule.deny_all for rule in rules), | ||
| allowed=tuple(sorted(v.removeprefix("is:") for v in allowed)), | ||
| denied=tuple(sorted(v.removeprefix("is:") for v in denied)), | ||
| ) |
There was a problem hiding this comment.
[Low] _restriction merges tag-conditioned policy.spec.rules (rule.condition.expression) into unconditional allow_all / deny_all / denied sets, and deploy/cloud_ops.py:_check_compliant attributes ancestor gcp.restrictServiceUsage denials to regime non-compliance when the folder policy already allows BigQuery. Filter out conditioned rules (or track them separately) and check whether the workload folder policy already permits the service before reporting regime non-compliance.
| Raises: | ||
| UnknownMembersError: If a binding's members, role or condition, an | ||
| additive member, or an audit config is unknown until apply, | ||
| because then the removed grants cannot be listed. | ||
| ValueError: If a change has a malformed role or condition. |
There was a problem hiding this comment.
[Low] _plan_change (lines 553-566) skips unknown created additive google_organization_iam_member values rather than raising UnknownMembersError (only unknown deletes raise, because a created additive member never removes existing grants).
| Raises: | |
| UnknownMembersError: If a binding's members, role or condition, an | |
| additive member, or an audit config is unknown until apply, | |
| because then the removed grants cannot be listed. | |
| ValueError: If a change has a malformed role or condition. | |
| Raises: | |
| UnknownMembersError: If a binding's members, role or condition, a | |
| deleted additive member, or an audit config is unknown until | |
| apply, because then the removed grants cannot be listed. | |
| ValueError: If a change has a malformed role or condition. |
Description
The
stellar deploycommand turns the DDG prerequisites, Stage 0, and Stage 1 into reviewed, resumable steps, so operators stop running Terraform, the state migration, and the helper scripts by hand.Part of #306. This pull request closes no issue on its own.
The branch has two commits on
main, and each passes the full gate on its own. The first adds the Terraform runner, the plan guards, and the organization IAM diff, and the second adds the deploy engine, the step catalog, the commands, and the docs.stellar guidecommand explains each of the 24 deploy steps and the manual commands that do the same, without reading Google Cloud.stellar statuscommand runs the check of every step, writes nothing, and exits with status 4 while a step is not done.stellar deploycommand skips the steps that are done and resumes at the first open step.--stage,--only, and--from-stepflags select the steps of one run.--yesflag answers only the routine questions, and manual steps, plan findings, and the organization IAM review always ask.plans/folder of the environment and applies only that saved plan.cli/src/stellar_engine/terraform/guards.pymodule refuse a bootstrap-phase plan after a final apply, an import that replaces its object, and the loss of a protected Stage 0 resource.google_organization_iam_bindingresources and the additivegoogle_organization_iam_memberresources of Stage 0, and compares members without case, as the provider does.overrides.0-bootstrap.iam_bindings_additiveanswer under the key that Stage 0 uses, so the other members of that role stay intact.stellar iam snapshot,stellar iam diff, andstellar iam restorecommands save, compare, and restore the organization IAM policy outside a deploy.TF_VAR_,TF_CLI_ARGS, impersonation, and credential override variables from the child environment, so the apply matches the reviewed plan.s0.migrate_statestep compares the lineage and serial of the local and remote Stage 0 states, and refuses to copy when it cannot tell which state is newer.--yesflag answers only for the bootstrap project and the projects of the deployment prefix.764086051850of the gcloud OAuth client, which appears in errors when user credentials lack a quota project. The number belongs to the gcloud client, not to any deployment.run-state.jsonfile of each environment records every step outcome, operator answer, and engine commit.deploy.lockfile of each environment keeps two deploys from running at once.shared.iamcheck now counts the grants that theoverrides.0-bootstrap.iamanswer keeps, and reports audit config overwrites.stellar initcommand refuses to movestage0_phasefromfinalback tobootstraponce run-state records a final Stage 0 apply.stellar rendercommand now points atstellar deployinstead of printing Terraform commands.cli/README.mdfile documents the deploy steps, the IAM commands, the exit codes, and the manual fallback.docs/ddg.md,docs/prerequisites.md, andscripts/README.mdfiles name the deploy step that replaces each helper script, and each script stays as the manual fallback for one release.docs/prerequisites.mdfile now names thesetIAM.shscript with its real file name instead ofsetIam.sh.cli/src/stellar_engine/render/imports.pymodule now also requiresdeploy_org_resources.org_policies, so an environment that leaves org policies unmanaged plans no import.Changes to existing commands
No Terraform variable, input or state key changes, so no upgrade note applies. Three existing commands behave differently.
stellar rendercommand now ends withNext: stellar deploy --stage <n>instead of the Terraform commands, and thestellar guidecommand lists those commands.stellar rendercommand writes the Stage 0 org policy import block only whendeploy_org_resources.org_policiesis true.stellar initcommand exits with status 2 when it would movestage0_phasefromfinalback tobootstrapafter run-state records a final Stage 0 apply.shared.iamcheck reads the full organization IAM policy, so theCloudApi.org_iam_bindingsmethod and theapi.IamBindingclass give way toget_org_iam_policy, which keeps conditions and audit configs.The new exit statuses 4 and 5 come only from the new
stellar deploy,stellar statusandstellar iamcommands.Type of Change
Deployment & Compliance Impact
Checklist
Code Quality & Reusability
modules/orfast/stages-aw/can be leveraged for this change.docs/naming-convention.md.Documentation
README.mdof the modified module or blueprint.Security
Testing
Testing Performed
uv run pytestfrom thecli/directory passes with 1940 tests and 16 syrupy snapshots, and the first commit alone passes with 1364 tests.uv run pytest -m terraformfrom thecli/directory passes against the real Terraform binary with 50 tests.cli/tests/test_terraform_runner.pyfile cover the runner argv order, the saved plan flow, the state migration, and the environment scrub:test_target_skips_an_import_outside_the_targettest_saved_plan_flow_and_show_json_shapetest_migrate_state_copies_once_then_does_nothingtest_force_copy_overwrites_a_newer_destinationtest_parent_variables_and_cli_args_do_not_leaktest_state_import_and_unlock_argvtest_fix_pulls_rewrites_and_pushes_with_terraformtest in thecli/tests/test_deploy_state_fix.pyfile pulls, rewrites, and pushes a real state for the log bucket fix.test_status_leaves_the_environment_byte_identicaltest checks thatstellar statuswrites nothing.FakeCloudclass and Terraform with theFakeTerraformclass, so no unit test touches the network.uv run ruff format --check,uv run ruff check, anduv run pyrightfrom thecli/directory report no issues.python3 tools/check_boilerplate.py cliand the rootpytestsuite pass from the repository root.abcin organization123456789012, where Stage 0 is applied in the final phase and an outside automation reverted 11 organization policies. Every Google Cloud call was a read, and a scripted prompter declined every apply and every remedy that writes.stellar render --diffreports only the new org policy import condition and an input thatmainadded since the last render.stellar statusshows every Stage 0 step done or not applicable excepts0.apply_final, which asks for a decision because run-state records no final apply. It exits with status 4 and writes nothing.stellar deploy --stage 0initializes the remote backend, plans to a saved file and writes one IAM review snapshot. The IAM review finds 0 removed grants and 1 added grant and passes without a question.stellar iam diff --planreports 0 removed, 1 added and 4 kept grants, andstellar iam diffagainst the snapshot reports 0 removed and 171 kept grants.s0.allow_bqstayed pending on a deployed environment, because the workload folder policy still blocks BigQuery for new enablements. The step now requires the BigQuery services of each Stage 0 project in the Stage 0 state, and a test covers it.