Repository navigation
feat(modules)!: sync Cloud Foundation Fabric modules to v60.0.0 and add sync tooling - #311
Conversation
033a1aa to
6c58100
Compare
|
The Trivy failure comes from vendored upstream code, not from changes in this pull request. All three alerts point at
The fix belongs upstream rather than in this branch. A local edit to a vendored file turns into a merge conflict on every weekly sync, and the next sync after the upstream fix brings the corrected lockfile here. The The recipe is a sample Cloud Function that no stage or blueprint deploys, so the alerts do not affect any deployment. A maintainer can dismiss the three code scanning alerts as vendored code, or leave them open until the sync brings the upstream fix. |
6c58100 to
5ebfdb1
Compare
There was a problem hiding this comment.
Trivy found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
5ebfdb1 to
97115de
Compare
|
Rebased onto Conflict resolutions
Verification
Open items
The branch stays a draft until a test organization confirms a fresh deployment and an upgrade plan. |
The sync adds modules/api-gateway/recipe-multi-region/function/package.json, which the Dependabot coverage test rejects. Upstream Dependabot already updates that manifest, and local updates would conflict with the next sync, so the test skips manifests in modules whose versions.tf file records a Fabric release.
The tools/sync-from-cff.sh script now also syncs the repository paths that the tools/cff-sync.bases file lists with a slash in the first field, as "<local path> <upstream path> <base release>". An entry is either a directory or a single file, and it gets the same three-way merge, base tracking and report rows as a module. The --modules option accepts these paths too, and the dirty-tree guard covers them. Python files outside modules/ follow the repository ruff format style while upstream uses 2-space indents. The script formats the base and the target blob with ruff before it compares and merges them, so only real upstream changes reach the local copy. A path that upstream left alone skips the formatting. The CFF_SYNC_PY_FORMAT variable overrides the formatter, which the tests use to stay hermetic, and the cff-sync.yml workflow now sets up uv for the default formatter. The tools/cff-sync-pr.sh script stages the tracked paths with the modules, so a release that changes only the tooling still opens a pull request, and its title and body name modules and tooling. The report and console output now print the full local path of each conflicting file instead of rebuilding it from the module name.
…59.2.0 The tools/tfdoc.py and tools/check_documentation.py scripts predated the vendored modules, so regenerating a synced README rewrote every table and the README check failed on 87 synced modules. - Three-way merge both scripts from Fabric v35.1.0, the closest upstream ancestor of the local copies, to v59.2.0, with both upstream sides in the repository ruff format style. The merge keeps the local fixes: the ruff style and lint fixes, missing parser contexts, non-greedy README markers, escaped pipes in output descriptions, the named failure labels and the checks that only run on READMEs with tables. - Drop the local tftest regular expression and the fallback for missing fixtures in favor of the upstream directive parser. Vendor the tests/examples/utils.py parser module, which the tfdoc.py script imports, in the repository Python style. - Vendor all 24 files under the tests/fixtures/ directory, which module README examples name and the tfdoc.py script links. The script now fails on an unknown fixture again, as upstream does. - Track the four paths in the tools/cff-sync.bases file at v59.2.0, so the weekly sync keeps them current, and list their local changes in the docs/upstream-sync.md guide. - Skip the tests/fixtures/ directory in the Trivy scan, because the fixtures only feed README examples. The tools/tflint.sh script already covers only fast/, blueprints/ and modules/. - Replace the tfdoc tests for the dropped expression with tests for the directive parser and the fixture links. Fixes google#319.
The synced modules/net-lb-ext/README.md file links to the net-cloud-armor module, which Stellar Engine did not vendor, so the link check failed. Vendoring the modules/net-cloud-armor/ module fixes the links, and the weekly sync keeps it current through the module_name in its versions.tf file. The OpenTofu versions.tofu file stays out, as the tools/cff-sync.ignore file requires.
Regenerate the 36 README files that the v59.2.0 tools/tfdoc.py script reports as stale. Variable tables drop the type tooltips that the old script rendered and upstream never renders, fixture lists link to the vendored tests/fixtures/ directory, and the modules/project-factory/ README drops its Contents TOC entry. Every regenerated synced module README now matches upstream v59.2.0 byte for byte, except the modules/datafusion-se/ copy.
3b8d742 to
8e59234
Compare
The rebase onto the Trivy severity filter from google#342 left two skip-dirs keys in the Trivy step of the ci.yml workflow. GitHub Actions rejects a workflow with a duplicate key, so the step lists the recipe directories from google#342 and the vendored tests/fixtures/ directory in one skip-dirs input.
…60.0.0 Generated by tools/sync-from-cff.sh.
# Conflicts: # blueprints/third-party-solutions/gitlab/README.md
a4327ab to
0e059f0
Compare
…ate Trivy ignores
0e059f0 to
290e35d
Compare
Description
The vendored modules under
modules/and the README tooling undertools/now match upstream Cloud Foundation Fabric v59.2.0, and new tooling keeps them current with a weekly sync pull request. The modules last synced from Fabric v38.0.0, 21 major releases ago, so stages and blueprints missed upstream fixes and the move to google provider 8.Fixes #308
Fixes #319
Sync tooling
tools/sync-from-cff.shscript three-way merges every file of every tracked module from its recorded base release to a target release, so Stellar Engine changes survive and conflicts surface instead of upstream overwriting them.module_nameof itsversions.tffile. The newtools/cff-sync.basesfile covers the modules that upstream ships without one, and the-secopies track their upstream modules through the same records.tools/cff-sync.basesfile also lists repository paths outsidemodules/, a directory or a single file, which the sync tracks the same way. The README tooling and the test fixtures use these entries.modules/keep the repositoryruff formatstyle. The script formats both upstream sides the same way before it merges them, so the indentation difference from upstream never conflicts.tools/cff-sync.ignoresuch asversions.tofu, and reports module calls to directories that do not exist..github/workflows/cff-sync.ymlworkflow runs the sync every Monday and keeps one pull request open through the newtools/cff-sync-pr.shscript. It opens the pull request as a draft when files conflict or a release needs a new provider major version.docs/upstream-sync.mdguide describes the procedure, the review checklist, the local changes that each sync preserves, the workflow setup and the cadence.tests/test_sync_from_cff.shandtests/test_cff_sync_pr.shsuites cover the merge paths, the tracked paths and the branch handling with throwaway repositories and a fakeghcommand. Thetools/run_tests.shscript and the CI shell job run everytests/test_*.shsuite since refactor(tests): flatten test suite into tests/ and expand unit coverage #299.Sync to v59.2.0
The commits separate the generated sync from the hand-written changes, so reviewers can check each one on its own.
chore(modules): sync Cloud Foundation Fabric modules to v59.2.0holds the script output, with conflicting files kept at their local version.fix(modules): resolve the v59.2.0 sync conflicts and keep Stellar Engine changesresolves the 72 conflicting files. It starts from upstream and keeps only the local behavior that upstream lacks, as the local changes table indocs/upstream-sync.mdlists. It also vendors thedataplex-aspect-typesmodule, which the syncedproject-factorymodule calls.chore(repo)!: raise the Terraform and provider baselinemovesdefault-versions.tf, the styleguide, the documentation, the Stellar Engine modules and the blueprints to Terraform 1.12.2 and the google providers>= 8.6.0, < 9.0.0.fix(fast)!andfix(blueprints)!adapt every caller to the new module interfaces and to provider 8.docs(modules)regenerates the README tables of the modules with local changes, using thetools/tfdoc.pyscript of Fabric v59.2.0.chore(tools)removes theUPSTREAM_PENDING_SYNClist from thetools/tflint.shscript, because the synced modules now pass the naming rule.test(cli)makes the preflight tools test read the Terraform floor from the modules instead of hardcoding 1.10.2.ci(deps)makes the Dependabot coverage test skip manifests in modules that the sync owns, such as the newmodules/api-gateway/recipe-multi-region/function/package.jsonfile. Upstream Dependabot updates them, and local updates would conflict with the next sync.README tooling
These commits fix #319 and get
tools/lint.shpassing again.feat(tools)teaches the sync script to track the repository paths intools/cff-sync.bases, with the Python formatting step and new shell tests. Thetools/cff-sync-pr.shscript commits those paths with the modules, and the weekly workflow sets up uv for the formatting step.chore(tools)three-way mergestools/tfdoc.pyandtools/check_documentation.pyfrom Fabric v35.1.0, the closest upstream ancestor of the local copies, to v59.2.0. The merge keeps the local fixes, whichdocs/upstream-sync.mdlists. The commit also vendors thetests/examples/utils.pydirective parser thattools/tfdoc.pyimports and all 24tests/fixtures/files that module README examples name, and records all four paths intools/cff-sync.bases.chore(modules)vendors thenet-cloud-armormodule, which the syncednet-lb-extREADME links to.docs(repo)regenerates the 36 README files that the syncedtools/tfdoc.pyscript reports as stale. Every regenerated synced module README now matches upstream byte for byte, apart from thedatafusion-secopy.ci(security)merges thetests/fixtures/entry into the Trivyskip-dirsinput from ci(security): enforce Trivy SARIF severity filter and suppress false positives #342. An earlier rebase left twoskip-dirskeys in the Trivy step, which GitHub Actions rejects as a duplicate key.Decisions for reviewers
cloud-run-v2-secopy now matches upstreamcloud-run-v2exactly. A follow-up change can point its callers tomodules/cloud-run-v2/and delete the copy.projectmodule keeps thegcloudstep that deletes the default network, because upstream still has no native equivalent.gke-cluster-standardandgke-cluster-standard-semodules keep the Shielded GKE Nodes default from fix(gke): enable shielded nodes by default and fix gke-hardened feature key #341, because upstream still defaults it to off.gke-cluster-standardandgke-cluster-autopilotmodules keep the public endpoint disabled by default. Upstream removed that default, which would give thefedramp-high/gitlabcluster a public control plane endpoint.net-vpccallers setdirectpath = false, because upstream now adds a DirectPath route to every VPC by default.fedramp-high/gitlabblueprint keeps the authorized range from fix(blueprints): apply fedramp-high/gitlab GKE control plane and node settings #345. The upstream GKE modules now omit master authorized networks when a caller passes no ranges, so every GKE caller in Stellar Engine passes at least one range.net-lb-ext,net-lb-intandnet-lb-proxy-intmodules take the upstream v59.2.0 files, which already contain the health check and backend service fixes from fix(modules): fix health check and backend service configs in net-lb-ext, net-lb-int, and net-lb-proxy-int #297.cnapblueprint lets IAP use a Google-managed OAuth client by default, and the new optionaliap_oauth2_clientvariable accepts a client that operators create in the console. Provider 8 removedgoogle_iap_clientafter Google shut down the IAP OAuth Admin API.vertex-mlopsblueprint drops theMANAGEDnotebook type, because Google-managed notebooks reached end of life and provider 8 removedgoogle_notebooks_runtime.modules/api-gateway/recipe-multi-region/function/package-lock.jsonfile. The file matches upstream v59.2.0 and upstreammainbyte for byte, so the fix belongs upstream and arrives through the weekly sync.compute-vmmodule, which moved from themain.tffile to theinstance.tffile upstream. Both alerts depend on caller inputs. The Stellar Engine GKE callers set authorized ranges, and only the stage 2 NVA and NGFW appliances enable IP forwarding, which routing requires andmainalready reports.tests/fixtures/directory to theskip-dirsinput from ci(security): enforce Trivy SARIF severity filter and suppress false positives #342, because those files only feed README examples. Thetools/tflint.shscript already covers onlyfast/,blueprints/andmodules/.datastoreandil5/private-service-connectblueprints, theprivate-service-connectmodule and thegemini-enterpriseblueprints keep their provider ranges. The first three wrap registry modules with their own constraints, and the last ship their own deployment tooling.Scope
The tooling and the sync land together on purpose, so reviewers see the script with its first real run. The commits keep the tooling separate, so a split into a tooling pull request and a sync pull request stays possible on request.
The README tooling from #319, the vendored test fixtures and the vendored
net-cloud-armormodule belong here too, because the README and link checks oftools/lint.shonly pass on the synced modules with them. They stay separate commits.Fixes carried over
The migration surfaced several existing defects. Their fixes merged into
mainfirst, and the rebased branch keeps each one on the v59.2.0 interfaces.blueprints/il5/gke-hardenedCIS node settings that never applied. The branch keeps itsnode_config.confidential_nodesinput as a local change to thegke-nodepoolmodule.blueprints/fedramp-high/gitlabGKE settings that Terraform discarded. The branch keeps its authorized range.bastion_vm_public_ipoutput ofblueprints/il5/gkereads theexternal_ipoutput of thecompute-vmmodule.blueprints/il5/gke-hardenedand defaults Shielded GKE Nodes to on in the cluster modules.blueprints/third-party-solutions/gitlabdatabase gets its private network through thepsa_configobject of thecloudsql-instancemodule.blueprints/fedramp-high/vertex-mlopsapplies the notebook owner to Workbench instances.fast/stages-aw/3-securityperimeter adds the log sink ingress policy only with a logging configuration.The branch also carries recent stage changes from
mainonto the new module interfaces.deploy_org_resourcestoggles from feat(0-bootstrap): add granular deploy_org_resources toggles for folder-scoped deployments #381 keep gating the contacts, custom roles, org policies and log sinks of the stage 0organizationmodule. The custom roles take the upstream{ permissions = [...] }object type, and theroot_nodecondition variable moves to the top-levelcontextinput of the module.compute.restrictVpcPeeringproducers from feat(0-bootstrap): make compute.restrictVpcPeering producers configurable #352 and the regime-gatedcompute.requireOsLoginpolicy from feat(0-bootstrap): gate compute.requireOsLogin by compliance regime #293 need no module change.Follow-up work tracks the cleanups that this sync enables.
modules/cloud-run-v2-se/with the upstream module.Breaking Changes
Existing deployments need Terraform 1.12.2, a provider upgrade, and a few state and configuration changes before the first plan. The
!in the title makes Release Please cut a new major release.Type of Change
Deployment & Compliance Impact
Checklist
Code Quality & Reusability
modules/orfast/stages-aw/can be leveraged for this change.docs/naming-convention.md.Documentation
README.mdof the modified module or blueprint.Security
Testing
Testing Performed
The checks below ran locally on the branch rebased onto
mainat e239604. This pull request stays a draft until a test organization confirms the deployment and upgrade paths, which no check here covers.bash tools/run_tests.shpasses with 122 Python tests and the shell suites, including 58 sync script tests and 14 pull request script tests.cli/project passespytestwith 969 tests,ruff check,ruff format --checkandpyright.bash tools/lint.shpasses 9 of its 10 checks, includingterraform fmt -recursive -check, the README check and the link check. The Terraform lint check did not run, because TFLint is not installed locally.tools/tflint.shwith TFLint v0.64.0 passed for all 155 directories, with no skipped modules, before this rebase. The rebase changes only the stage 0organization.tffile among the Terraform files of the branch.shellcheck0.11.0 passes for the sync scripts, their tests,tools/tflint.shandtools/run_tests.sh.zizmorandactionlintreport no findings for theci.ymlandcff-sync.ymlworkflows. Before theci(security)commit,actionlintreported the duplicate Trivyskip-dirskey.terraform init -backend=falseandterraform validatepass for 151 of the 155 Terraform directories undermodules/,fast/stages-aw/andblueprints/. Stages 1 to 3 fail only because thegoogle.billingprovider alias has no configuration outside a deployment, whichmainreports the same way. A local stub for that alias let them validate before this rebase.tools/sync-from-cff.sh --to v59.2.0 --conflict-style oursreported 72 conflicting files, and a dry run on the branch reports no changes for the four tracked paths. Both ran before this rebase, which touches no synced module file.-refresh=falseagainstmainand the branch show the same resources for the 3-security VPC Service Controls perimeter in dry-run and enforced modes, the 2-networking-b bastion, the 2-networking-a NVA templates, and theil5/gke-hardenedblueprint, apart from new provider fields such asdeletion_policy.terraform state rmremoves agoogle_iap_cliententry that provider 8 cannot read, while aremovedblock cannot.