Repository navigation
fix(rbac): read API credentials with the API reader, not a cluster-wide Secret cache - #40
Open
tameemakhtar wants to merge 1 commit into
Open
tameemakhtar wants to merge 1 commit into
tameemakhtar wants to merge 1 commit into
Conversation
…de Secret cache createLinodeClient read the credentials Secret through the manager's cached client. That lazily starts a Secret informer across all namespaces, so the controller needed cluster-wide get/list/watch on Secrets and kept every Secret in the cluster in memory. Read the Secret with mgr.GetAPIReader() instead: one GET per reconcile, which only needs `get` on the one Secret. The token is still re-read on every reconcile, so a rotated token is picked up as before. RBAC: - kubebuilder markers: drop the cluster-wide secrets rule and reduce the kube-system/linode rule to `get`; regenerate config/rbac/role.yaml. - config/rbac: bind the namespaced manager-role Role, which had no RoleBinding and was only effective through the cluster-wide grant. - helm: remove secrets from the ClusterRole and from the controller Role, and add a cloud-firewall-controller-credentials Role and RoleBinding in the namespace of the Secret the controller reads (linode-cfw in .Values.namespace when apiToken is set, otherwise kube-system/linode).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
createLinodeClientreads the credentials Secret through the manager's cached client (internal/controller/cloudfirewall_controller.go:747,r.Get(context.TODO(), ...)), andReconcilecalls it on every reconcile (L121). The first cached read of a Secret makes controller-runtime start a Secret informer.cmd/main.go:131sets noCache.DefaultNamespacesorByObject, so that informer lists and watches Secrets in every namespace and keeps all of them in memory.Because of this, the controller needs a cluster-wide grant on Secrets:
cloudfirewall_controller.go:70has//+kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch, next to the intended namespaced rule at L72 (namespace=kube-system,resourceNames=linode).config/rbac/role.yamlandhelm/controller/templates/rbac/ctrl-cluster-role.yamlboth grantget,listandwatchon all Secrets cluster-wide.The namespaced rules do not cover the Secret that is actually read in every case:
ctrl-role.ymlgrantslinodein.Values.namespace. WhenapiTokenis set,deployment.yamlpasses--linode-creds=linode-cfw --linode-creds-ns=<.Values.namespace>, so the Role names the wrong Secret. WhenapiTokenis not set, the Secret iskube-system/linode, which the Role covers only if.Values.namespaceiskube-system.manager-roleRole in kube-system has no RoleBinding.Today these cases work only because of the cluster-wide grant.
Change
APIReader client.Readerfield toCloudFirewallReconcilerand set it tomgr.GetAPIReader()incmd/main.go.createLinodeClientnow takes the reconcile context and reads the Secret throughAPIReaderwith a 30 second timeout. It still re-reads the Secret on every reconcile, so a rotated token is picked up straight from the API server.get.config/rbac/role.yamlis regenerated withmake manifests.config/rbac/role_binding.yaml: add a RoleBinding for the namespacedmanager-roleRole.ctrl-role.yml. Addtemplates/rbac/ctrl-creds-role.yaml, a Role and RoleBinding namedcloud-firewall-controller-credentials. They are created in the namespace of the Secret that the controller reads and grant onlygeton that Secret:linode-cfwin.Values.namespacewhenapiTokenis set, otherwiselinodeinkube-system. Unlike the other RBAC templates, they are not Helm hooks. They are ordinary release resources, so Helm removes them on uninstall, and removes the old copy when a change toapiTokenornamespacemoves the grant to another namespace. Helm still creates Roles and RoleBindings before the Deployment.Why APIReader
Another option is to keep the cached client and restrict the Secret informer with
cache.Options.ByObject, using the Secret's namespace and ametadata.namefield selector. That still needslistandwatch, plus extra cache setup. The API reader is simpler: one GET per reconcile, and it needs onlygeton one named Secret.Testing
internal/controller/credentials_test.go. The reconciler's cached client is a fake client with an interceptor that returns Forbidden for any Secret Get or List.APIReaderis a fake client that holds the Secret. The specs check that the read succeeds with atokenkey, and that an error is returned when the key is missing and when the Secret does not exist.r.Get(context.TODO(), ...)line restored, all three specs fail withsecrets is forbidden. With the change, all three pass.make test: the three new specs pass. The existing scaffold spec "should successfully reconcile the resource" fails withresource name may not be empty, which is the same failure as onmain(77b6da4) because that spec does not set the credential options. I only addedAPIReader: k8sClientto it so that it fails the same way and does not panic on a nil reader.golangci-lint run --new-from-rev=origin/main: 0 issues. The fullmake lintreports 46 issues on this branch and 48 onmain, all in code that I did not change.helm templatewith default values,--set apiToken=x,--set namespace=cfw-system, and both together: no ClusterRole has a Secrets rule, and the credentials Role is in the Secret's namespace with the correct resource name and onlyget.helm lintpasses.kustomize build config/default: the ClusterRole has no Secrets rule, and the kube-systemmanager-roleRole (linode,get) is now bound to the controller service account.Upgrade note
Helm users who set
apiTokennow get a namespacedgetgrant onlinode-cfwin.Values.namespace, instead of relying on the cluster-wide grant.The new RBAC needs a controller image that contains this change. An older image reads the Secret through the cache, its informer cannot list Secrets, and
Reconcileblocks waiting for the cache to sync. This can happen in three ways:image.tagis pinned to an older image.pullPolicyisIfNotPresent.helm rollbackto an older chart. The ClusterRole and the controller Role arepre-install,pre-upgradehooks, and a rollback does not run them, so they keep the narrowed rules. The rollback also deletes the credentials Role. After a rollback, the cluster-wide Secrets grant has to be restored by hand.This PR does not change
versionorappVersioninChart.yaml, following the repo's practice of bumping them in release commits. On merge,helm-release.yamlwould rebuild the image and push it asv0.2.1again, while chart-releaser skips the existing chart 0.2.1. I'd suggest releasing this as a new chart version andappVersion, so that the new image gets its own tag and ships with the new RBAC. I'm happy to add the bump to this PR if you prefer.The new image works with the old RBAC, so upgrading the image first is safe. After this change, the controller no longer has, or uses, read access to Secrets other than the one credentials Secret.