Skip to content

fix(rbac): read API credentials with the API reader, not a cluster-wide Secret cache - #40

Open
tameemakhtar wants to merge 1 commit into
linode:mainfrom
tameemakhtar:fix/secret-rbac-api-reader
Open

tameemakhtar wants to merge 1 commit into
linode:mainfrom
tameemakhtar:fix/secret-rbac-api-reader

Conversation

@tameemakhtar

Copy link
Copy Markdown

Problem

createLinodeClient reads the credentials Secret through the manager's cached client (internal/controller/cloudfirewall_controller.go:747, r.Get(context.TODO(), ...)), and Reconcile calls it on every reconcile (L121). The first cached read of a Secret makes controller-runtime start a Secret informer. cmd/main.go:131 sets no Cache.DefaultNamespaces or ByObject, so that informer lists and watches Secrets in every namespace and keeps all of them in memory.

Because of this, the controller needs a cluster-wide grant on Secrets:

  • cloudfirewall_controller.go:70 has //+kubebuilder:rbac:groups="",resources=secrets,verbs=get;list;watch, next to the intended namespaced rule at L72 (namespace=kube-system,resourceNames=linode).
  • config/rbac/role.yaml and helm/controller/templates/rbac/ctrl-cluster-role.yaml both grant get, list and watch on all Secrets cluster-wide.

The namespaced rules do not cover the Secret that is actually read in every case:

  • In Helm, ctrl-role.yml grants linode in .Values.namespace. When apiToken is set, deployment.yaml passes --linode-creds=linode-cfw --linode-creds-ns=<.Values.namespace>, so the Role names the wrong Secret. When apiToken is not set, the Secret is kube-system/linode, which the Role covers only if .Values.namespace is kube-system.
  • In kustomize, the generated manager-role Role in kube-system has no RoleBinding.

Today these cases work only because of the cluster-wide grant.

Change

  • Add an APIReader client.Reader field to CloudFirewallReconciler and set it to mgr.GetAPIReader() in cmd/main.go. createLinodeClient now takes the reconcile context and reads the Secret through APIReader with a 30 second timeout. It still re-reads the Secret on every reconcile, so a rotated token is picked up straight from the API server.
  • RBAC markers: remove the cluster-wide Secrets marker and reduce the kube-system/linode marker to get. config/rbac/role.yaml is regenerated with make manifests.
  • config/rbac/role_binding.yaml: add a RoleBinding for the namespaced manager-role Role.
  • Helm: remove the Secrets rule from the ClusterRole and from ctrl-role.yml. Add templates/rbac/ctrl-creds-role.yaml, a Role and RoleBinding named cloud-firewall-controller-credentials. They are created in the namespace of the Secret that the controller reads and grant only get on that Secret: linode-cfw in .Values.namespace when apiToken is set, otherwise linode in kube-system. Unlike the other RBAC templates, they are not Helm hooks. They are ordinary release resources, so Helm removes them on uninstall, and removes the old copy when a change to apiToken or namespace moves the grant to another namespace. Helm still creates Roles and RoleBindings before the Deployment.

Why APIReader

Another option is to keep the cached client and restrict the Secret informer with cache.Options.ByObject, using the Secret's namespace and a metadata.name field selector. That still needs list and watch, plus extra cache setup. The API reader is simpler: one GET per reconcile, and it needs only get on one named Secret.

Testing

  • New Ginkgo specs in internal/controller/credentials_test.go. The reconciler's cached client is a fake client with an interceptor that returns Forbidden for any Secret Get or List. APIReader is a fake client that holds the Secret. The specs check that the read succeeds with a token key, and that an error is returned when the key is missing and when the Secret does not exist.
  • With the old r.Get(context.TODO(), ...) line restored, all three specs fail with secrets is forbidden. With the change, all three pass.
  • make test: the three new specs pass. The existing scaffold spec "should successfully reconcile the resource" fails with resource name may not be empty, which is the same failure as on main (77b6da4) because that spec does not set the credential options. I only added APIReader: k8sClient to it so that it fails the same way and does not panic on a nil reader.
  • golangci-lint run --new-from-rev=origin/main: 0 issues. The full make lint reports 46 issues on this branch and 48 on main, all in code that I did not change.
  • helm template with default values, --set apiToken=x, --set namespace=cfw-system, and both together: no ClusterRole has a Secrets rule, and the credentials Role is in the Secret's namespace with the correct resource name and only get. helm lint passes.
  • kustomize build config/default: the ClusterRole has no Secrets rule, and the kube-system manager-role Role (linode, get) is now bound to the controller service account.

Upgrade note

Helm users who set apiToken now get a namespaced get grant on linode-cfw in .Values.namespace, instead of relying on the cluster-wide grant.

The new RBAC needs a controller image that contains this change. An older image reads the Secret through the cache, its informer cannot list Secrets, and Reconcile blocks waiting for the cache to sync. This can happen in three ways:

  • image.tag is pinned to an older image.
  • A node still has an older image cached under the same tag, because pullPolicy is IfNotPresent.
  • helm rollback to an older chart. The ClusterRole and the controller Role are pre-install,pre-upgrade hooks, and a rollback does not run them, so they keep the narrowed rules. The rollback also deletes the credentials Role. After a rollback, the cluster-wide Secrets grant has to be restored by hand.

This PR does not change version or appVersion in Chart.yaml, following the repo's practice of bumping them in release commits. On merge, helm-release.yaml would rebuild the image and push it as v0.2.1 again, while chart-releaser skips the existing chart 0.2.1. I'd suggest releasing this as a new chart version and appVersion, so that the new image gets its own tag and ships with the new RBAC. I'm happy to add the bump to this PR if you prefer.

The new image works with the old RBAC, so upgrading the image first is safe. After this change, the controller no longer has, or uses, read access to Secrets other than the one credentials Secret.

…de Secret cache

createLinodeClient read the credentials Secret through the manager's
cached client. That lazily starts a Secret informer across all
namespaces, so the controller needed cluster-wide get/list/watch on
Secrets and kept every Secret in the cluster in memory.

Read the Secret with mgr.GetAPIReader() instead: one GET per reconcile,
which only needs `get` on the one Secret. The token is still re-read on
every reconcile, so a rotated token is picked up as before.

RBAC:
- kubebuilder markers: drop the cluster-wide secrets rule and reduce the
  kube-system/linode rule to `get`; regenerate config/rbac/role.yaml.
- config/rbac: bind the namespaced manager-role Role, which had no
  RoleBinding and was only effective through the cluster-wide grant.
- helm: remove secrets from the ClusterRole and from the controller
  Role, and add a cloud-firewall-controller-credentials Role and
  RoleBinding in the namespace of the Secret the controller reads
  (linode-cfw in .Values.namespace when apiToken is set, otherwise
  kube-system/linode).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant