Repository navigation
Isolate full-deployment Kafka and ZooKeeper from tenant action pods on 0.9.0 #276
Description
Problem
In the 0.9.0 full deployment, ordinary tenant action pods can reach the shared Kafka and ZooKeeper endpoints. These are internal control-plane services and should not be accessible to tenant code. The deployment does not install a network boundary restricting those services to their intended clients.
This is an internal multi-tenant isolation problem. It does not establish an unauthenticated Internet-facing entry point, and slim installations without Kafka/ZooKeeper are outside this issue.
Attacker and risk
| Actor / position | Preconditions | Risk |
|---|---|---|
| Anonymous Internet client | Only the public HTTP ingress is exposed | No direct Kafka/ZooKeeper access demonstrated by these checks |
| Ordinary tenant/developer | Can deploy and invoke an action in their own namespace | Reachable unauthenticated shared messaging and coordination endpoints; potential cross-tenant confidentiality, integrity and availability impact |
| Host or Kubernetes administrator | Already controls the cluster | Outside the tenant-isolation boundary |
Expected behavior
Action pods must be denied access to Kafka and ZooKeeper by Service name, pod DNS name and pod IP. Controller/invoker traffic, broker coordination and monitoring must keep working.
Updating the operator on an existing full installation must install the protection automatically, including closing pre-existing messaging connections.
Proposed change
The accompanying operator PR targets 0.9.0 and adds narrow same-namespace ingress NetworkPolicies for Kafka and ZooKeeper. Trusted client selectors exclude every pod carrying user-action-pod, regardless of its value. Metrics access is limited to the installation's Prometheus pods.
The operator applies policies before workloads, rolls legacy messaging pods once, waits for completion, and reconciles enabled ZooKeeper then Kafka during resume and Whisk updates. Failures propagate; disabled components are skipped.
This requires a CNI that enforces NetworkPolicy. A policy object alone is not proof of isolation. The change preserves component versions, protocols and PVC contents; it does not add Kafka SASL/TLS or ZooKeeper authentication.
Validation
Local synthetic K3s lab:
- 16/16 tenant connection attempts were reachable in the legacy fixture and denied after automatic operator resume.
- Legitimate clients, Prometheus, action-labelled spoof checks and foreign-namespace checks passed.
- Repeating operator resume preserved messaging pod identities.
- Alpha/Beta application tests: 15/15; Redis Lua: 84/84; Streamer: 24/24.
- Scheduled actions continued working, and application source remained unchanged.
- Operator unit suite: 66/66, including 19 focused messaging tests.
The lab exercised operator image replacement/restart and the real Kopf resume handler. The exact ops update operator CLI command was not run with the unpublished local image.
Known limitation
Whisk uninstall still relies on owner garbage collection when the process-local saved component spec is absent. Policy-last teardown is therefore not guaranteed during uninstall. This was identified in the targeted review and accepted by the contributor as a non-blocking limitation of this proposal. It remains disclosed for maintainer assessment.
Activity