Skip to content

Isolate etcd and Milvus credentials and management endpoints on 0.9.0 - #141

Merged
sciabarracom merged 3 commits into
apache:0.9.0from
miki3421:security-0.9.0-etcd-milvus
Oct 20, 2026
Merged

sciabarracom merged 3 commits into
apache:0.9.0from
miki3421:security-0.9.0-etcd-milvus

Conversation

@miki3421

@miki3421 miki3421 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

The 0.9.0 deployment uses a repository-shipped etcd JWT signing key, exposes etcd administrator credentials in Milvus configuration, publishes Milvus metrics through a NodePort, and includes configuration values in DEBUG logs. This patch addresses these paths together while retaining the 0.9.0 etcd/Milvus engine versions and persistent data layout.

  • Generate and retain an installation-specific RSA-3072 signing key in a new Secret; refuse implicit replacement of invalid or missing signing state on an already patched installation. Exclude stale shared-key manifests in the persistent deploy directory and remove the old live Secret after etcd rollout.
  • Store full/slim Milvus configuration in a Secret and mount it read-only. Use a dedicated etcd identity with READWRITE permission only under milvus/, plus READ on the literal health key required by Milvus 2.5.2 startup. Refuse existing identities with unexpected roles or broader permissions. Remove the old ConfigMap only after the Secret-backed Deployment is ready.
  • Change Milvus Services to ClusterIP and add ingress NetworkPolicies: etcd clients are operator/Milvus/members, peer traffic is members only, and Milvus metrics are limited to operator/Prometheus. Authenticated Milvus application traffic remains available on port 19530. Add the namespaced operator permissions required to reconcile these policies.
  • Log configuration key names without values; apply Secret resources through a private wrapper that does not report credential-bearing payloads or stderr on failure.

The transition also verifies and preserves an existing Milvus system user instead of recreating it during reconciliation. It never resets that account after an authentication failure.

Validation:

  • 13 focused regression tests passed again locally; added to the pull-request Check workflow. Full/slim Secret rendering includes punctuation and newline credentials.
  • The complete isolated regression suite passes 33 tests, including six tests for temporary loopback-only Kubernetes forwarding, failure handling and cleanup. This supports external integration runners without reopening etcd or Milvus NodePorts. Full/slim integration clients connect to Milvus through a local tunnel after the actual deployment reconciliation completes.
  • Both existing full K3s labs passed 15 live security checks, tenant-runtime network denial checks and legitimate Prometheus/API connectivity checks.
  • Alpha/Beta: 15/15 application checks before and after on each lab, without editing or redeploying application code. Existing fixtures, PVC/PV bindings and Service identities were preserved; all three etcd members and Milvus were Ready at completion.
  • The adapted etcd client passed all 15 live security checks again in an isolated baseline operator process. The forwarding helper was also exercised from the VM host against the actual private Services, retaining authentication requirements. The Advanced port was exercised separately; this PR contains only the 0.9.0 backport.

Operational considerations: NetworkPolicies require an enforcing CNI (enforcement tested with K3s). The standard full/slim Milvus integration tests now also pass in the GitHub Kind workflow using private tunnels; that verifies compatibility rather than CNI enforcement. Kubernetes Secret-read and administrator privileges remain privileged. Baseline etcd transport remains unchanged. Operators of existing installations should rotate previously exposed etcd root and other affected credentials separately after reconciling etcd and Milvus; passwords were rotated and the etcd probe environments refreshed in both test labs. Automatic credential rotation or data migration is not part of this PR.

CI transparency: the existing task itest runner reports failures without a failing exit code. A final Check step now rejects any integration failure except the two reproduced in the unmodified 0.9.0 CI run 38893750250: tests/kind/ferretdb_test.ipy and tests/kind/runtimes_preloader_test.ipy. Those remain explicit warnings, not passing tests. In particular, failures of either full or slim Milvus now make the Check fail.

Final CI result for f06f3ec: Check run 39257637756 succeeded. All 33 regression tests passed. Both full/slim Milvus integration scenarios passed among 36 passing integration scripts. The only two remaining failures are the baseline failures listed above; the new regression gate passed with no unexpected failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants