Health check on the operator restarting if redis breaks #181

Activity

1. sciabarracom

   assigned

   ghost

   on Oct 31, 2025
2. miki3421 commented on Nov 1, 2026

   

   miki3421

   on Nov 1, 2026

   More actions

   Confirmed on the 0.9.0 baseline with a controlled Redis-only restart.

   Environment: Redis 8.2.1 and a persistent Redis PVC. The running openserverless/redis.py and openserverless/templates/redis-conf.yaml are byte-identical to Apache 0.9.0 commit 0205d8efedfbcda3c10a5f45718eaa32069dfc3b. The complete lab operator image includes other local fixes; this was not an unmodified upstream-image test.

   Reproduction:

   1. Provision two synthetic Redis identities through the operator's normal provisioning function. Verify own-key reads/writes and denial of peer-key reads.
   2. Explicitly save the synthetic data to RDB, independently of ACL state.
   3. Restart only the Redis pod, keeping the operator container unchanged.
   4. Retry the same credentials and inspect ACL membership and the persisted synthetic data.

   Result: both ACL users disappear and authentication is rejected, while their saved data survives. The existing synthetic Alpha/Beta tenant credentials also stop authenticating. Operator container identity/restart count, Redis configuration and PVC identity remain unchanged. Exact original ACL state was restored afterwards, Alpha/Beta authentication recovered, and temporary identities/data were removed.

   Root cause: _save_acl_if_configured() calls ACL SAVE only if CONFIG GET aclfile is nonempty, but the standard Redis template does not configure ACL storage. WhiskUser resume reconciliation does not cover a Redis-only restart. PR apache/openserverless-operator#140 retains this storage behavior. The same failure was observed on a legacy 0.1.0 lab installation.

   I am preparing a focused 0.9.0 operator fix for persistent Redis installations: bootstrap ACL storage from the currently authorized state, save supported account mutations, and load it on Redis restart. Existing credentials, key/channel permissions, quota read-only state and revocations must remain intact. New installations must start safely before the first ACL file exists; existing installations must preserve their current ACLs before changing the Redis startup command. Regression coverage will include Redis-only and operator restarts, tenant isolation and successful revocation remaining revoked. Ephemeral Redis retains its explicit nonpersistent storage semantics.

   This is an authentication availability regression; cross-tenant access was not demonstrated. RDB durability/shutdown behavior is a separate observation and is not claimed as the root cause of ACL loss.