Repository navigation
[0.9.0] Redis loses recent RDB writes during graceful pod termination #284
Description
A normal Redis pod replacement can lose acknowledged writes made since the last RDB snapshot, even though Redis persistence is enabled. The operator starts Redis under a shell without exec, so Redis does not receive the container stop signal and cannot perform its final save.
Affected configuration
Verified on the 0.9.0 K3s baseline with Redis 8.2.1, save 3600 1 300 100 60 10000, AOF disabled and a persistent PVC. The main-container command in both openserverless/templates/redis-set.yaml and deploy/redis/redis-set.yaml is:
command: ["/bin/sh", "-c", "redis-server /redis-master/redis.conf"]That command is present in Apache 0.9.0 commit 0205d8efedfbcda3c10a5f45718eaa32069dfc3b, before the ACL persistence change in apache/openserverless-operator#157.
Controlled reproduction
Four experiments used the same Redis image and persistence configuration, with isolated synthetic pods/PVCs. Each wrote one key and saved it, then acknowledged a second key without an intervening snapshot. The previously saved key survived every experiment.
| Stop path | Container exit | Latest write |
|---|---|---|
| Current command, ordinary pod deletion | 137 after 30.722 s | Lost |
Same command with exec, ordinary pod deletion |
0 after 0.699 s | Preserved |
Current command, explicit SHUTDOWN SAVE |
0 after 0.190 s | Preserved |
exec command, deliberate SIGKILL |
137 | Lost, expected with RDB |
With the current command, process 1 was sh and Redis was its child. Redis logged neither receipt of SIGTERM nor a final RDB save. With exec, Redis was process 1, logged SIGTERM and its final RDB snapshot, and retained the newest write. Original tenant workloads were not restarted or changed by this reproduction; temporary resources were removed.
Expected behavior and correction
Kubernetes sends the stop signal to process 1. With RDB save points configured, Redis performs its shutdown save when it receives SIGTERM. The ordinary pod-deletion case should therefore preserve the pending writes when sufficient shutdown time is available.
The focused correction is to use exec redis-server /redis-master/redis.conf in both manifests, with regression coverage for signal propagation and a real Redis restart. This leaves the image version, RDB/AOF policy and storage layout unchanged.
This is separate from ACL identity persistence in #181 / apache/openserverless-operator#157. A real crash or SIGKILL can still lose writes since the last RDB snapshot. The experiments used two small keys; the grace period needed to save a production-sized dataset was not measured.
Activity