Repository navigation
CrashLoopBackOff on Kubernetes: DuckDB WAL replay fails on fresh database initialization #36
Description
Environment
- Liwan version: 1.4 (image
ghcr.io/explodingcamera/liwan:1.4, SHAsha256:80c696af40b84abb1a008ee307e297a7722c178e654ab88dee5e953e2f93f661) - DuckDB version: 1.5.0 (bundled in liwan 1.4)
- Kubernetes: k3s v1.33.6+k3s1
- Container runtime: containerd 2.1.5-k3s1.33
- Host OS: Debian GNU/Linux 13 (trixie)
- Kernel: 6.12.38+deb13-cloud-amd64
- CPU: AMD EPYC with AVX2 support
- Filesystem: ext4
- Storage: local-path provisioner (k3s default), bind-mount into pod
Problem
Liwan crashes immediately (<1 second) on every startup inside a Kubernetes pod,
even on a completely clean /data directory. The error is:
WARN liwan::app::db: Failed to create DuckDB connection. If you've just upgraded
to Liwan 1.2, please downgrade to version 1.1.1 first, start and stop the server,
and then upgrade to 1.2 again.
Error: Failed to create DuckDB connection: INTERNAL Error: Failure while replaying
WAL file "/data/liwan-events.duckdb.wal": Calling DatabaseManager::GetDefaultDatabase
with no default database set
This error signals an assertion failure within DuckDB.
DuckDB creates the WAL file on fresh init, then immediately fails to replay it as
part of the initial checkpoint. The result is a CrashLoopBackOff with a deterministic
4494-byte WAL file created every time.
What We Tested
| Scenario | Result |
|---|---|
ctr run with overlay filesystem (no volume) |
✅ Works |
ctr run with bind-mount to the same PVC path (clean dir) |
✅ Works |
| Kubernetes pod, clean PVC, default security context | ❌ Crashes |
Kubernetes pod + seccompProfile: Unconfined |
❌ Crashes |
Kubernetes pod + capabilities: add: ["ALL"] |
❌ Crashes |
Kubernetes pod + runAsUser: 0 |
❌ Crashes |
| Kubernetes pod + memory limit 512Mi | ❌ Crashes |
Kubernetes pod + LIWAN_DUCKDB_THREADS=1 |
❌ Crashes |
The crash is deterministic and always produces the same 4494-byte WAL file before failing.
Additional Findings
LIWAN_LISTENcannot be set via environment variable — causesError: duplicate field 'listen', suggesting the distroless image includes a TOML config withlistenalready set.LIWAN_BASE_URLmust point to a resolvable domain before the pod starts, otherwise liwan
panics withfailed to lookup address information: Name does not resolve(src/web/mod.rs:143).- DuckDB 1.5.1 release notes mention a fix for "WAL corruption related to
MarkBlockAsCheckpointed on fresh database initialization", which matches this exact failure.
Liwan 1.4 bundles DuckDB 1.5.0.
Suspected Root Cause
DuckDB 1.5.0 bug in fresh database initialization, fixed in DuckDB 1.5.1. The bug surfaces
specifically in the Kubernetes pod execution context (possibly related to cgroup constraints or
subtle runtime differences vs. bare ctr run). Upgrading the bundled DuckDB to ≥1.5.1 would
likely fix the issue.
Workaround
None found. The application is currently not usable on Kubernetes with liwan 1.4.
Activity
explodingcamera commentedon Mar 25, 2026
Hi, thanks for report this! I sadly can't reproduce your issues with k3s on my local machine, but I'll probably push an update with duckdb 1.5.1 tomorrow.
The
duplicate fieldis most likely due to bothportandlistenbeing set somewhere in a config or env, I'll also include a fix to allow this in the next update by prioritizinglisten.As for the BASE_URL error, do you maybe have
listenset to a domain + port? Thefailed to lookup address information: Name does not resolveshould be coming from a listen address that can't be resolved to an ip address.marcochiodo commentedon Mar 26, 2026
Thanks for the quick response!
To clarify the
BASE_URLpanic: we did not haveLIWAN_LISTENset at all when that error occurred. Our only environment variable wasLIWAN_BASE_URL=mydomain.com. The listen variable was added later in an attempt to explicitly bind to0.0.0.0:9042— which is when we hit the duplicate field error.So it seems like liwan may be using the hostname from
LIWAN_BASE_URLas the listen address, or the config embedded in the distroless image has a non-standard default for listen that references the base URL. Either way, the panic atsrc/web/mod.rs:143was triggered purely byLIWAN_BASE_URLbeing set, with no explicit listen configuration on our side.Looking forward to the DuckDB 1.5.1 update — we'll test it as soon as it's out!
marcochiodo commentedon Mar 26, 2026
Would you be able to push a new image tag to GHCR with the DuckDB 1.5.1 fix before the official release? Even a 1.4.1 or edge tag would be enough for us to test it. Thanks!
explodingcamera commentedon Mar 26, 2026
@marcochiodo I just pushed a new image if you want to try it out (
edge/1.4.1-rc.0)marcochiodo commentedon Mar 26, 2026
Just tested edge/1.4.1-rc.0 — great progress!
One remaining issue: the pod still crashes on startup without explicitly setting
LIWAN_LISTEN=0.0.0.0:9042. The default listen address from the embedded config resolves to the node's external IP (193.203.15.59), which can't be bound from inside a container. SettingLIWAN_LISTEN=0.0.0.0:9042via environment variable fixes it.Overall liwan is now working on Kubernetes. Thanks for the quick turnaround!
explodingcamera commentedon May 10, 2026
Fixed in v1.5.0