0.9.0 operator: wait for the private SeaweedFS Filer before resume-time IAM reconciliation #286

Problem

On the 0.9.0 operator, resume-time SeaweedFS IAM reconciliation can run before the private Filer HTTP listener is available. The SeaweedFS StatefulSet currently has neither a startup probe nor a readiness probe, so Kubernetes can report Ready before the service is usable.

The fresh-install path already waits for the private Filer. The existing-installation seaweedfs_deploy.reconcile() path immediately runs anonymous-grant reconciliation and persisted-policy verification.

Observed evidence

On an existing single-node K3s installation inside WSL2, the retained startup logs show this sequence (CET, 2026-11-08):

• 11:20:33: SeaweedFS is marked Ready.
• 11:20:54.743: whisk_resume fails in SeaweedfsClient.verify_anonymous_access() with RuntimeError: Could not verify anonymous S3 web-only grants.
• 11:20:57.431: SeaweedFS starts its Filer HTTP listener on port 9090.
• 11:22:04.975: the subsequent whisk_resume attempt succeeds.

Current private Filer readback independently confirms the expected bucket-scoped anonymous grants. The original exception suppresses its underlying request error, so the precise failed request type/status cannot be recovered. This timing is strong evidence of a startup-order race for the observed episode; no controlled restart has yet been performed to reproduce it.

The relevant live source files match operator commit 93fc94ac2f69a1f52bd36e09e76344023460260b on 0.9.0. The workload uses SeaweedFS 3.97. An earlier clean laboratory installation did not reproduce this exception.

Proposed minimum correction

1. Wait for actual private Filer HTTP availability before resume-time IAM operations, with bounded retry and recovery of a port-forward lost during startup.
2. Add suitable SeaweedFS startup/readiness probes for new installations.
3. Reconcile missing probes on existing StatefulSets without replacing their image, command, environment, storage or custom probes; wait for the resulting rollout before using the service.
4. Keep persisted IAM verification fail-closed and propagate actual policy/bootstrap failures. Do not turn an IAM mismatch into a successful readiness retry.

Acceptance

• Delayed/unavailable Filer does not trigger premature IAM changes or ingress publication.
• A ready Filer allows both bootstrap and resume; an actual IAM failure remains an error.
• Existing installation settings and PVCs are preserved; applying the correction again does not restart an unchanged workload.
• Local regression tests, isolated 0.9.0 laboratory validation, then an operator-authorized WSL validation before a PR.