[polylane]

The E2E environment step sometimes failed while it registered cell clusters with the Karmada API server. The harness tunes the Karmada control plane by restarting it, then reads and writes through an API server that is still coming back, and the final write is not retried. Provisioning now waits for the control plane to finish starting before it registers cells.

flowchart TD
  A["e2e:karmada:install: helm install + tuning patches"] --> B["patch controller / etcd / apiserver pod templates; delete etcd-0"]
  B --> C["Karmada control plane starts rolling out"]
  D["wait --for=condition=Available returns from OLD pod"] --> E["e2e:karmada:configure (retry 60s)"]
  D --> F["e2e:karmada:join-clusters"]
  F --> G["patch-cluster-secret.sh: un-retried apply to 127.0.0.1:32443"]
  C -.-> H["API server unreachable: connection reset by peer / EOF"]
  H --> G
  H --> E
  G --> I["provisioning fails, run fails"]
  J["Fix: wait for rollout status of karmada deployments + etcd StatefulSet"] -.-> D

Loading

Why this fix

Five of the last 100 main E2E runs failed in Provision Kind + Karmada environment, each with the Karmada API server at 127.0.0.1:32443 unreachable (connection reset by peer / EOF); the other ten failures were in Run Chainsaw e2e suites, a separate class. The tuning steps patch the Karmada pod templates (disable-leader-election, disable-etcd-fsync deleting etcd-0, the API-server probe patch), so a rollout is in flight. The gate kubectl wait --for=condition=Available deploy --all returns from the old pod, so the harness configured the API and registered cells while the API server was restarting, and the un-retried patch-cluster-secret.sh failed on the first dropped connection. Waiting for the rollout matches the gate's stated intent, "converge before we build the kubeconfig and register cells".

Fix walkthrough

Watch the fix explained (1:34). Recorded for commit 863ac9021a17836b9277c0b44fe31514f658e01b.

Production impact

CI-only Taskfile change; no deployed resource behavior

The only changed file is Taskfile.yaml, the local E2E harness definition. It runs in the GitHub Actions runner, not in production. The compute repository's only deployment edge is an OCI image built from Dockerfile (ghcr.io/datum-cloud/compute), which this change does not touch. I checked the repository node's edges: it defines that image and is owned by teams, with no runtime resource that reads Taskfile.yaml.

Checked and not affected (1)

• compute

Observability

• The changed step is visible in the E2E Tests workflow under Provision Kind + Karmada environment; on failure the harness uploads e2e-diagnostics including karmada-pods.txt.
• The new waits emit kubectl rollout status output and the final line Karmada control plane is Available and rolled out.

Polylane assessed this pull request at 863ac90.

Causal chain

• Signal (alert): CI failing: E2E Tests on datum-cloud/compute@main
• Surfacing site: GitHub Actions E2E Tests workflow on datum-cloud/compute at datum-cloud/compute:.github/workflows/test-e2e.yml#Provision Kind + Karmada environment
• Mechanism: e2e:karmada:install rewrites the Karmada control-plane pod templates (leader-election, etcd fsync, apiserver probes) so the karmada-apiserver and etcd roll out. The gate kubectl wait --for=condition=Available deploy --all returns from the OLD pod, so the harness proceeds to e2e:karmada:configure and _e2e:karmada:join-cluster while the API server is restarting. The join step's patch-cluster-secret.sh runs an un-retried kubectl apply against the NodePort 127.0.0.1:32443 and fails with 'read: connection reset by peer' (or openapi 'EOF').
• Producer: compute e2e provisioning harness (Taskfile e2e:karmada:install readiness gate), at datum-cloud/compute:Taskfile.yaml#e2e:karmada:install
• What happens to the failed unit today: e2e:karmada:configure retries the federation apply 6x/10s and joins retry 5x/10s; patch-cluster-secret.sh and the cluster label step have no retry
• Cadence check: 5 provisioning failures in the last 100 main E2E runs (2026-10-07 to 2026-11-04)
• Blast radius: 0 other resource(s), unknown other tenant(s); data at risk: none named
• Producer evidence:
  • Taskfile.yaml e2e:karmada:install gates on 'kubectl -n karmada-system wait --for=condition=Available deploy --all --timeout=10m', which is satisfied by the old pod during a rollout
  • Taskfile.yaml _e2e:karmada:disable-etcd-fsync patches karmada-apiserver probe pod template and deletes etcd-0; _e2e:karmada:disable-leader-election patches three controller pod templates - all trigger rollouts just before join
  • Run 41745760160 log: karmadactl join succeeds at 00:21:15, patch-cluster-secret.sh apply gets 'read tcp 127.0.0.1:47556->127.0.0.1:32443: read: connection reset by peer' at 00:21:16
  • Runs 3150, 2954, 3025, 3023 failed in the same provisioning step on the same Karmada API server becoming unreachable

1 file changed (+17/-2)

• Taskfile.yaml: modified, +17/-2

Repository lint: make lint (declared in Makefile) did not finish within its time budget, so its verdict is not known; run it before merging.

---

Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment.