Skip to content

Feature: Teardown step for all tests #606

Description

@soenkeliebau

Problem

The dominant, platform-independent cause of the IT teardown-timeouts is a Kubernetes
namespace-deletion race: when kuttl runs kubectl delete namespace, the StatefulSet controller
recreates a Pod that GC removed while its StatefulSet was still active. The recreated Pod wedges on
generic-ephemeral-volume PVC ownership and/or the listener/secret CSI mount and holds
kubernetes.io/pvc-protection, so the namespace stays Terminating until the step times out.

The fix that removes this race is to stop the StatefulSets before the namespace is
deleted - but adding that step to every test by hand is unnecessary overhead and we want to avoid this.

Proposal

Add a single shared teardown step (via the beku shared/ directory from #1) that runs before
kuttl deletes the namespace:

  1. Delete all SDP custom resources in the namespace dynamically —
    kubectl api-resources --verbs=delete --namespaced -o name | grep '\.stackable\.tech' → delete
    each. Product-agnostic; no per-operator hardcoding.
  2. Wait for the StatefulSets to be gone (so nothing is recreated during namespace deletion).
  3. Force-delete any lingering Pods (--grace-period=0 --force) so we don't wait out long termination
    grace periods.

depends on stackabletech/beku.py#40

Activity

  1. razvan commented on Jul 9, 2026

    @razvan
    Member

    kubectl api-resources --verbs=delete --namespaced -o name | grep '.stackable.tech' → delete

    If zookeeper is deleted first, deleting hdfs, hbase, etc. is going to take a lot longer and we might still run into timeouts

  2. soenkeliebau commented on Jul 9, 2026

    @soenkeliebau
    MemberAuthor

    kubectl api-resources --verbs=delete --namespaced -o name | grep '.stackable.tech' → delete

    If zookeeper is deleted first, deleting hdfs, hbase, etc. is going to take a lot longer and we might still run into timeouts

    I'll create a PR with the script in a second, then we can discuss there. If we can come up with a sensible order to do things in, that can be added to the script of course. Otherwise I'd say after the wait is expired the force delete of all pods should catch those stragglers?

  3. soenkeliebau commented on Jul 9, 2026

    @soenkeliebau
    MemberAuthor

    PR with teardown script up: #607

  4. soenkeliebau commented on Aug 27, 2026

    @soenkeliebau
    MemberAuthor

    closing this due to moving away from beku and kuttl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions