Repository navigation
feat(fleet): fork isolated twin environments with database cloning and teardown (Phase 2A) - #24
Conversation
Extract Kubernetes workloads from production namespaces, resolving container image tags to pinned sha256 digests from live pod container statuses. Extract container resource requests and limits, environment variables (literal and valueFrom), and referenced ConfigMaps across env and volume mounts. Provides K8sWorkloadReader and FakeWorkloadReader conforming to the new WorkloadReader protocol, maintaining deterministic snapshots for testing and enabling namespace cloning in twin environments per ADR-008. Implements build-plan step A2.1.
Generate Kubernetes manifests for isolated twin environments in ust-twin-<incident>-<candidate>. Enforces twin egress isolation via twin NetworkPolicy, ServiceAccount/RoleBinding RBAC, UNDERSTUDY_ROLE=twin role injection, database DSN redirection to twin-postgres, and external URL redirection to egress-stub while preserving intra-twin service DNS. Implements build-plan step A2.2.
Implement the PostgreSQL snapshot refresher and isolated twin database
cloner for twin environments per ADR-009.
The snapshot refresher dumps the live ust_prod database from prod-postgres
into snapshot_staging on twin-postgres and swaps to snapshot_template via
rename, ensuring no persistent connections are ever held against the template.
The database cloner issues CREATE DATABASE ... TEMPLATE snapshot_template
to provide per-twin filesystem-level write isolation with sub-second fork times,
retrying on PostgreSQL concurrency error 55006 ("source database is being
accessed by other users") with exponential backoff and backend termination.
All database interactions execute inside the twin-postgres container
environment without introducing psycopg imports into fleet, keeping the
architectural import boundary intact.
Implements build-plan step A2.3.
Provide the Kubernetes-backed fleet controller and teardown manager implementing the FleetController protocol for isolated twin environments. The controller forks N twin environments concurrently per incident, rendering isolated manifests with redirected DSNs, cloning template databases on twin-postgres, waiting up to 120s for pod readiness, and asserting that twin NetworkPolicy is enforced before returning Ready twin handles (K6). The teardown manager provides idempotent, label-based cleanup of all twin namespaces and databases for an incident. Background and on-demand garbage collection reaps stale namespaces and orphaned databases via `ust fleet gc`. Process-level atexit and SIGTERM signal handlers ensure leaked twin resources are cleaned up upon unexpected process termination. Implements build-plan steps A2.4 and A2.5.
A code review of build-plan steps A2.1-A2.5 found fourteen defects, three of them blocking. Every fork failed for a real `inc_<ulid>` incident ID because twin namespaces were rendered with the underscore the Kubernetes API server rejects, and the snapshot refresher could silently publish an empty `snapshot_template` because a piped `pg_dump` failure was masked by psql's exit status and psql itself exits 0 without `ON_ERROR_STOP`. Twins cloned from that template would have broken the "twin item count equals prod" invariant with no error anywhere. - render: sanitize namespaces to DNS-1123, mirroring what `sanitize_database_name` already does for PostgreSQL. The `understudy.dev/incident` label keeps the raw ID, so teardown and GC selectors are unchanged. - database: `ON_ERROR_STOP=1` on every psql, dump to a file rather than a pipe (`/bin/sh` is dash, which has no `pipefail`), probe the restored staging database before the rename, and quote the `datname` as a SQL literal so the `pg_terminate_backend` safety net before `DROP DATABASE` actually runs. - k8s: serialize probes and volumes into camelCase manifest form so the public renderer output is valid without the controller patching it at apply time; translate downwardAPI and projected volumes with their items and modes, and refuse PVC or hostPath rather than emitting a sourceless volume. - k8s: pin digests from the Deployment's current ReplicaSet, since mid-rollout both revisions' pods match the selector and the new spec's repo could be paired with an old pod's digest. An image no pod can pin now degrades to the spec tag with `digest_pinned=False` instead of failing all N twins, because an all-Pending prod Deployment is the incident this project exists to fix. - controller: tear down the twins a partially failed fork already created, register the resolved teardown manager with the atexit and SIGTERM handlers instead of the usually-None constructor argument, and wait on only the pods the fork expects so one evicted leftover cannot burn the 120s timeout. - teardown: never reap an open incident, and gate the orphan-database sweep on both the retention threshold and in-flight clones. Twin databases are stamped with their creation time, since a fork clones its database before its namespace exists and "no namespace" alone does not mean abandoned. Fixes build-plan steps A2.1 through A2.5. Relies on ADR-008 (fork is a namespace clone at pinned digests) and ADR-009 (never hold a connection to the snapshot template). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mc7qMbFq3dGCiHcF2qNvP8
The live checkpoint assertions could pass without testing anything. The egress check asserted only that `kubectl exec ... curl` exited non-zero, which is also what a missing binary, a wrong pod, or an RBAC denial looks like -- and the service images are `python:3.12-slim`, which ships no curl, so it was almost certainly passing vacuously. Worse, it probed `edge-gateway.ust-prod:8000`, but that Service publishes 8080, so the connection fails for everyone: the same probe times out from a `ust-prod` pod. The fidelity check compared the twin against `snapshot_template`, making it tautological with respect to the cloner and unable to detect an empty or stale template. The probe is now stdlib sockets, always exits 0 so a broken exec fails the assertion, and runs behind two positive controls: the same probe must succeed in-namespace, and the same target must be reachable from a `ust-prod` pod. `auth-service.ust-prod:8000` and `prod-postgres.ust-prod:5432` are both refused from a twin while DNS still resolves, so the NetworkPolicy is what contains it. The fidelity baseline now comes from `ust_prod` on `prod-postgres.ust-prod`, and the write-isolation check asserts against production rather than the template. The incident ID is underscored like `new_incident_id()` so the namespace renderer is exercised end to end. docs/04-build-plan.md carried both errors in the documented Checkpoint A2 command, and CHECKLIST.md records what was actually observed on the live k3d cluster per AGENTS.md section 2.5. Verifies build-plan step A2.4. Relies on invariant K6 (twin egress containment). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mc7qMbFq3dGCiHcF2qNvP8
Code review of Phase 2A, and the fixes appliedA review of this branch found 14 defects; all are fixed in 67de9ca and 0b800bd. Three Blocking — the fleet could not work at all1. 2. 3. Verified live, both directions: Correctness4. 5. 6. 7. 8. Both are now checked end to end — the publicly rendered bundle is accepted by the API server: 9. 10. Garbage collection could destroy live work11. 12. Verified live — with an incident open, a zero-retention sweep touches nothing: The checkpoint was passing on assertions that could not fail13. DNS resolving while the connection is refused is what makes this a NetworkPolicy result rather 14. Notes for review
🤖 Generated with Claude Code |
`test_readiness_polling_api_error` constructed the controller with only `apps_api` injected, so `_wait_for_readiness` resolved the lazy `core_api` property and built a real Kubernetes client. On a developer machine that succeeds against the local kubeconfig and the test passes; CI has no kubeconfig, so it failed with `ConfigException: Invalid kube-config file`. The defect predates the code-review fixes -- the same eager `self.core_api` lookup is in the original polling loop -- and only surfaced now because this branch had never been pushed, so CI had never run these tests. Inject a fake `core_api`, and add `tests/unit/conftest.py` to make both config loaders raise. This mirrors what `tests/conftest.py` already does for `.env` and secrets: a unit test must not silently pick up developer-local state. The guard is scoped to `tests/unit`, since the live integration tests legitimately need a real cluster, and it turns a CI-only `ConfigException` into an immediate local failure naming the client to inject. Tests that deliberately exercise the client factories patch the loaders themselves, which wins inside their `with` block. Verified with `make check` both with a kubeconfig present and with `KUBECONFIG` pointing nowhere and a scratch `HOME`: 428 passed, 100% coverage. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Mc7qMbFq3dGCiHcF2qNvP8
Build-Plan Step Implemented
fleet/k8s.py,fleet/render.py,fleet/database.py,fleet/controller.py,fleet/teardown.py)Checkpoint Execution & Output
Checkpoint A2, run against the live
k3d-ustcluster. Thecurl localhost:8090/localhost:8080lines in the build plan need port-forwards, so the equivalent counts areread directly from
twin-postgresandprod-postgres; everything else is the documentedcommand. The egress check now carries two positive controls, because "the connection
failed" is also what a missing binary or an unreachable target looks like.
make check: 428 passed, 100% line and branch coverage, ruff + mypy --strict +import-linter + mock parity clean.
pytest -m integration tests/integration: 7 passed.ADRs Relied On
the Deployment's current ReplicaSet; when no running pod can supply an imageID the
snapshot degrades to the spec tag and records
digest_pinned=Falserather than failingthe fork, so an all-Pending prod Deployment (the incident this project exists to
remediate) can still be twinned. The weaker pinning is recorded, never hidden.
snapshot_template. The staging-and-renamepattern is unchanged; the refresh pipeline now fails loudly rather than renaming an empty
staging database over the template.
rather than assumed (invariant K6).
Mock Registry Disclosures
None. No
# MOCKED:markers added or modified;scripts/check_mocks.pyis green.Documentation Updates
docs/04-build-plan.md— the Checkpoint A2 command was unrunnable in two ways: itexpected the namespace
ust-twin-inc_test-0, which the Kubernetes API server cannotcreate, and it probed egress with
curl, which is absent frompython:3.12-slim, againstedge-gateway.ust-prod:8000, a port that Service does not publish (it publishes 8080).CHECKLIST.md— A2 checkpoint boxes re-confirmed against what was actually observed,with the fixes enumerated.
Definition of Done
make checkgreen (ruff, mypy --strict, import-linter, pytest)# MOCKED:marker without a row in docs/MOCKS.md