Skip to content

feat(fleet): fork isolated twin environments with database cloning and teardown (Phase 2A) - #24

Merged
Harikeshav-R merged 7 commits into
mainfrom
feat/phase-2a-fleet
Sep 13, 2026
Merged

Harikeshav-R merged 7 commits into
mainfrom
feat/phase-2a-fleet

Conversation

@Harikeshav-R

Copy link
Copy Markdown
Owner

Build-Plan Step Implemented

  • Step: Phase 2A — A2.1 through A2.5 (fleet/k8s.py, fleet/render.py, fleet/database.py, fleet/controller.py, fleet/teardown.py)

Checkpoint Execution & Output

Checkpoint A2, run against the live k3d-ust cluster. The curl localhost:8090 /
localhost:8080 lines in the build plan need port-forwards, so the equivalent counts are
read directly from twin-postgres and prod-postgres; everything else is the documented
command. The egress check now carries two positive controls, because "the connection
failed" is also what a missing binary or an unreachable target looks like.

$ ust fleet fork --incident inc_test --count 3
twin_id=twin_inc_test_0 namespace=ust-twin-inc-test-0 database=twin_inc_test_0 state=ready
twin_id=twin_inc_test_1 namespace=ust-twin-inc-test-1 database=twin_inc_test_1 state=ready
twin_id=twin_inc_test_2 namespace=ust-twin-inc-test-2 database=twin_inc_test_2 state=ready
$ kubectl get ns -l understudy.dev/incident=inc_test
NAME                  STATUS   AGE
ust-twin-inc-test-0   Active   20s
ust-twin-inc-test-1   Active   20s
ust-twin-inc-test-2   Active   20s

$ kubectl get pods -n ust-twin-inc-test-0
NAME                            READY   STATUS    RESTARTS   AGE
auth-service-6d9c46b6d6-mhxcr   1/1     Running   0          20s
data-service-784c9579cc-hgw6t   1/1     Running   0          20s
edge-gateway-57d588bc87-k5hkh   1/1     Running   0          20s
worker-579c977465-d6z9n         1/1     Running   0          20s

$ psql -h twin-postgres -c "\l" | grep twin_   (via kubectl exec)
twin_inc_test_0
twin_inc_test_1
twin_inc_test_2

# data isolation: twin item count vs PRODUCTION item count
prod  (ust_prod on prod-postgres.ust-prod): 207
twin  (twin_inc_test_0 on twin-postgres):    207

# write isolation: insert into the twin, then re-count production
INSERT 0 1
twin after write:  208
prod after write:  207
prod sees 'twin-only': 0

# egress denial, with both positive controls
control a) twin  -> auth-service:8000 (in-namespace):   OK
control b) prod  -> auth-service.ust-prod:8000:         OK
assert     twin  -> auth-service.ust-prod:8000:         BLOCKED:ConnectionRefusedError
assert     twin  -> prod-postgres.ust-prod:5432:        BLOCKED:ConnectionRefusedError
dns still resolves from the twin:                      10.43.49.93

$ ust fleet teardown --incident inc_test
fleet torn down for incident=inc_test

$ kubectl get ns -l understudy.dev/incident=inc_test
NAME                  STATUS        AGE
ust-twin-inc-test-0   Terminating   56s
ust-twin-inc-test-1   Terminating   56s
ust-twin-inc-test-2   Terminating   56s

$ psql ... | grep twin_
0
  (a few seconds later, after Kubernetes finishes finalizing)
ust-twin-inc-test-1   Terminating   63s
ust-twin-inc-test-2   Terminating   63s
  (polled until gone) all twin namespaces deleted

make check: 428 passed, 100% line and branch coverage, ruff + mypy --strict +
import-linter + mock parity clean.
pytest -m integration tests/integration: 7 passed.

ADRs Relied On

  • ADR-008 — forking is a namespace clone at pinned digests. Digests are now taken from
    the Deployment's current ReplicaSet; when no running pod can supply an imageID the
    snapshot degrades to the spec tag and records digest_pinned=False rather than failing
    the fork, so an all-Pending prod Deployment (the incident this project exists to
    remediate) can still be twinned. The weaker pinning is recorded, never hidden.
  • ADR-009 — never hold a connection to snapshot_template. The staging-and-rename
    pattern is unchanged; the refresh pipeline now fails loudly rather than renaming an empty
    staging database over the template.
  • ADR-014 — no direct egress from twins; the NetworkPolicy containment is now proven
    rather than assumed (invariant K6).

Mock Registry Disclosures

None. No # MOCKED: markers added or modified; scripts/check_mocks.py is green.

Documentation Updates

  • docs/04-build-plan.md — the Checkpoint A2 command was unrunnable in two ways: it
    expected the namespace ust-twin-inc_test-0, which the Kubernetes API server cannot
    create, and it probed egress with curl, which is absent from python:3.12-slim, against
    edge-gateway.ust-prod:8000, a port that Service does not publish (it publishes 8080).
  • CHECKLIST.md — A2 checkpoint boxes re-confirmed against what was actually observed,
    with the fixes enumerated.

Definition of Done

  • Implementation complete for the named build-plan step
  • Unit tests written; 100% line and branch coverage of the added lines
  • make check green (ruff, mypy --strict, import-linter, pytest)
  • Checkpoint command run and output matches, OR marked unverified with a reason
  • Docs updated where this change made them untrue (or "none required", with why)
  • No new # MOCKED: marker without a row in docs/MOCKS.md
  • Conventional commit(s), conventional branch, PR opened, never pushed to main

Harikeshav-R and others added 6 commits September 12, 2026 20:16
Extract Kubernetes workloads from production namespaces, resolving container
image tags to pinned sha256 digests from live pod container statuses. Extract
container resource requests and limits, environment variables (literal and
valueFrom), and referenced ConfigMaps across env and volume mounts.

Provides K8sWorkloadReader and FakeWorkloadReader conforming to the new
WorkloadReader protocol, maintaining deterministic snapshots for testing
and enabling namespace cloning in twin environments per ADR-008.

Implements build-plan step A2.1.
Generate Kubernetes manifests for isolated twin environments in
ust-twin-<incident>-<candidate>. Enforces twin egress isolation via
twin NetworkPolicy, ServiceAccount/RoleBinding RBAC, UNDERSTUDY_ROLE=twin
role injection, database DSN redirection to twin-postgres, and external
URL redirection to egress-stub while preserving intra-twin service DNS.

Implements build-plan step A2.2.
Implement the PostgreSQL snapshot refresher and isolated twin database
cloner for twin environments per ADR-009.

The snapshot refresher dumps the live ust_prod database from prod-postgres
into snapshot_staging on twin-postgres and swaps to snapshot_template via
rename, ensuring no persistent connections are ever held against the template.

The database cloner issues CREATE DATABASE ... TEMPLATE snapshot_template
to provide per-twin filesystem-level write isolation with sub-second fork times,
retrying on PostgreSQL concurrency error 55006 ("source database is being
accessed by other users") with exponential backoff and backend termination.

All database interactions execute inside the twin-postgres container
environment without introducing psycopg imports into fleet, keeping the
architectural import boundary intact.

Implements build-plan step A2.3.
Provide the Kubernetes-backed fleet controller and teardown manager implementing the FleetController protocol for isolated twin environments.

The controller forks N twin environments concurrently per incident, rendering isolated manifests with redirected DSNs, cloning template databases on twin-postgres, waiting up to 120s for pod readiness, and asserting that twin NetworkPolicy is enforced before returning Ready twin handles (K6).

The teardown manager provides idempotent, label-based cleanup of all twin namespaces and databases for an incident. Background and on-demand garbage collection reaps stale namespaces and orphaned databases via `ust fleet gc`. Process-level atexit and SIGTERM signal handlers ensure leaked twin resources are cleaned up upon unexpected process termination.

Implements build-plan steps A2.4 and A2.5.
A code review of build-plan steps A2.1-A2.5 found fourteen defects, three of
them blocking. Every fork failed for a real `inc_<ulid>` incident ID because
twin namespaces were rendered with the underscore the Kubernetes API server
rejects, and the snapshot refresher could silently publish an empty
`snapshot_template` because a piped `pg_dump` failure was masked by psql's exit
status and psql itself exits 0 without `ON_ERROR_STOP`. Twins cloned from that
template would have broken the "twin item count equals prod" invariant with no
error anywhere.

- render: sanitize namespaces to DNS-1123, mirroring what
  `sanitize_database_name` already does for PostgreSQL. The
  `understudy.dev/incident` label keeps the raw ID, so teardown and GC
  selectors are unchanged.
- database: `ON_ERROR_STOP=1` on every psql, dump to a file rather than a pipe
  (`/bin/sh` is dash, which has no `pipefail`), probe the restored staging
  database before the rename, and quote the `datname` as a SQL literal so the
  `pg_terminate_backend` safety net before `DROP DATABASE` actually runs.
- k8s: serialize probes and volumes into camelCase manifest form so the public
  renderer output is valid without the controller patching it at apply time;
  translate downwardAPI and projected volumes with their items and modes, and
  refuse PVC or hostPath rather than emitting a sourceless volume.
- k8s: pin digests from the Deployment's current ReplicaSet, since mid-rollout
  both revisions' pods match the selector and the new spec's repo could be
  paired with an old pod's digest. An image no pod can pin now degrades to the
  spec tag with `digest_pinned=False` instead of failing all N twins, because
  an all-Pending prod Deployment is the incident this project exists to fix.
- controller: tear down the twins a partially failed fork already created,
  register the resolved teardown manager with the atexit and SIGTERM handlers
  instead of the usually-None constructor argument, and wait on only the pods
  the fork expects so one evicted leftover cannot burn the 120s timeout.
- teardown: never reap an open incident, and gate the orphan-database sweep on
  both the retention threshold and in-flight clones. Twin databases are stamped
  with their creation time, since a fork clones its database before its
  namespace exists and "no namespace" alone does not mean abandoned.

Fixes build-plan steps A2.1 through A2.5. Relies on ADR-008 (fork is a
namespace clone at pinned digests) and ADR-009 (never hold a connection to the
snapshot template).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mc7qMbFq3dGCiHcF2qNvP8
The live checkpoint assertions could pass without testing anything. The egress
check asserted only that `kubectl exec ... curl` exited non-zero, which is also
what a missing binary, a wrong pod, or an RBAC denial looks like -- and the
service images are `python:3.12-slim`, which ships no curl, so it was almost
certainly passing vacuously. Worse, it probed `edge-gateway.ust-prod:8000`,
but that Service publishes 8080, so the connection fails for everyone: the same
probe times out from a `ust-prod` pod. The fidelity check compared the twin
against `snapshot_template`, making it tautological with respect to the cloner
and unable to detect an empty or stale template.

The probe is now stdlib sockets, always exits 0 so a broken exec fails the
assertion, and runs behind two positive controls: the same probe must succeed
in-namespace, and the same target must be reachable from a `ust-prod` pod.
`auth-service.ust-prod:8000` and `prod-postgres.ust-prod:5432` are both refused
from a twin while DNS still resolves, so the NetworkPolicy is what contains it.
The fidelity baseline now comes from `ust_prod` on `prod-postgres.ust-prod`,
and the write-isolation check asserts against production rather than the
template. The incident ID is underscored like `new_incident_id()` so the
namespace renderer is exercised end to end.

docs/04-build-plan.md carried both errors in the documented Checkpoint A2
command, and CHECKLIST.md records what was actually observed on the live k3d
cluster per AGENTS.md section 2.5.

Verifies build-plan step A2.4. Relies on invariant K6 (twin egress
containment).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mc7qMbFq3dGCiHcF2qNvP8
@Harikeshav-R

Copy link
Copy Markdown
Owner Author

Code review of Phase 2A, and the fixes applied

A review of this branch found 14 defects; all are fixed in 67de9ca and 0b800bd. Three
were blocking, and two were test-integrity problems that meant the A2 checkpoint boxes were
ticked on assertions that could not fail. Grouped by what they would have done in practice.

Blocking — the fleet could not work at all

1. fleet/render.py:56 — every fork failed for a real incident ID.
build_twin_namespace only did incident_id.strip(), so inc_<ulid> from new_incident_id()
produced ust-twin-inc_01m2cd…-0. Namespace names must be DNS-1123 labels, so
create_namespace returned 422, _apply_namespace re-raised (status ≠ 409), and the fork
died. The documented checkpoint command ust fleet fork --incident inc_test failed the same
way. Every unit test either mocked the renderer or never reached the API, so nothing caught it.
→ New sanitize_dns1123_label(), mirroring what sanitize_database_name() already does for
PostgreSQL. The understudy.dev/incident label keeps the raw ID (underscores are legal in
label values), so teardown and GC selectors are untouched.

2. fleet/database.py:189 — the pg_terminate_backend safety net had never run.
WHERE datname = "snapshot_template" uses double quotes, which is an identifier in
PostgreSQL, so the statement failed with column "snapshot_template" does not exist on every
refresh. With no ON_ERROR_STOP, psql's exit status came from the last -c, so the failure
was invisible — meaning stray connections were never killed before DROP DATABASE.

3. fleet/database.py:186 — the refresher could publish an empty snapshot template.
pg_dump … | psql -q takes its status from psql, and psql without ON_ERROR_STOP=1 exits 0
even when every statement of the restore fails. With prod-postgres unreachable, pg_dump
exits 1 and writes nothing, psql exits 0, the && chain proceeds, and the empty staging
database is renamed over snapshot_template. refresh_snapshot then reports success and
updates _last_snapshot_time. Every later twin clones an empty database, breaking "twin item
count equals prod item count" with no error anywhere.
→ ON_ERROR_STOP=1 on every psql, dump to a file instead of a pipe (/bin/sh is dash, no
pipefail), and a SELECT 1 FROM items LIMIT 1 guard against staging before the rename.

Verified live, both directions:

# unreachable source
snapshot_refresh_failed  returncode=1
  stderr='pg_dump: error: could not translate host name "no-such-postgres.ust-prod" …'
correctly failed: Snapshot refresh pipeline failed with exit code 1
template still has items: 207        <- previous template intact, not clobbered
last snapshot time: None             <- no false success recorded

# healthy source
snapshot_refresh_completed  refreshed_at=2026-09-13T04:00:30Z
template items: 207

Correctness

4. fleet/controller.py:208 — a partial fork leaked namespaces, databases, and tasks.
asyncio.gather(*tasks) propagates the first failure but does not cancel its siblings, and
fork had no compensating teardown. Candidate 2 timing out at 120 s left candidates 0 and 1
Ready and billed until the hourly GC, with "Task exception was never retrieved" noise behind
them. → return_exceptions=True plus teardown_incident on the way out.

5. fleet/k8s.py:358 — digests could be read from the wrong pod mid-rollout.
matching_pods selected on the Deployment's matchLabels alone, which both the old and new
ReplicaSet's pods satisfy, then took running_pods[0] in arbitrary API order.
resolve_image_digest combined the new spec's repo with whichever pod was picked, so a twin
could silently run the previous image while the report claimed it pinned current prod — during
a rollout, which is the canonical Understudy incident. → Pods are filtered to the ReplicaSet
carrying the Deployment's current deployment.kubernetes.io/revision, with an image-name
consistency check as the fallback.

6. fleet/k8s.py:398 — an all-Pending prod Deployment aborted the whole fork.
Pending pods have no imageID, so resolve_image_digest raised, failing read_workloads and
therefore all N twins — in exactly the ImagePullBackOff state this project exists to
remediate. → Degrade to the spec's tag and record digest_pinned=False on the snapshot, so
the twin forks and the provenance stays honest rather than silently weaker.

7. fleet/k8s.py:179 — probes were serialized in snake_case.
probe.to_dict() yields {'http_get': …, 'initial_delay_seconds': …}, and _extract_probe
only stripped top-level Nones and leading underscores. K8sFleetController._apply_deployment
patched this at apply time, so the bug was invisible through the controller but live for anyone
using render_twin_manifests + to_yaml() + kubectl apply — the API server rejects the probe
or drops it. → Fixed in the renderer, and the apply-time patch is deleted.

8. fleet/k8s.py:210 — unsupported volumes became sourceless volumes.
Only configMap/secret/emptyDir were handled, so a PVC or projected volume rendered as
{"name": "data"}, which the API server rejects outright. items/defaultMode were also
dropped, so a workload mounting one key got all of them at the wrong paths. → downwardAPI and
projected are translated with their items and modes; PVC and hostPath now fail loudly, since a
twin cannot honestly reproduce persistent storage and an empty substitute would be a fidelity
lie.

Both are now checked end to end — the publicly rendered bundle is accepted by the API server:

$ kubectl apply --dry-run=server -f twin.yaml
namespace/ust-twin-inc-yamlcheck-0 configured (server dry run)
serviceaccount/understudy-twin created (server dry run)
networkpolicy.networking.k8s.io/twin-egress-containment created (server dry run)
configmap/app-config created (server dry run)
service/{auth-service,data-service,edge-gateway,worker} created (server dry run)
deployment.apps/{auth-service,data-service,edge-gateway,worker} created (server dry run)

9. fleet/controller.py:522 — one leftover pod burned the full 120 s.
The deployment-level checks already confirmed ready_replicas/available_replicas, but the
pod loop then iterated every pod in the namespace and failed on the first that was not
Running — including Succeeded/Failed/evicted leftovers and pods still terminating from a
replace_namespaced_deployment. → Readiness waits only on pods matching an expected
deployment's selector, ignoring terminal and terminating ones.

10. fleet/controller.py:99 — the atexit/SIGTERM net ignored injected dependencies.
register_teardown_handlers(self._teardown_manager) passed the raw constructor argument,
normally None, so _GLOBAL_TEARDOWN_MANAGER stayed unset and the handlers built a fresh
FleetTeardownManager() from get_settings() defaults — discarding the injected settings,
core_api, and database_cloner, so cleanup could target the wrong cluster. → Registration
moved into fork(), passing the resolved manager.

Garbage collection could destroy live work

11. fleet/teardown.py:383 — GC reaped active incidents purely by age.
gc() deleted every namespace labelled understudy.dev/incident older than the threshold
without consulting _ACTIVE_INCIDENTS. An incident open longer than an hour — routine for a
human-in-the-loop remediation — had its live twins deleted under the running orchestrator, and
their databases dropped right after.

12. fleet/teardown.py:409 — the orphan-database sweep ignored the age threshold entirely
and raced in-flight forks.
_fork_one_twin clones the database (step 1) before creating the
namespace (step 3). In that window has_active_ns is False, so the sweep dropped the database
regardless of older_than_seconds, which was never consulted in that loop — a concurrent
ust fleet gc destroyed a twin database out from under an active fork. → Twin databases are
stamped with their creation time (COMMENT ON DATABASE), and the sweep now requires no
surviving namespace
and incident inactive and known age ≥ threshold; unknown age
is logged and left alone.

Verified live — with an incident open, a zero-retention sweep touches nothing:

gc reaped namespaces: []
gc dropped databases: []
namespace still alive: ust-twin-inc-gclive-0 Active
databases still present: ['twin_inc_gclive_0']
second gc dropped: []                      <- database cloned before its namespace existed
inflight db present: True
after teardown, databases: []

The checkpoint was passing on assertions that could not fail

13. tests/integration/fleet/test_controller_live.py:144 — the K6 egress assertion was
vacuous, twice over.
It asserted only egress_check.returncode != 0, which is equally
satisfied by curl missing from the image, a wrong pod, or an RBAC denial — and the service
images are python:3.12-slim, which ships no curl, so it was almost certainly passing
without ever opening a socket. While fixing it I found a second layer: the target was
edge-gateway.ust-prod:8000, but that Service publishes 8080 (targetPort 8000), so the
connection fails for everyone — I confirmed it also times out from a ust-prod pod. Swapping
curl for a socket probe would have kept the box green while still proving nothing.
→ The probe is stdlib sockets, always exits 0 so a broken kubectl exec fails the assertion,
and runs behind two positive controls: the same probe must succeed in-namespace, and the same
target must be reachable from a ust-prod pod.

control a) twin  -> auth-service:8000 (in-namespace):   OK
control b) prod  -> auth-service.ust-prod:8000:         OK
assert     twin  -> auth-service.ust-prod:8000:         BLOCKED:ConnectionRefusedError
assert     twin  -> prod-postgres.ust-prod:5432:        BLOCKED:ConnectionRefusedError
dns still resolves from the twin:                       10.43.49.93

DNS resolving while the connection is refused is what makes this a NetworkPolicy result rather
than a name-resolution accident. K6 now holds on evidence.

14. test_controller_live.py:102 — twin fidelity was compared against the template, not
prod.
prod_count was read from snapshot_template on twin-postgres, making the check
tautological with respect to the cloner and structurally unable to detect finding #3. The
CHECKLIST item it certified says "equals prod item count". → The baseline now comes from
ust_prod on prod-postgres.ust-prod, and write isolation is asserted against production
instead of the template.

Notes for review

🤖 Generated with Claude Code

`test_readiness_polling_api_error` constructed the controller with only
`apps_api` injected, so `_wait_for_readiness` resolved the lazy `core_api`
property and built a real Kubernetes client. On a developer machine that
succeeds against the local kubeconfig and the test passes; CI has no kubeconfig,
so it failed with `ConfigException: Invalid kube-config file`. The defect
predates the code-review fixes -- the same eager `self.core_api` lookup is in
the original polling loop -- and only surfaced now because this branch had never
been pushed, so CI had never run these tests.

Inject a fake `core_api`, and add `tests/unit/conftest.py` to make both config
loaders raise. This mirrors what `tests/conftest.py` already does for `.env` and
secrets: a unit test must not silently pick up developer-local state. The guard
is scoped to `tests/unit`, since the live integration tests legitimately need a
real cluster, and it turns a CI-only `ConfigException` into an immediate local
failure naming the client to inject. Tests that deliberately exercise the client
factories patch the loaders themselves, which wins inside their `with` block.

Verified with `make check` both with a kubeconfig present and with
`KUBECONFIG` pointing nowhere and a scratch `HOME`: 428 passed, 100% coverage.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Mc7qMbFq3dGCiHcF2qNvP8
@Harikeshav-R
Harikeshav-R merged commit 19831be into main Sep 13, 2026
2 checks passed
@Harikeshav-R
Harikeshav-R deleted the feat/phase-2a-fleet branch September 13, 2026 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant