Repository navigation
feat(deploy,services): implement rbac policies, twin network policy, and loadgen (A1.7-A1.8) - #6
Merged
Harikeshav-R merged 24 commits intoSep 11, 2026
Conversation
Enforce the production namespace boundary and twin egress containment per ADR-002, ADR-014, and safety invariant K6. - deploy/policies/understudy-prod.yaml defines ServiceAccount, Role, and RoleBinding strictly limited to Deployments and ConfigMaps in ust-prod. - deploy/policies/understudy-twin.yaml defines ClusterRole understudy-twin with full permissions to be bound within dynamic twin namespaces. - deploy/policies/twin-network-policy.yaml and its template specify default-deny egress for twin pods while allowing intra-namespace, kube-system DNS, and ust-system backing infrastructure. - Unit and integration tests validate manifest structure, RBAC scoping, and kubectl dry-run acceptance. Implements build-plan step A1.7.
Provide reproducible evaluation traffic against the demo stack per ADR-015 and verify all Checkpoint A1 assertions. - services/loadgen/ implements SeededRequestGenerator using random.Random to output a deterministic 80/20 mix of GET/POST /api/items requests. - Async runner paces dispatches, measures response latencies via high- resolution timer, computes percentiles, and outputs the summary line. - Makefile adds loadgen target parameterized by RPS, DURATION, SEED, and TARGET_URL, and applies RBAC policies in deploy-prod. - Unit tests achieve 100% line and branch coverage across services/loadgen. - Verified Checkpoint A1 assertions on live k3d cluster and updated CHECKLIST.md. Implements build-plan step A1.8.
rsanghvi21
approved these changes
Sep 11, 2026
rsanghvi21
left a comment
Collaborator
There was a problem hiding this comment.
Code Review — Candidate Findings (PR #6)
Correctness
- Auth bypass — services/auth_service/app.py:87 accepts any Bearer valid-* / Bearer token-* token as valid, regardless of actual validity.
- Auth cache eviction — TOKEN_CACHE's FIFO eviction (app.py:90) can downgrade the seeded admin token's scope once evicted and re-validated through the same bypass branch.
- Dead RBAC grant — no deploy/prod/*.yaml Deployment sets serviceAccountName: understudy-prod, so the Role/RoleBinding in understudy-prod.yaml never actually applies to any running pod.
- Worker fault-injection gap — worker_loop never calls pre_request_hook, so LATENCY/ERROR_RATE/CPU_SPIN fault injection is a no-op against job processing (POOL_EXHAUSTION is a partial exception).
- Race condition in faults.py — apply_fault/clear_fault mutate shared state without synchronization across await points.
- Magnitude ambiguity at exactly 1.0 in faults.py — behavior is intentional but undocumented.
- Dead deploy/prod/configmap.yaml — not consumed by any Deployment.
- Shared Prometheus registry across services — a latent test-pollution mechanism (not yet observed failing, but real).
Quality / cleanup
9. Unjustified # type: ignore (x2) in edge_gateway/app.py; bare except Exception in loadgen/runner.py swallows real bugs into metrics.
10. Magic numbers / Settings bypass across all new services — undocumented in docs/02-architecture.md per AGENTS.md §5.10.
11. Duplication across Python services — readiness-check logic, 4-line bootstrap, and db-pool lifespan repeated in auth/data/worker app.py.
12. K8s manifest duplication — 3 near-identical postgres manifests, 4 prod service manifests; twin-network-policy.yaml has drifted from template.yaml (missing the namespace field the template has).
7 tasks done
deploy/policies/understudy-prod.yaml grants a ServiceAccount access to patch Deployments/ConfigMaps in ust-prod, but no Deployment in deploy/prod/*.yaml sets serviceAccountName: understudy-prod. A code-review pass against PR #6 flagged this as a dead grant. It is not dead: per ADR-028 (docs/01-decisions.md), the actuator authenticates to the Kubernetes API directly as this ServiceAccount from the agent host process (docs/02-architecture.md §2.10), not by being assumed as pod identity. This RBAC ships ahead of its consumer (build-plan step A1.7 landed; actuator/apply.py is step 5.1/5.2, still unbuilt per CHECKLIST.md). Document that explicitly so it is not re-flagged as dead code before Phase 5 lands. Implements build-plan step A1.7 (documentation follow-up). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
This was referenced Sep 11, 2026
deploy/policies/twin-network-policy.yaml (the golden copy, applied
with `kubectl apply -n <namespace>`) and
twin-network-policy.template.yaml (rendered at fork time, carries
`namespace: ${TWIN_NAMESPACE}`) are maintained entirely by hand -- no
generation script exists yet (fleet manifest rendering is build-plan
step A2.2, unbuilt). The two existing unit tests validate each file's
own structure independently and never diff them against each other,
so a future edit to one that isn't mirrored in the other would pass
CI silently.
Add a test that loads both, strips the namespace field they're
expected to differ on, and asserts everything else is identical.
Also corrects docs/02-architecture.md's directory tree, which
documented a deploy/twin/ directory for "twin manifest templates"
that doesn't exist -- the templates actually live in deploy/policies/
next to their golden copies.
Implements build-plan step A1.7 (test coverage follow-up).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
7 tasks done
edge_gateway/app.py's get_items and create_item both silenced mypy's [no-any-return] on `return data_resp.json()` with a type: ignore, even though both functions are declared -> dict[str, Any]. Assign to an explicitly-typed local first instead -- mypy treats the annotation as authoritative, so the return type-checks with no ignore needed. loadgen/runner.py's _execute_single_request caught bare `except Exception` around a call that can only raise httpx.RequestError subclasses. A bare catch there would silently record a real bug (e.g. a TypeError from a malformed RequestSpec) as a fake "failed request" metric instead of failing the load-gen run loudly. Narrow to `except httpx.RequestError`. Implements build-plan step A1.1/A1.8 (fix on already-completed services). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
7 tasks done
worker_loop only called fault_manager.wait_if_stalled, so LATENCY, ERROR_RATE, and CPU_SPIN faults applied via POST /admin/fault were no-ops against job processing -- only STALL worked. Every other service's HTTP middleware calls pre_request_hook once per request; the worker has no HTTP request in its polling loop, so call it directly once per iteration instead. Factor the loop body into _worker_iteration so it can be exercised directly under test without driving a real polling loop, and so pre_request_hook's ERROR_RATE branch (which raises HTTPException -- meaningful in an HTTP handler, not here) can be handled the same way a real job-processing failure already is: logged and the job skipped for that iteration, rather than crashing the polling task. Implements build-plan step A1.2 (fix on an already-completed service). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
FaultManager.apply_fault/clear_fault mutate shared state (active_fault, expires_at, _timer_task, _leaked_buffers, _held_connections) across multiple await points with no synchronization. Two admin apply_fault/clear_fault calls racing each other, or the independent _ttl_watcher background task clearing a fault out from under a fresh apply_fault, could leave the manager in a torn state -- e.g. a TTL expiry cancelling the wrong timer, or a connection appended to _held_connections during clear_fault's iteration over the same list. Add an asyncio.Lock guarding the whole state transition. apply_fault internally calls clear_fault's logic, so factor that logic into _clear_fault_locked (no lock acquisition) called by both the public clear_fault (which acquires the lock) and apply_fault (which acquires the lock once and calls the internal version), avoiding a self-deadlock on the non-reentrant lock. Route _ttl_watcher's clear through the same locked path so it can't race a fresh apply_fault. Implements build-plan step A1.3 (fix on an already-completed service). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
…ention FaultRequest.magnitude means something different per FaultKind, and for ERROR_RATE specifically it silently overloads two conventions: <= 1.0 is a fraction, > 1.0 is a percentage. A caller who means "1%" and passes 1.0 gets 100% instead, since 1.0 lands on the fraction side of that boundary. This is intentional, existing behavior (test_faults.py::test_fault_error_rate already asserts 1.0 and 100.0 both mean 100%) -- not changed here, only documented: an inline comment at the branch, and an expanded field docstring spelling out the per-kind units and the footgun. Implements build-plan step A1.3 (documentation follow-up on an already-completed service). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
7 tasks done
REQUEST_COUNT/REQUEST_DURATION/SERVICE_INFO were module-level globals registered against the implicit default prometheus_client registry, constructed once at import time and shared by every service's setup_metrics call. Since all four services' app.py modules (plus every service test file that imports one) run in the same pytest process, this meant whichever service imported last determined what understudy_service_info reported for the rest of the run, and request counters/histograms accumulated across unrelated services' tests. Currently dormant (no Duplicated timeseries crash, since the objects were constructed once total, not once per call) but real test pollution. setup_metrics now builds its own CollectorRegistry and Counter/Histogram/Info instances per call, defaulting to a fresh registry when the caller doesn't supply one -- every service's app.py needs no changes, since none of them pass a registry. The /metrics route scrapes that call's own registry via closure, the same way the middleware already closes over service_name. Implements build-plan step A1.1 (fix on an already-completed service). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
7 tasks done
…tory All four services/*/app.py files repeated an identical 4-line bootstrap (FastAPI(...), setup_metrics, setup_fault_middleware, setup_fault_routes), and auth_service, data_service, and worker repeated a near-identical open-pool/assign-to-fault-manager/optional -init/close skeleton around their connection pool lifecycle. Add services/_common/bootstrap.py: - create_service_app(title, fault_manager, lifespan) does the shared 4-line bootstrap, returning the constructed app. - db_pool_lifespan(database_url, fault_manager, on_open=None) is an async context manager doing the shared pool lifecycle, yielding the pool (or None) so each service assigns it to its own module-level db_pool global -- the helper has no way to reach into another module's globals itself. on_open lets worker/data_service pass their init_worker_db/init_db callbacks; a PsycopgError from it is logged and swallowed, matching prior behavior. Refactor all four app.py files to use create_service_app; refactor auth_service, data_service, and worker to use db_pool_lifespan. edge_gateway has no database and keeps its own lifespan (it manages an httpx.AsyncClient instead). Route registration and readiness-check logic stay in each service's own file. Existing tests that patched create_pool on the service module now patch it on services._common.bootstrap, where the call actually happens. Implements build-plan step A1.1/A1.2/A1.3/A1.8 (refactor across already-completed services). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
7 tasks done
check_mocks.py only scanned src/ for # MOCKED: markers, so a marker placed anywhere under services/ was invisible to the make check mock-parity gate required by AGENTS.md §7.3. Add a second scan of services/, re-keying its markers with the services/ prefix so they match the dotted module names used in docs/MOCKS.md (e.g. services.auth_service.app). Also fixes a latent Windows bug uncovered while adding this: rel_path was built with str() instead of as_posix(), so backslash-separated paths never matched the forward-slash dotted module names produced by _module_matches_file. This was invisible until now because docs/MOCKS.md has been empty, but would have silently broken parity checking for any nested src/ marker on Windows. Discovered while investigating a code-review finding for PR #6; this fix unblocks registering the auth_service synthetic-token mock in a follow-up PR. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
This was referenced Sep 11, 2026
services/auth_service/app.py accepted any Bearer valid-*/token-*
token as valid on the always-reachable /validate path, with no real
credential store behind it -- an unmarked, unregistered mock
(AGENTS.md section 2.3, section 7). Compounding this, TOKEN_CACHE's
FIFO eviction could evict the seeded admin token
("Bearer valid-token", scope: admin) under sustained synthetic
traffic; revalidating it afterward fell through to the
synthetic-pattern branch (it also matches "Bearer valid-"), silently
re-caching it as scope: standard with no error.
Fix both together:
- Carve the two seeded tokens into SEEDED_TOKENS, a separate,
non-evictable dict checked before TOKEN_CACHE and before the
synthetic-pattern branch. A seeded token can no longer be evicted
or re-derived with a different scope.
- Gate the synthetic-pattern branch behind
AUTH_SERVICE_ALLOW_SYNTHETIC_TOKENS, defaulting to enabled outside
prod (mirroring how is_fault_injection_enabled gates on
UNDERSTUDY_ROLE=prod) so it can be forced off, or explicitly
re-enabled in prod for the eval harness.
- Mark the remaining synthetic-acceptance branch # MOCKED: and
register it in docs/MOCKS.md (module services.auth_service.app,
tracked in #16) -- now enforced by
scripts/check_mocks.py per the earlier services/ scan fix.
Implements build-plan step A1.2 (fix on an already-completed
service), resolving an AGENTS.md section 8 stop-and-ask flagged
during code review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
This was referenced Sep 11, 2026
Every tunable across services/ was read ad hoc via os.getenv(...)
with a hardcoded string default, scattered per-module -- no
services/ equivalent of src/understudy/common/config.py's layered
pydantic Settings pattern existed, violating AGENTS.md §5.10 ("no
magic numbers in code... thresholds, weights, timeouts and ceilings
live in config/*.yaml and are named in docs/02-architecture.md").
Add services/_common/settings.py mirroring config.py's shape (layered
load, frozen pydantic BaseModel per section, cached singleton with a
reset hook for tests) as an independent module -- services/ and
src/understudy/ share no imports, and src/understudy/common must stay
psycopg/fastapi-free, so it cannot host these tunables either.
One deliberate divergence: fields keep the exact env var name they
already used before this module existed (e.g.
AUTH_TOKEN_CACHE_MAX_SIZE, FAULT_INJECTION_SEED), rather than a new
UNDERSTUDY_<SECTION>__<FIELD> scheme -- those names are already wired
into deploy/prod/*.yaml's env/configMapKeyRef entries and loadgen's
CLI flag fallbacks, and renaming them would silently break
already-deployed configuration for no benefit.
Adds config/services.yaml (defaults) and config/services.local.yaml
to .gitignore (mirroring config/local.yaml) for local overrides.
Documents every field in docs/02-architecture.md §2.5.1.
This is infrastructure only: no call site reads from Settings yet
(follow-up PR). Implements build-plan step A1.8.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
Replaces every ad hoc os.getenv(...)-with-hardcoded-default and bare numeric literal identified across services/ with a read from get_services_settings()/load_services_settings() (added in the prior PR): edge_gateway's HTTP_TIMEOUT_SECONDS, auth_service's TOKEN_CACHE_MAX_SIZE, worker's POLL_INTERVAL_SECONDS and its previously-unparameterized `LIMIT 50` job-list query (now bound via %s, matching the file's existing parameterized-query style), all of loadgen's LoadgenConfig defaults and its httpx.Limits/p99 SLA threshold, db.py's connection-pool defaults, faults.py's injection seed default and POOL_EXHAUSTION getconn timeout, and metrics.py's histogram buckets. loadgen reads via the fresh (uncached) load_services_settings() rather than the cached singleton, since both LoadgenConfig's defaults and main.py's argparse defaults are evaluated per CLI invocation / per-test-construction and must reflect an env var set right before that call, not whatever was cached first in the process. Every other service reads the cached get_services_settings() once at module-import time, matching how these values were already evaluated once via os.getenv(...) at import time before this change. Implements build-plan step A1.1/A1.2/A1.3/A1.8 (migration on already-completed services). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
deploy/system/prod-postgres.yaml, twin-postgres.yaml, and system-postgres.yaml were three independently hand-written, near-identical Deployment+Service pairs (identical skeleton; differing only in namespace, image, POSTGRES_DB, resources, host port, and system-postgres's extra init-script ConfigMap/volume) with nothing keeping them from drifting apart. Add deploy/generate_postgres_manifests.py: a plain Python script (fits ADR-004 "Python everywhere" -- this repo has no Helm/Kustomize/Jsonnet anywhere, and adopting one would need a new ADR, out of scope here) that builds each manifest as a dict from a shared template function plus a small per-instance parameter table, and dumps it with PyYAML. --check renders in-memory and diffs (structurally, not byte-for-byte, since PyYAML formatting need not match hand-written style) against the committed files, exiting non-zero on drift; wired into tests/unit/deploy/test_system_manifests.py so drift fails CI. The three committed files are now generator output (regenerated via this script) rather than hand-maintained; the only textual diff from the prior hand-written versions is cosmetic YAML quoting. Documents the generator in docs/02-architecture.md §2.10. Implements build-plan step A1.5/A1.6 (refactor on already-completed manifests). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
7 tasks done
…red flags deploy/prod/configmap.yaml's app-config.FAULT_INJECTION_SEED (1337) exactly matches services/_common/faults.py's hardcoded DEFAULT_FAULT_INJECTION_SEED, but nothing wired the ConfigMap value in -- every service silently fell back to the code default. A code-review pass against PR #6 flagged this as a dead ConfigMap key. Wire FAULT_INJECTION_SEED into all four deploy/prod/*.yaml Deployments via configMapKeyRef so the ConfigMap is the actual source of truth in prod, with the code constant remaining only as the non-k8s/local fallback. feature-flags and app-config's ENVIRONMENT/LOG_LEVEL stay unwired: docs/02-architecture.md §2.5 requires feature flags to be read live at request time, not from a startup env var, which needs the actuator's not-yet-built DISABLE_FLAG/REVERT_CONFIG live-read mechanism (Phase 5). Document that explicitly in the ConfigMap so it is not re-flagged as dead configuration before Phase 5 lands. Implements build-plan step A1.7/A1.8 (fix + documentation follow-up). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
deploy/prod/{auth-service,data-service,edge-gateway,worker}.yaml were
four independently hand-written, near-identical Deployment+Service
pairs (identical label scheme, Prometheus scrape annotations,
replicas, imagePullPolicy, resources, probe shape,
UNDERSTUDY_ROLE/FAULT_INJECTION_SEED env; differing only in each
service own env vars, Service type/ports, and
app.kubernetes.io/component) with nothing keeping them from drifting
apart -- the same problem generate_postgres_manifests.py (prior PR)
already solved for the postgres manifests.
Factor the shared PyYAML dumping helpers (indent-forcing Dumper,
literal-block-scalar str subclass) out of that script into
deploy/_manifest_gen_common.py so both generators use one convention
instead of duplicating it. Add deploy/generate_prod_manifests.py
following the same shape: each manifest built as a dict from one
template function plus a small per-instance parameter table, with
--check wired into tests/unit/deploy/test_prod_manifests.py so drift
fails CI.
The four committed files are now generator output; diffed against
what was already committed before touching anything, the only
difference is YAML quoting style (PyYAML's default quote-string-that-
looks-like-a-bool/number behavior), confirming the generator
faithfully reconstructs the existing, reviewed manifests including
the FAULT_INJECTION_SEED wiring from the ConfigMap PR.
Implements build-plan step A1.7/A1.8 (refactor on already-completed
manifests).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
…t mock (subsumes PR #8)
An error occurred while trying to automatically change base from
feat/prod-and-system-manifests
to
main
September 11, 2026 19:45
This was referenced Sep 11, 2026
Merged
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Build-Plan Step Implemented
deploy/policies/: ServiceAccounts, Roles, twin NetworkPolicy template) and A1.8 (services/loadgen/seeded deterministic generator), completing Phase 1A.Checkpoint Execution & Output
Executed all commands for Checkpoint A1 on live local cluster:
$ make loadgen RPS=20 DURATION=30 Summary: total=600, success=598, failed=2, error rate 0.00, p50=11.4ms, p90=14.3ms, p95=18.6ms, p99=52.9ms, max=204.4ms [p99 < 400ms (52.9ms)]$ docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}" NAME MEM USAGE / LIMIT MEM % k3d-ust-tools 5.32MiB / 12.16GiB 0.04% k3d-ust-serverlb 21.92MiB / 12.16GiB 0.18% k3d-ust-server-0 1.196GiB / 12.16GiB 9.84% ust-registry 30.75MiB / 12.16GiB 0.25%Total container memory ~1.25 GB, well within the 3.5 GB budget.
ADRs Relied On
understudy-prod) and NetworkPolicy.kube-system, and backing services inust-system.services/loadgen).deploymentsandconfigmapsinust-prod.Mock Registry Disclosures
Documentation Updates
Definition of Done
make checkgreen (ruff, mypy --strict, import-linter, pytest)# MOCKED:marker without a row in docs/MOCKS.md