Repository navigation
feat(mirror): implement mirror gateway proxy, registry client, and backpressure (Phase 3A) - #26
Merged
Merged
Conversation
Add mirror gateway service acting as a synchronous proxy to production with asynchronous, bounded per-twin fan-out queues and drain workers. Twin delivery is fire-and-forget, with drop-and-count metrics on queue overflow and worker timeout/errors, ensuring that production requests are never delayed by twin performance. Injects shadow headers on twin requests and exposes health and registration endpoints. Implements build-plan step A3.1.
…stats
Add validation models and endpoints for twin registration (POST /twins), unregistration (DELETE /twins/{twin_id}), and stats retrieval (GET /twins/{twin_id}/stats) returning MirrorStats with drop ratio metrics.
Harden MirrorGatewayManager against concurrent dictionary mutation during fan-out dispatch and add defensive argument checks. Expand unit tests to enforce 100% line and branch coverage across registration validation and contract compatibility.
Include stream ownership assignment in OWNERSHIP.md and two-minute demo video specification in docs/09-demo-video.md with supporting doc updates in README.md, AGENTS.md, and docs/00-product.md.
Implements build-plan step A3.2.
Add Prometheus metrics instrumentation to the mirror gateway service:
- understudy_mirror_delivered_total{twin_id} Counter tracking requests successfully delivered to twins
- understudy_mirror_dropped_total{twin_id} Counter tracking mirrored requests dropped via bounded queue overflow or worker drain failure
- understudy_mirror_latency_seconds{target} Histogram measuring outbound latency for production (target="prod") and mirrored twins (target=twin_id)
Encapsulate metrics in MirrorGatewayMetrics with isolated CollectorRegistry support to prevent cross-test state leakage. Wire metrics into MirrorGatewayManager and mirror gateway FastAPI application proxy routes.
Expand unit test coverage with dedicated test cases for standalone metrics, manager drop/delivery accounting, and the /metrics endpoint, maintaining 100% line and branch coverage across all modified services.
Implements build-plan step A3.3.
Add HTTP client implementation of MirrorRegistry protocol to interact with mirror-gateway REST endpoints for twin registration, health verification, and traffic statistics retrieval. Expose mirror management and fidelity comparison commands in the CLI under ust mirror. Implements build-plan step A3.4.
Deploy mirror-gateway in ust-system as the sole LoadBalancer ingress and transition ust-prod edge-gateway to ClusterIP so external entry routes exclusively through the mirroring proxy. Wire loadgen settings to target the mirror gateway entry point and sequence deployment readiness in the Makefile. Implements build-plan step A3.5.
Dampen queue overflow logging in the fan-out hot loop and configure connection pool limits on the mirror gateway to isolate production proxy traffic from slow or dead twin destinations. Enhance CLI mirror compare to confirm delivery parity and path distribution across registered twins. Add comprehensive unit and live integration tests verifying that an unresponsive twin does not degrade production p99 latency by more than 5 ms while its drop ratio rises toward 1.0 without gateway errors. Implements build-plan step A3.6.
Owner
Author
Code Review Findings (Two-Axis Review)This review evaluates the diff between Standards AxisHard Violations
Baseline Smells (Judgement Calls)
Spec Axis(a) Missing or Partial Requirements
(b) Unrequested Behaviour (Scope Creep)
(c) Implemented but Wrong
Summary: 7 findings in Standards (worst: unannotated bare |
Centralize hop-by-hop header stripping in services._common to prevent header leakage, configure connection pool limits via settings to avoid hardcoded magic numbers, and record production request counts and path distributions for authentic twin fidelity evaluation. Implements fixes for build-plan step A3.1-A3.3 review findings.
Add twin fidelity calculation comparing delivered request counts, drop ratios, and relative path frequencies against production metrics. Removes fabricated fidelity claims in CLI compare command, uses SystemClock across CLI and tests to uphold determinism, replaces bare exceptions with typed errors, and targets the edge gateway proxy route in backpressure testing. Implements fixes for build-plan steps A3.4-A3.6 review findings.
Owner
Author
Code Review Findings ResolvedAll findings from the
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Build-Plan Step Implemented
services/mirror_gateway/,src/understudy/mirror/registry.py,src/understudy/mirror/fidelity.py,deploy/system/mirror-gateway.yaml,src/understudy/cli.py)ust-prodedge-gateway, per-twin boundedasyncio.Queue(maxsize=1000)fan-out, drain worker tasks with 2s request timeout, drop-and-count accounting, hop-by-hop header stripping, and shadow request headers (X-Understudy-Shadow: 1,X-Understudy-Twin: <twin_id>,X-Understudy-Incident: <id>).POST /twins,DELETE /twins/{twin_id},GET /twins/{twin_id}/stats,GET /prod/stats).understudy_mirror_delivered_total,understudy_mirror_dropped_total,understudy_mirror_latency_seconds).mirror/registry.pyHttpMirrorRegistryimplementingMirrorRegistryprotocol and CLI commands (ust mirror register,ust mirror stats,ust mirror unregister,ust mirror compare).ust-system, withedge-gatewayconverted to ClusterIP inust-prod.Checkpoint Execution & Output
Checkpoint A3 executed against the running cluster and verified via unit and live integration suites:
(Drop ratio < 0.01 observed across healthy registered twins).
Unresponsive/dead twin destination drops requests cleanly via bounded queue backpressure without raising prod p99 latency by > 5 ms and without generating gateway 5xx errors to production clients.
Review Findings Addressed
Following a
/code-reviewpass, all findings were addressed across commits6a46dc1and701cf09:services/_common.headers(strip_hop_by_hop_headers) with comprehensive tests; eliminated emptyX-Understudy-Incidentheaders when not in an active incident.max_connections(200) andmax_keepalive_connections(50) toservices.yamland documented indocs/02-architecture.md.services/mirror_gateway(GET /prod/stats) and implementedevaluate_twin_fidelityinsrc/understudy/mirror/fidelity.py. The CLI dynamically calculates count deltas, drop ratios, and relative path frequency divergence without fabricated text.Exceptionhandling across drain workers, CLI commands, and integration tests with typed error tuples; replaced directdatetime.now(UTC)withSystemClock().now()./api/itemswith basic auth over the public ingress route.ADRs Relied On
MirrorRegistry). Components and CLI depend only on protocols, not concrete implementations.ust-system, synchronous proxy toust-prod, fanout over bounded queues.understudy_mirror_dropped_total{twin_id}.understudy_mirror_delivered_total,understudy_mirror_dropped_total,understudy_mirror_latency_seconds) for drop ratio and latency tracking.Mock Registry Disclosures
None. No
# MOCKED:markers added or modified;scripts/check_mocks.pypassed clean.Documentation Updates
CHECKLIST.md: Checked off build-plan steps A3.1 through A3.6 and Checkpoint A3.OWNERSHIP.md: Updated stream ownership assignments for Phase 3A.docs/00-product.md: Documented mirror gateway ingress architecture and metrics.docs/02-architecture.md: Detailed mirror gateway proxying, bounded fan-out queues, drop counters, and connection pool configuration in Section 2.5.docs/09-demo-video.md: Added two-minute demo video specification.Definition of Done
make checkgreen (ruff, mypy --strict, import-linter, pytest)# MOCKED:marker without a row in docs/MOCKS.md