Skip to content

fix(services): isolate Prometheus registries per service instance - #14

Merged
Harikeshav-R merged 1 commit into
feat/policies-and-loadgenfrom
fix/isolate-prometheus-registries
Sep 11, 2026
Merged

Harikeshav-R merged 1 commit into
feat/policies-and-loadgenfrom
fix/isolate-prometheus-registries

Conversation

@rsanghvi21

Copy link
Copy Markdown
Collaborator

Build-Plan Step Implemented

  • Step: A1.1 (fix on an already-completed service).

Checkpoint Execution & Output

$ uv run pytest tests/unit/services/test_metrics.py -v
2 passed in 0.64s

$ uv run pytest tests/unit/services --ignore=tests/unit/services/test_edge_gateway.py -k "not test_run_loadgen_with_owned_client_and_zero_rps" -v
82 passed, 1 deselected

$ uv run pytest tests/unit --ignore=tests/unit/services/test_edge_gateway.py -k "not test_run_loadgen_with_owned_client_and_zero_rps" --cov=understudy --cov-branch --cov-fail-under=100
191 passed, 1 deselected, 100.00% coverage

$ uv run ruff check . && uv run ruff format --check . && uv run mypy --strict src tests && uv run lint-imports && uv run python scripts/check_mocks.py
All green.

Verified the installed prometheus_client version's Counter/Histogram/
Info all accept a registry= kwarg and CollectorRegistry() takes no
required args before relying on that API.

Excluded above: two pre-existing Windows-local-only items unrelated to this
change (real httpx.AsyncClient() construction crashes on this dev machine;
not present on CI, ubuntu-latest).

ADRs Relied On

  • None.

Mock Registry Disclosures

  • None.

Documentation Updates

  • None required -- internal test-hygiene/correctness fix, no documented
    contract or tunable changed. setup_metrics's new optional registry
    parameter is a test-only seam, documented in its own docstring.

Definition of Done

  • Implementation complete for the named build-plan step
  • Unit tests written; 100% line and branch coverage of the added lines
  • make check green (see checkpoint output; two pre-existing Windows-local-only items excluded, unrelated)
  • Checkpoint command run and output matches, OR marked unverified with a reason
  • Docs updated where this change made them untrue (none required)
  • No new # MOCKED: marker without a row in docs/MOCKS.md
  • Conventional commit(s), conventional branch, PR opened, never pushed to main

Context

A /code-review pass against PR #6 flagged services/_common/metrics.py's
module-level REQUEST_COUNT/REQUEST_DURATION/SERVICE_INFO globals as a
latent test-pollution mechanism: every service's app.py (and every test
file that imports one) shares the same Prometheus objects in-process.
Currently dormant since the objects are constructed once total rather than
duplicated per call, but real -- understudy_service_info reports whichever
service imported last, and counters accumulate across unrelated tests. Fixed
by having setup_metrics build its own registry and metric instances per
call. Added a regression test with two isolated registries confirming
neither service's metrics leak into the other's /metrics output.

Base branch is feat/policies-and-loadgen (PR #6, still open), since
services/ only exists on that branch's stack.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh

REQUEST_COUNT/REQUEST_DURATION/SERVICE_INFO were module-level globals
registered against the implicit default prometheus_client registry,
constructed once at import time and shared by every service's
setup_metrics call. Since all four services' app.py modules (plus
every service test file that imports one) run in the same pytest
process, this meant whichever service imported last determined what
understudy_service_info reported for the rest of the run, and
request counters/histograms accumulated across unrelated services'
tests. Currently dormant (no Duplicated timeseries crash, since the
objects were constructed once total, not once per call) but real
test pollution.

setup_metrics now builds its own CollectorRegistry and
Counter/Histogram/Info instances per call, defaulting to a fresh
registry when the caller doesn't supply one -- every service's
app.py needs no changes, since none of them pass a registry. The
/metrics route scrapes that call's own registry via closure, the
same way the middleware already closes over service_name.

Implements build-plan step A1.1 (fix on an already-completed
service).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
@Harikeshav-R
Harikeshav-R merged commit a27f2f7 into feat/policies-and-loadgen Sep 11, 2026
1 check passed
@Harikeshav-R
Harikeshav-R deleted the fix/isolate-prometheus-registries branch September 11, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants