Repository navigation
fix(services): isolate Prometheus registries per service instance - #14
Merged
Harikeshav-R merged 1 commit intoSep 11, 2026
Merged
Conversation
REQUEST_COUNT/REQUEST_DURATION/SERVICE_INFO were module-level globals registered against the implicit default prometheus_client registry, constructed once at import time and shared by every service's setup_metrics call. Since all four services' app.py modules (plus every service test file that imports one) run in the same pytest process, this meant whichever service imported last determined what understudy_service_info reported for the rest of the run, and request counters/histograms accumulated across unrelated services' tests. Currently dormant (no Duplicated timeseries crash, since the objects were constructed once total, not once per call) but real test pollution. setup_metrics now builds its own CollectorRegistry and Counter/Histogram/Info instances per call, defaulting to a fresh registry when the caller doesn't supply one -- every service's app.py needs no changes, since none of them pass a registry. The /metrics route scrapes that call's own registry via closure, the same way the middleware already closes over service_name. Implements build-plan step A1.1 (fix on an already-completed service). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Build-Plan Step Implemented
Checkpoint Execution & Output
Verified the installed
prometheus_clientversion'sCounter/Histogram/Infoall accept aregistry=kwarg andCollectorRegistry()takes norequired args before relying on that API.
Excluded above: two pre-existing Windows-local-only items unrelated to this
change (real
httpx.AsyncClient()construction crashes on this dev machine;not present on CI,
ubuntu-latest).ADRs Relied On
Mock Registry Disclosures
Documentation Updates
contract or tunable changed.
setup_metrics's new optionalregistryparameter is a test-only seam, documented in its own docstring.
Definition of Done
make checkgreen (see checkpoint output; two pre-existing Windows-local-only items excluded, unrelated)# MOCKED:marker without a row in docs/MOCKS.mdContext
A
/code-reviewpass against PR #6 flaggedservices/_common/metrics.py'smodule-level
REQUEST_COUNT/REQUEST_DURATION/SERVICE_INFOglobals as alatent test-pollution mechanism: every service's
app.py(and every testfile that imports one) shares the same Prometheus objects in-process.
Currently dormant since the objects are constructed once total rather than
duplicated per call, but real --
understudy_service_inforeports whicheverservice imported last, and counters accumulate across unrelated tests. Fixed
by having
setup_metricsbuild its own registry and metric instances percall. Added a regression test with two isolated registries confirming
neither service's metrics leak into the other's
/metricsoutput.Base branch is
feat/policies-and-loadgen(PR #6, still open), sinceservices/only exists on that branch's stack.🤖 Generated with Claude Code
https://claude.ai/code/session_01CzBeBUsZmE38K1kWeLMhNh