Skip to content

Record HTTP metrics once when both OpenTelemetry metrics and Prometheus are configured - #365

Merged
lesnik512 merged 1 commit into
mainfrom
http-metrics-once
Oct 29, 2026
Merged

lesnik512 merged 1 commit into
mainfrom
http-metrics-once

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Closes #312.

On FastAPI, Litestar and FastMCP, HTTP request metrics are now recorded once. When opentelemetry_metrics_endpoint installs a MeterProvider, the Prometheus instrument no longer adds its HTTP middleware, so OpenTelemetry's HTTP metrics are the only set. The metrics route is still served, for the user's own collectors and FastMCP's tool-call metrics.

Decision

The issue asked to settle which side wins before writing a PR. The maintainer picked OpenTelemetry. opentelemetry_metrics_endpoint is an explicit opt-in, while Prometheus is on whenever its extra is installed, so the setting the user turned on by hand wins. FastStream stays as it is: both of its broker middlewares are opt-in (opentelemetry_middleware_cls, prometheus_middleware_cls), so a service with both asked for both.

The measurement below does not back the issue's other argument, that OpenTelemetry's is the cheaper set. On the M2 host, OTel's HTTP metrics cost 34 to 36 µs per request and Prometheus's 18 to 25 µs. The issue's Linux run had OTel at about 12 µs. I haven't found the cause. The candidates are the OS, or opentelemetry-sdk 1.45.1 against 1.44.0. On this host, keeping Prometheus would have saved more. Switching the rule is a small change if you'd rather go that way.

Changes

  • PrometheusHttpConfig in prometheus_instrument.py subclasses PrometheusConfig and adds prometheus_http_metrics_enabled: bool | None = None. FastAPIConfig, LitestarConfig and FastMcpConfig inherit it, and FastStream keeps plain PrometheusConfig. is_prometheus_http_metrics_on(opentelemetry_records_http_metrics=...) resolves the field:
    • None is on unless OpenTelemetry records HTTP metrics.
    • True keeps both sets, which is the field the issue asks for.
    • False turns them off even without OpenTelemetry. This one wasn't requested. It comes free with the bool | None shape, the same shape sentry_auto_enabling_integrations uses.
  • OpenTelemetryInstrument.is_recording_metrics(config) is true when the endpoint is set, the framework's OTel instrument has its dependencies, and the OTLP metric exporter for opentelemetry_exporter_protocol is installed. With the exporter missing, bootstrap() installs no MeterProvider and only warns, so Prometheus keeps its HTTP metrics and the service doesn't lose both. Each framework's Prometheus instrument asks its own OTel instrument class, because Litestar's and FastMCP's dependency checks differ.
  • FastAPI calls Instrumentator(...).expose without .instrument. Litestar adds the controller without PrometheusMiddleware. FastMCP skips the http_app() wrapping and keeps the tool-call middleware and the route.
  • In tests, the FastMCP fixture's registry cleanup moved to conftest.unregister_prometheus_request_metrics so the FastAPI tests can use it. conftest also has in_memory_opentelemetry_metric_reader and two small readers for each side.

Tests

Written first. One request is sent and the test checks which set moved: an in-memory metric reader for OpenTelemetry, and the request's unique path in prometheus_client.generate_latest() for Prometheus. The same parametrized test runs for FastAPI, Litestar and FastMCP. These cases failed on main:

  • both configured: only OpenTelemetry moves (FastAPI also over the HTTP protocol)
  • both configured with prometheus_http_metrics_enabled=True: both move
  • prometheus_http_metrics_enabled=False: neither moves

These passed on main and pin the "each signal alone" check: Prometheus alone, OpenTelemetry alone, the metrics route still served (and FastMCP's tool-call middleware still added), and a missing OTLP metric exporter leaving Prometheus's HTTP metrics in place.

The full suite passes, 574 tests at 100% coverage (just test-ci). just lint (including ty), just adr-check and just docs-build pass too. No floor changes, so I left the floors to CI.

Measurement

In-process on the M2 host (macOS, CPython 3.14.7, opentelemetry-sdk 1.45.1, instrumentation-asgi and -fastapi 0.66b1, prometheus-client 0.26.0, prometheus-fastapi-instrumentator 8.1.0, fastapi 0.162.0, litestar 2.25.0). Figures are µs over each suite's bare app, and every row's rounds were within 2%:

scenario FastAPI Litestar
otel_0: OTel at 0% +47.4 +38.8
otel_0_metrics: + OTel HTTP metrics +83.3 +72.8
otel_0_prom: + Prometheus +71.8 +56.4
otel_0_metrics_prom_both: 1.x behaviour +106.4 +90.2
otel_0_metrics_prom: the 2.0 default +84.5 +73.4

Configuring both now costs 21.9 µs less on FastAPI and 16.8 µs less on Litestar. The issue asked for a stack_scenarios.py run before and after. With run.py stack async, at 100% sampling over otel at 109.0 µs, OTel HTTP metrics cost +44.4, both sets +71.8, and the 2.0 default +48.1. otel_metrics, otel_metrics_prom and otel_metrics_prom_both are new stack scenarios, and otel_0_metrics_prom_both is new in the FastAPI and Litestar suites. I didn't measure FastMCP: its noise, about ±15% over a 620 µs baseline, is larger than the saving.

Docs

  • configuration.md:
    • A new "Prometheus HTTP metrics" section gives the rule, the field and when OpenTelemetry counts as recording.
    • The OpenTelemetry "Metrics" paragraph that called double recording "supported" now points to it and says why FastStream differs.
    • The FastMCP Prometheus section notes that its HTTP metrics are left out.
  • performance.md: the "savings applied for you" paragraph now has seven savings.
  • benchmarks/README.md §4d gets the rerun, including the reversed cost on this host, and §9 gets the stack command.

Breaking changes

FastAPI, Litestar and FastMCP services that set opentelemetry_metrics_endpoint and also serve Prometheus metrics lose the instrumentator's or Litestar's HTTP series on /metrics, for example http_requests_total and litestar_requests_total. Set prometheus_http_metrics_enabled=True to keep them.

@lesnik512
lesnik512 merged commit 57597f9 into main Oct 29, 2026
37 checks passed
@lesnik512
lesnik512 deleted the http-metrics-once branch October 29, 2026 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record HTTP metrics once when both OpenTelemetry metrics and Prometheus are configured

1 participant