Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 38 additions & 3 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ structlog 26.1.0. In-process, baseline 17.7 µs per request. The handler does no
| OTel 0% + OTel HTTP metrics (`opentelemetry_metrics_endpoint`) | +78.9 | +12.3 over OTel 0% |
| OTel 0% + Prometheus | +91.5 | +24.9 over OTel 0% |
| OTel 0% + OTel HTTP metrics + Prometheus | +104.9 | +38.3 over OTel 0% |
| Sentry, lite-bootstrap defaults (tracing off) | +59.0 to +59.7 | two runs |
| Sentry, lite-bootstrap 1.x defaults (tracing off) | +59.0 to +59.7 | two runs |
| Sentry, `sentry_auto_session_tracking=False` | +51.8 | |
| Sentry, `auto_enabling_integrations=False` | +0.1 | the Starlette and FastAPI integrations are auto-enabling, so nothing captures request errors |
| health check `/health/` | +22.6 | noisy, 16% spread |
Expand All @@ -219,6 +219,23 @@ structlog 26.1.0. In-process, baseline 17.7 µs per request. The handler does no
| full: OTel + Prometheus + logging + Sentry + health, defaults | +259.6 | |
<!-- --8<-- [end:fastapilinux] -->

Since 2.0, lite-bootstrap reports FastAPI request errors itself while Sentry tracing is off, without
`StarletteIntegration` and `FastApiIntegration`. Rerun in-process on the M2 host (macOS, CPython
3.14.7, fastapi 0.162.0, starlette 1.10.0,
`bench.py fastapi bare,sentry,sentry_sampled_trace,sentry_nointegration,sentry_integration,sentry_integration_sampled_trace --repeat 2`):

| scenario | sentry-sdk 2.73.0 | 3.0.0b2 |
|---|---:|---:|
| `sentry`: lean capture, the 2.0 default | +6.1 | +5.1 |
| `sentry_sampled_trace`: the same, request carries a sampled `sentry-trace` | +11.5 | +10.9 |
| `sentry_nointegration`: both integrations disabled, no request capture at all | +0.1 | -0.2 |
| `sentry_integration`: `sentry_lean_error_capture=False` | +57.1 | +47.0 |
| `sentry_integration_sampled_trace` | +59.2 | +113.2 |

The last row on 3.0.0b2 varied by 40% between rounds. The table above was measured before 2.0. Its
`sentry_nosess` row now has the same config as `sentry`, and `sentry_noauto` no longer stops request
capture, because lean capture needs no auto-enabling integration.

FastAPI real-server runs under uvicorn in the container were unstable (bare repeat medians of 8235
and 6341 RPS, over 100% spread on the full stack), so the Linux real-server numbers below are
Litestar on Granian only.
Expand Down Expand Up @@ -398,7 +415,9 @@ Since 2.0, lite-bootstrap reports Litestar request errors itself while Sentry tr
| `sentry_integration_sampled_trace` | +74.4 | +147.4 |

On 3.0 an incoming sampled `sentry-trace` makes `LitestarIntegration` record a real span even with
tracing off. The table above was measured before 2.0. Its `sentry_nosess` scenario now has the same
tracing off. `sentry_nointegration`, which disables `LitestarIntegration` and turns lean capture off,
so nothing captures request errors, measured +0.3 and +0.0 in a later run that put `sentry` at
+6.7 and +6.8. The table above was measured before 2.0. Its `sentry_nosess` scenario now has the same
config as `sentry`, because sessions are off by default since 2.0.

lite-bootstrap's Litestar defaults with no endpoint configured (Prometheus, health, logging and
Expand Down Expand Up @@ -498,14 +517,30 @@ it FastMCP and the MCP SDK. Noise on this suite is about ±30 µs even with 3 re
| OpenTelemetry, 100% | +272 | 3 ASGI spans plus FastMCP's own server span |
| OpenTelemetry, 100%, FastMCP's own spans replaced by a no-op tracer | +226 | FastMCP's spans cost 46 |
| OpenTelemetry, 10% | +120 | |
| Sentry, defaults | +100 | the Starlette integration |
| Sentry, 1.x defaults | +100 | the Starlette integration |
| Sentry, MCPIntegration disabled | +99 | MCPIntegration costs nothing measurable with tracing off |
| Sentry, `auto_enabling_integrations=False` | +3 | no Starlette integration, so no request capture |
| full: OTel + Prometheus + logging + Sentry + health | +521 | |
<!-- --8<-- [end:fastmcp] -->

The no-op tracer row patches FastMCP to attribute the cost; it is not a configuration.

Since 2.0, lite-bootstrap reports errors in every application `http_app()` builds itself while
Sentry tracing is off, without `StarletteIntegration`. Rerun in-process on the M2 host (macOS,
CPython 3.14.7, fastmcp 4.1.4, starlette 1.10.0,
`bench.py fastmcp bare,sentry,sentry_sampled_trace,sentry_nointegration,sentry_integration,sentry_integration_sampled_trace --repeat 3 --ops 1500 --warmup 300`),
over a baseline of about 620 µs per tool call:

| scenario | sentry-sdk 2.73.0 | 3.0.0b2 |
|---|---:|---:|
| `sentry`: lean capture, the 2.0 default | +20.6 | +15.9 |
| `sentry_sampled_trace` | +16.1 | +18.9 |
| `sentry_nointegration`: `StarletteIntegration` disabled, no request capture at all | -0.3 | +1.8 |
| `sentry_integration`: `sentry_lean_error_capture=False` | +82.2 | +65.0 |
| `sentry_integration_sampled_trace` | +82.3 | +216.4 |

Rounds differed by up to 15%, so the lean rows are not resolved from each other.

### 8f. Logging, per call

The per-call chain is the same on every framework: structlog's stdlib `BoundLogger`, lite-bootstrap's
Expand Down
3 changes: 3 additions & 0 deletions benchmarks/frameworks/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -167,3 +167,6 @@ def sentry(**extra_params: object) -> dict[str, typing.Any]:
LOG: typing.Final = {"logging_enabled": True}
CORS: typing.Final = {"cors_allowed_origins": ["https://example.com"], "cors_allowed_methods": ["GET"]}
ORIGIN: typing.Final[Headers] = ((b"origin", b"https://example.com"),)
SAMPLED_SENTRY_TRACE: typing.Final[Headers] = (
(b"sentry-trace", b"771a43a4192642f0b136d5159a501700-1234567890abcdef-1"),
)
13 changes: 13 additions & 0 deletions benchmarks/frameworks/fastapi_suite.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
OTEL,
OTEL_METRICS,
PROM,
SAMPLED_SENTRY_TRACE,
ASGIApp,
Cleanup,
Op,
Expand All @@ -22,6 +23,8 @@
start_lifespan,
)
from fastapi import FastAPI
from sentry_sdk.integrations.fastapi import FastApiIntegration
from sentry_sdk.integrations.starlette import StarletteIntegration

from lite_bootstrap import FastAPIBootstrapper, FastAPIConfig

Expand All @@ -44,6 +47,16 @@
"sentry": Scenario(config=base_config(**sentry())),
"sentry_nosess": Scenario(config=base_config(**sentry(), sentry_auto_session_tracking=False)),
"sentry_noauto": Scenario(config=base_config(**sentry(auto_enabling_integrations=False))),
"sentry_nointegration": Scenario(
config=base_config(
**sentry(disabled_integrations=[StarletteIntegration, FastApiIntegration]), sentry_lean_error_capture=False
)
),
"sentry_integration": Scenario(config=base_config(**sentry(), sentry_lean_error_capture=False)),
"sentry_sampled_trace": Scenario(config=base_config(**sentry()), headers=SAMPLED_SENTRY_TRACE),
"sentry_integration_sampled_trace": Scenario(
config=base_config(**sentry(), sentry_lean_error_capture=False), headers=SAMPLED_SENTRY_TRACE
),
"health": Scenario(config=base_config(**HEALTH), path="/health/"),
"health_otel": Scenario(config=base_config(**HEALTH, **OTEL), path="/health/"),
"health_otel_nospan": Scenario(
Expand Down
12 changes: 11 additions & 1 deletion benchmarks/frameworks/fastmcp_suite.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
LOG,
OTEL,
PROM,
SAMPLED_SENTRY_TRACE,
Cleanup,
Op,
Scenario,
Expand All @@ -25,6 +26,7 @@
from fastmcp import FastMCP
from opentelemetry.trace import NoOpTracer, Tracer
from sentry_sdk.integrations.mcp import MCPIntegration
from sentry_sdk.integrations.starlette import StarletteIntegration

from lite_bootstrap import FastMcpBootstrapper, FastMcpConfig

Expand All @@ -49,6 +51,14 @@ def no_fastmcp_spans() -> None:
"sentry": Scenario(config=base_config(**sentry())),
"sentry_nomcp": Scenario(config=base_config(**sentry(disabled_integrations=[MCPIntegration]))),
"sentry_noauto": Scenario(config=base_config(**sentry(auto_enabling_integrations=False))),
"sentry_nointegration": Scenario(
config=base_config(**sentry(disabled_integrations=[StarletteIntegration]), sentry_lean_error_capture=False)
),
"sentry_integration": Scenario(config=base_config(**sentry(), sentry_lean_error_capture=False)),
"sentry_sampled_trace": Scenario(config=base_config(**sentry()), headers=SAMPLED_SENTRY_TRACE),
"sentry_integration_sampled_trace": Scenario(
config=base_config(**sentry(), sentry_lean_error_capture=False), headers=SAMPLED_SENTRY_TRACE
),
"full": Scenario(config=base_config(**OTEL, **PROM, **LOG, **sentry(), health_checks_enabled=True)),
}

Expand All @@ -75,5 +85,5 @@ def add(a: int, b: int) -> int:
application = FastMcpBootstrapper(FastMcpConfig(application=mcp, **scenario.config)).bootstrap()
http_app = application.http_app(stateless_http=True, json_response=True)
stop = await start_lifespan(http_app)
scope = http_scope("/mcp", method="POST", extra_headers=HEADERS)
scope = http_scope("/mcp", method="POST", extra_headers=(*HEADERS, *scenario.headers))
return asgi_op(http_app, scope, body=CALL_BODY), stop
6 changes: 4 additions & 2 deletions benchmarks/frameworks/litestar_suite.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
OTEL,
OTEL_METRICS,
PROM,
SAMPLED_SENTRY_TRACE,
ASGIApp,
Cleanup,
Op,
Expand All @@ -38,7 +39,6 @@
HEALTH: typing.Final = {"health_checks_enabled": True}
LITESTAR_DEFAULTS: typing.Final = {**PROM, **HEALTH, **LOG, "swagger_path": "/docs"}
FULL: typing.Final = {**OTEL, **PROM, **LOG, **sentry(), **HEALTH}
SAMPLED_SENTRY_TRACE: typing.Final = ((b"sentry-trace", b"771a43a4192642f0b136d5159a501700-1234567890abcdef-1"),)

SCENARIOS: typing.Final = {
"bare": Scenario(config=base_config()),
Expand All @@ -63,7 +63,9 @@
"sentry_nosess": Scenario(config=base_config(**sentry(), sentry_auto_session_tracking=False)),
"sentry_noauto": Scenario(config=base_config(**sentry(auto_enabling_integrations=False))),
"sentry_nodefault": Scenario(config=base_config(**sentry(), sentry_default_integrations=False)),
"sentry_nointegration": Scenario(config=base_config(**sentry(disabled_integrations=[LitestarIntegration]))),
"sentry_nointegration": Scenario(
config=base_config(**sentry(disabled_integrations=[LitestarIntegration]), sentry_lean_error_capture=False)
),
"sentry_integration": Scenario(config=base_config(**sentry(), sentry_lean_error_capture=False)),
"sentry_sampled_trace": Scenario(config=base_config(**sentry()), headers=SAMPLED_SENTRY_TRACE),
"sentry_integration_sampled_trace": Scenario(
Expand Down
58 changes: 58 additions & 0 deletions docs/integrations/fastapi.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,64 @@ application = bootstrapper.bootstrap()

Read more about available configuration options [here](../introduction/configuration.md).

## Sentry

While Sentry tracing is off, lite-bootstrap reports request errors itself instead of through
sentry-sdk's `StarletteIntegration` and `FastApiIntegration`. Tracing counts as off when
`sentry_traces_sample_rate` is unset and `sentry_additional_params` sets none of
`traces_sample_rate`, `traces_sampler` and `enable_tracing`. With tracing off, the integrations
still build a transaction for every request and then discard it
([getsentry/sentry-python#7400](https://github.com/getsentry/sentry-python/issues/7400)).

`sentry_lean_error_capture` picks the path:

- `None`, the default: lean capture while tracing is off and `sentry_integrations` holds neither
integration, the integrations otherwise.
- `True`: lean capture always. lite-bootstrap warns when tracing is on, since no request starts a
transaction, and when `sentry_integrations` holds either integration, which gets dropped.
- `False`: the integrations always.

Lean capture passes both integrations in `disabled_integrations` and adds one pure-ASGI middleware.
It gives each HTTP and WebSocket request its own Sentry scope, and its own release-health session
when `sentry_auto_session_tracking` is on. It continues an incoming `sentry-trace` and `baggage`,
and attaches the request to every event sent while the request runs.

The request data follows the same [`data_collection`](../introduction/configuration.md#sentry-data-collection)
settings the integrations read. Method, URL, query string, headers and cookies are filtered by
their denylists. The event's transaction is the route template, such as `/items/{item_id}`. The
client address and the user that Starlette's `AuthenticationMiddleware` sets are attached only
when `user_info` is on. A body is attached only when `http_bodies` includes `"incoming_request"`
and the handler has read all of it: lean capture copies what the application reads and never
reads the stream itself. JSON and URL-encoded form bodies are attached parsed, a multipart body is
replaced by a placeholder, and any other body is attached as text. On sentry-sdk 2.x the
integrations read the body before the handler runs.

What is reported matches the integrations. An exception that escapes the application is reported.
An exception that an exception handler answers is reported only when it carries a `status_code`
from 500 to 599, as `HTTPException(503)` does, whatever the handler responds. So a 404 from
`HTTPException` is not reported, and neither is a custom exception your handler turns into a 503
unless the exception itself has a 5xx `status_code`. To find those handlers, lean capture looks
through the middleware below it for Starlette's `ExceptionMiddleware`. A middleware added before
bootstrap that keeps its inner application anywhere but `.app` hides it, and then only escaping
exceptions are reported.

uvicorn and Granian log an exception that escapes the application. sentry-sdk's dedupe integration
drops that log event, because it carries the same exception, so each failing request sends one
event either way.

Measured in-process on a do-nothing handler, over the same app without Sentry:

| Sentry setup | sentry-sdk 2.73.0 | 3.0.0b2 |
|---|---:|---:|
| lean capture, the default | +6.1 µs | +5.1 µs |
| the same, request carries a sampled `sentry-trace` | +11.5 µs | +10.9 µs |
| `sentry_lean_error_capture=False` | +57.1 µs | +47.0 µs |
| the same, request carries a sampled `sentry-trace` | +59.2 µs | +113.2 µs |

sentry-sdk records installed integrations for the whole process, and `disabled_integrations`
cannot remove one that an earlier `sentry_sdk.init` installed. If that happened, lite-bootstrap
warns that errors may be reported twice.

## Logging

Structlog is configured process-wide, so `structlog.get_logger()` works in any route handler.
Expand Down
22 changes: 22 additions & 0 deletions docs/integrations/fastmcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,28 @@ nanoseconds. A message that raises is logged at exception level and the exceptio

Set `health_checks_enabled=False` to omit the health route.

## Sentry

While Sentry tracing is off, lite-bootstrap reports errors in every application
`application.http_app()` builds itself, instead of through sentry-sdk's `StarletteIntegration`.
FastMCP's HTTP application is a Starlette application, so it works as it does
[on FastAPI](fastapi.md#sentry): the same `sentry_lean_error_capture` rule, the same middleware and
the same request data. `MCPIntegration` is left alone.

A tool that raises does not fail the request: FastMCP answers with an error result and logs the
exception, and the logging integration turns that record into one event with the `/mcp` request
attached, with lean capture on or off.

Measured in-process per `tools/call` over stateless JSON HTTP, over the same app without Sentry:

| Sentry setup | sentry-sdk 2.73.0 | 3.0.0b2 |
|---|---:|---:|
| lean capture, the default | +20.6 µs | +15.9 µs |
| `sentry_lean_error_capture=False` | +82.2 µs | +65.0 µs |
| the same, request carries a sampled `sentry-trace` | +82.3 µs | +216.4 µs |

A tool call costs about 620 µs before Sentry, and rows differ by up to 15% between runs.

## Metrics

With the `fastmcp-metrics` extra installed, the registry is served at `prometheus_metrics_path`,
Expand Down
7 changes: 4 additions & 3 deletions docs/introduction/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,13 @@ Additional parameters can also be supplied through the settings object:
- `sentry_additional_params` - additional params, which will be passed to `sentry_sdk.init`, overriding any of the above that map to the same keyword
- `sentry_default_integrations` - whether to use sentry's default integrations (default: `True`)
- `sentry_before_send` - optional callback chained after the built-in structlog enricher, passed to `sentry_sdk.init(before_send=...)`
- `sentry_lean_error_capture` - Litestar only: report request errors without Sentry's `LitestarIntegration` (default: `None`, meaning on while Sentry tracing is off). See [Litestar](../integrations/litestar.md#sentry).
- `sentry_lean_error_capture` - Litestar, FastAPI and FastMCP: report request errors without sentry-sdk's framework integration (default: `None`, meaning on while Sentry tracing is off). See [Litestar](../integrations/litestar.md#sentry), [FastAPI](../integrations/fastapi.md#sentry) and [FastMCP](../integrations/fastmcp.md#sentry).

Read more about sentry_sdk params [here](https://docs.sentry.io/platforms/python/configuration/options/).

Sentry is the second most expensive instrument in the stack. Its cost comes almost entirely from the ASGI
integration; see [Performance](performance.md).
Before 2.0, Sentry was the second most expensive instrument in the stack, and its cost came almost
entirely from sentry-sdk's framework integration. Since 2.0, lite-bootstrap replaces that integration on
Litestar, FastAPI and FastMCP while Sentry tracing is off; see [Performance](performance.md).

### Sentry logging integration

Expand Down
17 changes: 12 additions & 5 deletions docs/introduction/performance.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,10 @@ Four things here are worth knowing before you tune anything:

`attach_stacktrace`, `max_breadcrumbs` and dropping the default integrations all measure inside
noise. Essentially the entire cost is the ASGI integration, and over half of that is a `Transaction`
built and discarded because `sentry_traces_sample_rate` is unset.
built and discarded because `sentry_traces_sample_rate` is unset. That table configures
sentry-sdk directly. Since 2.0, lite-bootstrap leaves the framework integration out while tracing
is off and reports request errors itself, which costs about 6 µs per request on FastAPI
([FastAPI: Sentry](../integrations/fastapi.md#sentry)).

## If you are RPS-constrained, start here

Expand All @@ -65,13 +68,15 @@ request, every request then pays the per-record cost measured here:

--8<-- "benchmarks/README.md:logging"

lite-bootstrap already applies three savings for you. It turns Sentry Logs off in the logging
lite-bootstrap already applies four savings for you. It turns Sentry Logs off in the logging
integration, because it never enables them and on 2.x the handler formats every record before
checking whether they are enabled; that one costs nothing. Since 2.0 it also leaves release-health
sessions off and starts breadcrumbs at `WARNING`, so an `INFO` record skips the breadcrumb handler.
Those two cost you release health and `INFO` breadcrumbs on error events; set
`sentry_auto_session_tracking=True` or `sentry_logging_breadcrumb_level=logging.INFO` to get them
back. The numbers on this page were measured before 2.0, against the old defaults.
back. And while Sentry tracing is off it reports request errors without sentry-sdk's framework
integration, which costs you request transactions and bodies the handler never read; set `sentry_lean_error_capture=False` to keep the integration. The numbers on this page were
measured before 2.0, against the old defaults.

## What each saving costs you

Expand All @@ -92,7 +97,8 @@ The same instruments one at a time, in-process, in the Linux container:
--8<-- "benchmarks/README.md:fastapilinux"

The ordering matches macOS: OpenTelemetry first, then Sentry, then Prometheus, with logging free
until something logs.
until something logs. Since 2.0, Sentry with tracing off drops from about +59 to about +6 µs per
request, below Prometheus ([FastAPI: Sentry](../integrations/fastapi.md#sentry)).

## Litestar

Expand Down Expand Up @@ -131,7 +137,8 @@ Per tool call, through the stateless JSON HTTP app:
--8<-- "benchmarks/README.md:fastmcp"

The baseline is large, so the instruments are a smaller share here than on the HTTP frameworks.
OpenTelemetry is still the largest item.
OpenTelemetry is still the largest item. Since 2.0, Sentry with tracing off drops from about +82 to
about +21 µs per tool call ([FastMCP: Sentry](../integrations/fastmcp.md#sentry)).

## Unsampled OpenTelemetry is not free

Expand Down
Loading
Loading