Repository navigation
Benchmark docs: add Litestar, FastStream and FastMCP tables, the unsampled-OTel floor and the macOS caveat #335
Description
docs/introduction/performance.md and benchmarks/README.md measure FastAPI only (macOS, Apple M2, uvicorn). A runtime survey on 2026-10-24 at 705d8a3 measured all four frameworks per request, per message and per tool call on Linux with 2 CPUs (an OrbStack container on the same M2, CPython 3.14.9, same dependency versions as the README plus litestar 2.24.0, faststream 0.7.4, fastmcp 3.4.6, granian 2.8.5). It also re-checked the README's FastAPI per-instrument table on current dependencies on macOS: bare 16.4 µs, Prometheus +17.3 (README +17.7), Sentry +57.1 (+57.6), OTel +122.4 (+121.1), all four +228.2 (+230.3). The README numbers hold.
Several 2.0 issues quote these measurements (#307, #309, #310, #311, #312, #313). Publishing them gives those issues a source in the repo.
Three findings the docs should carry beyond the tables:
- Unsampled OpenTelemetry is not free. At a 0% ratio sampler a request still costs +57 to +67 µs with the ASGI
receive/sendspans (Litestar +56.8 and +60.7 in two runs, FastAPI +66.6), and about +39 µs on Litestar without them (1% sampling with the exclusion). FastStream's unsampled floor is +25.4 µs per message. lite-bootstrap leaves the sampler to the SDK (ADR-0008), whose default samples every root trace unlessOTEL_TRACES_SAMPLERsays otherwise, so a user who samples down needs this floor to know what it buys. - macOS inflates anything that yields to the event loop. One loop iteration costs about 12 µs on macOS because
KqueueSelector.select(0)is a 12.2 µs syscall, against 0.31 µs forEpollSelector.select(0)in the Linux container;asyncio.gatheron one coroutine costs 42 µs against 5.0 µs. FastMCP's baseline per tool call is 670 µs on macOS and 517 µs on Linux. Anyone reproducing on a Mac misranks the health check and FastMCP items. - On FastAPI under uvicorn in the container, real-server runs were unstable (bare repeat medians 8235 against 6341 RPS), so the Linux real-server numbers are Litestar on Granian only.
Work
- Commit the harness and scenario suites the survey used to
benchmarks/: one process per scenario, an in-process driver after a real lifespan startup, a Granian real-server runner, a FastStream driver that calls the subscriber'sprocess_messageon a prebuilt raw message against a realRedisBroker(TestRedisBrokerreplaces the broker's logger with aMagicMockand hides FastStream's per-message log calls), and a FastMCP driver that poststools/calltohttp_app(stateless_http=True, json_response=True). Extend section 8, "Reproducing", with the commands. benchmarks/README.md: add per-instrument tables for Litestar (in-process and Granian), FastStream (per consumed message and per publish) and FastMCP (per tool call, with its ±30 µs noise stated), each with its environment line and a note that the endpoint is a do-nothing handler.docs/introduction/performance.md: a short section per framework that pulls in the new tables, the unsampled-OTel floor, and a "Reproducing on macOS" caveat.- Keep the FastAPI macOS tables as they are; add the Linux FastAPI in-process numbers next to them rather than replacing them.
Verify: just docs-build passes with --strict and renders the new snippets, and the links job passes.
Not breaking.
Activity