Skip to content

Benchmark docs: add Litestar, FastStream and FastMCP tables, the unsampled-OTel floor and the macOS caveat #335

Closed

Description

@lesnik512

docs/introduction/performance.md and benchmarks/README.md measure FastAPI only (macOS, Apple M2, uvicorn). A runtime survey on 2026-10-24 at 705d8a3 measured all four frameworks per request, per message and per tool call on Linux with 2 CPUs (an OrbStack container on the same M2, CPython 3.14.9, same dependency versions as the README plus litestar 2.24.0, faststream 0.7.4, fastmcp 3.4.6, granian 2.8.5). It also re-checked the README's FastAPI per-instrument table on current dependencies on macOS: bare 16.4 µs, Prometheus +17.3 (README +17.7), Sentry +57.1 (+57.6), OTel +122.4 (+121.1), all four +228.2 (+230.3). The README numbers hold.

Several 2.0 issues quote these measurements (#307, #309, #310, #311, #312, #313). Publishing them gives those issues a source in the repo.

Three findings the docs should carry beyond the tables:

  • Unsampled OpenTelemetry is not free. At a 0% ratio sampler a request still costs +57 to +67 µs with the ASGI receive/send spans (Litestar +56.8 and +60.7 in two runs, FastAPI +66.6), and about +39 µs on Litestar without them (1% sampling with the exclusion). FastStream's unsampled floor is +25.4 µs per message. lite-bootstrap leaves the sampler to the SDK (ADR-0008), whose default samples every root trace unless OTEL_TRACES_SAMPLER says otherwise, so a user who samples down needs this floor to know what it buys.
  • macOS inflates anything that yields to the event loop. One loop iteration costs about 12 µs on macOS because KqueueSelector.select(0) is a 12.2 µs syscall, against 0.31 µs for EpollSelector.select(0) in the Linux container; asyncio.gather on one coroutine costs 42 µs against 5.0 µs. FastMCP's baseline per tool call is 670 µs on macOS and 517 µs on Linux. Anyone reproducing on a Mac misranks the health check and FastMCP items.
  • On FastAPI under uvicorn in the container, real-server runs were unstable (bare repeat medians 8235 against 6341 RPS), so the Linux real-server numbers are Litestar on Granian only.

Work

  • Commit the harness and scenario suites the survey used to benchmarks/: one process per scenario, an in-process driver after a real lifespan startup, a Granian real-server runner, a FastStream driver that calls the subscriber's process_message on a prebuilt raw message against a real RedisBroker (TestRedisBroker replaces the broker's logger with a MagicMock and hides FastStream's per-message log calls), and a FastMCP driver that posts tools/call to http_app(stateless_http=True, json_response=True). Extend section 8, "Reproducing", with the commands.
  • benchmarks/README.md: add per-instrument tables for Litestar (in-process and Granian), FastStream (per consumed message and per publish) and FastMCP (per tool call, with its ±30 µs noise stated), each with its environment line and a note that the endpoint is a do-nothing handler.
  • docs/introduction/performance.md: a short section per framework that pulls in the new tables, the unsampled-OTel floor, and a "Reproducing on macOS" caveat.
  • Keep the FastAPI macOS tables as they are; add the Linux FastAPI in-process numbers next to them rather than replacing them.

Verify: just docs-build passes with --strict and renders the new snippets, and the links job passes.

Not breaking.

Activity

  1. added
    documentationImprovements or additions to documentation
    needs-triageMaintainer needs to evaluate this issue
    on Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationneeds-triageMaintainer needs to evaluate this issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions