From 35851a3d54e71c8875dcdf90131437ad9b4e7a66 Mon Sep 17 00:00:00 2001 From: Jonas Daniels Date: Wed, 28 Oct 2026 15:03:01 -0700 Subject: [PATCH] docs: bring docs up to date with 1.11.3, interning and current benchmarks Co-Authored-By: Claude Opus 5.5 --- CONTRIBUTING.md | 7 ++++++- README.md | 22 +++++++++++++++++++--- bench/__init__.py | 3 ++- docs/compat.md | 8 ++++++-- docs/plan.md | 9 +++++---- 5 files changed, 38 insertions(+), 11 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ae72679..b83b11c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -37,4 +37,9 @@ uv run --group e2e python e2e/run.py ## Bumping Zero -Change `ZERO_VERSION`, the `@rocicorp/zero` version in `harness/package.json` and `e2e/package.json`, and the `rocicorp/zero` image in `e2e/docker-compose.yml`. Then run the full suite and fix what it finds. +Change `ZERO_VERSION`, the `@rocicorp/zero` version in `harness/package.json` and `e2e/package.json`, the `rocicorp/zero` image in `e2e/docker-compose.yml`, and the version in `README.md`. Then run the full suite and fix what it finds: + +- Port what changed upstream, and move each ported module's docstring to the new tag and commit. +- The line ranges in `tests/compat_scope.py` are lines of the old TypeScript. Remap them to the new version, which `tests.compat` reports as covered or uncovered regions. +- Regenerate the benchmark fixtures with `uv run python -m bench.capture`; `tests/wire/test_bench_fixtures.py` fails until you do. +- Record any new divergence in [`docs/compat.md`](docs/compat.md#known-differences). diff --git a/README.md b/README.md index 1c10d4b..f1f9e52 100644 --- a/README.md +++ b/README.md @@ -81,6 +81,7 @@ Point `ZERO_QUERY_URL` and `ZERO_MUTATE_URL` at `/query` and `/mutate`. With Fas - **Re-running a mutator.** `tx.retry_on(predicate)` asks `predicate(error, attempt)`, sync or async, whether to run the mutator again in a fresh transaction when the run fails, including at `COMMIT`. Only `True` means yes. A mutator runs at most 5 times per push, and `tx` is unusable once its transaction has ended, inside the predicate too. - **Isolation level.** Each adapter takes `isolation_level="read committed" | "repeatable read" | "serializable"` for the transactions mutations run in. Unset, the database's default applies. - **Validation.** `@define_query(Model)` and `@define_mutator(Model)` parse args with a Pydantic model or `TypeAdapter` first; a failure reaches the client as upstream's validation error. +- **Queries are shared.** Like upstream's, the builder hands back the query it built before when a step repeats one, and the query endpoint reuses a query's mapped AST and its JSON across responses. Treat a built query and its AST as immutable. - **Strict building.** `create_builder(schema, strict=True)` refuses an unknown column when the query is built. Upstream only refuses it when the endpoint maps the query, failing the whole batch, so this is for your tests. ## How compatibility is checked @@ -90,8 +91,7 @@ The published npm package is the definition of correct. Every test diffs Python - **Wire.** `harness/` runs the real `handleMutateRequest`, `handleQueryRequest`, `PushProcessor` and the query builder under Node. Hypothesis generates pushes (with duplicates, gaps, refusals and cleanup mutations) and builder programs, sends each through both sides against identical Postgres databases, and diffs the responses and every row written. - **HTTP.** The raw bytes of a request go to upstream's handlers as a Fetch `Request` and to the Starlette routes, so query-string parsing, body decoding and `JSON.parse`'s error messages are compared too, along with the response as `JSON.stringify` writes it. - **SQL.** The same generated queries run through upstream's `ZQLDatabase.run` and through `tx.run` against one seeded database, and the result rows are compared, never the SQL text. - -The wire and SQL tiers run against every adapter. +- **Adapters.** The wire and SQL tiers run against every adapter. - **End to end.** `e2e/run.py` starts Postgres, two `rocicorp/zero` containers at the pinned version, the TypeScript reference app server and the Python one. An unmodified Zero client runs the same script through each, and the run fails if the client ends up with different data, the databases differ, or the reference run strays from `e2e/expected.json`. Run them with `uv run pytest` (needs Node 24 and Docker, or `ZERO_SERVER_PG` pointing at a Postgres) and `uv run --group e2e python e2e/run.py`. @@ -100,7 +100,23 @@ A diff that passes only covers the inputs tried, so `uv run python -m tests.comp ## Performance -`bench/` runs the same workloads through zero-server and the published package. The CPU-only cases (the query endpoint, request parsing, the mutate endpoint over an in-memory database, z2s) are [CodSpeed](https://codspeed.io) benchmarks on both sides, and CodSpeed flags a pull request that makes any of them slower. `uv run python -m bench` times every case on the wall clock, including those that wait on Postgres (`tx.run`, mutate, `PushProcessor`, per adapter), and prints each case's Python/TypeScript ratio. CI shows that table without gating on it. `bench/__init__.py` has the commands. +`bench/` runs the same workloads through zero-server and the published package. The CPU-only cases (the query endpoint, building and name-mapping queries, request parsing, the mutate endpoint over an in-memory database, z2s) are [CodSpeed](https://codspeed.io) benchmarks on both sides, and CodSpeed flags a pull request that makes any of them slower. `uv run python -m bench` times every case on the wall clock, including those that wait on Postgres (`tx.run`, mutate, `PushProcessor`, per adapter), and prints each case's Python/TypeScript ratio. CI shows that table without gating on it. `bench/__init__.py` has the commands. + +Python/TypeScript ratios at ec0768a against `@rocicorp/zero` 1.11.3, on an Apple Silicon laptop (median of the rounds; below 1 means Python is faster): + +| case | Py/TS | +| --- | --- | +| Query endpoint with JSON, 1 to 100 queries (`query_json`) | 1.3 to 1.7 | +| Query endpoint, parsed body in, objects out (`query`) | 1.3 to 2.4 | +| Building a query (`build`), mapping its AST (`map`) | 6 to 14 | +| Parsing 1000-item bodies from JSON (`parse_json`) | 1.4 to 1.5 | +| Mutate endpoint over an in-memory database (`mutate_cpu`) | 2.1 to 4.0 | +| Compiling ZQL to SQL (`z2s`) | 0.8 to 1.1 | +| `tx.run` on Postgres, every adapter (`run`) | 0.8 to 1.1 | +| Mutate endpoint and `PushProcessor` on Postgres, every adapter | 0.7 to 1.0 | +| Process start with imports, asyncpg / psycopg / SQLAlchemy | 1.1 / 1.5 / 2.0 | + +Building and mapping cost microseconds on either side; the endpoint reuses both for a repeated query. Wherever Postgres is involved, the database dominates, and Python is within 10% of npm or faster. ## Known differences diff --git a/bench/__init__.py b/bench/__init__.py index 0dc26cf..f4fbda0 100644 --- a/bench/__init__.py +++ b/bench/__init__.py @@ -4,7 +4,8 @@ - CPU-only cases: `bench/cpu.py` and `harness/bench/cpu_cases.ts`, with the same case names. The query endpoint at batches of 1 and 100 (`query`, and `query_json` with the JSON parse and - serialization an HTTP endpoint adds), parsing 1000-item push and transform bodies (`parse`, + serialization an HTTP endpoint adds), its two stages on their own (building a query, `build`, and + mapping its AST to server names, `map`), parsing 1000-item push and transform bodies (`parse`, `parse_json`), the mutate endpoint over an in-memory database (`mutate_cpu`), and compiling ZQL to SQL (`z2s`). - Cases that wait on Postgres: `tx.run` (`run`), the mutate endpoint (`mutate`) and `PushProcessor` diff --git a/docs/compat.md b/docs/compat.md index fce72c4..88b3152 100644 --- a/docs/compat.md +++ b/docs/compat.md @@ -70,13 +70,17 @@ These are the cases where the proof holds Python to something narrower than "ide - **Interned queries.** Upstream's builder hands back the query it built before when a step repeats one, keyed so that some values JavaScript's types rule out share a key: a `flip` of `'yes'` with `true`, any `orderBy` direction but `'asc'` with `'desc'`. In one process, such a value can come back as whichever - was built first. Python interns as well, but by exact keys, so it builds what it is given. The harness gives each program a fresh schema, which - upstream interns by, so that no comparison depends on the programs before it. + was built first. Python interns as well, but by exact keys, so it builds what it is given. The harness + gives each program a fresh schema, which upstream interns by, so that no comparison depends on the + programs before it. - **A mutator returning `null`.** Upstream records a mutator's return value as result data unless it is `undefined`, so a TypeScript mutator can return `null` as data. Python has only `None`, which the port takes as returning nothing, as the tests' mutators never return `null`. - **The JavaScript object model.** Upstream's registries also resolve `~`, their marker key, and names every object inherits such as `toString`. A Python registry has neither. +- **Logging.** Upstream logs through a `LogSink` and log level passed in. Python logs through the stdlib + `logging` loggers `zero_server.query`, `zero_server.mutate` and `zero_server.adapters`, so the wording + and fields of log lines are not compared. - **The runtime.** The oracle runs on Node 24, and JSON parse messages are V8's. Bun or Deno word them differently. - **The driver.** Upstream hands some values to its JavaScript driver as they are, so what gets bound diff --git a/docs/plan.md b/docs/plan.md index e876959..301303a 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -3,8 +3,8 @@ ## Goal A Python backend can replace a TypeScript backend behind an unmodified Zero client, for one pinned -`@rocicorp/zero` version at a time, and later releases fast-follow each upstream release. The first pin is -**1.10.0**. +`@rocicorp/zero` version at a time, and later releases fast-follow each upstream release. The first pin was +**1.10.0**; the current one is in `ZERO_VERSION`. ## Scope @@ -30,7 +30,7 @@ The published npm package at the pinned version is the definition of correct. Th ships no `.test.ts` for zql, z2s, zero-server, zero-protocol or zero-cache, so there are no upstream tests to port. Three tiers of differential tests, each comparing Python against the real package: -1. **Wire.** A small bun runner calls the real handlers. Hypothesis generates query args and mutation +1. **Wire.** A small Node runner (`harness/`) calls the real handlers. Hypothesis generates query args and mutation sequences, including gaps, duplicates, refusals and the `_zero_cleanupResults` mutation. Diff the responses and the rows written to the upstream schema (`zero_0.clients`, `zero_0.mutations`). 2. **SQL (z2s).** The same ZQL through TS and Python against the same Postgres. Compare result rows, never @@ -45,7 +45,8 @@ tests alone must reach every line and branch of the cited upstream modules insid ## Phases -Each phase is a PR. Its completion criterion is what makes it done. +Each phase is a PR. Its completion criterion is what makes it done. Phases 1 to 7 are done, and 1.11.3 +was ported by hand. Of phase 8, publishing on a tag is done; watching npm and porting are not. 1. **Scaffold.** `ZERO_VERSION` (1.10.0), CI running ruff, ty, pytest with coverage. Done when CI is green on an empty test suite.