From c6d914c22de55e92da664b4ed1aaa3f228cb0b86 Mon Sep 17 00:00:00 2001 From: Will Manning Date: Sat, 7 Nov 2026 19:30:35 -0500 Subject: [PATCH] bench docs: name the script that refreshes the cache vortex-data/vortex#11192 moved the cache revalidate out of scripts/post-ingest.py into scripts/refresh-bench-site.py, which a reusable workflow runs once per CI run. Update the contract, the architecture docs, the secrets runbook, and two code comments to name the new caller and describe when it runs and how it handles failure. No behaviour changes. Signed-off-by: Will Manning --- docs/bench/CONTRACT.md | 22 ++++++++++++++----- docs/bench/README.md | 2 ++ docs/bench/architecture/data-pipeline.md | 7 +++--- docs/bench/architecture/performance.md | 19 +++++++++------- docs/bench/architecture/read-path.md | 4 ++-- docs/bench/runbooks/deploy-secrets-setup.md | 6 ++--- src/app/(bench)/bench/api/revalidate/route.ts | 5 +++-- src/bench/lib/data-cache.ts | 7 +++--- 8 files changed, 45 insertions(+), 27 deletions(-) diff --git a/docs/bench/CONTRACT.md b/docs/bench/CONTRACT.md index 2772105c..1101e654 100644 --- a/docs/bench/CONTRACT.md +++ b/docs/bench/CONTRACT.md @@ -12,10 +12,11 @@ and are unchanged by this repository; this repo owns the ingest *contract*, the read service. > **Scope.** There is one ingest path: the monorepo's `scripts/post-ingest.py --postgres` writes -> emitter records directly into the hosted RDS Postgres and then pings this repo's read service -> to flush its cache. The retired v3 HTTP ingest path (`POST /api/ingest` into a Rust server) is -> described in [`docs/legacy.md`](legacy.md); the record shapes on the wire are unchanged -> from that era, which is why the producer flag is still spelled `--gh-json-v3`. +> emitter records directly into the hosted RDS Postgres. Then, once per CI run, the monorepo's +> `scripts/refresh-bench-site.py` pings this repo's read service to flush its cache. The retired +> v3 HTTP ingest path (`POST /api/ingest` into a Rust server) is described in +> [`docs/legacy.md`](legacy.md); the record shapes on the wire are unchanged from that era, which +> is why the producer flag is still spelled `--gh-json-v3`. ## Versioning: `SCHEMA_VERSION` @@ -80,8 +81,9 @@ The `reopen` mode does not clear the OS page cache. ## The ingest path — direct Postgres write + cache revalidate -The monorepo's `scripts/post-ingest.py --postgres` runs in benchmark CI (required, not -best-effort — a failure fails the bench job and alerts): +The monorepo's `scripts/post-ingest.py --postgres` runs in each benchmark CI job and does steps 1 +and 2 (required, not best-effort — a failure fails the bench job and alerts). The monorepo's +`scripts/refresh-bench-site.py` does step 3, once per CI run: 1. **Commit upsert:** the writer builds the commit row from `git show` and upserts it (`ON CONFLICT (commit_sha) DO UPDATE`) before applying any record. @@ -96,6 +98,14 @@ best-effort — a failure fails the bench job and alerts): `Authorization: Bearer $BENCH_REVALIDATE_TOKEN`, to mark the Next.js Data Cache stale so it refreshes against freshly written data: the next read serves the previous data once while each entry refreshes in the background, and later reads get the new data. + The reusable workflow `.github/workflows/refresh-bench-site.yml` runs the script in its own + job, after the ingest jobs of a `develop-bench.yml`, `nightly-bench.yml`, or + `commit-metadata.yml` run finish. In `develop-bench.yml` and `nightly-bench.yml` it runs even + when some benchmark jobs fail. The script does nothing when `BENCH_SITE_BASE_URL` or the + token is unset. It retries a transport error, `429`, or `5xx` up to three times (after 5, 15, + and 45 seconds). A revalidate that still fails, or fails with another status such as `401`, + fails the refresh job but not the ingest jobs. After a successful revalidate, the script + warms `/` and each group's default `n=100` bundle. A failed warm request only logs a warning. ### `measurement_id`: the frozen hash contract diff --git a/docs/bench/README.md b/docs/bench/README.md index 67313094..067a43cc 100644 --- a/docs/bench/README.md +++ b/docs/bench/README.md @@ -36,6 +36,8 @@ the site reads straight from it. │ benchmark jobs │ │ vortex-bench ─▶ JSONL records │ │ scripts/post-ingest.py │ +│ once per run, after the jobs │ +│ scripts/refresh-bench-site.py │ └───────┬──────────────────┬───────┘ │ upsert rows │ POST /api/revalidate │ (direct SQL) │ (flush the read cache) diff --git a/docs/bench/architecture/data-pipeline.md b/docs/bench/architecture/data-pipeline.md index 02513b30..78c041ab 100644 --- a/docs/bench/architecture/data-pipeline.md +++ b/docs/bench/architecture/data-pipeline.md @@ -88,9 +88,10 @@ every CI ingest run fails. Full procedure in `post-ingest.py --postgres` (required in monorepo benchmark CI) writes the records **directly to RDS** as the least-privilege `bench_ingest` IAM role -(`INSERT … ON CONFLICT (measurement_id) DO UPDATE`), then pings -`POST {BENCH_SITE_BASE_URL}/api/revalidate` to flush the Next.js read cache. -Each JSONL file applies in one transaction. The retired v3 HTTP path +(`INSERT … ON CONFLICT (measurement_id) DO UPDATE`). Each JSONL file applies in +one transaction. After the ingest jobs of a CI run finish, the monorepo's +`scripts/refresh-bench-site.py` sends one `POST {BENCH_SITE_BASE_URL}/api/revalidate` +to flush the Next.js read cache. The retired v3 HTTP path (`POST /api/ingest` into the Rust server) is described in [`../legacy.md`](../legacy.md). diff --git a/docs/bench/architecture/performance.md b/docs/bench/architecture/performance.md index 9522fd9c..cb858088 100644 --- a/docs/bench/architecture/performance.md +++ b/docs/bench/architecture/performance.md @@ -61,24 +61,27 @@ app, all sharing the `bench-data` tag. The backstop bounds staleness; `POST /api/revalidate` ([`src/app/(bench)/bench/api/revalidate/route.ts`](../../../src/app/(bench)/bench/api/revalidate/route.ts)) -*removes* it. On a successful ingest the emitter calls it and it runs +*removes* it. After the ingest jobs of a benchmark CI run finish, the monorepo's +`scripts/refresh-bench-site.py` calls it once, and it runs `revalidateTag('bench-data', 'max')`, which marks the entries stale rather than dropping them: the next read of each serves the previous data once and refreshes it in the background, and the read after that gets the fresh data, instead of waiting out the 24-hour cap. No read waits on the database because of an ingest; new results can take about one more CDN cycle (five minutes) to show. Dropping the entries instead (`{ expire: 0 }`) failed in production: every fill then -shared one advisory lock, the emitter revalidates after every benchmark job +shared one advisory lock, the emitter then revalidated after every benchmark job (about once a minute during a develop run), and the reads that waited on the lock ran out of retries. A failed render of `/` then stayed in the CDN for five minutes. Each entry now fills under its own lock (see -[read-path.md](read-path.md#database-connection)). The +[read-path.md](read-path.md#database-connection)), and the monorepo now +revalidates once per run, not once per job. The endpoint **fails closed** — a missing `BENCH_REVALIDATE_TOKEN` is a `503`, never -a silent accept — and the post-ingest hook is **best-effort**: every failure in -it is caught, logged, and swallowed so a cache-refresh problem can never change -an ingest's exit code. That is why the whole path is safe to ship before the -emitter cutover wires the token (it is inert until both `BENCH_SITE_BASE_URL` -and the token are set). +a silent accept. The refresh runs in its own CI job, apart from the ingest jobs, +so a cache-refresh problem can never change an ingest's exit code. The script +retries a transport error, `429`, or `5xx` up to three times. A revalidate that +still fails fails only the refresh job. The refresh is inert until both +`BENCH_SITE_BASE_URL` and the token are set, which is why the path was safe to +ship before the emitter cutover wired the token. ### Keeping the function and its connections warm diff --git a/docs/bench/architecture/read-path.md b/docs/bench/architecture/read-path.md index 0ac984d3..c7c1afac 100644 --- a/docs/bench/architecture/read-path.md +++ b/docs/bench/architecture/read-path.md @@ -68,8 +68,8 @@ by two independent layers, so the common case is cheap and ingest can flush both are never cached. **Freshness propagation.** The primary mechanism is -`POST /api/revalidate` (token-gated; `src/app/(bench)/bench/api/revalidate/route.ts`): on a -successful ingest, `scripts/post-ingest.py` calls it and it runs +`POST /api/revalidate` (token-gated; `src/app/(bench)/bench/api/revalidate/route.ts`): once a +benchmark CI run's ingest jobs finish, `scripts/refresh-bench-site.py` calls it, and it runs `revalidateTag('bench-data', 'max')`, the stale-while-revalidate profile: the next read of each Layer 1 entry serves the previous data once and refreshes the entry in the background, so no page waits on the database because of an ingest, diff --git a/docs/bench/runbooks/deploy-secrets-setup.md b/docs/bench/runbooks/deploy-secrets-setup.md index 26826203..cc186d98 100644 --- a/docs/bench/runbooks/deploy-secrets-setup.md +++ b/docs/bench/runbooks/deploy-secrets-setup.md @@ -142,9 +142,9 @@ team's secret store. **Never paste a value into this document.** | `BENCH_DB_POOL_MAX` | Max connections in the pg pool (default `8`; omit to use default) | Existing monorepo Vercel project env | | `BENCH_DB_IDLE_TIMEOUT_MS` | Pool idle-connection timeout in ms (default `5000`; positive integer; omit to use default) | Existing monorepo Vercel project env | | `BENCH_DB_CONNECTION_TIMEOUT_MS` | New-connection and pool-slot timeout in ms (default `5000`; positive integer; omit to use default) | Existing monorepo Vercel project env | -| `BENCH_REVALIDATE_TOKEN` | Bearer token for `POST /api/revalidate` — must match the value the monorepo's emitter caller (`post-ingest.py`) sends | Existing monorepo Vercel project env; coordinate with the monorepo caller (see note below) | +| `BENCH_REVALIDATE_TOKEN` | Bearer token for `POST /api/revalidate` — must match the value the monorepo's caller (`refresh-bench-site.py`) sends | Existing monorepo Vercel project env; coordinate with the monorepo caller (see note below) | -> **`BENCH_REVALIDATE_TOKEN` coordination:** this token authenticates the monorepo's `post-ingest.py` +> **`BENCH_REVALIDATE_TOKEN` coordination:** this token authenticates the monorepo's `refresh-bench-site.py` > revalidation caller against `POST /api/revalidate`. Setting it on the new project only makes the new > project ACCEPT a matching bearer — it does **not** redirect the emitter's traffic here. The emitter > POSTs to whatever base URL it is configured with (currently the monorepo-owned project's domain), so @@ -152,7 +152,7 @@ team's secret store. **Never paste a value into this document.** > emitter's configured URL), revalidation traffic continues to hit the **old** project — that is > expected during the Phase-4 setup window. Use the SAME token value the monorepo caller sends; if you > rotate it, update it in both places: this Vercel project and the monorepo's secret that feeds -> `post-ingest.py`. +> `refresh-bench-site.py`. > **`BENCH_DATA_TAG` is not env-configurable:** the Next.js Data Cache tag is a compile-time constant > (`export const BENCH_DATA_TAG = 'bench-data'` in `src/bench/lib/data-cache.ts`), imported directly by diff --git a/src/app/(bench)/bench/api/revalidate/route.ts b/src/app/(bench)/bench/api/revalidate/route.ts index 013cd044..33c9fef7 100644 --- a/src/app/(bench)/bench/api/revalidate/route.ts +++ b/src/app/(bench)/bench/api/revalidate/route.ts @@ -10,8 +10,9 @@ import { BENCH_DATA_TAG } from '@/bench/lib/data-cache'; /** * `POST /api/revalidate` marks the [`BENCH_DATA_TAG`] Data Cache entries stale, so - * they refresh against freshly ingested data (see below). `scripts/post-ingest.py` - * calls this after a successful Postgres write. Auth is a bearer token compared + * they refresh against freshly ingested data (see below). The monorepo's + * `scripts/refresh-bench-site.py` calls this once per benchmark CI run, after the + * run's Postgres writes finish. Auth is a bearer token compared * in constant time against `BENCH_REVALIDATE_TOKEN`; a missing env var fails * closed with `503` so an unconfigured deployment never silently accepts * unauthenticated revalidation. The response is never CDN-cached. diff --git a/src/bench/lib/data-cache.ts b/src/bench/lib/data-cache.ts index e9e03b25..e0a2c5eb 100644 --- a/src/bench/lib/data-cache.ts +++ b/src/bench/lib/data-cache.ts @@ -33,7 +33,8 @@ export const BENCH_DATA_TAG = 'bench-data'; /** * Backstop revalidation interval (seconds) for every cached read. The - * post-ingest revalidate hook is the primary freshness mechanism; this bound + * revalidate hook that runs after each benchmark CI run is the primary + * freshness mechanism; this bound * caps staleness at twenty-four hours if that hook ever fails to fire, so the * layer degrades to bounded staleness rather than serving stale data forever. * @@ -41,8 +42,8 @@ export const BENCH_DATA_TAG = 'bench-data'; * site: a longer backstop keeps the default last-100 window warm across * overnight idle gaps, so a CDN miss reads this Data Cache instead of paying * the multi-second cold database fill. The freshness cost is bounded by the - * revalidate hook, which marks the tag stale on every ingest once its env is - * set. + * revalidate hook, which marks the tag stale after every benchmark CI run once + * its env is set. */ export const DATA_CACHE_BACKSTOP_SECONDS = 86400;