Skip to content

perf(d1): route request reads through D1 read replicas - #303

Merged
harlan-zw merged 1 commit into
mainfrom
perf/d1-read-replication
Sep 29, 2026
Merged

harlan-zw merged 1 commit into
mainfrom
perf/d1-read-replication

Conversation

@harlan-zw

Copy link
Copy Markdown
Collaborator

❓ Type of change

  • 📖 Documentation
  • 🐞 Bug fix
  • 👌 Enhancement
  • ✨ New feature
  • 🧹 Chore
  • ⚠️ Breaking change

📚 Description

Crawler bursts overload the single D1 primary. On 2026-09-29 at 07:50 UTC about 520 skill page requests made about 4,900 D1 reads in two minutes. The baseline is 150 to 200 reads a minute. The burst queued on the primary and threw D1_ERROR: D1 DB is overloaded. Requests queued for too long. about 200 times (SKILLD-2Y, 1F, 27, 2P, 1S). The same error hit on 09-22 and 09-24.

Each request already opened one D1 session, pinned to first-primary. Now the session constraint depends on the request:

  • GET, HEAD, and OPTIONS start first-unconstrained, so they can run on the nearest replica.
  • POST, PUT, PATCH, and DELETE start first-primary, so a read before a write sees every other writer.
  • After a write, the session bookmark goes into an httpOnly d1-bookmark cookie for one hour. The visitor's next safe request starts from that bookmark, so they read their own write. The OAuth callback is a GET that writes, and it is covered the same way.

The cookie is set only on a response whose Cache-Control says private or no-store. It is never set on anything Workers Cache or the CDN can share. A cache fill that writes inside waitUntil after the response has left sets nothing.

Cron tasks, queue consumers, and ai-ready's db0 connector use the raw env.DB binding, so they stay on the primary.

With replication off, D1 serves all three constraints from the primary. So this can merge and deploy before the setting changes, and it behaves exactly as before.

Loose ends:

  • An internal $fetch that writes during SSR sets its cookie on the internal response, and that cookie is lost. I found no SSR path that writes and then needs to read the write in a later request.
  • One hour is a guess. Replica lag is seconds, and a stale bookmark costs nothing because every replica is past it.

📝 Migration

Production replication is disabled today (read from the API on 2026-09-29; the primary is in OC). This PR does nothing until an operator turns it on. Run this after the deploy:

curl -X PUT "https://api.cloudflare.com/client/v4/accounts/5904138d55ca25d5670dca6adf99894e/d1/database/a5e53f35-f5e5-4987-8c67-c0175addc7cc" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"read_replication": {"mode": "auto"}}'

The dashboard path is D1 > skilld-db > Settings > Enable Read Replication. To roll back, send "mode": "disabled". Cloudflare says a full disable can take up to 24 hours, and the code stays correct during that time. Billing does not change: it is still rows_read and rows_written.

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

Crawler bursts queue every read on the single D1 primary and fail with 'D1 DB is overloaded'. A per-request session lets safe reads start on any replica, keeps mutating requests on the primary, and hands a write's bookmark to the visitor's next request.
@harlan-zw

Copy link
Copy Markdown
Collaborator Author

Checked by hand on a local wrangler dev of the production build (.output/server/wrangler.json), with a copy of the local D1 migrated to 0123:

  • The skill page, /api/skills/vercel-labs/skills/find-skills, /api/skill-related/..., and /api/_auth/session all returned 200 with no d1-bookmark cookie.
  • POST /api/cli/device/start wrote its row and returned Set-Cookie: d1-bookmark=00000000-000058ed-00000000-000...; Max-Age=3600; HttpOnly; Secure; SameSite=Lax with Cache-Control: private, no-store.
  • The follow-up POST /api/cli/device/poll found the row (pending). A GET that carried the cookie returned 200. A forged d1-bookmark=first-primary was ignored, and the request still returned 200.
  • The server log showed no D1_ERROR.

Local D1 has no replicas, so this run shows that nothing regressed. It does not show reads spreading across replicas. Check that after the operator step with meta.served_by_primary in Sentry, or with the D1 read metrics.

@harlan-zw
harlan-zw merged commit 883499b into main Sep 29, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant