Skip to content

docs(work): add the cost and D1 overload follow-up brief #310

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
harlan-zw merged 1 commit into from
Sep 29, 2026
Merged

docs(work): add the cost and D1 overload follow-up brief #310

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions docs/work/EXECUTE-cost-followup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Cost and D1 overload follow-up

Status: open · 2026-09-29 · fixes shipped in #295 to #307; re-measure due 2026-10-06

**Next move:** Blocked until 2026-10-06. The check needs one full week of data after the fixes: 2026-09-30 to 2026-10-06. After that, run the checklist below and compare against the baseline table.

Done means: every metric in the baseline table meets its target for the week 2026-09-30 to 2026-10-06, or the miss has its own ledger item.

## Ledger

- [ ] Measure the week against the baseline table below
- [ ] Review both WAF rules' matched requests for false positives, including Harlan's own Unlighthouse and nuxtseo crawls
- [ ] Stop `pnpm install` from rewriting `minimumReleaseAgeExclude`. It dropped the comment on the `skilld` CLI exemptions twice on 2026-09-29.
- [ ] Re-time uncached skill pages (2.4 to 3.8 s before #305 and #306)
- [ ] Move this brief to `shipped/` once every target is met

## What shipped on 2026-09-29

| Change | Metric it moves |
| --- | --- |
| #295 icon collections resolve during SSR | Workers Logs events |
| #296 watchdog query seeks by index; `scheduled_runs` pruned | D1 rows read |
| #297 per-skill cache moved from KV to the Cache API | KV_CACHE writes |
| #298 malformed `/gh` segments return a 9 B 404 | CPU and requests |
| #299 and #305 indexes and merged render queries | D1 rows read per request |
| #303 D1 Sessions API; replication set to `auto` | Primary read share; "D1 overloaded" |
| #306 related skills load in the browser | Reads per SSR render |
| Zone WAF: block `47.79.0.0/16`; 50 requests per 10 s per IP on `/gh/*` | Crawler bursts |

## Baseline and targets

| Metric | Baseline | Target for the week |
| --- | --- | --- |
| Workers Logs events | ~1.3M/day | under 300k/day |
| KV_CACHE writes and deletes | 45k to 65k/day | under 8k/day |
| skilld-db rows read | ~550M/day | under 250M/day |
| D1 rows read per request | ~820 (09-29 07:00 UTC) | under 400 |
| Primary share of D1 reads | 100% | under 20% |
| Workers CPU | ~8.7M ms/day | under 6M ms/day |
| [Sentry](https://sentry.io) "D1 DB is overloaded" | ~290 events in September | 0 |
| Account D1 reads vs the 25B allowance | ~25.2B/month | under |

## How to measure

The `.env` `CLOUDFLARE_API_TOKEN` reads GraphQL Analytics and Workers Observability. It has no billing or WAF scope. Run `cf` from outside the repository, so it uses the OAuth login, which can read WAF rules. Account `5904138d55ca25d5670dca6adf99894e`; D1 `a5e53f35-f5e5-4987-8c67-c0175addc7cc`; KV_CACHE `187e636458cb49faa2ae14743adc7736`.

1. [GraphQL](https://graphql.org) `workersInvocationsAdaptive`, script `skilld-dev`: requests and `cpuTimeUs` per day.
2. GraphQL `d1AnalyticsAdaptiveGroups`: `readQueries` and `rowsRead` by `databaseRole`.
3. GraphQL `d1QueriesAdaptiveGroups` ordered by `sum_rowsRead_DESC`: check the top 10 for new offenders.
4. GraphQL `kvOperationsAdaptiveGroups` for KV_CACHE: writes and deletes per day.
5. Telemetry query `POST /accounts/<id>/workers/observability/telemetry/query`: event count per day, grouped by `$metadata.message`.
6. Sentry project `skilld`: search `overloaded`, `edge-cache-write` and `traceLifecycle`.
7. Zone security events for the two WAF rules.

To roll back replication, send `PUT /accounts/<id>/d1/database/<id>`{lang="html"} with `{"read_replication":{"mode":"disabled"}}`. It can take up to 24 hours.

## Log

- 2026-09-29 Audit found skilld at ~$70/month gross. Workers Logs was the largest line, and ~87% of it was SSR icon warnings. Crawler bursts (09-22, 09-24, 09-29) overloaded the single D1 primary.
- 2026-09-29 One hour after replication: 90% of reads on replicas, and 0 Sentry events at 3.5 times normal traffic. Rows read per request fell from ~820 to ~340.
1 change: 1 addition & 0 deletions docs/work/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ progress to the `## Log`. Never open a separate progress file.
- [EXECUTE-github-pivot.md](EXECUTE-github-pivot.md): Phase 4 cleanup and the mail deploy prerequisites.
- [EXECUTE-seo-keyword-rework.md](EXECUTE-seo-keyword-rework.md): the keyword-led rework of categories, collections and editorial pages.
- [EXECUTE-homepage-rebuild.md](EXECUTE-homepage-rebuild.md): the cluster grid, and the classifier that has to become a job.
- [EXECUTE-cost-followup.md](EXECUTE-cost-followup.md): the week-later check on the 2026-09-29 cost and D1 overload fixes.

The roadmap below sequences them.

Expand Down