perf: answer entry listings from entry_list_values (#110)

.changeset/entry-list-values-cloudflare.md

@@ -0,0 +1,5 @@
+---
+"@cloudhal/cloudflare": patch
+---
+
+The admin Worker's minute tick gains a `lists.backfill` system job (every five minutes): a budgeted `rebuildListValues` for listed collections that are new, changed, reset by a race or older than migration 0029, sized to the tick plan. Its declared cost (16 statements on the free plan, 150 on paid) is the run's statement budget, which the backfill enforces on every statement, discovery included, and each run starts at another waiting collection. On the paid plan it also removes rows of collections that list no fields any more, never those of a collection listed while it ran.

.changeset/entry-list-values-core.md

@@ -0,0 +1,9 @@
+---
+"@cloudhal/core-ts": patch
+---
+
+RFC-0042 §4, #110: `queryContentEntries` (the API's `GET /entries?where=…`, Studio's lists, MCP `list_entries`) answers `where`, `orderBy`, status, locale, `limit` and `offset` in SQL from the new `entry_list_values` side table once a collection is ready, instead of reading and evaluating the whole collection; results are the shared evaluator's, which stays the fallback (term filters, collections not ready, operands SQL cannot compare exactly) and the oracle of a seeded property suite on SQLite, D1 and Durable Object SQLite. On the 10,000-entry benchmark the store path drops from 124 ms CPU / 10,001 rows (SQLite) and 176 ms / 10,003 rows read (D1) to 0.6 ms / 22 rows and 5.3 ms / 5,068 rows read; a page at offset 4,980 costs about the same as the first.
+
+The store keeps the table exact. Entry creates, updates, moves and deletes write an entry's rows (its `listFields` values as `resolveFields` reads them) in their own batch, and an original's write rewrites its same-site translations' rows; site writes and field operations that change a collection's list definitions reset its readiness in the same batch, and imports reset the site's. A race never refuses a write: it marks the collection not ready. New `rebuildListValues(sql, options)` is the resumable, budgeted backfill (guarded per chunk, safe to run concurrently; `budget.statements` bounds every statement it issues, `start` rotates where a run begins, `tidy` removes rows of unlisted collections); `updateSite`, finished field operations and `importSite` run it for what they reset, and `openLocalStore` runs it on open. A field operation's revert resets every listed collection whose values its restored bodies change. `deleteContentEntry` behaves as before.
+
+`revertTagsToTerms` now leaves an entry that was edited after the migration's before-snapshot (by a concurrent write, or before a resumed run) as it is and reports it in `edited`, instead of restoring its earlier fields, status or collection.

.changeset/entry-list-values-mcp.md

@@ -0,0 +1,5 @@
+---
+"@cloudhal/mcp-server": patch
+---
+
+`list_entries` with `orderBy` and `limit` lets the store page (in SQL once the collection's list values are ready, RFC-0042 §4, #110) instead of loading every match. Arguments, cursors and output are unchanged; without `orderBy` the list keeps its last-updated order.

.changeset/entry-list-values-storage.md

@@ -0,0 +1,5 @@
+---
+"@cloudhal/storage": patch
+---
+
+Migration `0029_entry_list_values` (RFC-0042 §4, #110): `entry_list_values`, the indexed list-field values of entries (field, presence, number, text, collection-sort key, creation time, node, locale and status) with a query index and a per-node index, and `entry_list_state`, each listed collection's readiness (definition fingerprint, ready flag, backfill epoch and cursor). DDL only; existing data is untouched.

apps/studio/vitest.config.ts

@@ -13,6 +13,9 @@ export default defineConfig({
     // Dialog/menu flows take ~0.5 s alone but several seconds when the whole monorepo tests in
     // parallel (turbo); 5 s made them flaky.
     testTimeout: 20_000,
+    // setup.ts's beforeAll compiles every language's catalogs and lazy namespaces; on a CI runner
+    // busy with other packages it passed the default 10 s (#134).
+    hookTimeout: 60_000,
     // Vitest ≥5.0.4 caches transformed modules on disk by default. Its key does not cover the
     // catalogs() plugin's code (packages/messages), so a compiler change could test stale catalogs.
     fsModuleCache: false,

docs/adr/0044-collections-stage-3.md

@@ -37,7 +37,7 @@ The open stage 3 items from RFC-0040 and RFC-0041 address these: structured data
    - Sorting only by number, date or datetime.
    - `now` is the only relative value (no offsets).
    - One in-memory evaluator (`fields/query.ts`) is shared by themes (`queryEntries`), `ContentList`, the API, MCP and Studio.
-   - No indexes in stage 3. A side table follows only if the 10,000-entry benchmark shows more than 50 ms CPU or more than 10,000 rows read per list.
+   - Step 3's benchmark exceeded the threshold (more than 50 ms CPU or 10,000 rows read per list) on the store, API and MCP path, so `entry_list_values` landed in stage 3 (#110): rows of resolved list-field values, written in the entry's batch; per-collection readiness with a fingerprint and a resumable backfill; the in-memory evaluator stays the fallback and the oracle. Themes and snapshots are unchanged.
    - Studio list columns and filters come from `listFields`.
 5. **Index pages and feeds:**
    - A collection's index page is an ordinary page made by "Create listing page", with a `contentList`; there is no `indexPath`.
@@ -56,9 +56,11 @@ The open stage 3 items from RFC-0040 and RFC-0041 address these: structured data
   - Collections get index pages and feeds.
 - Negative:
   - The structured-data table must follow Google's documentation.
-  - Unindexed listings scan a collection in memory.
+  - Listings scan a collection in memory until its list index is ready.
+  - Every entry write carries two list-value housekeeping statements.
   - Collection settings grow again.
 - Risks:
   - **JSON-LD injection:** mitigated by `jsonLdText` and plain-text constants.
   - **Gates on old site Workers:** guarded by the `gates: 2` feature flag.
   - **Mapping drift:** if a mapped field changes, the operations and `list_field_in_use` refuse or report it.
+  - **A stale list index:** never read; every statement checks readiness under the current definitions' fingerprint.

docs/rfcs/0042-collections-stage-3.md

@@ -245,6 +245,27 @@ The preset table, abridged (R = Google-required; a value source in parentheses i
   - **Next step, if needed:** an `entry_list_values (site_id, collection, locale, field, num, text, node_id)` side table, written in the entry's own batch (rule 4) for `listFields` only, with one composite index.
 
   Unmeasured today; the benchmark decides whether that side table lands in stage 3 or later.
+
+  **Implementation note (2026-10, #110).** Step 3's benchmark (#105, `bench:lists`, 10,000 events) put the store path above open question 10's threshold: `queryContentEntries` took 124 ms CPU / 10,001 rows on SQLite and 176 ms / 10,003 rows read on D1 (the warm snapshot path: 22 ms, 0 rows). So the side table landed in stage 3:
+  - **Shape** (migration 0029). `entry_list_values` keeps the sketched columns and adds `has` (presence; 2 marks one row per option a multiple select holds), `tie` (the collection sort's key; for `title`, mapped so that SQLite's BINARY order is JavaScript's), `created` (the store order) and `status`. `locale` is a trailing covered column, because Studio, the API and MCP send no locale and a locale-first index could neither range-scan nor order those lists. One composite index `(site_id, collection, field, has, num, text, tie, node_id, created, locale, status)` serves every filter and sort; a second, `(node_id, field, has, num, text)`, serves per-entry deletes and the probes of further conditions. `entry_list_state` holds each listed collection's readiness: the fingerprint of its listed definitions and sort, a ready flag, and the backfill's epoch and cursor.
+  - **Rows** are computed in JS from `resolveFields` (a translation reads its original's non-localized values), one per entry and listed field, and written in the entry's own batch; an original's write rewrites its same-site translations' rows. A race never refuses a write: it marks the collection not ready. Numbers that are not safe integers are bound as parameters, because D1's `json_each` does not read every double back exactly.
+  - **Readiness.** A site write that changes a collection's fingerprint resets it in the same batch; a resumable, budgeted backfill (`rebuildListValues`) makes it ready again, run inline after settings changes, field operations and imports, by `openLocalStore`, and by the admin `lists.backfill` job. Field-operation passes and reverts reset every collection that lists a key their bodies change (a revert's restored body can carry an edit made meanwhile); a tags revert leaves an entry edited during the migration alone. The backfill's budget counts statements too: discovery is two (the sites, then their state rows), every later unit asks first, and a chunk whose batch would not fit is cut short, so the job declares its statement budget as its tick cost and starts each run at another waiting collection. On the paid plan the job also removes rows of collections that list no fields, in bounded statements guarded on the site being as read and the collection having no state row, so a collection listed while it ran keeps its rows.
+  - **Queries.** While the collection is ready, `queryContentEntries` runs `where`, `orderBy`, status, locale, `limit` and `offset` in SQL; every statement re-checks readiness under the fingerprint of the site it parsed against. An ordered page is a count and a page per segment, both following the index; an unordered page reads each match's id once, in the store order (sorted without a `LIMIT`: a top-N sort re-reads its sorter for every row on D1), then the page's entries by id, each re-checked against the query; an unpaged answer is one statement. Term filters, collections that are not ready and text operands that are not well-formed UTF-16 run on the evaluator over today's scan. The evaluator is no longer the only executor, but it stays the definition: a seeded property suite compares the two on random data and queries on SQLite, D1 and Durable Object SQLite.
+  - **Cursors** stay offsets. A page in a segment's second half is read from its end, so a deep page costs about what the first one does. Keyset cursors are deferred: MCP's cursor is at most 32 characters, and `total` already reads every match.
+  - **Known limits.** `total` counts every match, so a list whose matches are the whole collection (an order alone, `ne`, `unset`) still reads about N index rows, though with far less CPU; an unordered page also returns each match's id to the Worker. An unpaged answer (the API without `limit`; MCP `list_entries` without `orderBy`, which keeps its last-updated order and so pages itself) returns every match's body, about three rows read per match on D1: above about a third of the collection it reads more rows than the scan did, with less CPU. An ordered page whose first value repeats many times sorts that group when the order's direction and the collection sort's differ. Per-collection counters would be a later step.
+  - **Before and after** (`bench:lists`; on SQLite the bench counts rows returned, on D1 `rows_read`):
+
+    | Path | SQLite before | SQLite after | D1 before | D1 after |
+    |---|---|---|---|---|
+    | `queryContentEntries` (`startsAt gte now`, ordered, 20) | 124 ms / 10,001 | 0.6 ms / 22 | 176 ms / 10,003 | 5.3 ms / 5,068 |
+    | the same at offset 4,980 | – | 0.5 ms / 22 | – | 8.8 ms / 5,088 |
+    | unordered, `price gte 150` (7,000 match), 20 | – | 3.5 ms / 7,021 | – | 10.9 ms / 7,147 |
+    | `startsAt gte now`, ordered, unpaged (5,000 bodies) | – | 20 ms / 5,001 | – | 62 ms / 15,006 |
+    | `listContentEntries`, no `where` (unchanged) | 53 ms / 10,000 | 52 ms / 10,000 | 100 ms / 10,002 | 123 ms / 10,002 |
+    | warm snapshot (unchanged) | 22 ms / 0 | 21 ms / 0 | – | 22 ms / 0 |
+    | rebuild of 10,000 entries (once) | – | 235 ms | – | 761 ms CPU, 3.7 s wall |
+
+    Before the unordered page read ids and the unpaged answer dropped its count, the same two lists read 21,029 and 20,006 rows on D1 (#110's review).
 - **API and MCP:**
   - `GET …/entries` and `list_entries` gain `where` (JSON), `orderBy` and `limit`/`cursor`. Without `limit`, today's unpaged answer stays.
   - `list_collections` returns `listFields` and, per field, the allowed ops.
@@ -393,5 +414,5 @@ Step 3 ships before step 4, so agents never set a rule the live site ignores. St
 7. **Index pages:** "Create listing page" makes an ordinary page; no `indexPath` setting. *Recommended.*
 8. **Per-collection feeds** via middleware after a theme 404, in stage 3 (step 6). *Recommended.*
 9. **Google Indexing API for job removals:** out. It needs a Google service account per site; Google's page recommends it for faster recrawls, but `validThrough` in the past is sufficient. *Recommended.*
-10. **Side table for list values:** only if step 3's benchmark shows a 10,000-entry list above 50 ms CPU or 10,000 rows read per request. *Recommended.*
+10. **Side table for list values:** only if step 3's benchmark shows a 10,000-entry list above 50 ms CPU or 10,000 rows read per request. *Recommended.* Measured in step 3 (#105): above the threshold on the store path; the side table landed in #110 (§4 implementation note).
 11. **References and repeaters:** stage 4, its own RFC. *Recommended.*

packages/astro-renderer/bench/entry-list.test.ts

@@ -1,20 +1,27 @@
 /**
- * RFC-0042 §4 "Indexing: none in stage 3" — the trigger for a side table: a list of a
- * 10,000-entry collection by field, measured on SQLite and on D1 (Miniflare). The RFC's threshold
- * (open question 10): more than 50 ms CPU or more than 10,000 rows read per list means
- * `entry_list_values` lands; below it, in-memory evaluation over today's collection scan stays.
+ * RFC-0042 §4 — a list of a 10,000-entry collection by field, measured on SQLite and on D1
+ * (Miniflare). The RFC's threshold (open question 10): more than 50 ms CPU or more than 10,000
+ * rows read per list. Step 3 measured the store path above it (#105), so `entry_list_values`
+ * landed (#110): `queryContentEntries` (the API, Studio, MCP) now answers from the side table
+ * once the collection is ready, and this asserts it stays under the threshold on both backends.
  *
  * Not part of `pnpm test` (it seeds 10,000 entries): `pnpm --filter @cloudhal/astro-renderer
- * bench:lists`. It prints a table and fails only when a path crosses the threshold.
+ * bench:lists`. It prints a table and fails only when a per-request path crosses the threshold.
  *
  * - CPU is this process's (`process.cpuUsage`): the store's SQL on SQLite, the evaluator, JSON
  *   parsing. On D1 the database runs in workerd, so CPU here is what a Worker spends besides D1.
  * - Rows: rows the driver returned (SQLite) and D1's own `meta.rows_read` (D1).
+ * - Per request too: an unordered page of a broad filter (70% of the collection), which reads
+ *   each match's id once and the page's entries by id.
+ * - Informational: the rebuild of the side table (the backfill a migration or settings change
+ *   needs, writes not counted), a deep page (offset 4,980) and an unpaged answer (every match's
+ *   body, so about three rows read per match on D1).
  */
 import type { Site } from "@cloudhal/core-contracts";
 import {
   GATES_ENFORCED,
   queryContentEntries,
+  rebuildListValues,
   SiteGraphStore,
   seedStarterSite,
 } from "@cloudhal/core-ts";
@@ -41,6 +48,14 @@ const QUERY = {
   limit: 20,
 } satisfies FieldEntryQuery;
 
+/** Unordered, 70% of the collection matches (`price` is `i % 500`), first page. */
+const BROAD = {
+  collection: "events",
+  where: [{ field: "price", op: "gte", value: "150" }],
+  limit: 20,
+} satisfies FieldEntryQuery;
+const { limit: _limit, ...UNPAGED } = QUERY;
+
 /** Counts rows: returned rows, or D1's `rows_read` when the driver reports it. */
 class Counting implements SqlDriver {
   readonly name: string;
@@ -172,6 +187,34 @@ async function seedSqlite(): Promise<{ store: SiteGraphStore; site: Site; sql: S
 }
 
 const samples: Sample[] = [];
+const API_PATH = "store queryContentEntries (API)";
+const BROAD_PATH = "store queryContentEntries, unordered 70%";
+
+/** The side table is ready for `events` (else the store path would measure the scan). */
+async function expectReady(sql: SqlDriver, siteId: string) {
+  const state = await sql.get<{ ready: number }>(
+    "SELECT ready FROM entry_list_state WHERE site_id = ? AND collection = 'events'",
+    [siteId],
+  );
+  expect(state?.ready, "events list values ready").toBe(1);
+}
+
+/** One timed rebuild of the site's list values (rows: reads only; writes are D1's rows_written). */
+async function rebuild(backend: string, counter: Counting, siteId: string): Promise<Sample> {
+  counter.rows = 0;
+  const c0 = process.cpuUsage();
+  const t0 = performance.now();
+  const r = await rebuildListValues(counter, { siteId });
+  const c = process.cpuUsage(c0);
+  return {
+    backend,
+    path: "rebuildListValues (backfill, once)",
+    cpuMs: (c.user + c.system) / 1000,
+    wallMs: performance.now() - t0,
+    rows: counter.rows,
+    matched: r.collections.reduce((n, x) => n + x.written, 0),
+  };
+}
 
 async function runPaths(backend: string, store: SiteGraphStore, counter: Counting, siteId: string) {
   // Today's scan without field conditions, for comparison: what every list already costs.
@@ -181,10 +224,25 @@ async function runPaths(backend: string, store: SiteGraphStore, counter: Countin
     })),
   );
   samples.push(
-    await measure(backend, "store queryContentEntries (API)", counter, () =>
+    await measure(backend, API_PATH, counter, () =>
       queryContentEntries(store, siteId, { ...QUERY, now: NOW }),
     ),
   );
+  samples.push(
+    await measure(backend, "store queryContentEntries, offset 4980", counter, () =>
+      queryContentEntries(store, siteId, { ...QUERY, offset: 4_980, now: NOW }),
+    ),
+  );
+  samples.push(
+    await measure(backend, BROAD_PATH, counter, () =>
+      queryContentEntries(store, siteId, { ...BROAD, now: NOW }),
+    ),
+  );
+  samples.push(
+    await measure(backend, "store queryContentEntries, unpaged", counter, () =>
+      queryContentEntries(store, siteId, { ...UNPAGED, now: NOW }),
+    ),
+  );
   const live = new LiveSource(store, siteId, { includeDrafts: false, now: () => NOW });
   samples.push(
     await measure(backend, "LiveSource.queryEntries", counter, () => live.queryEntries(QUERY)),
@@ -226,7 +284,12 @@ describe(`listing a ${N}-entry collection by field (RFC-0042 §4)`, () => {
     );
     const counter = new Counting(sql);
     const counted = new SiteGraphStore({ sql: counter, blobs: store.blobs, gates: GATES_ENFORCED });
+    await expectReady(sql, site.id);
     await runPaths("sqlite", counted, counter, site.id);
+    // Informational: the same rows rebuilt from scratch, as after a migration.
+    await sql.run("DELETE FROM entry_list_state WHERE site_id = ?", [site.id]);
+    samples.push(await rebuild("sqlite", counter, site.id));
+    await expectReady(sql, site.id);
 
     // The same site on D1 (Miniflare): copied with a bundle import.
     d1 = await createMiniflareD1();
@@ -248,6 +311,9 @@ describe(`listing a ${N}-entry collection by field (RFC-0042 §4)`, () => {
         rows.slice(i, i + 50).map((r) => ({ sql: insert, params: cols.map((c) => r[c] ?? null) })),
       );
     const d1counter = new Counting(d1sql, d1.db);
+    // The raw copy bypassed every write path: build the side table as the backfill job would.
+    samples.push(await rebuild("d1 (miniflare)", d1counter, site.id));
+    await expectReady(d1sql, site.id);
     const d1store = new SiteGraphStore({
       sql: d1counter,
       blobs: target.blobs,
@@ -270,7 +336,10 @@ describe(`listing a ${N}-entry collection by field (RFC-0042 §4)`, () => {
       expect(s.matched, `${s.backend} ${s.path}`).toBeGreaterThan(0);
       if (s.path.startsWith("SnapshotSource warm"))
         expect(s.cpuMs, `${s.backend} ${s.path} cpu`).toBeLessThan(CPU_LIMIT_MS);
+      if (s.path === API_PATH || s.path === BROAD_PATH) {
+        expect(s.cpuMs, `${s.backend} ${s.path} cpu`).toBeLessThan(CPU_LIMIT_MS);
+        expect(s.rows, `${s.backend} ${s.path} rows`).toBeLessThan(ROWS_LIMIT);
+      }
     }
-    expect(ROWS_LIMIT).toBe(10_000);
   });
 });