diff --git a/.changeset/entry-list-values-cloudflare.md b/.changeset/entry-list-values-cloudflare.md new file mode 100644 index 00000000..e1a92559 --- /dev/null +++ b/.changeset/entry-list-values-cloudflare.md @@ -0,0 +1,5 @@ +--- +"@cloudhal/cloudflare": patch +--- + +The admin Worker's minute tick gains a `lists.backfill` system job (every five minutes): a budgeted `rebuildListValues` for listed collections that are new, changed, reset by a race or older than migration 0029, sized to the tick plan. Its declared cost (16 statements on the free plan, 150 on paid) is the run's statement budget, which the backfill enforces on every statement, discovery included, and each run starts at another waiting collection. On the paid plan it also removes rows of collections that list no fields any more, never those of a collection listed while it ran. diff --git a/.changeset/entry-list-values-core.md b/.changeset/entry-list-values-core.md new file mode 100644 index 00000000..7626a8b4 --- /dev/null +++ b/.changeset/entry-list-values-core.md @@ -0,0 +1,9 @@ +--- +"@cloudhal/core-ts": patch +--- + +RFC-0042 §4, #110: `queryContentEntries` (the API's `GET /entries?where=…`, Studio's lists, MCP `list_entries`) answers `where`, `orderBy`, status, locale, `limit` and `offset` in SQL from the new `entry_list_values` side table once a collection is ready, instead of reading and evaluating the whole collection; results are the shared evaluator's, which stays the fallback (term filters, collections not ready, operands SQL cannot compare exactly) and the oracle of a seeded property suite on SQLite, D1 and Durable Object SQLite. On the 10,000-entry benchmark the store path drops from 124 ms CPU / 10,001 rows (SQLite) and 176 ms / 10,003 rows read (D1) to 0.6 ms / 22 rows and 5.3 ms / 5,068 rows read; a page at offset 4,980 costs about the same as the first. + +The store keeps the table exact. Entry creates, updates, moves and deletes write an entry's rows (its `listFields` values as `resolveFields` reads them) in their own batch, and an original's write rewrites its same-site translations' rows; site writes and field operations that change a collection's list definitions reset its readiness in the same batch, and imports reset the site's. A race never refuses a write: it marks the collection not ready. New `rebuildListValues(sql, options)` is the resumable, budgeted backfill (guarded per chunk, safe to run concurrently; `budget.statements` bounds every statement it issues, `start` rotates where a run begins, `tidy` removes rows of unlisted collections); `updateSite`, finished field operations and `importSite` run it for what they reset, and `openLocalStore` runs it on open. A field operation's revert resets every listed collection whose values its restored bodies change. `deleteContentEntry` behaves as before. + +`revertTagsToTerms` now leaves an entry that was edited after the migration's before-snapshot (by a concurrent write, or before a resumed run) as it is and reports it in `edited`, instead of restoring its earlier fields, status or collection. diff --git a/.changeset/entry-list-values-mcp.md b/.changeset/entry-list-values-mcp.md new file mode 100644 index 00000000..c72e3750 --- /dev/null +++ b/.changeset/entry-list-values-mcp.md @@ -0,0 +1,5 @@ +--- +"@cloudhal/mcp-server": patch +--- + +`list_entries` with `orderBy` and `limit` lets the store page (in SQL once the collection's list values are ready, RFC-0042 §4, #110) instead of loading every match. Arguments, cursors and output are unchanged; without `orderBy` the list keeps its last-updated order. diff --git a/.changeset/entry-list-values-storage.md b/.changeset/entry-list-values-storage.md new file mode 100644 index 00000000..65b703fb --- /dev/null +++ b/.changeset/entry-list-values-storage.md @@ -0,0 +1,5 @@ +--- +"@cloudhal/storage": patch +--- + +Migration `0029_entry_list_values` (RFC-0042 §4, #110): `entry_list_values`, the indexed list-field values of entries (field, presence, number, text, collection-sort key, creation time, node, locale and status) with a query index and a per-node index, and `entry_list_state`, each listed collection's readiness (definition fingerprint, ready flag, backfill epoch and cursor). DDL only; existing data is untouched. diff --git a/apps/studio/vitest.config.ts b/apps/studio/vitest.config.ts index 92878ca7..f18c61c2 100644 --- a/apps/studio/vitest.config.ts +++ b/apps/studio/vitest.config.ts @@ -13,6 +13,9 @@ export default defineConfig({ // Dialog/menu flows take ~0.5 s alone but several seconds when the whole monorepo tests in // parallel (turbo); 5 s made them flaky. testTimeout: 20_000, + // setup.ts's beforeAll compiles every language's catalogs and lazy namespaces; on a CI runner + // busy with other packages it passed the default 10 s (#134). + hookTimeout: 60_000, // Vitest ≥5.0.4 caches transformed modules on disk by default. Its key does not cover the // catalogs() plugin's code (packages/messages), so a compiler change could test stale catalogs. fsModuleCache: false, diff --git a/docs/adr/0044-collections-stage-3.md b/docs/adr/0044-collections-stage-3.md index e35bc504..f6c6e80f 100644 --- a/docs/adr/0044-collections-stage-3.md +++ b/docs/adr/0044-collections-stage-3.md @@ -37,7 +37,7 @@ The open stage 3 items from RFC-0040 and RFC-0041 address these: structured data - Sorting only by number, date or datetime. - `now` is the only relative value (no offsets). - One in-memory evaluator (`fields/query.ts`) is shared by themes (`queryEntries`), `ContentList`, the API, MCP and Studio. - - No indexes in stage 3. A side table follows only if the 10,000-entry benchmark shows more than 50 ms CPU or more than 10,000 rows read per list. + - Step 3's benchmark exceeded the threshold (more than 50 ms CPU or 10,000 rows read per list) on the store, API and MCP path, so `entry_list_values` landed in stage 3 (#110): rows of resolved list-field values, written in the entry's batch; per-collection readiness with a fingerprint and a resumable backfill; the in-memory evaluator stays the fallback and the oracle. Themes and snapshots are unchanged. - Studio list columns and filters come from `listFields`. 5. **Index pages and feeds:** - A collection's index page is an ordinary page made by "Create listing page", with a `contentList`; there is no `indexPath`. @@ -56,9 +56,11 @@ The open stage 3 items from RFC-0040 and RFC-0041 address these: structured data - Collections get index pages and feeds. - Negative: - The structured-data table must follow Google's documentation. - - Unindexed listings scan a collection in memory. + - Listings scan a collection in memory until its list index is ready. + - Every entry write carries two list-value housekeeping statements. - Collection settings grow again. - Risks: - **JSON-LD injection:** mitigated by `jsonLdText` and plain-text constants. - **Gates on old site Workers:** guarded by the `gates: 2` feature flag. - **Mapping drift:** if a mapped field changes, the operations and `list_field_in_use` refuse or report it. + - **A stale list index:** never read; every statement checks readiness under the current definitions' fingerprint. diff --git a/docs/rfcs/0042-collections-stage-3.md b/docs/rfcs/0042-collections-stage-3.md index 52e8ea34..0f837eab 100644 --- a/docs/rfcs/0042-collections-stage-3.md +++ b/docs/rfcs/0042-collections-stage-3.md @@ -245,6 +245,27 @@ The preset table, abridged (R = Google-required; a value source in parentheses i - **Next step, if needed:** an `entry_list_values (site_id, collection, locale, field, num, text, node_id)` side table, written in the entry's own batch (rule 4) for `listFields` only, with one composite index. Unmeasured today; the benchmark decides whether that side table lands in stage 3 or later. + + **Implementation note (2026-10, #110).** Step 3's benchmark (#105, `bench:lists`, 10,000 events) put the store path above open question 10's threshold: `queryContentEntries` took 124 ms CPU / 10,001 rows on SQLite and 176 ms / 10,003 rows read on D1 (the warm snapshot path: 22 ms, 0 rows). So the side table landed in stage 3: + - **Shape** (migration 0029). `entry_list_values` keeps the sketched columns and adds `has` (presence; 2 marks one row per option a multiple select holds), `tie` (the collection sort's key; for `title`, mapped so that SQLite's BINARY order is JavaScript's), `created` (the store order) and `status`. `locale` is a trailing covered column, because Studio, the API and MCP send no locale and a locale-first index could neither range-scan nor order those lists. One composite index `(site_id, collection, field, has, num, text, tie, node_id, created, locale, status)` serves every filter and sort; a second, `(node_id, field, has, num, text)`, serves per-entry deletes and the probes of further conditions. `entry_list_state` holds each listed collection's readiness: the fingerprint of its listed definitions and sort, a ready flag, and the backfill's epoch and cursor. + - **Rows** are computed in JS from `resolveFields` (a translation reads its original's non-localized values), one per entry and listed field, and written in the entry's own batch; an original's write rewrites its same-site translations' rows. A race never refuses a write: it marks the collection not ready. Numbers that are not safe integers are bound as parameters, because D1's `json_each` does not read every double back exactly. + - **Readiness.** A site write that changes a collection's fingerprint resets it in the same batch; a resumable, budgeted backfill (`rebuildListValues`) makes it ready again, run inline after settings changes, field operations and imports, by `openLocalStore`, and by the admin `lists.backfill` job. Field-operation passes and reverts reset every collection that lists a key their bodies change (a revert's restored body can carry an edit made meanwhile); a tags revert leaves an entry edited during the migration alone. The backfill's budget counts statements too: discovery is two (the sites, then their state rows), every later unit asks first, and a chunk whose batch would not fit is cut short, so the job declares its statement budget as its tick cost and starts each run at another waiting collection. On the paid plan the job also removes rows of collections that list no fields, in bounded statements guarded on the site being as read and the collection having no state row, so a collection listed while it ran keeps its rows. + - **Queries.** While the collection is ready, `queryContentEntries` runs `where`, `orderBy`, status, locale, `limit` and `offset` in SQL; every statement re-checks readiness under the fingerprint of the site it parsed against. An ordered page is a count and a page per segment, both following the index; an unordered page reads each match's id once, in the store order (sorted without a `LIMIT`: a top-N sort re-reads its sorter for every row on D1), then the page's entries by id, each re-checked against the query; an unpaged answer is one statement. Term filters, collections that are not ready and text operands that are not well-formed UTF-16 run on the evaluator over today's scan. The evaluator is no longer the only executor, but it stays the definition: a seeded property suite compares the two on random data and queries on SQLite, D1 and Durable Object SQLite. + - **Cursors** stay offsets. A page in a segment's second half is read from its end, so a deep page costs about what the first one does. Keyset cursors are deferred: MCP's cursor is at most 32 characters, and `total` already reads every match. + - **Known limits.** `total` counts every match, so a list whose matches are the whole collection (an order alone, `ne`, `unset`) still reads about N index rows, though with far less CPU; an unordered page also returns each match's id to the Worker. An unpaged answer (the API without `limit`; MCP `list_entries` without `orderBy`, which keeps its last-updated order and so pages itself) returns every match's body, about three rows read per match on D1: above about a third of the collection it reads more rows than the scan did, with less CPU. An ordered page whose first value repeats many times sorts that group when the order's direction and the collection sort's differ. Per-collection counters would be a later step. + - **Before and after** (`bench:lists`; on SQLite the bench counts rows returned, on D1 `rows_read`): + + | Path | SQLite before | SQLite after | D1 before | D1 after | + |---|---|---|---|---| + | `queryContentEntries` (`startsAt gte now`, ordered, 20) | 124 ms / 10,001 | 0.6 ms / 22 | 176 ms / 10,003 | 5.3 ms / 5,068 | + | the same at offset 4,980 | – | 0.5 ms / 22 | – | 8.8 ms / 5,088 | + | unordered, `price gte 150` (7,000 match), 20 | – | 3.5 ms / 7,021 | – | 10.9 ms / 7,147 | + | `startsAt gte now`, ordered, unpaged (5,000 bodies) | – | 20 ms / 5,001 | – | 62 ms / 15,006 | + | `listContentEntries`, no `where` (unchanged) | 53 ms / 10,000 | 52 ms / 10,000 | 100 ms / 10,002 | 123 ms / 10,002 | + | warm snapshot (unchanged) | 22 ms / 0 | 21 ms / 0 | – | 22 ms / 0 | + | rebuild of 10,000 entries (once) | – | 235 ms | – | 761 ms CPU, 3.7 s wall | + + Before the unordered page read ids and the unpaged answer dropped its count, the same two lists read 21,029 and 20,006 rows on D1 (#110's review). - **API and MCP:** - `GET …/entries` and `list_entries` gain `where` (JSON), `orderBy` and `limit`/`cursor`. Without `limit`, today's unpaged answer stays. - `list_collections` returns `listFields` and, per field, the allowed ops. @@ -393,5 +414,5 @@ Step 3 ships before step 4, so agents never set a rule the live site ignores. St 7. **Index pages:** "Create listing page" makes an ordinary page; no `indexPath` setting. *Recommended.* 8. **Per-collection feeds** via middleware after a theme 404, in stage 3 (step 6). *Recommended.* 9. **Google Indexing API for job removals:** out. It needs a Google service account per site; Google's page recommends it for faster recrawls, but `validThrough` in the past is sufficient. *Recommended.* -10. **Side table for list values:** only if step 3's benchmark shows a 10,000-entry list above 50 ms CPU or 10,000 rows read per request. *Recommended.* +10. **Side table for list values:** only if step 3's benchmark shows a 10,000-entry list above 50 ms CPU or 10,000 rows read per request. *Recommended.* Measured in step 3 (#105): above the threshold on the store path; the side table landed in #110 (§4 implementation note). 11. **References and repeaters:** stage 4, its own RFC. *Recommended.* diff --git a/packages/astro-renderer/bench/entry-list.test.ts b/packages/astro-renderer/bench/entry-list.test.ts index 7473db86..e0ad9cf2 100644 --- a/packages/astro-renderer/bench/entry-list.test.ts +++ b/packages/astro-renderer/bench/entry-list.test.ts @@ -1,20 +1,27 @@ /** - * RFC-0042 §4 "Indexing: none in stage 3" — the trigger for a side table: a list of a - * 10,000-entry collection by field, measured on SQLite and on D1 (Miniflare). The RFC's threshold - * (open question 10): more than 50 ms CPU or more than 10,000 rows read per list means - * `entry_list_values` lands; below it, in-memory evaluation over today's collection scan stays. + * RFC-0042 §4 — a list of a 10,000-entry collection by field, measured on SQLite and on D1 + * (Miniflare). The RFC's threshold (open question 10): more than 50 ms CPU or more than 10,000 + * rows read per list. Step 3 measured the store path above it (#105), so `entry_list_values` + * landed (#110): `queryContentEntries` (the API, Studio, MCP) now answers from the side table + * once the collection is ready, and this asserts it stays under the threshold on both backends. * * Not part of `pnpm test` (it seeds 10,000 entries): `pnpm --filter @cloudhal/astro-renderer - * bench:lists`. It prints a table and fails only when a path crosses the threshold. + * bench:lists`. It prints a table and fails only when a per-request path crosses the threshold. * * - CPU is this process's (`process.cpuUsage`): the store's SQL on SQLite, the evaluator, JSON * parsing. On D1 the database runs in workerd, so CPU here is what a Worker spends besides D1. * - Rows: rows the driver returned (SQLite) and D1's own `meta.rows_read` (D1). + * - Per request too: an unordered page of a broad filter (70% of the collection), which reads + * each match's id once and the page's entries by id. + * - Informational: the rebuild of the side table (the backfill a migration or settings change + * needs, writes not counted), a deep page (offset 4,980) and an unpaged answer (every match's + * body, so about three rows read per match on D1). */ import type { Site } from "@cloudhal/core-contracts"; import { GATES_ENFORCED, queryContentEntries, + rebuildListValues, SiteGraphStore, seedStarterSite, } from "@cloudhal/core-ts"; @@ -41,6 +48,14 @@ const QUERY = { limit: 20, } satisfies FieldEntryQuery; +/** Unordered, 70% of the collection matches (`price` is `i % 500`), first page. */ +const BROAD = { + collection: "events", + where: [{ field: "price", op: "gte", value: "150" }], + limit: 20, +} satisfies FieldEntryQuery; +const { limit: _limit, ...UNPAGED } = QUERY; + /** Counts rows: returned rows, or D1's `rows_read` when the driver reports it. */ class Counting implements SqlDriver { readonly name: string; @@ -172,6 +187,34 @@ async function seedSqlite(): Promise<{ store: SiteGraphStore; site: Site; sql: S } const samples: Sample[] = []; +const API_PATH = "store queryContentEntries (API)"; +const BROAD_PATH = "store queryContentEntries, unordered 70%"; + +/** The side table is ready for `events` (else the store path would measure the scan). */ +async function expectReady(sql: SqlDriver, siteId: string) { + const state = await sql.get<{ ready: number }>( + "SELECT ready FROM entry_list_state WHERE site_id = ? AND collection = 'events'", + [siteId], + ); + expect(state?.ready, "events list values ready").toBe(1); +} + +/** One timed rebuild of the site's list values (rows: reads only; writes are D1's rows_written). */ +async function rebuild(backend: string, counter: Counting, siteId: string): Promise { + counter.rows = 0; + const c0 = process.cpuUsage(); + const t0 = performance.now(); + const r = await rebuildListValues(counter, { siteId }); + const c = process.cpuUsage(c0); + return { + backend, + path: "rebuildListValues (backfill, once)", + cpuMs: (c.user + c.system) / 1000, + wallMs: performance.now() - t0, + rows: counter.rows, + matched: r.collections.reduce((n, x) => n + x.written, 0), + }; +} async function runPaths(backend: string, store: SiteGraphStore, counter: Counting, siteId: string) { // Today's scan without field conditions, for comparison: what every list already costs. @@ -181,10 +224,25 @@ async function runPaths(backend: string, store: SiteGraphStore, counter: Countin })), ); samples.push( - await measure(backend, "store queryContentEntries (API)", counter, () => + await measure(backend, API_PATH, counter, () => queryContentEntries(store, siteId, { ...QUERY, now: NOW }), ), ); + samples.push( + await measure(backend, "store queryContentEntries, offset 4980", counter, () => + queryContentEntries(store, siteId, { ...QUERY, offset: 4_980, now: NOW }), + ), + ); + samples.push( + await measure(backend, BROAD_PATH, counter, () => + queryContentEntries(store, siteId, { ...BROAD, now: NOW }), + ), + ); + samples.push( + await measure(backend, "store queryContentEntries, unpaged", counter, () => + queryContentEntries(store, siteId, { ...UNPAGED, now: NOW }), + ), + ); const live = new LiveSource(store, siteId, { includeDrafts: false, now: () => NOW }); samples.push( await measure(backend, "LiveSource.queryEntries", counter, () => live.queryEntries(QUERY)), @@ -226,7 +284,12 @@ describe(`listing a ${N}-entry collection by field (RFC-0042 §4)`, () => { ); const counter = new Counting(sql); const counted = new SiteGraphStore({ sql: counter, blobs: store.blobs, gates: GATES_ENFORCED }); + await expectReady(sql, site.id); await runPaths("sqlite", counted, counter, site.id); + // Informational: the same rows rebuilt from scratch, as after a migration. + await sql.run("DELETE FROM entry_list_state WHERE site_id = ?", [site.id]); + samples.push(await rebuild("sqlite", counter, site.id)); + await expectReady(sql, site.id); // The same site on D1 (Miniflare): copied with a bundle import. d1 = await createMiniflareD1(); @@ -248,6 +311,9 @@ describe(`listing a ${N}-entry collection by field (RFC-0042 §4)`, () => { rows.slice(i, i + 50).map((r) => ({ sql: insert, params: cols.map((c) => r[c] ?? null) })), ); const d1counter = new Counting(d1sql, d1.db); + // The raw copy bypassed every write path: build the side table as the backfill job would. + samples.push(await rebuild("d1 (miniflare)", d1counter, site.id)); + await expectReady(d1sql, site.id); const d1store = new SiteGraphStore({ sql: d1counter, blobs: target.blobs, @@ -270,7 +336,10 @@ describe(`listing a ${N}-entry collection by field (RFC-0042 §4)`, () => { expect(s.matched, `${s.backend} ${s.path}`).toBeGreaterThan(0); if (s.path.startsWith("SnapshotSource warm")) expect(s.cpuMs, `${s.backend} ${s.path} cpu`).toBeLessThan(CPU_LIMIT_MS); + if (s.path === API_PATH || s.path === BROAD_PATH) { + expect(s.cpuMs, `${s.backend} ${s.path} cpu`).toBeLessThan(CPU_LIMIT_MS); + expect(s.rows, `${s.backend} ${s.path} rows`).toBeLessThan(ROWS_LIMIT); + } } - expect(ROWS_LIMIT).toBe(10_000); }); }); diff --git a/packages/cloudflare/src/list-values-job.ts b/packages/cloudflare/src/list-values-job.ts new file mode 100644 index 00000000..eda49b02 --- /dev/null +++ b/packages/cloudflare/src/list-values-job.ts @@ -0,0 +1,46 @@ +/** + * RFC-0042 §4, #110: the admin Worker's `lists.backfill` system job. Every five minutes it makes + * listed collections' `entry_list_values` ready again where a settings change, a field + * operation, an import, a race or a deployment from before migration 0029 left them not ready + * (`rebuildListValues`, resumable and guarded, so a run cut short loses nothing), within a budget + * that fits the tick plan. On the paid plan it also removes rows of collections that list no + * fields any more. Until a collection is ready, its listings scan as before. + * + * The declared `cost` is the run's statement budget, which `rebuildListValues` enforces on every + * statement it issues (discovery, chunks and tidy alike), so the tick's meter never sees more. + * Each run starts at another not-ready collection, so a busy one cannot hold the rest back. + */ +import { rebuildListValues } from "@cloudhal/core-ts"; +import type { SystemJobDefinition } from "@cloudhal/schedules"; +import type { SqlDriver } from "@cloudhal/storage"; + +const EVERY_MS = 5 * 60_000; + +/** The next five-minute mark strictly after `after` (UTC). */ +export const everyFiveMinutes = (after: Date): Date => + new Date((Math.floor(after.getTime() / EVERY_MS) + 1) * EVERY_MS); + +export function listValuesJob( + sql: SqlDriver, + o: { paid: boolean; log: (line: string) => void }, +): SystemJobDefinition { + // Discovery is two statements, a chunk of 100 entries about five; the free tick allows 40 in all. + const budget = o.paid + ? { entries: 2000, ms: 20_000, statements: 150 } + : { entries: 200, ms: 5_000, statements: 16 }; + return { + name: "lists.backfill", + cost: budget.statements, + next: everyFiveMinutes, + async run({ now }) { + const start = Math.floor(now.getTime() / EVERY_MS); + const r = await rebuildListValues(sql, { budget, tidy: o.paid, start }); + const written = r.collections.reduce((n, c) => n + c.written, 0); + const waiting = r.collections.filter((c) => !c.ready).length; + if (written + r.tidied > 0 || r.exhausted) + o.log( + `lists backfill: ${written} entr${written === 1 ? "y" : "ies"} indexed, ${r.tidied} row(s) tidied, ${waiting} collection(s) not ready${r.exhausted ? " (more next run)" : ""}`, + ); + }, + }; +} diff --git a/packages/cloudflare/src/scheduler.ts b/packages/cloudflare/src/scheduler.ts index 5841dd75..e8834be5 100644 --- a/packages/cloudflare/src/scheduler.ts +++ b/packages/cloudflare/src/scheduler.ts @@ -63,6 +63,7 @@ import type { SqlDriver } from "@cloudhal/storage"; import { D1Driver } from "@cloudhal/storage/d1"; import { R2BlobStore } from "@cloudhal/storage/r2"; import { flushCommentPurges, runCommentsHourly } from "./comments.ts"; +import { listValuesJob } from "./list-values-job.ts"; import { PURGE_MAX_REQUESTS, PURGE_TAGS_PER_REQUEST } from "./purge.ts"; import { attachedSite, bindingFor, routedSite } from "./site-routes.ts"; @@ -270,6 +271,7 @@ export function adminSystemJobs( log(`schedules retention: ${r.runs} run(s), ${r.releases} release(s) deleted`); }, }, + listValuesJob(sql, { paid: env.CLOUDHAL_TICK_BUDGET === "paid", log }), ]; if (o.commentsOn) jobs.push({ diff --git a/packages/cloudflare/test/comments.test.ts b/packages/cloudflare/test/comments.test.ts index 64f56557..02d83b5e 100644 --- a/packages/cloudflare/test/comments.test.ts +++ b/packages/cloudflare/test/comments.test.ts @@ -25,6 +25,7 @@ import { flushCommentPurges, runCommentsHourly, } from "../src/comments.ts"; +import { everyFiveMinutes } from "../src/list-values-job.ts"; import { purgeSiteCache } from "../src/purge.ts"; const T0 = new Date("2026-10-12T12:00:00.000Z"); @@ -344,7 +345,9 @@ describe("availability and the Worker", () => { true, ); expect(lines.some((l) => /^analytics rollup \(none\)/.test(l))).toBe(true); - expect(lines.some((l) => /^schedules tick \(cron\): 2 ran/.test(l))).toBe(true); + // `lists.backfill` (every five minutes, #110) is due too unless the first tick came after :15. + const ran = everyFiveMinutes(new Date(first)).getTime() <= Date.parse(at17) ? 3 : 2; + expect(lines.some((l) => l.startsWith(`schedules tick (cron): ${ran} ran`))).toBe(true); // The watermark moved to the next hour: the next minute does not run it again. const n = lines.length; await tick(Date.parse(at17) + 64_100); diff --git a/packages/cloudflare/test/list-values-job.test.ts b/packages/cloudflare/test/list-values-job.test.ts new file mode 100644 index 00000000..80fd45f3 --- /dev/null +++ b/packages/cloudflare/test/list-values-job.test.ts @@ -0,0 +1,179 @@ +/** RFC-0042 §4, #110: the admin Worker's `lists.backfill` system job on D1. */ +import { SiteGraphStore } from "@cloudhal/core-ts"; +import { MemoryBlobStore, type SqlDriver } from "@cloudhal/storage"; +import { D1Driver } from "@cloudhal/storage/d1"; +import { createMiniflareD1 } from "@cloudhal/storage/testing"; +import { afterAll, describe, expect, it } from "vitest"; +import { everyFiveMinutes, listValuesJob } from "../src/list-values-job.ts"; + +let d1: Awaited> | undefined; +afterAll(async () => { + await d1?.dispose(); +}); + +/** `sql`, counting statements as the tick's meter does (a batch counts each of its statements). */ +function counting(sql: SqlDriver) { + const c = { n: 0, sql: undefined as unknown as SqlDriver }; + c.sql = { + name: `${sql.name}+count`, + get: async (q, p) => { + c.n++; + return sql.get(q, p); + }, + all: async (q, p) => { + c.n++; + return sql.all(q, p); + }, + run: async (q, p) => { + c.n++; + return sql.run(q, p); + }, + batch: async (st) => { + c.n += st.length; + return sql.batch(st); + }, + exec: (q) => sql.exec(q), + close: async () => {}, + }; + return c; +} + +const author = { kind: "human" as const, id: "user_01HZX8K3Q9CD", name: "H" }; +const price = [ + { key: "price", type: "number", labels: { en: { label: "Price" } }, localized: false }, +]; + +describe("lists.backfill", () => { + it("runs on five-minute marks", () => { + expect(everyFiveMinutes(new Date("2026-10-26T10:03:10Z")).toISOString()).toBe( + "2026-10-26T10:05:00.000Z", + ); + expect(everyFiveMinutes(new Date("2026-10-26T10:05:00Z")).toISOString()).toBe( + "2026-10-26T10:10:00.000Z", + ); + }); + + it("makes a collection without a state row ready, and logs only when it did work", async () => { + d1 = await createMiniflareD1(); + const sql = new D1Driver(d1.db); + const store = new SiteGraphStore({ sql, blobs: new MemoryBlobStore() }); + await store.init(); + const project = await store.createProject({ + name: "J", + slug: "j", + ownerId: "user_01HZX8K3Q9CD", + siteIds: [], + }); + const fields = [ + { key: "price", type: "number", labels: { en: { label: "Price" } }, localized: false }, + ]; + const site = await store.createSite({ + projectId: project.id, + name: "J", + slug: "j", + defaultLocale: "en", + locales: ["en"], + domains: [], + pageIds: [], + theme: { name: "blog" }, + collections: { events: { fields, listFields: ["price"] } } as never, + }); + await store.createContentEntry({ + siteId: site.id, + collection: "events", + slug: "a", + locale: "en", + status: "draft", + title: "A", + body: { format: "markdown", value: "A" }, + fields: { price: 1.5 }, + tags: [], + author: { kind: "human", id: "user_01HZX8K3Q9CD", name: "H" }, + }); + // A deployment from before migration 0029 has rows of nothing and no state. + await sql.run("DELETE FROM entry_list_state"); + await sql.run("DELETE FROM entry_list_values"); + const lines: string[] = []; + const job = listValuesJob(sql, { paid: false, log: (l) => lines.push(l) }); + expect(job.name).toBe("lists.backfill"); + await job.run({ now: new Date(), trigger: "cron" }); + expect( + await sql.get("SELECT ready FROM entry_list_state WHERE site_id = ?", [site.id]), + ).toEqual({ ready: 1 }); + expect(await sql.get("SELECT num FROM entry_list_values WHERE site_id = ?", [site.id])).toEqual( + { num: 1.5 }, + ); + expect(lines).toEqual([expect.stringMatching(/^lists backfill: 1 entry indexed/)]); + await job.run({ now: new Date(), trigger: "cron" }); + expect(lines).toHaveLength(1); + }); + + it("never issues more statements than its declared cost, however many collections wait", async () => { + d1 ??= await createMiniflareD1(); + const sql = new D1Driver(d1.db); + const store = new SiteGraphStore({ sql, blobs: new MemoryBlobStore() }); + await store.init(); + await sql.run("DELETE FROM entry_list_state"); + await sql.run("DELETE FROM entry_list_values"); + const sites: string[] = []; + for (let i = 0; i < 4; i++) { + const project = await store.createProject({ + name: `C${i}`, + slug: `c${i}`, + ownerId: author.id, + siteIds: [], + }); + const collections = Object.fromEntries( + ["a", "b", "c", "d", "e"].map((c) => [c, { fields: price, listFields: ["price"] }]), + ); + const site = await store.createSite({ + projectId: project.id, + name: `C${i}`, + slug: `c${i}`, + defaultLocale: "en", + locales: ["en"], + domains: [], + pageIds: [], + theme: { name: "blog" }, + collections: { ...collections, gone: { fields: price } } as never, + }); + sites.push(site.id); + for (const c of ["a", "c"]) + await store.createContentEntry({ + siteId: site.id, + collection: c, + slug: "x", + locale: "en", + status: "draft", + title: "X", + body: { format: "markdown", value: "X" }, + fields: { price: i + 0.5 }, + tags: [], + author, + }); + } + // As after an upgrade to migration 0029: twenty listed collections, none with a state row. + await sql.run("DELETE FROM entry_list_state"); + for (const paid of [false, true]) { + const counted = counting(sql); + const job = listValuesJob(counted.sql, { paid, log: () => {} }); + let runs = 0; + for (; runs < 40; runs++) { + counted.n = 0; + await job.run({ now: new Date(Date.UTC(2026, 9, 26, 10, 5 * runs)), trigger: "cron" }); + expect(counted.n, `run ${runs}`).toBeLessThanOrEqual(job.cost); + const ready = await sql.get<{ n: number }>( + "SELECT count(*) AS n FROM entry_list_state WHERE ready = 1 AND site_id IN (SELECT value FROM json_each(?))", + [JSON.stringify(sites)], + ); + if (Number(ready?.n) === 20) break; + } + expect(runs).toBeLessThan(40); + // Nothing left: discovery alone, two statements (and, paid, one tidy for every site). + counted.n = 0; + await job.run({ now: new Date(), trigger: "cron" }); + expect(counted.n).toBe(paid ? 3 : 2); + await sql.run("DELETE FROM entry_list_state"); + } + }); +}); diff --git a/packages/core-ts/src/entry-query.ts b/packages/core-ts/src/entry-query.ts index 22fdece0..07efd602 100644 --- a/packages/core-ts/src/entry-query.ts +++ b/packages/core-ts/src/entry-query.ts @@ -1,12 +1,17 @@ /** * RFC-0042 §4 "API and MCP": a collection's entries filtered and sorted by its list fields, over * the store's collection prefilter (`listContentEntries`: collection, locale, status, the RFC-0002 - * term filter). Field conditions run in memory with the shared evaluator (`fields/query.ts`) on - * resolved values (a translation reads its original's non-localized ones), so SQL and memory - * cannot drift. Ties fall back to the collection's `sort`, then `id`; entries without a value sort - * last. `now` is injected (epoch ms), never read from a clock here. + * term filter). The shared evaluator (`fields/query.ts`) defines the answer, on resolved values (a + * translation reads its original's non-localized ones): ties fall back to the collection's + * `sort`, then `id`; entries without a value sort last. `now` is injected (epoch ms), never read + * from a clock here. + * + * #110: while the collection's `entry_list_values` rows are ready, the query runs in SQL + * (`list-values/query.ts`): where, orderBy, status, locale, limit and offset. It falls back to + * the in-memory scan (`scanFieldQuery`, the oracle a property suite compares it with) for a term + * filter, a collection that is not ready, or an operand SQL cannot compare exactly. */ -import type { ContentEntry, FieldCondition, FieldOrder } from "@cloudhal/core-contracts"; +import type { ContentEntry, FieldCondition, FieldOrder, Site } from "@cloudhal/core-contracts"; import { InvariantError, NotFoundError } from "./errors.ts"; import { compareByField, @@ -18,6 +23,7 @@ import { type ResolvedFields, resolveFields, } from "./fields/index.ts"; +import { queryListValues } from "./list-values/query.ts"; import type { SiteGraphStore } from "./store.ts"; import type { EntryQuery } from "./taxonomy.ts"; @@ -108,6 +114,44 @@ export function applyFieldQuery( .map((e) => e.entry); } +const slicePage = (entries: ContentEntry[], q: FieldEntryListQuery): FieldEntryList => { + const start = q.offset ?? 0; + const end = q.limit === undefined ? undefined : start + q.limit; + return { entries: entries.slice(start, end), total: entries.length }; +}; + +/** The prefiltered `entries` resolved (originals read as the store holds them) and evaluated. */ +async function evaluate( + store: SiteGraphStore, + site: Site, + entries: readonly ContentEntry[], + q: { collection: string; query: ParsedFieldQuery; now: number }, +): Promise { + const byId = await originals(store, entries); + const rows = entries.map((entry) => { + const group = entry.translation?.group; + const original = group && group !== entry.id ? byId.get(group) : undefined; + return { entry, resolved: resolveFields(site, entry, original) }; + }); + const sort = site.collections?.[q.collection]?.sort ?? "newest"; + return applyFieldQuery(rows, q.query, { now: q.now, sort }); +} + +/** + * The in-memory answer: the store's prefilter, then the evaluator over every entry it returns. + * The definition `queryContentEntries` answers by, and the pushdown's oracle in tests. + */ +export async function scanFieldQuery( + store: SiteGraphStore, + site: Site, + query: ParsedFieldQuery, + q: FieldEntryListQuery & { collection: string }, +): Promise { + const { where: _w, orderBy: _o, limit: _l, offset: _f, now, ...prefilter } = q; + const entries = await store.listContentEntries(site.id, prefilter); + return slicePage(await evaluate(store, site, entries, { ...q, query, now }), q); +} + /** * The entries of `q.collection` matching `q.where`, ordered by `q.orderBy` (else the store's * order), paged by `limit`/`offset`. Refuses as `invalid_query` (with paths) when the query does @@ -120,23 +164,19 @@ export async function queryContentEntries( ): Promise { const bounds = checkBounds(q); if (bounds.length > 0) refuse(bounds); - const { where, orderBy, limit, offset, now, ...prefilter } = q; - let entries = await store.listContentEntries(siteId, prefilter); - if (q.collection !== undefined && (where !== undefined || orderBy !== undefined)) { - const site = await store.getSite(siteId); - if (!site) throw new NotFoundError("site", siteId); - const parsed = parseFieldQuery(site, q.collection, { where, orderBy }); - if (!parsed.ok) return refuse(parsed.issues); - const byId = await originals(store, entries); - const rows = entries.map((entry) => { - const group = entry.translation?.group; - const original = group && group !== entry.id ? byId.get(group) : undefined; - return { entry, resolved: resolveFields(site, entry, original) }; - }); - const sort = site.collections?.[q.collection]?.sort ?? "newest"; - entries = applyFieldQuery(rows, parsed.query, { now, sort }); - } - const start = offset ?? 0; - const page = limit === undefined ? entries.slice(start) : entries.slice(start, start + limit); - return { entries: page, total: entries.length }; + const { where, orderBy, limit: _l, offset: _f, now, ...prefilter } = q; + const { collection } = q; + if (collection === undefined || (where === undefined && orderBy === undefined)) + return slicePage(await store.listContentEntries(siteId, prefilter), q); + // A term filter is compiled by the store's scan, and refused there first, as before. + const listed = + prefilter.filter === undefined ? undefined : await store.listContentEntries(siteId, prefilter); + const site = await store.getSite(siteId); + if (!site) throw new NotFoundError("site", siteId); + const parsed = parseFieldQuery(site, collection, { where, orderBy }); + if (!parsed.ok) return refuse(parsed.issues); + const query = parsed.query; + if (listed) return slicePage(await evaluate(store, site, listed, { collection, query, now }), q); + const pushed = await queryListValues(store.sql, site, { ...q, collection, query }); + return pushed ?? scanFieldQuery(store, site, query, { ...q, collection }); } diff --git a/packages/core-ts/src/fields-operations/revert.ts b/packages/core-ts/src/fields-operations/revert.ts index 13f8e158..19c2dabc 100644 --- a/packages/core-ts/src/fields-operations/revert.ts +++ b/packages/core-ts/src/fields-operations/revert.ts @@ -26,6 +26,7 @@ import { readOperation, refuseRunning, requireSite, + settleLists, writeSite, } from "./state.ts"; import { @@ -38,6 +39,7 @@ import { type OperationContext, operationContext, operationKey, + operationKeys, } from "./types.ts"; /** The marker that reads the operation's end state as the earlier one's source. */ @@ -217,6 +219,7 @@ export async function revertFieldOperation( guard: marked ? markerGuard(siteId, op.collection, operationId) : undefined, freshAlso: (sourceId) => (earlier.has(sourceId) ? snaps.before.get(sourceId) : undefined), done: out.restored + out.edited.length, + lists: { collection: op.collection, keys: operationKeys(op) }, }); for (const w of r.written) if (w.decision?.replace) out.restored++; @@ -239,5 +242,6 @@ export async function revertFieldOperation( label: labelOf("revert ", op), metadata: { [META.kind]: FIELD_MIGRATION, [META.id]: operationId, [META.phase]: "reverted" }, }); + await settleLists(ctx, siteId, out.restored + out.edited.length); return { ...out, done: true, skipped: [], snapshotId: snapshot.id }; } diff --git a/packages/core-ts/src/fields-operations/rewrite.ts b/packages/core-ts/src/fields-operations/rewrite.ts index 50ef4922..a2c29278 100644 --- a/packages/core-ts/src/fields-operations/rewrite.ts +++ b/packages/core-ts/src/fields-operations/rewrite.ts @@ -14,7 +14,9 @@ import { } from "@cloudhal/core-contracts"; import type { SqlStatement } from "@cloudhal/storage"; import type { ValueChange } from "../fields/migration.ts"; +import { sameJson } from "../fields/types.ts"; import { nodeHash } from "../hash.ts"; +import { passResetStatement } from "../list-values/state-sql.ts"; import { fieldAssetRefs } from "../media/field-refs.ts"; import { and, type Cond, casUpdateStatement, hashIs } from "../node-sql.ts"; import { fieldRefStatements } from "../store/entry-fields.ts"; @@ -76,6 +78,13 @@ export interface PassInput { readonly freshAlso?: ((sourceId: string) => string | undefined) | undefined; /** Entries this call already rewrote (the budget counts across passes). */ readonly done: number; + /** + * RFC-0042 §4: the collection and the field keys the operation rewrites. Each batch resets + * (`passResetStatement`) the list values of collections listing one of them, or any other key a + * body of the batch changes (a revert's restored body can carry an edit made while the + * operation ran); the backfill rebuilds them once the operation is done. + */ + readonly lists: { readonly collection: string; readonly keys: readonly string[] }; } export interface PassResult { @@ -152,6 +161,17 @@ async function statementsOf(input: PassInput, p: Planned): Promise done >= ctx.budget.entries || Date.now() - ctx.started >= ctx.budget.ms; +/** The operation's keys, and every other field key a planned body changes. */ +function resetKeys(keys: readonly string[], planned: readonly Planned[]): string[] { + const out = new Set(keys); + for (const p of planned) { + const [was, now] = [p.stored.entry.fields ?? {}, p.next.fields ?? {}]; + for (const k of new Set([...Object.keys(was), ...Object.keys(now)])) + if (!out.has(k) && !sameJson(was[k], now[k])) out.add(k); + } + return [...out]; +} + /** One pass over the rows: plan, then write in batches until done or out of budget. */ export async function rewritePass(input: PassInput): Promise { const { ctx } = input; @@ -171,6 +191,13 @@ export async function rewritePass(input: PassInput): Promise { heads.push({ index: statements.length, p }); statements.push(...(await statementsOf(input, p))); } + const batch = heads.map((h) => h.p); + const ids = batch.map((p) => p.next.id); + const { collection } = input.lists; + const keys = resetKeys(input.lists.keys, batch); + statements.push( + passResetStatement({ siteId: input.site.id, ids, at: ctx.stamp(), collection, keys }), + ); const results = await ctx.store.sql.batch(statements); for (const { index, p } of heads) { const id = p.next.id; diff --git a/packages/core-ts/src/fields-operations/run.ts b/packages/core-ts/src/fields-operations/run.ts index ab176b6e..c98f6da8 100644 --- a/packages/core-ts/src/fields-operations/run.ts +++ b/packages/core-ts/src/fields-operations/run.ts @@ -30,6 +30,7 @@ import { readOperation, refuseRunning, requireSite, + settleLists, writeSite, } from "./state.ts"; import { @@ -42,6 +43,7 @@ import { META, type OperationContext, operationContext, + operationKeys, } from "./types.ts"; /** Passes per call while concurrent edits keep entries skipped. */ @@ -240,6 +242,7 @@ async function proceed( decide: decider(site, marked.def, op, (eid, hash) => hashes.get(eid) === hash), guard: markerGuard(siteId, op.collection, id), done: rewritten, + lists: { collection: op.collection, keys: operationKeys(op) }, }); rewritten += r.rewritten; skipped = r.skipped; @@ -260,6 +263,7 @@ async function proceed( label: labelOf("", op), metadata: { [META.kind]: FIELD_MIGRATION, [META.id]: id, [META.phase]: "after" }, }); + await settleLists(ctx, siteId, rewritten); return result({ before, after }, report, { rewritten, skipped: [], done: true }); } diff --git a/packages/core-ts/src/fields-operations/state.ts b/packages/core-ts/src/fields-operations/state.ts index 2e0074a8..122c4509 100644 --- a/packages/core-ts/src/fields-operations/state.ts +++ b/packages/core-ts/src/fields-operations/state.ts @@ -6,9 +6,13 @@ * writes may set or drop one. They still pass the precompiled validator and the definition rules. */ import { assertValid, type Site, type Snapshot } from "@cloudhal/core-contracts"; +import type { SqlStatement } from "@cloudhal/storage"; import { ConflictError, InvariantError, NotFoundError } from "../errors.ts"; import { checkFieldDefinitions, refuseFieldIssues } from "../fields/index.ts"; +import { nodeHash } from "../hash.ts"; import { ID_PREFIX, newId } from "../ids.ts"; +import { settleListValues } from "../list-values/backfill.ts"; +import { planListState } from "../list-values/site-plan.ts"; import { type Cond, casUpdateStatement, OpBatch, runOpBatch } from "../node-sql.ts"; import type { PageRewrite } from "./references.ts"; import { @@ -89,12 +93,20 @@ export async function writeSite( assertValid("site", next); refuseFieldIssues(checkFieldDefinitions(next)); const rewrites = (await pages?.(site)) ?? []; + // RFC-0042 §4: a marker on a listed field changes its fingerprint (phases A and C both). + const lists = (await planListState(site, next)).statements( + await nodeHash(next), + next.updatedAt, + ); if (rewrites.length > 0) { - if (await writeWithPages(ctx, { site, next, guard, rewrites })) return "written"; + if (await writeWithPages(ctx, { site, next, guard, rewrites, lists })) return "written"; } else { const s = await casUpdateStatement("site", next, site.rev, guard); - const r = await ctx.store.sql.run(s.sql, s.params ?? []); - if (r.changes === 1) return "written"; + const sql = ctx.store.sql; + const r = lists.length + ? (await sql.batch([s, ...lists]))[0] + : await sql.run(s.sql, s.params ?? []); + if (r?.changes === 1) return "written"; } rev = (await ctx.store.getSite(siteId))?.rev; if (rev === site.rev && rewrites.length === 0) return "guarded"; @@ -108,7 +120,13 @@ export async function writeSite( */ async function writeWithPages( ctx: OperationContext, - write: { site: Site; next: Site; guard: Cond | undefined; rewrites: readonly PageRewrite[] }, + write: { + site: Site; + next: Site; + guard: Cond | undefined; + rewrites: readonly PageRewrite[]; + lists: readonly SqlStatement[]; + }, ): Promise { const batch = new OpBatch(newId(ID_PREFIX.operation), write.site.id, write.next.updatedAt); await batch.update("site", write.site, write.next, write.guard); @@ -116,6 +134,7 @@ async function writeWithPages( assertValid("page", p.next); await batch.update("page", p.prev, p.next); } + batch.append(...write.lists); try { await runOpBatch(ctx.store.sql, batch); return true; @@ -139,3 +158,14 @@ export function refuseRunning(collection: string, id: string): never { { operationId: id, collection }, ); } + +/** + * RFC-0042 §4: once an operation is done, the list values it reset are rebuilt with what is left + * of its budget (`used` entries so far); the admin job finishes larger collections. + */ +export async function settleLists(ctx: OperationContext, siteId: string, used: number) { + const ms = ctx.budget.ms - (Date.now() - ctx.started); + const entries = ctx.budget.entries - used; + if (entries <= 0 || ms <= 0) return; + await settleListValues(ctx.store.sql, siteId, { budget: { entries, ms } }); +} diff --git a/packages/core-ts/src/fields-operations/types.ts b/packages/core-ts/src/fields-operations/types.ts index b3c555df..eab5aef9 100644 --- a/packages/core-ts/src/fields-operations/types.ts +++ b/packages/core-ts/src/fields-operations/types.ts @@ -173,3 +173,8 @@ export const operationKey = (op: FieldOperation): string => /** The field key once the operation is done. */ export const finalKey = (op: FieldOperation): string => (op.op === "rename_field" ? op.to : op.key); + +/** Every field key the operation rewrites in entries (a rename's both). */ +export const operationKeys = (op: FieldOperation): string[] => [ + ...new Set([operationKey(op), finalKey(op)]), +]; diff --git a/packages/core-ts/src/fields/query.ts b/packages/core-ts/src/fields/query.ts index e706c816..2ebdeb69 100644 --- a/packages/core-ts/src/fields/query.ts +++ b/packages/core-ts/src/fields/query.ts @@ -1,8 +1,8 @@ /** * RFC-0042 §4: listings by field. One evaluator for every caller — themes (`queryEntries`), * `contentList` sections, the API, MCP and Studio's list columns — following `filter.ts`'s pattern - * (one parser, one meaning). It runs in memory after the store's collection prefilter, so SQL and - * in-memory results cannot drift. Pure, Worker-safe and Temporal-free. + * (one parser, one meaning). It defines every answer: the store's SQL over `entry_list_values` + * (#110) is checked against it, and it runs whenever that cannot. Pure, Worker-safe, Temporal-free. * * | Field type | `where` ops | `orderBy` | `value` | * |---|---|---|---| @@ -251,7 +251,7 @@ export function parseFieldQuery( } /** A comparable form of a resolved value: a number, a day string, or undefined. */ -function sortKey(def: FieldDefinition, value: unknown): number | string | undefined { +export function sortKey(def: FieldDefinition, value: unknown): number | string | undefined { if (value === undefined || value === null) return undefined; if (def.type === "number") return typeof value === "number" ? value : undefined; if (def.type === "date") return typeof value === "string" ? value : undefined; @@ -264,7 +264,7 @@ function sortKey(def: FieldDefinition, value: unknown): number | string | undefi } /** An operand's comparable form at `now`. */ -function operandKey( +export function operandKey( o: FieldOperand, def: FieldDefinition, now: number, diff --git a/packages/core-ts/src/index.ts b/packages/core-ts/src/index.ts index 5ae75490..f4e9fe39 100644 --- a/packages/core-ts/src/index.ts +++ b/packages/core-ts/src/index.ts @@ -47,6 +47,13 @@ export * from "./hash.ts"; export type { NodeHold } from "./holds.ts"; export * from "./hosts.ts"; export * from "./ids.ts"; +export { + type ListValuesBudget, + type RebuildListValuesOptions, + type RebuildListValuesResult, + type RebuiltCollection, + rebuildListValues, +} from "./list-values/backfill.ts"; export * from "./locale-paths.ts"; export { importMediaUrls, diff --git a/packages/core-ts/src/list-values/backfill.ts b/packages/core-ts/src/list-values/backfill.ts new file mode 100644 index 00000000..c2b303e8 --- /dev/null +++ b/packages/core-ts/src/list-values/backfill.ts @@ -0,0 +1,346 @@ +/** + * RFC-0042 §4, #110: the resumable rebuild of `entry_list_values`. A collection's rows are read + * only while its `entry_list_state` row is ready under the current fingerprint; this makes it so. + * + * Discovery is two statements whatever the deployment's size: the sites, then their state rows; + * collections already ready under their fingerprint cost nothing more. Per collection that is + * not: the state row is (re)started when it is missing (a deployment from before migration 0029) + * or was built under other definitions; then entries are read in key order, 100 at a time with + * their originals, their rows computed in JS (`listValueRows`, the writers' function) and written + * in one atomic batch per chunk. Every statement of a chunk is guarded on the state row being + * exactly as read (fingerprint, `epoch`, `cursor`, not ready), so two runners never interleave + * and a reset in between makes the rest of the run a no-op; each node's rows also need the node, + * and its original, to still have the hash they were computed from. The cursor moves only when + * every node of the chunk did, so a node a concurrent writer changed is read again. The last + * chunk sets the state ready. No transactions (rule 4); progress lives in the state row, so a run + * that stops anywhere resumes. + * + * The budget (`meter.ts`) is asked before each unit of work, statements included: a chunk whose + * batch would not fit is cut short (the cursor stops at its last written node). + * + * Worker-safe: it takes a raw `SqlDriver`, as `importSite` does. + */ +import type { ContentEntry, Site } from "@cloudhal/core-contracts"; +import type { SqlDriver } from "@cloudhal/storage"; +import { hasListedMarker, listedCollections, listFingerprint } from "./fingerprint.ts"; +import { type ListValuesBudget, ListValuesMeter } from "./meter.ts"; +import { dependsOnOriginal, listValueRows, originalIdOf } from "./rows.ts"; +import { + advanceStatement, + newEpoch, + type StateView, + staleStateStatement, + stateIs, +} from "./state-sql.ts"; +import { type NodeValues, nodeRowsStatements, rowStatementCount } from "./statements.ts"; +import { tidyRows } from "./tidy.ts"; + +export type { ListValuesBudget } from "./meter.ts"; + +/** What a settings change rebuilds before `updateSite` returns; the rest is the job's. */ +export const INLINE_LIST_BUDGET: ListValuesBudget = { entries: 500, ms: 2_000 }; + +export interface RebuildListValuesOptions { + /** One site (absent: every site). */ + readonly siteId?: string | undefined; + /** Only these collections of it. */ + readonly collections?: readonly string[] | undefined; + /** Absent: until every collection is ready. */ + readonly budget?: ListValuesBudget | undefined; + /** Also remove rows of collections that list no fields any more (the admin job). */ + readonly tidy?: boolean | undefined; + /** + * Where the work starts in the list of collections that are not ready (taken modulo its + * length): a job passes a number that changes between runs, so a collection writers keep busy + * cannot hold the others back. + */ + readonly start?: number | undefined; +} + +export interface RebuiltCollection { + readonly siteId: string; + readonly collection: string; + readonly ready: boolean; + /** Entries whose rows this call wrote. */ + readonly written: number; + /** Why it is not ready: a field operation rewrites a listed field, the budget ran out, the definitions changed meanwhile, or writers kept changing its entries. */ + readonly reason?: "operation_running" | "budget" | "changed" | "busy"; +} + +export interface RebuildListValuesResult { + readonly collections: RebuiltCollection[]; + /** The budget ran out before every collection was ready. */ + readonly exhausted: boolean; + /** Rows of unlisted collections removed (`tidy`). */ + readonly tidied: number; +} + +/** Chunk batches in a row that changed nothing before a collection is left for the next run. */ +const MISSES = 20; +/** Statements a state (re)start may take: the upsert, and a read when it changed nothing. */ +const START_COST = 2; +/** Statements a chunk needs at least: its read, its originals, a DELETE, an INSERT, the advance. */ +const CHUNK_COST = 5; + +type StateRow = { + fingerprint: string; + epoch: string; + ready: number; + cursor: string | null; +}; + +interface SiteRead { + readonly site: Site; + readonly hash: string; +} + +/** A listed collection that is not ready (or has no state row) under `fingerprint`. */ +interface Work extends SiteRead { + readonly collection: string; + readonly fingerprint: string; + /** Its state row as discovery read it, when that row has the fingerprint. */ + readonly found: StateRow | undefined; +} + +const stamp = () => new Date().toISOString(); +const stateKey = (siteId: string, collection: string) => `${siteId}\n${collection}`; + +async function readState(sql: SqlDriver, siteId: string, collection: string) { + return sql.get( + "SELECT fingerprint, epoch, ready, cursor FROM entry_list_state WHERE site_id = ? AND collection = ?", + [siteId, collection], + ); +} + +async function readSites(sql: SqlDriver, siteId: string | undefined): Promise { + const rows = siteId + ? await sql.all<{ hash: string; body: string }>( + "SELECT hash, body FROM nodes WHERE id = ? AND kind = 'site'", + [siteId], + ) + : await sql.all<{ hash: string; body: string }>( + "SELECT hash, body FROM nodes WHERE kind = 'site' ORDER BY created_at, id", + ); + return rows.map((r) => ({ site: JSON.parse(r.body) as Site, hash: r.hash })); +} + +/** Every state row of the site (or of every site), in one statement. */ +async function readStates(sql: SqlDriver, siteId: string | undefined) { + const columns = "site_id, collection, fingerprint, epoch, ready, cursor"; + const rows = siteId + ? await sql.all( + `SELECT ${columns} FROM entry_list_state WHERE site_id = ?`, + [siteId], + ) + : await sql.all( + `SELECT ${columns} FROM entry_list_state`, + ); + return new Map( + rows.map((r) => [ + stateKey(r.site_id, r.collection), + { fingerprint: r.fingerprint, epoch: r.epoch, ready: r.ready, cursor: r.cursor }, + ]), + ); +} + +/** A chunk: the entries after a key in a collection's key range (`collection:locale:slug`). */ +export const CHUNK_SQL = `SELECT key, hash, body FROM nodes WHERE site_id = ? AND kind = 'contentEntry' + AND key > ? AND key < ? ORDER BY key LIMIT ?`; + +/** The chunk after `cursor`. */ +async function readChunk(sql: SqlDriver, w: Work, cursor: string | null, limit: number) { + const rows = await sql.all<{ key: string; hash: string; body: string }>(CHUNK_SQL, [ + w.site.id, + cursor ?? `${w.collection}:`, + `${w.collection};`, + limit, + ]); + return rows.map((r) => ({ key: r.key, hash: r.hash, entry: JSON.parse(r.body) as ContentEntry })); +} + +/** + * The originals a chunk's translations read, by id. Driven by the id list (`CROSS JOIN`): left to + * the planner, `site_id = ? AND kind = ?` made it walk the site's entries for every chunk. + */ +export const ORIGINALS_SQL = `SELECT n.id AS id, n.hash AS hash, n.body AS body FROM json_each(?) j + CROSS JOIN nodes n ON n.id = j.value WHERE n.site_id = ? AND n.kind = 'contentEntry'`; + +/** The chunk's rows, each translation's with its original as stored now. */ +async function chunkValues( + sql: SqlDriver, + site: Site, + chunk: readonly { hash: string; entry: ContentEntry }[], +): Promise { + const dep = (e: ContentEntry) => (dependsOnOriginal(site, e) ? originalIdOf(e) : undefined); + const ids = [...new Set(chunk.map((r) => dep(r.entry)).filter((id) => id !== undefined))]; + const originals = new Map(); + if (ids.length > 0) { + const rows = await sql.all<{ id: string; hash: string; body: string }>(ORIGINALS_SQL, [ + JSON.stringify(ids), + site.id, + ]); + for (const r of rows) + originals.set(r.id, { hash: r.hash, entry: JSON.parse(r.body) as ContentEntry }); + } + return chunk.map((r) => { + const id = dep(r.entry); + const o = id === undefined ? undefined : originals.get(id); + return { + id: r.entry.id, + hash: r.hash, + original: id === undefined ? undefined : { id, hash: o?.hash ?? null }, + values: listValueRows(site, r.entry, o?.entry), + }; + }); +} + +/** How many of `values` (a prefix) a batch of at most `room` statements holds with its advance. */ +function fitting(values: readonly NodeValues[], room: number): number { + if (rowStatementCount(values) + 1 <= room) return values.length; + let lo = 0; + let hi = values.length; + while (lo < hi) { + const mid = Math.ceil((lo + hi) / 2); + if (rowStatementCount(values.slice(0, mid)) + 1 <= room) lo = mid; + else hi = mid - 1; + } + return lo; +} + +/** The state row restarted for `w.fingerprint` (missing, or from other definitions). */ +async function startState(sql: SqlDriver, w: Work): Promise { + const view = { + siteId: w.site.id, + collection: w.collection, + fingerprint: w.fingerprint, + epoch: newEpoch(), + }; + const s = staleStateStatement(view, { id: w.site.id, hash: w.hash }, stamp()); + const r = await sql.run(s.sql, s.params ?? []); + if (r.changes === 1) + return { fingerprint: w.fingerprint, epoch: view.epoch, ready: 0, cursor: null }; + const now = await readState(sql, w.site.id, w.collection); + return now?.fingerprint === w.fingerprint ? now : undefined; +} + +async function rebuildCollection(meter: ListValuesMeter, w: Work): Promise { + const { sql } = meter; + const siteId = w.site.id; + const result = (ready: boolean, written: number, reason?: RebuiltCollection["reason"]) => ({ + siteId, + collection: w.collection, + ready, + written, + ...(reason ? { reason } : {}), + }); + let state = w.found; + if (!state) { + if (meter.out(START_COST)) return result(false, 0, "budget"); + state = await startState(sql, w); + if (!state) return result(false, 0, "changed"); + } + let written = 0; + let misses = 0; + while (state.ready !== 1) { + if (meter.out(CHUNK_COST)) return result(false, written, "budget"); + const view: StateView = { + siteId, + collection: w.collection, + fingerprint: w.fingerprint, + epoch: state.epoch, + cursor: state.cursor, + }; + const limit = meter.room(); + const chunk = await readChunk(sql, w, state.cursor, limit); + const all = await chunkValues(sql, w.site, chunk); + const k = fitting(all, meter.left()); + const values = all.slice(0, k); + const last = k === chunk.length && chunk.length < limit; + const cursor: string | null = chunk[k - 1]?.key ?? state.cursor; + const results = await sql.batch([ + ...nodeRowsStatements(siteId, values, stateIs(view)), + advanceStatement(view, values, { cursor, ready: last, at: stamp() }), + ]); + meter.take(Math.max(1, k)); + if (results.at(-1)?.changes === 1) { + written += k; + misses = 0; + state = { ...state, cursor, ready: last ? 1 : 0 }; + continue; + } + // A writer reset the state, another runner moved it, or an entry of the chunk changed. + if (meter.left() < 1) return result(false, written, "budget"); + const now = await readState(sql, siteId, w.collection); + if (now?.fingerprint !== w.fingerprint) return result(false, written, "changed"); + if (++misses > MISSES) return result(false, written, "busy"); + state = now; + } + return result(true, written); +} + +/** Each listed collection of `sites`: settled now (ready, or under an operation), or work. */ +async function discover( + sites: readonly SiteRead[], + states: ReadonlyMap, + only: readonly string[] | undefined, +): Promise<(RebuiltCollection | Work)[]> { + const out: (RebuiltCollection | Work)[] = []; + for (const read of sites) + for (const collection of listedCollections(read.site)) { + if (only && !only.includes(collection)) continue; + const siteId = read.site.id; + // Phase B still rewrites values; phase C changes the fingerprint and resets the state. + if (hasListedMarker(read.site, collection)) { + out.push({ siteId, collection, ready: false, written: 0, reason: "operation_running" }); + continue; + } + const fingerprint = await listFingerprint(read.site, collection); + const s = states.get(stateKey(siteId, collection)); + const found = s?.fingerprint === fingerprint ? s : undefined; + if (found?.ready === 1) out.push({ siteId, collection, ready: true, written: 0 }); + else out.push({ ...read, collection, fingerprint, found }); + } + return out; +} + +const isWork = (x: RebuiltCollection | Work): x is Work => "fingerprint" in x; + +/** + * Makes every listed collection's `entry_list_values` complete and ready (of one site, or all), + * within `budget`. Safe to run concurrently with writers and with itself; idempotent. + */ +export async function rebuildListValues( + sql: SqlDriver, + options: RebuildListValuesOptions = {}, +): Promise { + const meter = new ListValuesMeter(sql, options.budget); + if (meter.left() < 2) return { collections: [], exhausted: true, tidied: 0 }; + const sites = await readSites(meter.sql, options.siteId); + const states = await readStates(meter.sql, options.siteId); + const found = await discover(sites, states, options.collections); + const work = found.flatMap((x, i) => (isWork(x) ? [i] : [])); + const n = work.length; + const from = n ? ((Math.trunc(options.start ?? 0) % n) + n) % n : 0; + for (const i of [...work.slice(from), ...work.slice(0, from)]) + found[i] = await rebuildCollection(meter, found[i] as Work); + const collections = found as RebuiltCollection[]; + const tidied = options.tidy ? await tidyRows(meter, sites) : 0; + return { collections, exhausted: collections.some((c) => c.reason === "budget"), tidied }; +} + +/** `rebuildListValues` for the collections a write reset; never throws (the job repairs). */ +export async function settleListValues( + sql: SqlDriver, + siteId: string, + options: { collections?: readonly string[]; budget?: ListValuesBudget } = {}, +): Promise { + try { + return await rebuildListValues(sql, { + siteId, + collections: options.collections, + budget: options.budget ?? INLINE_LIST_BUDGET, + }); + } catch { + return undefined; + } +} diff --git a/packages/core-ts/src/list-values/fingerprint.ts b/packages/core-ts/src/list-values/fingerprint.ts new file mode 100644 index 00000000..99efcfb1 --- /dev/null +++ b/packages/core-ts/src/list-values/fingerprint.ts @@ -0,0 +1,95 @@ +/** + * RFC-0042 §4, #110: what a collection's `entry_list_values` rows were built from. Rows are valid + * only under the fingerprint they were built with; every site write that changes it resets the + * collection's readiness in the same batch (`statements.ts`), and the backfill rebuilds it. + * + * The fingerprint is a SHA-256 over the collection sort, the listed keys (sorted: reordering + * Studio's columns rebuilds nothing) and each listed definition minus the properties that never + * change a resolved value (`EXCLUDED`). While a listed definition carries a migration marker, the + * whole definition and the site's conversion context count (a `text → select` conversion reads + * option labels), and the collection is never answered from the table anyway. + */ +import type { FieldDefinition } from "@cloudhal/core-contracts"; +import { convertContext } from "../fields/migration.ts"; +import type { SiteFields } from "../fields/types.ts"; +import { canonicalJson, sha256Hex } from "../hash.ts"; + +/** The fingerprint of a collection without list fields. */ +export const NO_LIST = "none"; + +/** + * Definition properties left out of the fingerprint: they never change a resolved value (labels, + * help, publish rules, site search). Every other property counts, including future ones (a unit + * test walks the schema and fails on a property that is neither here nor fingerprinted). + */ +export const EXCLUDED: readonly string[] = ["labels", "requiredToPublish", "searchable"]; + +/** One listed definition as fingerprinted: options keep their values, not their labels. */ +function shape(def: FieldDefinition): Record { + const out: Record = { ...def }; + for (const k of EXCLUDED) delete out[k]; + if (def.type === "select") out.options = def.options.map((o) => o.value); + return out; +} + +/** True while a listed field of the collection is being changed by a field operation. */ +export function hasListedMarker(site: SiteFields, collection: string): boolean { + const settings = site.collections?.[collection]; + const keys: readonly string[] = settings?.listFields ?? []; + return (settings?.fields ?? []).some((d) => d.migration !== undefined && keys.includes(d.key)); +} + +/** The collection's list fingerprint under `site` (`NO_LIST` when it lists no fields). */ +export async function listFingerprint(site: SiteFields, collection: string): Promise { + const settings = site.collections?.[collection]; + const keys = settings?.listFields; + if (!keys?.length) return NO_LIST; + const listFields = [...keys].sort(); + const marked = hasListedMarker(site, collection); + const defs = listFields.map((k) => { + const def = settings?.fields?.find((d) => d.key === k); + return def === undefined ? null : marked ? def : shape(def); + }); + return sha256Hex( + canonicalJson({ + format: 1, + sort: settings?.sort ?? "newest", + listFields, + defs, + ...(marked ? { convert: convertContext(site) } : {}), + }), + ); +} + +/** The collections of a site that list fields. */ +export const listedCollections = (site: SiteFields): string[] => + Object.entries(site.collections ?? {}) + .filter(([, s]) => (s?.listFields?.length ?? 0) > 0) + .map(([c]) => c); + +/** + * Collections whose fingerprint differs between `stored` (absent: a new site) and `next`, with + * the new fingerprint. Empty without hashing when the settings and zone are unchanged. + */ +export async function changedFingerprints( + stored: SiteFields | undefined, + next: SiteFields, +): Promise> { + const out = new Map(); + const same = + stored !== undefined && + stored.timeZone === next.timeZone && + stored.defaultLocale === next.defaultLocale && + canonicalJson(stored.collections ?? {}) === canonicalJson(next.collections ?? {}); + if (same) return out; + const keys = new Set([ + ...Object.keys(stored?.collections ?? {}), + ...Object.keys(next.collections ?? {}), + ]); + for (const c of keys) { + const after = await listFingerprint(next, c); + const before = stored ? await listFingerprint(stored, c) : NO_LIST; + if (before !== after) out.set(c, after); + } + return out; +} diff --git a/packages/core-ts/src/list-values/meter.ts b/packages/core-ts/src/list-values/meter.ts new file mode 100644 index 00000000..13dc90d0 --- /dev/null +++ b/packages/core-ts/src/list-values/meter.ts @@ -0,0 +1,90 @@ +/** + * RFC-0042 §4, #110: what one `rebuildListValues` call may spend. Entries and milliseconds bound + * the chunk work; `statements` bounds every SQL statement the call issues, counted as a tick + * meter counts them (`get`, `all` and `run` are one each, a `batch()` counts each of its + * statements), so a scheduled job can declare it as its cost. Work asks before it starts: + * nothing is refused mid-way. + */ +import type { SqlDriver } from "@cloudhal/storage"; + +/** Work one call may do before it stops (the state rows keep the progress). */ +export interface ListValuesBudget { + readonly entries: number; + readonly ms: number; + /** SQL statements the call may issue (absent: no bound). */ + readonly statements?: number | undefined; +} + +/** Entries one chunk reads at most. */ +export const CHUNK = 100; + +export class ListValuesMeter { + /** `inner`, counting every statement into this meter. */ + readonly sql: SqlDriver; + readonly #budget: ListValuesBudget | undefined; + readonly #started = Date.now(); + #entries = 0; + #statements = 0; + + constructor(inner: SqlDriver, budget: ListValuesBudget | undefined) { + this.#budget = budget; + const note = (n: number) => { + this.#statements += n; + }; + this.sql = { + name: inner.name, + get: async (sql, params) => { + note(1); + return inner.get(sql, params); + }, + all: async (sql, params) => { + note(1); + return inner.all(sql, params); + }, + run: async (sql, params) => { + note(1); + return inner.run(sql, params); + }, + batch: async (statements) => { + note(statements.length); + return inner.batch(statements); + }, + exec: (sql) => inner.exec(sql), + close: () => inner.close(), + }; + } + + /** Statements issued so far. */ + get statements(): number { + return this.#statements; + } + + /** Statements still allowed. */ + left(): number { + const max = this.#budget?.statements; + return max === undefined ? Number.POSITIVE_INFINITY : max - this.#statements; + } + + /** Fewer than `n` statements are left, or the time is spent (what tidying checks). */ + short(n: number): boolean { + const b = this.#budget; + return !!b && (this.left() < n || Date.now() - this.#started >= b.ms); + } + + /** No more chunk work may start: entries or time spent, or fewer than `n` statements left. */ + out(n: number): boolean { + const b = this.#budget; + return !!b && (this.#entries >= b.entries || this.short(n)); + } + + /** Counts `n` entries of chunk work. */ + take(n: number): void { + this.#entries += n; + } + + /** Entries the next chunk may read (at most `CHUNK`, at least 1). */ + room(): number { + const b = this.#budget; + return b ? Math.max(1, Math.min(CHUNK, b.entries - this.#entries)) : CHUNK; + } +} diff --git a/packages/core-ts/src/list-values/query-sql.ts b/packages/core-ts/src/list-values/query-sql.ts new file mode 100644 index 00000000..d5496975 --- /dev/null +++ b/packages/core-ts/src/list-values/query-sql.ts @@ -0,0 +1,332 @@ +/** + * RFC-0042 §4, #110: a parsed field query (`fields/query.ts`) compiled to SQL over + * `entry_list_values`. Pure: it builds statements, `query.ts` runs them. Every statement checks + * inside itself that the collection is ready under the fingerprint the caller computed, so an + * answer never mixes a stale table with the current definitions. + * + * One condition drives the scan through `entry_list_values_query` (site, collection, field, has, + * num, text, …): the field the list is ordered by, else the most selective condition. It reads + * one row per entry (a field row; or, for a multiple select's `eq`, the one option row an entry + * can hold for that value), so `count(*)` is the number of matches. Every other condition is an + * `EXISTS` probe through `entry_list_values_node` (node_id, field, has, num, text). Operands are + * the evaluator's own (`operandKey`, then `encodeKey`, as the rows' `sortKey`), always bound as + * parameters (D1 reads doubles inside JSON inexactly); option lists travel as JSON strings. + * + * Order: with `orderBy`, entries with a value (`has = 1`) by `num`, then the collection sort's + * `tie`, then `node_id`; then those without one (`has = 0`) by `tie`, then `node_id`, which is + * `compareByField` then `bySort` then `id`. Without it, `created, node_id`: the store's + * `created_at, id`. A page in a segment's second half is read from its end in the reverse order + * (then put back in order), so a deep page reads the rows after it, not every row before it. + * + * Statement shapes, by what the index can do: + * - ordered and paged: the count, then a page per segment (both follow the index); + * - unordered and paged: the index does not hold `created` first, so `matches` reads each match's + * id and `created` once, unsorted (D1 counts a sorter's rows as read, a top-N sort's again for + * every row), the caller orders them, and `bodies` reads the window's entries by id, + * re-checking each against the query; + * - unpaged: `all`, every match's body in one statement (no count: the total is the rows + * returned; `ready` tells an empty answer from a collection that is not ready), in answer order + * with `orderBy`, else ordered by the caller as `matches` is. + */ +import type { FieldDefinition } from "@cloudhal/core-contracts"; +import type { SqlStatement, SqlValue } from "@cloudhal/storage"; +import { + type FieldOperand, + operandKey, + type ParsedCondition, + type ParsedFieldQuery, +} from "../fields/query.ts"; +import { encodeKey } from "./rows.ts"; + +type Sort = "newest" | "oldest" | "title"; + +/** What to compile: the parsed query, the collection, its prefilter and the readiness to check. */ +export interface ListQuerySpec { + readonly siteId: string; + readonly collection: string; + /** `listFingerprint` of the site the query was parsed against. */ + readonly fingerprint: string; + /** The collection's listed keys: a query without conditions reads the first one's rows. */ + readonly listFields: readonly string[]; + readonly query: ParsedFieldQuery; + readonly sort: Sort; + readonly locale?: string | undefined; + readonly status?: readonly string[] | undefined; + readonly now: number; +} + +/** A page's part: with `orderBy`, entries with a value, then those without; else one part. */ +export type Segment = "valued" | "missing" | "all"; + +/** The compiled query: one count, then a page statement per segment the window touches. */ +export interface CompiledListQuery { + /** One row: `ok` (1 when ready under the fingerprint), `total`, `valued` (has = 1). */ + readonly count: SqlStatement; + /** The segments in answer order. */ + readonly segments: readonly Segment[]; + /** + * `limit` rows of `segment` from `offset` (counted from its end when `reverse`): one `body` + * column, in answer order either way. + */ + page( + segment: Segment, + window: { limit: number; offset: number; reverse?: boolean }, + ): SqlStatement; + /** Every match's `body` (with its `id` and `created`): in answer order with `orderBy`, else in none. */ + readonly all: SqlStatement; + /** Without `orderBy`: every match's node `id` and `created`, in no order. */ + readonly matches: SqlStatement | undefined; + /** The `id` and `body` of each of `ids` that still matches (in no order). */ + bodies(ids: readonly string[]): SqlStatement; + /** One row while the collection is ready under the fingerprint. */ + readonly ready: SqlStatement; +} + +interface Frag { + readonly sql: string; + readonly params: readonly SqlValue[]; +} + +/** One condition: a predicate on its field row, or on its option rows (a multiple select). */ +interface Compiled { + readonly key: string; + readonly def: FieldDefinition; + /** `row`: the field row; `some`/`none`: an option row exists / does not. */ + readonly on: "row" | "some" | "none"; + /** Lower drives first (an equality seeks; `ne` reads every row). */ + readonly rank: number; + /** Can drive as an option row: one per entry at most (a single value). */ + readonly unique: boolean; + pred(alias: string): Frag; +} + +const frag = (sql: string, ...params: SqlValue[]): Frag => ({ sql, params }); +const join = (parts: readonly Frag[], sep = " AND "): Frag => ({ + sql: parts.map((p) => p.sql).join(sep), + params: parts.flatMap((p) => p.params), +}); + +const COMPARE: Record = { lt: "<", lte: "<=", gt: ">", gte: ">=" }; +/** Field types whose rows never hold `num` (so `num IS NULL` lets a seek reach `text`). */ +const TEXTUAL = new Set(["select", "text", "url", "richText", "image"]); + +function presence(c: ParsedCondition): Omit | undefined { + if (c.op === "set") return { on: "row", rank: 5, pred: (a) => frag(`${a}.has = 1`) }; + if (c.op === "unset") return { on: "row", rank: 6, pred: (a) => frag(`${a}.has = 0`) }; + return undefined; +} + +function ordered(c: ParsedCondition, k: number): Omit { + if (c.op === "eq") return { on: "row", rank: 1, pred: (a) => frag(`${a}.num = ?`, k) }; + if (c.op === "ne") return { on: "row", rank: 9, pred: (a) => frag(`${a}.num IS NOT ?`, k) }; + const op = COMPARE[c.op] as string; + return { on: "row", rank: 3, pred: (a) => frag(`${a}.num ${op} ?`, k) }; +} + +function select(c: ParsedCondition, values: string[]): Omit { + const multiple = c.def.type === "select" && c.def.multiple === true; + const one = values[0] as string; + const test = (a: string) => + c.op === "in" + ? frag(`${a}.text IN (SELECT value FROM json_each(?))`, JSON.stringify(values)) + : frag(`${a}.text = ?`, one); + if (multiple) + return { on: c.op === "ne" ? "none" : "some", rank: c.op === "ne" ? 9 : 1, pred: test }; + if (c.op === "ne") return { on: "row", rank: 9, pred: (a) => frag(`${a}.text IS NOT ?`, one) }; + return { on: "row", rank: c.op === "in" ? 2 : 1, pred: test }; +} + +/** One condition compiled, or why it stays with the evaluator. */ +function compileCondition(c: ParsedCondition, now: number, zone: string): Compiled | string { + const base = { key: c.key, def: c.def, unique: c.op === "eq" }; + const bare = presence(c); + if (bare) return { ...base, ...bare }; + const operands = c.operands as FieldOperand[]; + const strings = operands.map((o) => (o.kind === "string" ? o.s : undefined)); + switch (c.def.type) { + case "number": + case "date": + case "datetime": { + const o = operands[0]; + if (!o) return "an operator without a value"; + return { ...base, ...ordered(c, encodeKey(operandKey(o, c.def, now, zone))) }; + } + case "boolean": { + const yes = strings[0] === "true"; + return { + ...base, + on: "row", + rank: yes ? 4 : 7, + pred: (a) => frag(`${a}.num = ?`, yes ? 1 : 0), + }; + } + case "select": + if (strings.some((s) => s === undefined)) return "a select operand that is not a string"; + return { ...base, ...select(c, strings as string[]) }; + case "text": + case "url": { + const s = strings[0]; + if (c.op !== "eq" || s === undefined) return `${c.op} on a ${c.def.type} field`; + // A lone surrogate cannot be compared exactly through UTF-8; rows store NULL for it. + if (!s.isWellFormed()) return "a string operand that is not well formed"; + return { ...base, on: "row", rank: 1, pred: (a) => frag(`${a}.text = ?`, s) }; + } + default: + return `${c.op} on a ${c.def.type} field`; + } +} + +/** The scan's driver: a field (its field rows, or one value's option rows) and its predicates. */ +interface Driver { + readonly key: string; + readonly options: boolean; + readonly textual: boolean; + readonly preds: readonly Frag[]; + readonly probes: readonly Compiled[]; +} + +function chooseDriver(spec: ListQuerySpec, compiled: readonly Compiled[]): Driver | string { + const orderKey = spec.query.orderBy?.def.key; + let lead: Compiled | undefined; + if (orderKey === undefined) + for (const c of compiled) + if ((c.on === "row" || (c.on === "some" && c.unique)) && (!lead || c.rank < lead.rank)) + lead = c; + const key = orderKey ?? lead?.key ?? spec.listFields[0]; + if (key === undefined) return "a collection without list fields"; + if (lead?.on === "some") + return { + key, + options: true, + textual: true, + preds: [lead.pred("d")], + probes: compiled.filter((c) => c !== lead), + }; + const mine = (c: Compiled) => c.key === key && c.on === "row"; + const def = spec.query.orderBy?.def ?? lead?.def; + return { + key, + options: false, + textual: def !== undefined && TEXTUAL.has(def.type), + preds: compiled.filter(mine).map((c) => c.pred("d")), + probes: compiled.filter((c) => !mine(c)), + }; +} + +function probe(c: Compiled, i: number): Frag { + const p = `p${i}`; + const rows = c.on === "row" ? `${p}.has IN (0, 1)` : `${p}.has = 2 AND ${p}.num IS NULL`; + const test = c.pred(p); + return { + sql: `${c.on === "none" ? "NOT " : ""}EXISTS (SELECT 1 FROM entry_list_values ${p} + WHERE ${p}.node_id = d.node_id AND ${p}.field = ? AND ${rows} AND ${test.sql})`, + params: [c.key, ...test.params], + }; +} + +const READY = + "EXISTS (SELECT 1 FROM entry_list_state s WHERE s.site_id = ? AND s.collection = ? AND s.ready = 1 AND s.fingerprint = ?)"; + +/** The driver rows a segment reads. */ +function rowsOf(driver: Driver, segment: Segment | "count"): string { + if (driver.options) return "d.has = 2 AND d.num IS NULL"; + if (segment === "valued") return "d.has = 1"; + if (segment === "missing") return "d.has = 0 AND d.num IS NULL AND d.text IS NULL"; + return driver.textual ? "d.has IN (0, 1) AND d.num IS NULL" : "d.has IN (0, 1)"; +} + +/** A segment's order; `reverse` flips every term (a deep page read from the segment's end). */ +function orderOf(spec: ListQuerySpec, segment: Segment, at: { alias: string; reverse: boolean }) { + const [up, down] = at.reverse ? ["DESC", "ASC"] : ["ASC", "DESC"]; + const a = at.alias; + const tie = spec.sort === "newest" ? down : up; + if (segment === "all") return `${a}.created ${up}, ${a}.node_id ${up}`; + if (segment === "missing") return `${a}.tie ${tie}, ${a}.node_id ${up}`; + const dir = spec.query.orderBy?.direction === "desc" ? down : up; + return `${a}.num ${dir}, ${a}.text ${dir}, ${a}.tie ${tie}, ${a}.node_id ${up}`; +} + +/** + * An ordered answer's order in one statement: the valued segment's order after `has DESC` (the + * missing rows' `num` and `text` are NULL, so it is their segment's order too). + */ +const answerOrder = (spec: ListQuerySpec): string => + `d.has DESC, ${orderOf(spec, "valued", { alias: "d", reverse: false })}`; + +/** `all`, every match (sorted only with `orderBy`); `matches`, unordered queries' ids, unsorted. */ +function wholeStatements(spec: ListQuerySpec, m: { where: string; params: readonly SqlValue[] }) { + const order = spec.query.orderBy ? ` ORDER BY ${answerOrder(spec)}` : ""; + return { + all: { + sql: `SELECT n.body AS body, d.node_id AS id, d.created AS created FROM entry_list_values d + CROSS JOIN nodes n ON n.id = d.node_id AND n.kind = 'contentEntry' + WHERE ${m.where}${order}`, + params: m.params, + }, + matches: spec.query.orderBy + ? undefined + : { + sql: `SELECT d.node_id AS id, d.created AS created FROM entry_list_values d WHERE ${m.where}`, + params: m.params, + }, + }; +} + +/** Compiles `spec`, or says why the evaluator must answer it (a reason string). */ +export function compileListQuery(spec: ListQuerySpec): CompiledListQuery | string { + const compiled: Compiled[] = []; + for (const c of spec.query.where) { + const out = compileCondition(c, spec.now, spec.query.timeZone); + if (typeof out === "string") return out; + compiled.push(out); + } + const driver = chooseDriver(spec, compiled); + if (typeof driver === "string") return driver; + const ready = frag(READY, spec.siteId, spec.collection, spec.fingerprint); + const rest = [ + ...driver.preds, + ...(spec.locale !== undefined ? [frag("d.locale = ?", spec.locale)] : []), + ...(spec.status !== undefined + ? [frag("d.status IN (SELECT value FROM json_each(?))", JSON.stringify(spec.status))] + : []), + ...driver.probes.map(probe), + ready, + ]; + const at = [spec.siteId, spec.collection, driver.key]; + const filter = join([frag("d.site_id = ? AND d.collection = ? AND d.field = ?", ...at), ...rest]); + // By node: `+` keeps the planner off the query index, so each id is one seek of the node index. + const byNode = join([ + frag("+d.site_id = ? AND +d.collection = ? AND d.field = ?", ...at), + ...rest, + ]); + const where = (segment: Segment | "count") => `${rowsOf(driver, segment)} AND ${filter.sql}`; + return { + ...wholeStatements(spec, { where: where("count"), params: filter.params }), + bodies: (ids) => ({ + sql: `SELECT n.id AS id, n.body AS body FROM json_each(?) j + CROSS JOIN entry_list_values d ON d.node_id = j.value + CROSS JOIN nodes n ON n.id = d.node_id AND n.kind = 'contentEntry' + WHERE ${rowsOf(driver, "count")} AND ${byNode.sql}`, + params: [JSON.stringify(ids), ...byNode.params], + }), + ready: { sql: `SELECT 1 AS ok WHERE ${READY}`, params: ready.params }, + count: { + sql: `SELECT (SELECT count(*) FROM entry_list_state s WHERE s.site_id = ? AND s.collection = ? + AND s.ready = 1 AND s.fingerprint = ?) AS ok, + count(*) AS total, coalesce(sum(d.has = 1), 0) AS valued + FROM entry_list_values d WHERE ${where("count")}`, + params: [...ready.params, ...filter.params], + }, + segments: spec.query.orderBy ? ["valued", "missing"] : ["all"], + page: (segment, window) => ({ + sql: `SELECT n.body AS body FROM ( + SELECT d.node_id AS node_id, d.num AS num, d.text AS text, d.tie AS tie, d.created AS created + FROM entry_list_values d WHERE ${where(segment)} + ORDER BY ${orderOf(spec, segment, { alias: "d", reverse: window.reverse === true })} + LIMIT ? OFFSET ? + ) p JOIN nodes n ON n.id = p.node_id AND n.kind = 'contentEntry' + ORDER BY ${orderOf(spec, segment, { alias: "p", reverse: false })}`, + params: [...filter.params, window.limit, window.offset], + }), + }; +} diff --git a/packages/core-ts/src/list-values/query.ts b/packages/core-ts/src/list-values/query.ts new file mode 100644 index 00000000..fb848deb --- /dev/null +++ b/packages/core-ts/src/list-values/query.ts @@ -0,0 +1,181 @@ +/** + * RFC-0042 §4, #110: `queryContentEntries` answered from `entry_list_values`. Used only while the + * collection's rows are complete and current: no listed field is being changed by a field + * operation, and every statement finds the state row ready under the fingerprint of the site the + * query was parsed against. Otherwise, and whenever a page comes back shorter or longer than the + * count promised (a write landed between the two statements, seen when it removes a row of the + * page or adds one at a segment's end), it answers `undefined` and the caller runs the in-memory + * evaluator, which stays the definition (the property suite compares the two on random data and + * queries). + * + * Statements (`query-sql.ts` says why each shape): + * - ordered and paged: the count (`total`, and how many have a value), then one page statement + * per segment the window touches; bodies come from the page statement itself; + * - unordered and paged: every match's id and creation time (the total), put in the store's order + * here, then the window's bodies by id, each re-checked against the query (one that stopped + * matching sends the caller to the scan); + * - unpaged: one statement with every match's body (put in the store's order here when + * unordered); when it returns none, a readiness check. + * Under concurrent writes the total and the page can come from different moments, as offset + * paging across requests always can. + */ +import type { ContentEntry, Site } from "@cloudhal/core-contracts"; +import type { SqlDriver, SqlStatement, SqlValue } from "@cloudhal/storage"; +import type { ParsedFieldQuery } from "../fields/query.ts"; +import { parseRows } from "../node-sql.ts"; +import { hasListedMarker, listFingerprint, NO_LIST } from "./fingerprint.ts"; +import { type CompiledListQuery, compileListQuery, type Segment } from "./query-sql.ts"; +import { listedKeys } from "./rows.ts"; + +export interface ListValuesQuery { + readonly collection: string; + /** Parsed against the same `site` (`parseFieldQuery`). */ + readonly query: ParsedFieldQuery; + readonly locale?: string | undefined; + readonly status?: readonly string[] | undefined; + /** Absent: every match from `offset`. */ + readonly limit?: number | undefined; + readonly offset?: number | undefined; + readonly now: number; +} + +export interface ListValuesAnswer { + readonly entries: ContentEntry[]; + readonly total: number; +} + +interface Head { + readonly total: number; + readonly valued: number; +} + +/** A segment's part of the page; `end`: it runs to the segment's last row. */ +export interface PageWindow { + readonly segment: Segment; + /** The segment's row count. */ + readonly size: number; + readonly offset: number; + readonly limit: number; + readonly end: boolean; +} + +/** The parts of each segment inside `[offset, offset + limit)`, in answer order. */ +export function pageWindows( + sizes: readonly { segment: Segment; size: number }[], + offset: number, + limit: number | undefined, +): PageWindow[] { + const stop = limit === undefined ? Number.POSITIVE_INFINITY : offset + limit; + const out: PageWindow[] = []; + let start = 0; + for (const { segment, size } of sizes) { + const lo = Math.max(offset, start); + const hi = Math.min(stop, start + size); + if (hi > lo) + out.push({ segment, size, offset: lo - start, limit: hi - lo, end: hi === start + size }); + start += size; + } + return out; +} + +const sizesOf = (plan: CompiledListQuery, head: Head) => + plan.segments.map((segment) => ({ + segment, + size: + segment === "valued" + ? head.valued + : segment === "missing" + ? head.total - head.valued + : head.total, + })); + +/** + * A window's statement. In a segment's second half it counts from the segment's end (a deep + * page reads what follows it); at the end of a segment read forwards, one row more, to see a row + * that came after the count. + */ +function pageStatement(plan: CompiledListQuery, w: PageWindow): [string, SqlValue[]] { + const reverse = w.offset > w.size - w.offset - w.limit; + const page = reverse + ? plan.page(w.segment, { offset: w.size - w.offset - w.limit, limit: w.limit, reverse }) + : plan.page(w.segment, { offset: w.offset, limit: w.limit + (w.end ? 1 : 0) }); + return [page.sql, [...(page.params ?? [])]]; +} + +type Match = { id: string; created: string }; + +/** + * The store's order, `created_at, id`: both are ASCII (ISO instants, generated ids), where + * JavaScript's `<` is SQLite's BINARY order. + */ +const byCreated = (a: Match, b: Match) => + a.created < b.created ? -1 : a.created > b.created ? 1 : a.id < b.id ? -1 : a.id > b.id ? 1 : 0; + +async function isReady(sql: SqlDriver, plan: CompiledListQuery): Promise { + return (await sql.get(plan.ready.sql, plan.ready.params)) !== undefined; +} + +/** Every match in one statement; an unordered one is put in the store's order, then the offset. */ +async function unpaged(sql: SqlDriver, plan: CompiledListQuery, offset: number) { + const rows = await sql.all(plan.all.sql, plan.all.params); + if (rows.length === 0 && !(await isReady(sql, plan))) return undefined; + if (plan.matches) rows.sort(byCreated); + return { entries: parseRows(rows.slice(offset)), total: rows.length }; +} + +/** Unordered: the matches' ids (ordered here), then the window's bodies by id. */ +async function byIds( + sql: SqlDriver, + plan: CompiledListQuery, + matches: SqlStatement, + window: { offset: number; limit: number }, +): Promise { + const all = await sql.all(matches.sql, matches.params); + const ids = all.sort(byCreated).map((r) => r.id); + if (ids.length === 0 && !(await isReady(sql, plan))) return undefined; + const page = ids.slice(window.offset, window.offset + window.limit); + if (page.length === 0) return { entries: [], total: ids.length }; + const bodies = plan.bodies(page); + const rows = await sql.all<{ id: string; body: string }>(bodies.sql, bodies.params); + const byId = new Map(rows.map((r) => [r.id, r.body])); + if (rows.length !== page.length || byId.size !== page.length) return undefined; + const entries = page.map((id) => JSON.parse(byId.get(id) ?? "null") as ContentEntry | null); + if (entries.some((e) => e === null)) return undefined; + return { entries: entries as ContentEntry[], total: ids.length }; +} + +/** The query's answer from the table, or `undefined` when the evaluator must answer it. */ +export async function queryListValues( + sql: SqlDriver, + site: Site, + q: ListValuesQuery, +): Promise { + if (hasListedMarker(site, q.collection)) return undefined; + const fingerprint = await listFingerprint(site, q.collection); + if (fingerprint === NO_LIST) return undefined; + const plan = compileListQuery({ + siteId: site.id, + collection: q.collection, + fingerprint, + listFields: listedKeys(site, q.collection), + query: q.query, + sort: site.collections?.[q.collection]?.sort ?? "newest", + locale: q.locale, + status: q.status, + now: q.now, + }); + if (typeof plan === "string") return undefined; + const offset = q.offset ?? 0; + if (q.limit === undefined) return unpaged(sql, plan, offset); + if (plan.matches) return byIds(sql, plan, plan.matches, { offset, limit: q.limit }); + const head = await sql.get>(plan.count.sql, plan.count.params); + if (head?.ok !== 1) return undefined; + const counts: Head = { total: Number(head.total), valued: Number(head.valued) }; + const entries: ContentEntry[] = []; + for (const w of pageWindows(sizesOf(plan, counts), offset, q.limit)) { + const rows = await sql.all<{ body: string }>(...pageStatement(plan, w)); + if (rows.length !== w.limit) return undefined; + entries.push(...parseRows(rows)); + } + return { entries, total: counts.total }; +} diff --git a/packages/core-ts/src/list-values/rows.ts b/packages/core-ts/src/list-values/rows.ts new file mode 100644 index 00000000..d23d680f --- /dev/null +++ b/packages/core-ts/src/list-values/rows.ts @@ -0,0 +1,171 @@ +/** + * RFC-0042 §4, #110: the rows of `entry_list_values` for one entry. One row per listed field, + * whether or not the entry has a value (so `ne`, `unset`, boolean `eq false` and missing values + * last are answerable), plus one row per option a multiple select holds. Values are + * `resolveFields`, the evaluator's own resolution (a translation reads its original's + * non-localized values), and numbers, days and instants are `sortKey`'s, so the table means what + * `fields/query.ts` reads. Pure and Worker-safe. + */ +import type { ContentEntry, FieldDefinition } from "@cloudhal/core-contracts"; +import { FIELD_QUERY_LIMITS, sortKey } from "../fields/query.ts"; +import { type FieldsOriginal, hasFieldValue, resolveFields } from "../fields/resolve.ts"; +import type { SiteFields } from "../fields/types.ts"; + +/** One row: `has` is 0/1 on a field row and 2 on an option row of a multiple select. */ +export interface ListValueRow { + readonly field: string; + readonly has: 0 | 1 | 2; + readonly num: number | null; + readonly text: string | null; +} + +/** An entry's rows with the columns they share. */ +export interface EntryListValues { + readonly nodeId: string; + readonly collection: string; + readonly locale: string; + readonly status: string; + readonly tie: string; + readonly rows: readonly ListValueRow[]; +} + +/** What rows read of an entry. */ +export type ListEntry = Pick< + ContentEntry, + "id" | "collection" | "locale" | "status" | "title" | "publishedAt" | "createdAt" | "fields" +> & { translation?: ContentEntry["translation"] | undefined }; + +/** A `YYYY-MM-DD` day as the integer YYYYMMDD, built from its digits (same order as the string). */ +export function dayNumber(day: string): number { + return ( + Number(day.slice(0, 4)) * 10_000 + Number(day.slice(5, 7)) * 100 + Number(day.slice(8, 10)) + ); +} + +/** `num` of a number, date or datetime sort key (`sortKey`/`operandKey`); -0 is stored as 0. */ +export function encodeKey(key: number | string): number { + if (typeof key === "string") return dayNumber(key); + return key === 0 ? 0 : key; +} + +/** + * A string whose SQLite BINARY order (UTF-8, by code point) is JavaScript's `<` order (by UTF-16 + * unit): every unit below U+D800 stays, every other unit u becomes code point u + 0x800, so lone + * surrogates and U+E000–U+FFFF keep their places relative to astral characters. + */ +export function utf16Key(s: string): string { + let out = ""; + let from = 0; + for (let i = 0; i < s.length; i++) { + const u = s.charCodeAt(i); + if (u < 0xd800) continue; + out += s.slice(from, i) + String.fromCodePoint(u + 0x800); + from = i + 1; + } + return from === 0 ? s : out + s.slice(from); +} + +/** A text or url value as stored in `text`: itself when exact through UTF-8 and short, else NULL. */ +export function textKey(value: string): string | null { + return value.length <= FIELD_QUERY_LIMITS.valueLength && value.isWellFormed() ? value : null; +} + +type Sort = "newest" | "oldest" | "title"; + +/** The collection sort's key of an entry; the direction is applied at query time. */ +export function tieOf(sort: Sort | undefined, entry: ListEntry): string { + if (sort === "title") return utf16Key(entry.title); + return entry.publishedAt ?? entry.createdAt; +} + +const row = (field: string, has: 0 | 1 | 2, num: number | null, text: string | null) => ({ + field, + has, + num, + text, +}); + +/** The rows of one listed field from its resolved value. */ +export function fieldRows( + key: string, + def: FieldDefinition | undefined, + value: unknown, +): ListValueRow[] { + if (!def) return [row(key, 0, null, null)]; + const has = hasFieldValue(value) ? 1 : 0; + switch (def.type) { + case "number": + case "date": + case "datetime": { + const k = sortKey(def, value); + return [row(key, has, k === undefined ? null : encodeKey(k), null)]; + } + case "boolean": + return [row(key, has, value === true ? 1 : 0, null)]; + case "select": { + if (!def.multiple) return [row(key, has, null, typeof value === "string" ? value : null)]; + const held = Array.isArray(value) ? value.filter((v) => typeof v === "string") : []; + return [row(key, has, null, null), ...[...new Set(held)].map((o) => row(key, 2, null, o))]; + } + case "text": + case "url": + return [row(key, has, null, typeof value === "string" ? textKey(value) : null)]; + default: + return [row(key, has, null, null)]; + } +} + +/** The listed keys of a collection (empty when it lists none). */ +export const listedKeys = (site: SiteFields, collection: string): readonly string[] => + site.collections?.[collection]?.listFields ?? []; + +/** + * The entry's rows under `site`, `original` being its translation group's original (read only + * for a same-site translation, as `queryContentEntries` does); undefined when its collection + * lists no fields. + */ +export function listValueRows( + site: SiteFields, + entry: ListEntry, + original?: FieldsOriginal, +): EntryListValues | undefined { + const settings = site.collections?.[entry.collection]; + const keys = settings?.listFields; + if (!keys?.length) return undefined; + const resolved = resolveFields(site, entry, original); + return { + nodeId: entry.id, + collection: entry.collection, + locale: entry.locale, + status: entry.status, + tie: tieOf(settings?.sort, entry), + rows: keys.flatMap((k) => fieldRows(k, resolved[k]?.def, resolved[k]?.value)), + }; +} + +/** The original's id when `entry` is a translation whose original would be in the same site. */ +export function originalIdOf(entry: ListEntry): string | undefined { + const t = entry.translation; + if (!t || t.group === entry.id || t.source?.siteId !== undefined) return undefined; + return t.group; +} + +function ownKey(fields: ListEntry["fields"], def: FieldDefinition): boolean { + if (!fields) return false; + const from = def.migration?.fromKey; + return Object.hasOwn(fields, def.key) || (from !== undefined && Object.hasOwn(fields, from)); +} + +/** + * True when one of the entry's listed values comes from its original: a non-localized listed + * field, or a localized one without its own key (`resolve.ts`'s fallback). + */ +export function dependsOnOriginal(site: SiteFields, entry: ListEntry): boolean { + if (originalIdOf(entry) === undefined) return false; + const settings = site.collections?.[entry.collection]; + const defs = settings?.fields ?? []; + return (settings?.listFields ?? []).some((k) => { + const def = defs.find((d) => d.key === k); + return def !== undefined && (!def.localized || !ownKey(entry.fields, def)); + }); +} diff --git a/packages/core-ts/src/list-values/site-plan.ts b/packages/core-ts/src/list-values/site-plan.ts new file mode 100644 index 00000000..1afde176 --- /dev/null +++ b/packages/core-ts/src/list-values/site-plan.ts @@ -0,0 +1,57 @@ +/** + * RFC-0042 §4, #110: what a site write adds to its batch for `entry_list_values`. Every write of + * the site node that changes a collection's list fingerprint (`updateSite`, `createSite`, a field + * operation's phases) resets that collection's readiness in the same batch, anchored on the + * site's new hash, so rows built under other definitions are never read; the backfill rebuilds + * them. Writes that leave the definitions alone (publish, rollback, page lists) add nothing. + */ +import type { SqlStatement } from "@cloudhal/storage"; +import type { SiteFields } from "../fields/types.ts"; +import { hashIs } from "../node-sql.ts"; +import { changedFingerprints, listedCollections, listFingerprint } from "./fingerprint.ts"; +import { type ImportedList, importResetStatements, siteStateStatements } from "./state-sql.ts"; + +export interface ListStatePlan { + /** Collections whose fingerprint changes (the backfill's work after the write). */ + readonly changed: readonly string[]; + /** The state statements, once the site's new hash is known. */ + statements(hash: string, at: string): SqlStatement[]; +} + +/** The readiness changes of writing `next` over `stored` (absent: a new site). */ +export async function planListState( + stored: SiteFields | undefined, + next: SiteFields & { id: string }, +): Promise { + const changed = await changedFingerprints(stored, next); + return { + changed: [...changed.keys()], + statements: (hash, at) => + changed.size === 0 + ? [] + : siteStateStatements(next.id, changed, { anchor: hashIs(next.id, hash), at }), + }; +} + +/** + * Import (`importSite`, push, pull, `cloudhal import`) replaces the node set in one batch: these + * statements reset the site's list values in it, a listed collection that holds no imported entry + * being ready at once. The rows are rebuilt after the batch (`rebuildListValues`), so the + * batch's statement count stays fixed whatever the values are. + */ +export async function importListStatements( + siteId: string, + rows: readonly { id: string; kind: string; key: string; body: string }[], + at: string, +): Promise { + const row = rows.find((n) => n.kind === "site" && n.id === siteId); + const site = row ? (JSON.parse(row.body) as SiteFields) : undefined; + const listed: ImportedList[] = []; + for (const collection of site ? listedCollections(site) : []) + listed.push({ + collection, + fingerprint: await listFingerprint(site as SiteFields, collection), + empty: !rows.some((n) => n.kind === "contentEntry" && n.key.startsWith(`${collection}:`)), + }); + return importResetStatements(siteId, listed, at); +} diff --git a/packages/core-ts/src/list-values/state-sql.ts b/packages/core-ts/src/list-values/state-sql.ts new file mode 100644 index 00000000..d97e2f20 --- /dev/null +++ b/packages/core-ts/src/list-values/state-sql.ts @@ -0,0 +1,277 @@ +/** + * RFC-0042 §4, #110: the SQL of `entry_list_state`, a listed collection's readiness. Builders + * only: an entry write's state check, a site write's resets, import's restart, a field + * operation pass's reset, and the backfill's guards and progress. Every reset sets a new `epoch`, + * which the backfill's chunks are guarded on, so a reset in the middle of a run makes the rest of + * it a no-op. + */ +import type { SqlStatement, SqlValue } from "@cloudhal/storage"; +import type { Cond } from "../node-sql.ts"; +import { + at, + head, + NODE_MATCH, + type NodeValues, + ORIGINAL_OK, + type OriginalRead, + TRANSLATIONS_OF, +} from "./statements.ts"; + +/** A fresh token for `entry_list_state.epoch` (every reset changes it). */ +export const newEpoch = (): string => crypto.randomUUID(); + +/** What an entry write or delete checks, to degrade instead of leaving rows stale. */ +export interface StateCheck { + readonly siteId: string; + /** + * The fingerprints the write was planned under (`none` for a collection without list fields); + * `"none"` alone: the site listed no fields at all, so any state row is newer than the plan. + */ + readonly seen: ReadonlyMap | "none"; + /** The entry's rows read its original as `original`. */ + readonly own?: { readonly collection: string; readonly original: OriginalRead } | undefined; + /** The entry is an original whose same-site translations were read as `read`. */ + readonly translations?: + | { + readonly group: string; + readonly read: readonly { id: string; hash: string; collection: string }[]; + } + | undefined; + readonly anchor: Cond; + readonly at: string; +} + +const STATE_RESET = + "UPDATE entry_list_state SET ready = 0, cursor = NULL, epoch = ?, updated_at = ?"; + +/** + * Sets the affected collections not ready when the write was planned under other definitions + * (`seen`), or when the original or translations it read changed before it committed. Runs only + * once the write landed (`anchor`). Statements of the write that compare hashes come before + * anything that changes the translations' hashes (source-forward). + */ +export function stateCheckStatement(c: StateCheck): SqlStatement { + const params: SqlValue[] = [newEpoch(), c.at, c.siteId, ...c.anchor.params]; + if (c.seen === "none") + return { sql: `${STATE_RESET} WHERE site_id = ? AND ${c.anchor.sql}`, params }; + const or: string[] = [ + `EXISTS (SELECT 1 FROM json_each(?) j WHERE json_extract(j.value, '$[0]') = entry_list_state.collection + AND json_extract(j.value, '$[1]') <> entry_list_state.fingerprint)`, + ]; + params.push(JSON.stringify([...c.seen])); + if (c.own) { + const o = c.own.original; + or.push( + o.hash === null + ? "(entry_list_state.collection = ? AND EXISTS (SELECT 1 FROM nodes o WHERE o.id = ? AND o.site_id = ?))" + : "(entry_list_state.collection = ? AND NOT EXISTS (SELECT 1 FROM nodes o WHERE o.id = ? AND o.hash = ?))", + ); + params.push(c.own.collection, o.id, o.hash ?? c.siteId); + } + if (c.translations) { + const { group, read } = c.translations; + or.push(`((entry_list_state.collection IN (SELECT value FROM json_each(?)) + OR entry_list_state.collection IN (SELECT json_extract(t.body, '$.collection') FROM nodes t WHERE ${TRANSLATIONS_OF})) + AND ((SELECT count(*) FROM nodes t WHERE ${TRANSLATIONS_OF}) <> ? + OR EXISTS (SELECT 1 FROM json_each(?) j WHERE NOT EXISTS (SELECT 1 FROM nodes t + WHERE t.id = json_extract(j.value, '$[0]') AND t.hash = json_extract(j.value, '$[1]')))))`); + params.push( + JSON.stringify([...new Set(read.map((t) => t.collection))]), + c.siteId, + group, + group, + c.siteId, + group, + group, + read.length, + JSON.stringify(read.map((t) => [t.id, t.hash])), + ); + } + return { + sql: `${STATE_RESET} WHERE site_id = ? AND ${c.anchor.sql} AND (${or.join(" OR ")})`, + params, + }; +} + +/** + * A site write's readiness changes, anchored on the site's new hash: every collection whose + * fingerprint changed is reset (ready at once when it holds no entry: keys are + * `collection:locale:slug` and collections are slugs, so the key range is exactly the + * collection), and collections that list no fields any more lose their state row. Rows are left + * in place: every entry write replaces its own, and the backfill rewrites a collection's. + */ +export function siteStateStatements( + siteId: string, + changed: ReadonlyMap, + write: { anchor: Cond; at: string }, +): SqlStatement[] { + const listed = [...changed].filter(([, fp]) => fp !== "none"); + const dropped = [...changed].filter(([, fp]) => fp === "none").map(([c]) => c); + const out: SqlStatement[] = []; + if (listed.length > 0) + out.push({ + sql: `INSERT INTO entry_list_state (site_id, collection, fingerprint, epoch, ready, cursor, updated_at) + SELECT ?, json_extract(j.value, '$[0]'), json_extract(j.value, '$[1]'), ?, + NOT EXISTS (SELECT 1 FROM nodes n WHERE n.site_id = ? AND n.kind = 'contentEntry' + AND n.key > json_extract(j.value, '$[0]') || ':' AND n.key < json_extract(j.value, '$[0]') || ';'), + NULL, ? + FROM json_each(?) j WHERE ${write.anchor.sql} + ON CONFLICT (site_id, collection) DO UPDATE SET fingerprint = excluded.fingerprint, + epoch = excluded.epoch, ready = excluded.ready, cursor = NULL, updated_at = excluded.updated_at`, + params: [ + siteId, + newEpoch(), + siteId, + write.at, + JSON.stringify(listed), + ...write.anchor.params, + ], + }); + if (dropped.length > 0) + out.push({ + sql: `DELETE FROM entry_list_state WHERE site_id = ? + AND collection IN (SELECT value FROM json_each(?)) AND ${write.anchor.sql}`, + params: [siteId, JSON.stringify(dropped), ...write.anchor.params], + }); + return out; +} + +/** One listed collection of an imported site: its fingerprint, and whether it holds no entry. */ +export interface ImportedList { + readonly collection: string; + readonly fingerprint: string; + readonly empty: boolean; +} + +/** + * Import replaces a site's node set in one batch: its rows and states go, and each listed + * collection starts over (ready at once when empty). Rows are rebuilt after the batch. + */ +export function importResetStatements( + siteId: string, + listed: readonly ImportedList[], + at: string, +): SqlStatement[] { + const out: SqlStatement[] = [ + { sql: "DELETE FROM entry_list_values WHERE site_id = ?", params: [siteId] }, + { sql: "DELETE FROM entry_list_state WHERE site_id = ?", params: [siteId] }, + ]; + if (listed.length > 0) + out.push({ + sql: `INSERT INTO entry_list_state (site_id, collection, fingerprint, epoch, ready, cursor, updated_at) + SELECT ?, json_extract(j.value, '$[0]'), json_extract(j.value, '$[1]'), ?, json_extract(j.value, '$[2]'), NULL, ? + FROM json_each(?) j`, + params: [ + siteId, + newEpoch(), + at, + JSON.stringify(listed.map((l) => [l.collection, l.fingerprint, l.empty ? 1 : 0])), + ], + }); + return out; +} + +/** + * A field operation's pass rewrote `ids` (phase B, or a revert's restore) without touching their + * rows: the collections whose listed values may have moved are reset in the same batch. That is + * the operation's collection and those of the rewritten entries' same-site translations, each + * only while it lists one of `keys` (the operation's, and any other the batch's bodies change; + * the site as it is when the batch runs). + */ +export function passResetStatement(p: { + siteId: string; + collection: string; + ids: readonly string[]; + keys: readonly string[]; + at: string; +}): SqlStatement { + return { + sql: `${STATE_RESET} WHERE site_id = ? + AND (collection = ? OR collection IN (SELECT json_extract(t.body, '$.collection') FROM nodes t + WHERE t.site_id = ? AND t.kind = 'contentEntry' + AND json_extract(t.body, '$.translation.group') IN (SELECT value FROM json_each(?)) + AND json_extract(t.body, '$.translation.source.siteId') IS NULL)) + AND EXISTS (SELECT 1 FROM nodes s, + json_each(s.body, '$.collections."' || entry_list_state.collection || '".listFields') f + WHERE s.id = ? AND s.kind = 'site' AND f.value IN (SELECT value FROM json_each(?)))`, + params: [ + newEpoch(), + p.at, + p.siteId, + p.collection, + p.siteId, + JSON.stringify(p.ids), + p.siteId, + JSON.stringify(p.keys), + ], + }; +} + +/** A backfill run's view of a state row: chunks are written only while it is still this. */ +export interface StateView { + readonly siteId: string; + readonly collection: string; + readonly fingerprint: string; + readonly epoch: string; + readonly cursor: string | null; +} + +/** The state row is still `v` and not ready (every backfill statement's guard). */ +export function stateIs(v: StateView): Cond { + return { + sql: `EXISTS (SELECT 1 FROM entry_list_state s WHERE s.site_id = ? AND s.collection = ? + AND s.fingerprint = ? AND s.epoch = ? AND s.cursor IS ? AND s.ready = 0)`, + params: [v.siteId, v.collection, v.fingerprint, v.epoch, v.cursor], + }; +} + +/** + * The backfill's progress, last in a chunk's batch: the cursor moves to `cursor` (and `ready` + * becomes 1 after the last chunk) only while the state is `v` and every node of the chunk that + * still exists has the hash, and original, its rows were computed from. + */ +export function advanceStatement( + v: StateView, + chunk: readonly NodeValues[], + next: { cursor: string | null; ready: boolean; at: string }, +): SqlStatement { + return { + sql: `UPDATE entry_list_state SET cursor = ?, ready = ?, updated_at = ? + WHERE site_id = ? AND collection = ? AND fingerprint = ? AND epoch = ? AND cursor IS ? AND ready = 0 + AND NOT EXISTS (SELECT 1 FROM json_each(?) e + WHERE EXISTS (SELECT 1 FROM nodes n WHERE n.id = ${at(0)}) + AND NOT (${NODE_MATCH} AND ${ORIGINAL_OK}))`, + params: [ + next.cursor, + next.ready ? 1 : 0, + next.at, + v.siteId, + v.collection, + v.fingerprint, + v.epoch, + v.cursor, + JSON.stringify(chunk.map(head)), + v.siteId, + ], + }; +} + +/** + * The backfill's start on a collection without a state row, or with one from other + * definitions: not ready, from the beginning, under `v.epoch` (a fresh one, so one changed row + * means the state is exactly `v` with no cursor), while the site still has the hash it was read at. + */ +export function staleStateStatement( + v: Omit, + site: { id: string; hash: string }, + at: string, +): SqlStatement { + return { + sql: `INSERT INTO entry_list_state (site_id, collection, fingerprint, epoch, ready, cursor, updated_at) + SELECT ?, ?, ?, ?, 0, NULL, ? WHERE EXISTS (SELECT 1 FROM nodes WHERE id = ? AND hash = ?) + ON CONFLICT (site_id, collection) DO UPDATE SET fingerprint = excluded.fingerprint, + epoch = excluded.epoch, ready = 0, cursor = NULL, updated_at = excluded.updated_at + WHERE entry_list_state.fingerprint <> excluded.fingerprint`, + params: [v.siteId, v.collection, v.fingerprint, v.epoch, at, site.id, site.hash], + }; +} diff --git a/packages/core-ts/src/list-values/statements.ts b/packages/core-ts/src/list-values/statements.ts new file mode 100644 index 00000000..0292a783 --- /dev/null +++ b/packages/core-ts/src/list-values/statements.ts @@ -0,0 +1,165 @@ +/** + * RFC-0042 §4, #110: the SQL of `entry_list_values` rows (readiness is `state-sql.ts`). Builders + * only; the entry write paths (`write-plan.ts`) and the backfill put these statements in their + * own atomic batches, guarded on the hashes they were planned from (AGENTS.md rule 4). + * + * Rows travel as one JSON payload per statement (`json_each`): strings, and numbers that are safe + * integers (instants in ms, days as YYYYMMDD, 0/1). Other numbers are bound as parameters through + * a `VALUES` table: D1's JSON parser does not read every double back exactly. The INSERT's + * `CROSS JOIN`s keep the payload the outer loop (SQLite keeps their order): left to the planner, + * `n.site_id = ?` made it scan the site's nodes once per statement. + */ +import type { SqlStatement } from "@cloudhal/storage"; +import type { Cond } from "../node-sql.ts"; +import type { EntryListValues } from "./rows.ts"; + +/** The original a translation's rows were computed with: its hash as read, `null` when absent. */ +export interface OriginalRead { + readonly id: string; + readonly hash: string | null; +} + +/** One node's rows with what they were computed from. */ +export interface NodeValues { + readonly id: string; + /** The node's hash the rows belong to: as read, or its new hash in its own write. */ + readonly hash: string; + /** Present when a listed value came from (or would come from) the original. */ + readonly original?: OriginalRead | undefined; + /** Absent: the node gets no rows (its collection lists no fields). */ + readonly values: EntryListValues | undefined; +} + +/** Bound numbers per statement (D1 allows 100 parameters). */ +const MAX_BOUND = 80; +/** JSON characters per payload. */ +const MAX_PAYLOAD = 200_000; + +export const at = (i: number) => `json_extract(e.value, '$[${i}]')`; +/** The payload element's node still has the hash its rows were computed for. */ +export const NODE_MATCH = `EXISTS (SELECT 1 FROM nodes n WHERE n.id = ${at(0)} AND n.hash = ${at(1)})`; +/** The element's original is as read (one `?`: the site id). */ +export const ORIGINAL_OK = `(${at(2)} IS NULL OR CASE WHEN ${at(3)} IS NULL + THEN NOT EXISTS (SELECT 1 FROM nodes o WHERE o.id = ${at(2)} AND o.site_id = ?) + ELSE EXISTS (SELECT 1 FROM nodes o WHERE o.id = ${at(2)} AND o.hash = ${at(3)}) END)`; + +const tail = (guard: Cond | undefined) => (guard ? ` AND ${guard.sql}` : ""); +export const head = (n: NodeValues) => [ + n.id, + n.hash, + n.original?.id ?? null, + n.original?.hash ?? null, +]; + +function element(n: NodeValues, bound: number[]): unknown[] { + const v = n.values as EntryListValues; + const rows = v.rows.map((r) => { + if (r.num === null || Number.isSafeInteger(r.num)) return [r.field, r.has, r.num, r.text, null]; + bound.push(r.num); + return [r.field, r.has, null, r.text, bound.length - 1]; + }); + return [...head(n), v.collection, v.locale, v.status, v.tie, rows]; +} + +function insertStatement(siteId: string, nodes: readonly NodeValues[], guard?: Cond) { + const bound: number[] = []; + const payload = JSON.stringify(nodes.map((n) => element(n, bound))); + const num = bound.length + ? "COALESCE(json_extract(r.value, '$[2]'), (SELECT b.n FROM b WHERE b.i = json_extract(r.value, '$[4]')))" + : "json_extract(r.value, '$[2]')"; + const values = bound.length + ? `WITH b(i, n) AS (VALUES ${bound.map((_, i) => `(${i}, ?)`).join(", ")}) ` + : ""; + return { + sql: `${values}INSERT INTO entry_list_values + (site_id, collection, field, has, num, text, tie, node_id, created, locale, status) + SELECT n.site_id, ${at(4)}, json_extract(r.value, '$[0]'), json_extract(r.value, '$[1]'), + ${num}, json_extract(r.value, '$[3]'), ${at(7)}, n.id, n.created_at, ${at(5)}, ${at(6)} + FROM json_each(?) e + CROSS JOIN nodes n ON n.id = ${at(0)} AND n.hash = ${at(1)} + CROSS JOIN json_each(e.value, '$[8]') r + WHERE n.site_id = ? AND ${ORIGINAL_OK}${tail(guard)}`, + params: [...bound, payload, siteId, siteId, ...(guard?.params ?? [])], + }; +} + +/** Nodes in groups that fit one INSERT (bound numbers and payload size). */ +function groups(nodes: readonly NodeValues[]): NodeValues[][] { + const out: NodeValues[][] = []; + let group: NodeValues[] = []; + let bound = 0; + let size = 0; + for (const n of nodes) { + const rows = n.values?.rows ?? []; + const b = rows.filter((r) => r.num !== null && !Number.isSafeInteger(r.num)).length; + const s = JSON.stringify(rows).length + 200; + if (group.length > 0 && (bound + b > MAX_BOUND || size + s > MAX_PAYLOAD)) { + out.push(group); + group = []; + bound = 0; + size = 0; + } + group.push(n); + bound += b; + size += s; + } + if (group.length > 0) out.push(group); + return out; +} + +/** + * The rows of `nodes` replaced: one DELETE of their rows, then INSERTs of the new ones, each node + * only while it (and its original) still has the hash its rows were computed from, and only when + * `guard` holds. A node whose `values` is absent loses its rows. + */ +export function nodeRowsStatements( + siteId: string, + nodes: readonly NodeValues[], + guard?: Cond, +): SqlStatement[] { + if (nodes.length === 0) return []; + const del: SqlStatement = { + sql: `DELETE FROM entry_list_values WHERE node_id IN (SELECT ${at(0)} FROM json_each(?) e + WHERE ${NODE_MATCH} AND ${ORIGINAL_OK})${tail(guard)}`, + params: [JSON.stringify(nodes.map(head)), siteId, ...(guard?.params ?? [])], + }; + return [del, ...rowInserts(siteId, nodes, guard)]; +} + +const withRows = (nodes: readonly NodeValues[]) => + nodes.filter((n) => (n.values?.rows.length ?? 0) > 0); + +/** The INSERTs of `nodeRowsStatements` alone (after a DELETE of the caller's own). */ +export function rowInserts( + siteId: string, + nodes: readonly NodeValues[], + guard?: Cond, +): SqlStatement[] { + return groups(withRows(nodes)).map((g) => insertStatement(siteId, g, guard)); +} + +/** How many statements `nodeRowsStatements` makes of `nodes` (without building them). */ +export function rowStatementCount(nodes: readonly NodeValues[]): number { + return nodes.length === 0 ? 0 : 1 + groups(withRows(nodes)).length; +} + +/** Every row of a node, once the node has `hash` (its own write's first statement). */ +export function dropOwnRows(id: string, hash: string): SqlStatement { + return { + sql: "DELETE FROM entry_list_values WHERE node_id = ? AND EXISTS (SELECT 1 FROM nodes WHERE id = ? AND hash = ?)", + params: [id, id, hash], + }; +} + +/** Every row of a node, once the node is gone (a delete). */ +export function dropGoneRows(id: string): SqlStatement { + return { + sql: "DELETE FROM entry_list_values WHERE node_id = ? AND NOT EXISTS (SELECT 1 FROM nodes WHERE id = ?)", + params: [id, id], + }; +} + +/** Same-site translations of `group` (the store's own `nodes_translation_locale` index). */ +export const TRANSLATIONS_OF = `t.site_id = ? AND t.kind = 'contentEntry' + AND json_extract(t.body, '$.translation.group') = ? AND t.id <> ? + AND json_extract(t.body, '$.translation.source.siteId') IS NULL`; diff --git a/packages/core-ts/src/list-values/tidy.ts b/packages/core-ts/src/list-values/tidy.ts new file mode 100644 index 00000000..823e22e5 --- /dev/null +++ b/packages/core-ts/src/list-values/tidy.ts @@ -0,0 +1,58 @@ +/** + * RFC-0042 §4, #110: the admin job's tidy. A site write that unlists a collection drops its state + * row and leaves its rows (an entry write replaces only its own); this removes them, in bounded + * statements within the run's budget. + * + * Each DELETE covers many (site, collection) pairs and is safe inside itself: a row goes only + * while its collection has no state row (a collection gets one in the batch that lists it, and + * loses it in the batch that unlists it) and its site still has the hash the run read the pairs + * from. So a collection listed (and rebuilt) after the run read its site keeps every row. + */ +import type { Site } from "@cloudhal/core-contracts"; +import type { SqlStatement } from "@cloudhal/storage"; +import type { ListValuesMeter } from "./meter.ts"; + +/** Rows one statement removes at most. */ +export const TIDY_CHUNK = 2000; +/** (site, collection) pairs one statement looks at. */ +const PAIRS = 200; + +/** Up to `TIDY_CHUNK` rows of `pairs` (`[siteId, collection, siteHash]`), each guarded as above. */ +export function tidyStatement(pairs: readonly (readonly [string, string, string])[]): SqlStatement { + return { + sql: `DELETE FROM entry_list_values WHERE rowid IN (SELECT v.rowid FROM json_each(?) j + CROSS JOIN entry_list_values v + WHERE v.site_id = json_extract(j.value, '$[0]') AND v.collection = json_extract(j.value, '$[1]') + AND EXISTS (SELECT 1 FROM nodes s WHERE s.id = json_extract(j.value, '$[0]') + AND s.hash = json_extract(j.value, '$[2]')) + AND NOT EXISTS (SELECT 1 FROM entry_list_state t WHERE t.site_id = v.site_id + AND t.collection = v.collection) + LIMIT ${TIDY_CHUNK})`, + params: [JSON.stringify(pairs)], + }; +} + +/** The collections of `sites` that list no fields, with the site hash each was read at. */ +function unlisted(sites: readonly { site: Site; hash: string }[]) { + return sites.flatMap(({ site, hash }) => + Object.entries(site.collections ?? {}) + .filter(([, s]) => !s?.listFields?.length) + .map(([c]) => [site.id, c, hash] as const), + ); +} + +/** Removes rows of unlisted collections while the meter allows; the number removed. */ +export async function tidyRows( + meter: ListValuesMeter, + sites: readonly { site: Site; hash: string }[], +): Promise { + const pairs = unlisted(sites); + let removed = 0; + for (let i = 0; i < pairs.length && !meter.short(1); ) { + const s = tidyStatement(pairs.slice(i, i + PAIRS)); + const r = await meter.sql.run(s.sql, s.params ?? []); + removed += r.changes; + if (r.changes < TIDY_CHUNK) i += PAIRS; + } + return removed; +} diff --git a/packages/core-ts/src/list-values/write-plan.ts b/packages/core-ts/src/list-values/write-plan.ts new file mode 100644 index 00000000..9329be71 --- /dev/null +++ b/packages/core-ts/src/list-values/write-plan.ts @@ -0,0 +1,188 @@ +/** + * RFC-0042 §4, #110: what an entry write or delete adds to its own batch so `entry_list_values` + * stays exact (AGENTS.md rule 4). + * + * - The entry's rows: all of them go (whatever collection they were in), then its listed values + * come back, anchored on its new hash. A translation's rows read its original as stored, and + * only while the original still has that hash. + * - An original whose values changed (or that is created or deleted) rewrites the rows of its + * same-site translations that read a listed value from it, each anchored on the translation's + * hash as read. + * - Last, the state check: a race (definitions changed under the plan, the original or the + * translations moved) never refuses the write; it sets the affected collections not ready, + * and the backfill rebuilds them. Every write rewrites its own rows: no "unchanged" shortcut, + * which a running backfill would turn into stale rows. + * + * The statements come before source-forward's, which change the translations' hashes. + */ +import type { ContentEntry, Site } from "@cloudhal/core-contracts"; +import type { SqlDriver, SqlStatement } from "@cloudhal/storage"; +import { sameJson } from "../fields/types.ts"; +import { type Cond, hashIs } from "../node-sql.ts"; +import { listedCollections, listFingerprint } from "./fingerprint.ts"; +import { dependsOnOriginal, listValueRows, originalIdOf } from "./rows.ts"; +import { stateCheckStatement } from "./state-sql.ts"; +import { + dropGoneRows, + dropOwnRows, + type NodeValues, + nodeRowsStatements, + type OriginalRead, + rowInserts, + TRANSLATIONS_OF, +} from "./statements.ts"; + +/** A node as read, with the hash its rows are computed against. */ +export interface ReadEntry { + readonly entry: ContentEntry; + readonly hash: string; +} + +/** Statements built once the entry's new hash is known. */ +export type ListStatements = (hash: string) => SqlStatement[]; + +const parse = (r: { hash: string; body: string }): ReadEntry => ({ + entry: JSON.parse(r.body) as ContentEntry, + hash: r.hash, +}); + +/** The node `id` when it is an entry of `siteId` (an original lives in the same site). */ +export async function readEntry( + sql: SqlDriver, + siteId: string, + id: string, +): Promise { + const row = await sql.get<{ hash: string; body: string }>( + "SELECT hash, body FROM nodes WHERE id = ? AND site_id = ? AND kind = 'contentEntry'", + [id, siteId], + ); + return row ? parse(row) : undefined; +} + +/** The same-site translations of `group` (through the translation index). */ +export async function readTranslations( + sql: SqlDriver, + siteId: string, + group: string, +): Promise { + const rows = await sql.all<{ hash: string; body: string }>( + `SELECT t.hash, t.body FROM nodes t WHERE ${TRANSLATIONS_OF} ORDER BY t.id`, + [siteId, group, group], + ); + return rows.map(parse); +} + +async function fingerprints(site: Site, collections: Iterable) { + const out = new Map(); + for (const c of new Set(collections)) out.set(c, await listFingerprint(site, c)); + return out; +} + +/** The translations of an original that read a listed value from it, recomputed. */ +function translationValues( + site: Site, + translations: readonly ReadEntry[], + original: ContentEntry | undefined, +): NodeValues[] { + return translations + .filter((t) => dependsOnOriginal(site, t.entry)) + .map((t) => ({ + id: t.entry.id, + hash: t.hash, + values: listValueRows(site, t.entry, original), + })); +} + +const readSet = (group: string, translations: readonly ReadEntry[]) => ({ + group, + read: translations.map((t) => ({ id: t.entry.id, hash: t.hash, collection: t.entry.collection })), +}); + +/** The statements of `EntryEngine.write` for `entry` (its final body), `stored` being what it replaces. */ +export async function planListValues( + sql: SqlDriver, + site: Site, + write: { stored: ContentEntry | undefined; entry: ContentEntry; at: string }, +): Promise { + const { stored, entry, at } = write; + if (listedCollections(site).length === 0) + return (hash) => { + const anchor = hashIs(entry.id, hash); + return [ + dropOwnRows(entry.id, hash), + stateCheckStatement({ siteId: site.id, seen: "none", anchor, at }), + ]; + }; + const groupId = originalIdOf(entry); + const reads = groupId !== undefined && dependsOnOriginal(site, entry); + const original = reads ? await readEntry(sql, site.id, groupId) : undefined; + const own: OriginalRead | undefined = reads + ? { id: groupId, hash: original?.hash ?? null } + : undefined; + const values = listValueRows(site, entry, original?.entry); + // RFC-0041 §5: translations read an original's values live, so their rows follow it. + const changed = !stored || !sameJson(stored.fields ?? {}, entry.fields ?? {}); + const translations = changed ? await readTranslations(sql, site.id, entry.id) : []; + const others = translationValues(site, translations, entry); + const seen = await fingerprints(site, [ + entry.collection, + ...translations.map((t) => t.entry.collection), + ]); + return (hash) => { + const anchor = hashIs(entry.id, hash); + return [ + dropOwnRows(entry.id, hash), + ...rowInserts(site.id, [{ id: entry.id, hash, original: own, values }]), + ...nodeRowsStatements(site.id, others, anchor), + stateCheckStatement({ + siteId: site.id, + seen, + own: own && values ? { collection: entry.collection, original: own } : undefined, + translations: changed ? readSet(entry.id, translations) : undefined, + anchor, + at, + }), + ]; + }; +} + +/** `entry` is gone (rule 4's anchor of a delete). */ +const gone = (id: string): Cond => ({ + sql: "NOT EXISTS (SELECT 1 FROM nodes WHERE id = ?)", + params: [id], +}); + +/** The statements of `EntryEngine.delete`: its rows go, its translations fall back to their own. */ +export async function planListValuesDelete( + sql: SqlDriver, + site: Site | undefined, + entry: ContentEntry, + at: string, +): Promise { + const anchor = gone(entry.id); + if (!site || listedCollections(site).length === 0) + return [ + dropGoneRows(entry.id), + stateCheckStatement({ siteId: entry.siteId, seen: "none", anchor, at }), + ]; + const translations = await readTranslations(sql, site.id, entry.id); + const others = translationValues(site, translations, undefined).map((n) => ({ + ...n, + original: { id: entry.id, hash: null }, + })); + const seen = await fingerprints( + site, + translations.map((t) => t.entry.collection), + ); + return [ + dropGoneRows(entry.id), + ...nodeRowsStatements(site.id, others, anchor), + stateCheckStatement({ + siteId: site.id, + seen, + translations: readSet(entry.id, translations), + anchor, + at, + }), + ]; +} diff --git a/packages/core-ts/src/local.ts b/packages/core-ts/src/local.ts index dc9708be..93ef2857 100644 --- a/packages/core-ts/src/local.ts +++ b/packages/core-ts/src/local.ts @@ -5,6 +5,7 @@ import { FsBlobStore } from "@cloudhal/storage/fs"; import { BetterSqlite3Driver } from "@cloudhal/storage/sqlite"; import { GATES_ENFORCED } from "./availability.ts"; import type { BlockRegistry, PropsSchema } from "./blocks.ts"; +import { rebuildListValues } from "./list-values/backfill.ts"; import { SiteGraphStore } from "./store.ts"; export interface LocalStore { @@ -37,6 +38,8 @@ export async function openLocalStore( await mkdir(path.join(dir, "blobs"), { recursive: true }); const sql = await BetterSqlite3Driver.open(path.join(dir, "data.db")); await migrate(sql, migrations); + // RFC-0042 §4: list values for collections listed before migration 0029 or left unfinished. + await rebuildListValues(sql); const blobs = new FsBlobStore(path.join(dir, "blobs")); // The renderer that serves this store runs in the same process and enforces gates (S15). const store = new SiteGraphStore({ diff --git a/packages/core-ts/src/site-delete.ts b/packages/core-ts/src/site-delete.ts index 9e5dbe9e..39a92dc3 100644 --- a/packages/core-ts/src/site-delete.ts +++ b/packages/core-ts/src/site-delete.ts @@ -20,6 +20,8 @@ export const SITE_TABLES = [ "search_index", "search_index_cjk", "search_state", + "entry_list_values", + "entry_list_state", "ai_usage", "ai_settings", "body_proposals", diff --git a/packages/core-ts/src/store.ts b/packages/core-ts/src/store.ts index 63484bf5..a0a273b3 100644 --- a/packages/core-ts/src/store.ts +++ b/packages/core-ts/src/store.ts @@ -498,17 +498,8 @@ export class SiteGraphStore { return out; } - async deleteContentEntry(entry: ContentEntry): Promise { - await assertUnlocked(this.#sql, entry.siteId, [entry.id]); - const [r] = await this.#sql.batch([ - { - sql: "DELETE FROM nodes WHERE id = ? AND rev = ? AND kind = 'contentEntry' AND site_id = ?", - params: [entry.id, entry.rev, entry.siteId], - }, - this.#assets.dropRefsStatement(entry.id), - ]); - if (r?.changes !== 1) - throw new ConflictError("contentEntry", entry.id, entry.rev, await this.#revOf(entry.id)); + deleteContentEntry(entry: ContentEntry): Promise { + return this.#entries.delete(entry); } // ───────────────────────────── taxonomies (RFC-0002) ───────────────────────────── diff --git a/packages/core-ts/src/store/entries.ts b/packages/core-ts/src/store/entries.ts index 66dd6c49..fc360d70 100644 --- a/packages/core-ts/src/store/entries.ts +++ b/packages/core-ts/src/store/entries.ts @@ -6,6 +6,7 @@ import { ConflictError, NotFoundError } from "../errors.ts"; import { nodeHash } from "../hash.ts"; import { assertUnlocked } from "../holds.ts"; import { ID_PREFIX } from "../ids.ts"; +import { planListValues, planListValuesDelete } from "../list-values/write-plan.ts"; import { fillMissingSrc } from "../media/refs.ts"; import { and, casUpdateStatement, insertNodeStatement, OpBatch, runOpBatch } from "../node-sql.ts"; import { uniquenessKey } from "../nodes.ts"; @@ -104,7 +105,10 @@ export class EntryEngine { const hash = await nodeHash(entry); // RFC-0041 §5: a change to non-localized values alone leaves translations fresh. const forward = await planSourceForward(this.#ctx, site, { stored, entry, ts }); - const extra = [...refs.statements(hash), ...fields.statements(hash), ...(await forward(hash))]; + // RFC-0042 §4: list values before source-forward, which changes the translations' hashes. + const lists = await planListValues(this.#ctx.sql, site, { stored, entry, at: ts }); + const own = [...refs.statements(hash), ...fields.statements(hash), ...lists(hash)]; + const extra = [...own, ...(await forward(hash))]; const key = uniquenessKey("contentEntry", entry); const sameLocale = stored?.locale === entry.locale; const plan = await this.#ctx.tax.planPathChange(site, ts, { @@ -153,4 +157,21 @@ export class EntryEngine { }); return entry; } + + /** CAS delete with its asset references and list values (RFC-0042 §4), in one batch. */ + async delete(entry: ContentEntry): Promise { + await assertUnlocked(this.#ctx.sql, entry.siteId, [entry.id]); + const site = await this.#ctx.graph.getSite(entry.siteId); + const lists = await planListValuesDelete(this.#ctx.sql, site, entry, this.#ctx.stamp()); + const [r] = await this.#ctx.sql.batch([ + { + sql: "DELETE FROM nodes WHERE id = ? AND rev = ? AND kind = 'contentEntry' AND site_id = ?", + params: [entry.id, entry.rev, entry.siteId], + }, + this.#ctx.assets.dropRefsStatement(entry.id), + ...lists, + ]); + if (r?.changes !== 1) + throw new ConflictError("contentEntry", entry.id, entry.rev, await this.#ctx.revOf(entry.id)); + } } diff --git a/packages/core-ts/src/store/sites.ts b/packages/core-ts/src/store/sites.ts index 205e3231..846d29e6 100644 --- a/packages/core-ts/src/store/sites.ts +++ b/packages/core-ts/src/store/sites.ts @@ -7,6 +7,8 @@ import { ConflictError, InvariantError, isOpGuardViolation, NotFoundError } from import { nodeHash } from "../hash.ts"; import { checkHosts, hostStatements, hostsOf } from "../hosts.ts"; import { ID_PREFIX } from "../ids.ts"; +import { settleListValues } from "../list-values/backfill.ts"; +import { planListState } from "../list-values/site-plan.ts"; import { fillMissingSrc } from "../media/refs.ts"; import { and, casUpdateStatement, insertNodeStatement } from "../node-sql.ts"; import { type Creatable, uniquenessKey } from "../nodes.ts"; @@ -45,11 +47,14 @@ export class SiteEngine { // RFC-0034 §4: a hostname belongs to one site of the installation. const hosts = hostsOf(site); await checkHosts(this.#ctx.sql, hosts); + const hash = await nodeHash(site); + // RFC-0042 §4: a new site's listed collections are empty, so their list values are ready. + const lists = await planListState(undefined, site); await this.#ctx.runUnique( await insertNodeStatement("site", site, ts), "site", uniquenessKey("site", site), - hostStatements(hosts, { siteId: site.id, hash: await nodeHash(site) }), + [...hostStatements(hosts, { siteId: site.id, hash }), ...lists.statements(hash, ts)], ); return site; } @@ -114,6 +119,8 @@ export class SiteEngine { const refs = await this.#ctx.assets.planRefs("site", site.id, next, stored); const written = refs.node; if (options.dryRun) return written; + // RFC-0042 §4: a collection whose list definitions change is not ready until rebuilt. + const lists = await planListState(stored, written); const s = await casUpdateStatement( "site", written, @@ -126,6 +133,7 @@ export class SiteEngine { ...(extra?.({ siteId: site.id, hash }, written.updatedAt) ?? []), ...refs.statements(hash), ...((await permalinks?.statements(hash)) ?? []), + ...lists.statements(hash, written.updatedAt), ]; const r = await this.#run(s, more); if (r.changes !== 1) { @@ -139,6 +147,8 @@ export class SiteEngine { } throw new ConflictError("site", site.id, site.rev, rev); } + if (lists.changed.length > 0) + await settleListValues(this.#ctx.sql, site.id, { collections: lists.changed }); return written; } diff --git a/packages/core-ts/src/sync-plan.ts b/packages/core-ts/src/sync-plan.ts index 8250b015..da411635 100644 --- a/packages/core-ts/src/sync-plan.ts +++ b/packages/core-ts/src/sync-plan.ts @@ -15,6 +15,7 @@ import type { GateSupport } from "./availability.ts"; import { InvariantError } from "./errors.ts"; import { canonicalJson, sha256Hex } from "./hash.ts"; import { checkHosts, hostStatements, hostsOf, type SiteHost } from "./hosts.ts"; +import { importListStatements } from "./list-values/site-plan.ts"; import { refsRebuild } from "./media/refs-rebuild.ts"; import { type BundleMedia, @@ -281,7 +282,11 @@ function nodeBatch( bundle: SiteBundle, siteRow: NodeRow, changes: NodeChanges, - extra: { project: SqlStatement | undefined; hosts: readonly SiteHost[] }, + extra: { + project: SqlStatement | undefined; + hosts: readonly SiteHost[]; + lists: readonly SqlStatement[]; + }, ): Pick { const { siteId } = bundle; const batch: SqlStatement[] = extra.project ? [extra.project] : []; @@ -297,6 +302,7 @@ function nodeBatch( }); inserted++; } + batch.push(...extra.lists); } // The derived reference index follows the node set in the same batch (RFC-0004 §4.4). batch.push(...refsRebuild(siteId, changes.rows)); @@ -345,7 +351,9 @@ export async function prepareImport( const snapshots = await snapshotsToAdd(to, bundle); const project = await projectToAdd(to, bundle); const changes = await nodeChanges(to, bundle, mode, options); - const nodes = nodeBatch(bundle, siteRow, changes, { project, hosts }); + // RFC-0042 §4: the list values start over with the node set (rebuilt after the batch). + const lists = await importListStatements(siteId, changes.rows, new Date().toISOString()); + const nodes = nodeBatch(bundle, siteRow, changes, { project, hosts, lists }); const ledger = await planLedger(to, bundle, { refs, written: new Map(blobs.map((b) => [b.hash, b.body])), diff --git a/packages/core-ts/src/sync.ts b/packages/core-ts/src/sync.ts index a163ce92..9c931947 100644 --- a/packages/core-ts/src/sync.ts +++ b/packages/core-ts/src/sync.ts @@ -20,6 +20,7 @@ import type { Asset } from "@cloudhal/core-contracts"; import type { BlobStore, SqlDriver, SqlStatement } from "@cloudhal/storage"; import { mediaBlobKey } from "./assets.ts"; import { InvariantError } from "./errors.ts"; +import { rebuildListValues } from "./list-values/backfill.ts"; import { type BundleMedia, type CopySiteOptions, @@ -38,6 +39,9 @@ export * from "./sync-bundle.ts"; export { type ImportPlan, type ImportPlanOptions, planImport } from "./sync-plan.ts"; export { restoreBundle } from "./sync-restore.ts"; +/** The list-value rebuild an import runs after its batch; the admin job does the rest. */ +const IMPORT_LIST_BUDGET = { entries: 5000, ms: 15_000 }; + /** Splits statements into batches that respect D1's per-batch limit. */ async function runBatches(sql: SqlDriver, statements: SqlStatement[], size = 500): Promise { for (let i = 0; i < statements.length; i += size) await sql.batch(statements.slice(i, i + size)); @@ -166,6 +170,11 @@ export async function importSite( // 4. The media ledger and `snapshot_blobs`; taken-down bytes never survive a copy. await runBatches(to.sql, writes.ledger); for (const key of writes.takedowns) await to.blobs.delete(key); + // 5. RFC-0042 §4: list values of the new node set (until then, listings scan; the job resumes). + if (!plan.unchanged) + await rebuildListValues(to.sql, { siteId: plan.siteId, budget: IMPORT_LIST_BUDGET }).catch( + (e: unknown) => log(`list values: not rebuilt yet (${(e as Error).message})`), + ); log( `snapshots: ${writes.snapshots.length} added; nodes: ${plan.unchanged ? "unchanged" : `${writes.inserted} written, ${writes.deleted} replaced`}`, ); diff --git a/packages/core-ts/src/tags-migration.ts b/packages/core-ts/src/tags-migration.ts index 9c676aa9..5c8de783 100644 --- a/packages/core-ts/src/tags-migration.ts +++ b/packages/core-ts/src/tags-migration.ts @@ -13,6 +13,7 @@ import { } from "@cloudhal/core-contracts"; import type { SqlDriver, SqlStatement } from "@cloudhal/storage"; import { NotFoundError } from "./errors.ts"; +import { sameJson } from "./fields/types.ts"; import { ID_PREFIX } from "./ids.ts"; import { humanizeSlug } from "./names.ts"; import { type Cond, casUpdateStatement, OpBatch, runOpBatch } from "./node-sql.ts"; @@ -67,6 +68,16 @@ export interface TagsRevertResult { readonly snapshotId: string; } +/** + * The entry without what the migration writes (`tags`, the tag terms) and the stamps: a body the + * migration alone changed has the same rest before and after it. + */ +function untagged(entry: ContentEntry) { + const { tags: _t, terms, rev: _r, updatedAt: _u, ...rest } = entry; + const { tag: _g, ...others } = terms ?? {}; + return { ...rest, terms: others }; +} + async function migrationSnapshots(store: SiteGraphStore, siteId: string): Promise { return (await store.listSnapshots(siteId, 1000)).filter( (s) => s.metadata?.[KEY] === TAGS_MIGRATION, @@ -336,7 +347,9 @@ export async function revertTagsToTerms( `migration ${migrationId} did not finish; run it again to complete it, then revert`, ); - // 1. Entries still exactly as the migration left them get their earlier body back. + // 1. Entries still exactly as the migration left them get their earlier body back. One edited + // after the before-snapshot but before its batch (or a resumed run) counts as edited, so the + // restore never rolls back fields, status or collection (nor the list rows that follow them). const current = new Map((await store.currentNodeRefs(siteId)).map((r) => [r.id, r])); const beforeRefs = new Map(before.nodes.map((r) => [r.id, r])); const edited: string[] = []; @@ -351,7 +364,10 @@ export async function revertTagsToTerms( if (!prev.tags?.length) continue; const now = current.get(ref.id); if (!now) continue; - if (now.hash !== ref.hash) { + if ( + now.hash !== ref.hash || + !sameJson(untagged(prev), untagged(await store.readSnapshotNode(ref))) + ) { edited.push(ref.id); continue; } diff --git a/packages/core-ts/src/testing-list-values-kit.ts b/packages/core-ts/src/testing-list-values-kit.ts new file mode 100644 index 00000000..79073942 --- /dev/null +++ b/packages/core-ts/src/testing-list-values-kit.ts @@ -0,0 +1,263 @@ +/** + * RFC-0042 §4, #110: what the `entry_list_values` behavioural suites share — a fresh store per + * test on the factory's backend, a site with two listed collections, entries and translations, + * and `expectExact`, which recomputes every entry's rows from the nodes (the evaluator's + * resolution) and compares them, as a multiset, with the table (and the listed collections' + * states with their fingerprints). `listValuesKit` registers its hooks in the calling `describe`. + */ +import type { + Actor, + CollectionSettings, + ContentEntry, + FieldDefinition, + FieldOption, + Site, + Translation, +} from "@cloudhal/core-contracts"; +import type { BlobStore, SqlDriver, SqlStatement, SqlValue } from "@cloudhal/storage"; +import { dropAllTables } from "@cloudhal/storage/testing"; +import { afterAll, beforeEach, expect } from "vitest"; +import { GATES_ENFORCED } from "./availability.ts"; +import { listedCollections, listFingerprint } from "./list-values/fingerprint.ts"; +import { listValueRows, originalIdOf } from "./list-values/rows.ts"; +import { SiteGraphStore } from "./store.ts"; + +export const person: Actor = { kind: "human", id: "user_01HZX8K3Q9CD", name: "Halil" }; +const L = (label: string) => ({ en: { label } }); +const KINDS: [FieldOption, ...FieldOption[]] = [ + { value: "talk", labels: { en: "Talk" } }, + { value: "workshop", labels: { en: "Workshop" } }, + { value: "social", labels: { en: "Social" } }, +]; + +export const FIELDS: [FieldDefinition, ...FieldDefinition[]] = [ + { key: "startsAt", type: "datetime", labels: L("Starts"), localized: false }, + { key: "price", type: "number", labels: L("Price"), localized: false }, + { + key: "kind", + type: "select", + labels: L("Kind"), + localized: false, + options: KINDS, + multiple: true, + }, + { key: "venue", type: "text", labels: L("Venue"), localized: true }, + { key: "day", type: "date", labels: L("Day"), localized: true }, + { key: "free", type: "boolean", labels: L("Free"), localized: false }, + { key: "notes", type: "richText", labels: L("Notes"), localized: true }, +]; +export const EVENTS: CollectionSettings = { + fields: FIELDS, + listFields: ["startsAt", "price", "kind", "venue"], +}; +export const TALKS: CollectionSettings = { + fields: FIELDS, + listFields: ["price", "free", "day"], + sort: "title", +}; +export const at = (iso: string) => ({ at: iso, timeZone: "Europe/Istanbul" }); + +export interface ListValuesFactory { + create(): Promise<{ sql: SqlDriver; blobs: BlobStore }>; + dispose?(): Promise; +} + +/** A driver that records every statement it runs (to see what a write sent). */ +export function recording(sql: SqlDriver, seen: string[]): SqlDriver { + return { + name: `${sql.name}+rec`, + get: >(q: string, p?: readonly SqlValue[]) => + sql.get(q, p), + all: >(q: string, p?: readonly SqlValue[]) => + sql.all(q, p), + run: (q: string, p?: readonly SqlValue[]) => { + seen.push(q); + return sql.run(q, p); + }, + batch: (s: readonly SqlStatement[]) => { + seen.push(...s.map((x) => x.sql)); + return sql.batch(s); + }, + exec: (q: string) => sql.exec(q), + close: async () => {}, + }; +} + +type StateRow = { fingerprint: string; epoch: string; ready: number; cursor: string | null }; + +export interface ListValuesKit { + /** This test's driver, blobs and store (new in every test). */ + readonly sql: SqlDriver; + readonly blobs: BlobStore; + readonly store: SiteGraphStore; + /** A store over `driver` with the suite's clock and gates. */ + open(driver: SqlDriver): SiteGraphStore; + setup(collections?: Record): Promise; + entry( + site: Site, + slug: string, + fields: Record, + extra?: Partial, + on?: SiteGraphStore, + ): Promise; + translate(site: Site, original: ContentEntry, over: Partial): Promise; + fresh(site: Site): Promise; + get(id: string): Promise; + hashOf(id: string): Promise; + state(siteId: string, collection: string): Promise; + expectExact(siteId: string, options?: { ready?: boolean }): Promise; +} + +/** The rows `listValueRows` gives every entry of the site, as comparable strings. */ +async function wantedRows(kit: ListValuesKit, site: Site): Promise { + const nodes = await kit.sql.all<{ created_at: string; body: string }>( + "SELECT created_at, body FROM nodes WHERE site_id = ? AND kind = 'contentEntry'", + [site.id], + ); + const entries = nodes.map((r) => ({ + created: r.created_at, + entry: JSON.parse(r.body) as ContentEntry, + })); + const byId = new Map(entries.map((e) => [e.entry.id, e.entry])); + const want: string[] = []; + for (const { created, entry: e } of entries) { + const groupId = originalIdOf(e); + const v = listValueRows(site, e, groupId ? byId.get(groupId) : undefined); + for (const r of v?.rows ?? []) + want.push( + [ + e.id, + v?.collection, + r.field, + r.has, + r.num, + r.text, + v?.tie, + created, + e.locale, + e.status, + ].join("|"), + ); + } + return want; +} + +async function storedRows(sql: SqlDriver, siteId: string): Promise { + const rows = await sql.all>( + "SELECT node_id, collection, field, has, num, text, tie, created, locale, status FROM entry_list_values WHERE site_id = ?", + [siteId], + ); + const columns = ["node_id", "collection", "field", "has", "num", "text", "tie", "created"]; + return rows.map((r) => [...columns, "locale", "status"].map((c) => r[c]).join("|")); +} + +export function listValuesKit(factory: ListValuesFactory): ListValuesKit { + let sql: SqlDriver; + let blobs: BlobStore; + let store: SiteGraphStore; + let tick = 0; + let n = 0; + const open = (driver: SqlDriver) => + new SiteGraphStore({ + sql: driver, + blobs, + now: () => new Date(Date.UTC(2026, 9, 26, 9, 0, tick++)), + gates: GATES_ENFORCED, + }); + + beforeEach(async () => { + ({ sql, blobs } = await factory.create()); + await dropAllTables(sql); + tick = 0; + store = open(sql); + await store.init(); + }); + + afterAll(async () => { + await sql?.close(); + await factory.dispose?.(); + }); + + const kit: ListValuesKit = { + get sql() { + return sql; + }, + get blobs() { + return blobs; + }, + get store() { + return store; + }, + open, + async setup(collections = { events: EVENTS, talks: TALKS }) { + n++; + const project = await store.createProject({ + name: "Lists", + slug: `lists-${n}`, + ownerId: person.id, + siteIds: [], + }); + return store.createSite({ + projectId: project.id, + name: "Lists", + slug: `lists-${n}`, + defaultLocale: "en", + locales: ["en", "tr", "de"], + domains: [], + pageIds: [], + theme: { name: "blog" }, + timeZone: "Europe/Istanbul", + collections, + }); + }, + entry: (site, slug, fields, extra = {}, on = store) => + on.createContentEntry({ + siteId: site.id, + collection: "events", + slug, + locale: "en", + status: "draft", + title: slug, + body: { format: "markdown", value: "Body" }, + fields, + tags: [], + author: person, + ...extra, + } as never), + async translate(site, original, over) { + const translation = { + group: original.id, + method: "human", + source: { + id: original.id, + locale: original.locale, + rev: original.rev, + hash: await kit.hashOf(original.id), + }, + } as Translation; + const extra = { locale: "tr", title: `${original.title}-tr`, translation, ...over }; + return kit.entry(site, original.slug, {}, extra); + }, + fresh: async (site) => (await store.getSite(site.id)) as Site, + get: async (id) => (await store.getContentEntry(id)) as ContentEntry, + hashOf: async (id) => + (await sql.get<{ hash: string }>("SELECT hash FROM nodes WHERE id = ?", [id])) + ?.hash as string, + state: (siteId, collection) => + sql.get( + "SELECT fingerprint, epoch, ready, cursor FROM entry_list_state WHERE site_id = ? AND collection = ?", + [siteId, collection], + ), + async expectExact(siteId, options = {}) { + const site = (await store.getSite(siteId)) as Site; + expect((await storedRows(sql, siteId)).sort()).toEqual((await wantedRows(kit, site)).sort()); + if (options.ready === false) return; + for (const c of listedCollections(site)) + expect(await kit.state(siteId, c), c).toMatchObject({ + fingerprint: await listFingerprint(site, c), + ready: 1, + }); + }, + }; + return kit; +} diff --git a/packages/core-ts/src/testing-list-values-query.ts b/packages/core-ts/src/testing-list-values-query.ts new file mode 100644 index 00000000..78f0a1be --- /dev/null +++ b/packages/core-ts/src/testing-list-values-query.ts @@ -0,0 +1,408 @@ +/** + * RFC-0042 §4, #110 property suite: `queryContentEntries` answered from `entry_list_values` is + * the in-memory evaluator's answer (`scanFieldQuery`, the oracle), on random sites, data and + * queries. Seeded (`mulberry32`), so a failure names its seed and query. Each seed builds a site + * in a random zone with two collections listing four random fields of every type (and a random + * sort), writes originals and translations (in their collection and another, localized values + * present or falling back), then alternates rounds of random writes (values, status, title, + * `publishedAt` ties, collection and locale moves, deletes, settings that make stored values + * unreadable) with random queries: up to four conditions of every operator, `now`, orderBy in + * both directions, status and locale prefilters, limits and offsets. The pushdown must answer + * (except for an operand SQL cannot compare exactly) and equal the oracle; so must + * `queryContentEntries`. Also: what makes it fall back. Runs on better-sqlite3, Miniflare D1 and + * Durable Object SQLite (`test/list-values-query.test.ts`). + */ +import type { + Actor, + CollectionSettings, + ContentEntry, + FieldCondition, + FieldDefinition, + FieldOrder, + Site, + Translation, +} from "@cloudhal/core-contracts"; +import type { BlobStore, SqlDriver, SqlValue } from "@cloudhal/storage"; +import { dropAllTables } from "@cloudhal/storage/testing"; +import { afterAll, beforeEach, describe, expect, it } from "vitest"; +import { GATES_ENFORCED } from "./availability.ts"; +import { type FieldEntryListQuery, queryContentEntries, scanFieldQuery } from "./entry-query.ts"; +import { parseFieldQuery } from "./fields/index.ts"; +import { rebuildListValues } from "./list-values/backfill.ts"; +import { listedCollections, listFingerprint } from "./list-values/fingerprint.ts"; +import { queryListValues } from "./list-values/query.ts"; +import { makeJpeg } from "./media/fixtures.ts"; +import { SiteGraphStore } from "./store.ts"; +import type { ListValuesFactory } from "./testing-list-values.ts"; +import { + type Gen, + mulberry32, + pick, + randomCollections, + randomFields, + randomQuery, + type World, +} from "./testing-list-values-random.ts"; + +const person: Actor = { kind: "human", id: "user_01HZX8K3Q9CD", name: "Halil" }; +/** 2026-11-05 23:00 in Istanbul, 2026-11-06 10:00 in Kiritimati. */ +const NOW = Date.parse("2026-11-05T20:00:00Z"); +const SEEDS = [11, 23, 37, 41]; +const ORIGINALS = 36; +const TRANSLATIONS = 14; +const ROUNDS = 3; +const MUTATIONS = 12; +const QUERIES = 50; +/** A seed takes ~5 s locally and over 30 s on a 4-vCPU CI runner with other packages testing. */ +const SEED_TIMEOUT = 180_000; +const STATUSES = ["draft", "review", "published", "archived"] as const; +const TITLES = ["Alpha", "alpha", "Été", "𝒜lpha", "x", "￿z", "Zeta", "Ａ", "b"]; +const PUBLISHED = ["2026-10-01T00:00:00.000Z", "2026-10-02T00:00:00.000Z", undefined]; + +/** A driver that runs `hook` once, before the first read `when` matches (to race a read). */ +function beforeRead( + sql: SqlDriver, + when: (q: string) => boolean, + hook: () => Promise, +): SqlDriver { + let fired = false; + return { + name: `${sql.name}+race`, + get: >(q: string, p?: readonly SqlValue[]) => + sql.get(q, p), + all: async >(q: string, p?: readonly SqlValue[]) => { + if (!fired && when(q)) { + fired = true; + await hook(); + } + return sql.all(q, p); + }, + run: (q: string, p?: readonly SqlValue[]) => sql.run(q, p), + batch: (s) => sql.batch(s), + exec: (q: string) => sql.exec(q), + close: async () => {}, + }; +} + +/** A page statement of the count-then-page shape (ordered and paged). */ +const isPage = (q: string) => q.includes("LIMIT ? OFFSET ?") && q.includes("entry_list_values"); +/** The bodies-by-id statement (unordered and paged). */ +const isBodies = (q: string) => q.includes("CROSS JOIN entry_list_values d ON d.node_id"); + +/** A driver that records the reads it runs. */ +function watching(sql: SqlDriver, reads: string[]): SqlDriver { + return { + name: `${sql.name}+watch`, + get: >(q: string, p?: readonly SqlValue[]) => { + reads.push(q); + return sql.get(q, p); + }, + all: >(q: string, p?: readonly SqlValue[]) => { + reads.push(q); + return sql.all(q, p); + }, + run: (q: string, p?: readonly SqlValue[]) => sql.run(q, p), + batch: (s) => sql.batch(s), + exec: (q: string) => sql.exec(q), + close: async () => {}, + }; +} + +export function listValuesQuerySuite(label: string, factory: ListValuesFactory): void { + describe(`RFC-0042 §4 list-value queries equal the evaluator: ${label}`, () => { + let sql: SqlDriver; + let blobs: BlobStore; + let store: SiteGraphStore; + let tick = 0; + let n = 0; + + beforeEach(async () => { + ({ sql, blobs } = await factory.create()); + await dropAllTables(sql); + tick = 0; + store = new SiteGraphStore({ + sql, + blobs, + now: () => new Date(Date.UTC(2026, 9, 26, 9, 0, 0, tick++ * 250)), + gates: GATES_ENFORCED, + }); + await store.init(); + }); + + afterAll(async () => { + await sql?.close(); + await factory.dispose?.(); + }); + + async function setup(collections: Record, zone: string) { + n++; + const project = await store.createProject({ + name: "Query", + slug: `query-${n}`, + ownerId: person.id, + siteIds: [], + }); + return store.createSite({ + projectId: project.id, + name: "Query", + slug: `query-${n}`, + defaultLocale: "en", + locales: ["en", "tr", "de"], + domains: [], + pageIds: [], + theme: { name: "blog" }, + timeZone: zone, + collections, + }); + } + + const fresh = async (siteId: string) => (await store.getSite(siteId)) as Site; + const hashOf = async (id: string) => + (await sql.get<{ hash: string }>("SELECT hash FROM nodes WHERE id = ?", [id]))?.hash; + + /** Every listed collection is ready under the current fingerprint (writes and settles kept it). */ + async function expectReady(siteId: string) { + const site = await fresh(siteId); + for (const c of listedCollections(site)) { + const state = await sql.get<{ fingerprint: string; ready: number }>( + "SELECT fingerprint, ready FROM entry_list_state WHERE site_id = ? AND collection = ?", + [siteId, c], + ); + expect(state, c).toEqual({ fingerprint: await listFingerprint(site, c), ready: 1 }); + } + } + + function draft(world: World, g: Gen, over: Partial) { + const collection = over.collection ?? pick(g, ["events", "talks"]); + const status = pick(g, STATUSES); + const publishedAt = pick(g, PUBLISHED); + return { + siteId: world.site.id, + collection, + slug: `e-${world.slugs++}`, + locale: "en", + status, + title: `${pick(g, TITLES)}${g() < 0.5 ? "" : " 2"}`, + body: { format: "markdown", value: "Body" }, + fields: randomFields(g, world, collection), + tags: [], + author: person, + ...(publishedAt ? { publishedAt } : {}), + ...over, + }; + } + + async function translate(world: World, g: Gen, original: ContentEntry) { + const translation = { + group: original.id, + method: "human", + source: { + id: original.id, + locale: original.locale, + rev: original.rev, + hash: (await hashOf(original.id)) as string, + }, + } as Translation; + const collection = g() < 0.7 ? original.collection : pick(g, ["events", "talks"]); + const fields = randomFields(g, world, collection, { absent: 0.5, translation: true }); + return store.createContentEntry({ + ...draft(world, g, { collection }), + slug: original.slug, + locale: pick(g, ["tr", "de"]), + fields, + translation, + } as never); + } + + async function populate(world: World, g: Gen) { + const originals: ContentEntry[] = []; + for (let i = 0; i < ORIGINALS; i++) + originals.push(await store.createContentEntry(draft(world, g, {}) as never)); + let made = 0; + for (let i = 0; i < TRANSLATIONS; i++) + // A locale its group already has is refused (RFC-0033's index): try another original. + made += await translate(world, g, pick(g, originals)).then( + () => 1, + () => 0, + ); + expect(made).toBeGreaterThan(TRANSLATIONS / 2); + } + + /** One random write; refused ones (a key or translation locale taken) change nothing. */ + async function mutate(world: World, g: Gen) { + const rows = await sql.all<{ body: string }>( + "SELECT body FROM nodes WHERE site_id = ? AND kind = 'contentEntry' ORDER BY id", + [world.site.id], + ); + const e = JSON.parse(pick(g, rows).body) as ContentEntry; + const r = g(); + if (r < 0.1) return store.deleteContentEntry(e); + const next: ContentEntry = { ...e }; + const translation = e.translation !== undefined && e.translation.group !== e.id; + if (r < 0.45) next.fields = randomFields(g, world, e.collection, { translation }); + else if (r < 0.55) next.status = pick(g, STATUSES); + else if (r < 0.65) next.title = pick(g, TITLES); + else if (r < 0.75) next.publishedAt = pick(g, PUBLISHED.slice(0, 2)) as string; + else if (r < 0.88) next.collection = e.collection === "events" ? "talks" : "events"; + else next.locale = pick(g, ["en", "tr", "de"]); + return store.updateContentEntry(next); + } + + /** Settings that make stored values unreadable, or list other fields (a rebuild). */ + async function reshape(world: World, g: Gen) { + const site = await fresh(world.site.id); + const collections = randomCollections(g, world, site.collections ?? {}); + world.site = await store.updateSite({ ...site, collections }); + } + + async function check(world: World, q: FieldEntryListQuery & { collection: string }) { + const what = JSON.stringify({ seed: world.seed, ...q }); + const site = await fresh(world.site.id); + const parsed = parseFieldQuery(site, q.collection, q); + if (!parsed.ok) throw new Error(`${what}: ${JSON.stringify(parsed.issues)}`); + const oracle = await scanFieldQuery(store, site, parsed.query, q); + const pushed = await queryListValues(sql, site, { ...q, query: parsed.query }); + const exact = (q.where ?? []).every((c) => c.value?.isWellFormed() ?? true); + if (exact) expect(pushed, what).toBeDefined(); + else expect(pushed, what).toBeUndefined(); + const ids = (r: { entries: ContentEntry[] }) => r.entries.map((e) => e.id); + if (pushed) { + expect([ids(pushed), pushed.total], what).toEqual([ids(oracle), oracle.total]); + expect(pushed.entries, what).toEqual(oracle.entries); + } + expect(await queryContentEntries(store, site.id, q), what).toEqual(oracle); + return oracle.total; + } + + for (const seed of SEEDS) + it( + `seed ${seed}: random data and queries`, + async () => { + const g = mulberry32(seed); + const zone = pick(g, ["UTC", "Europe/Istanbul", "Pacific/Kiritimati"]); + const proto: World = { seed, site: undefined as never, slugs: 0, image: "", now: NOW }; + const site = await setup(randomCollections(g, proto, {}), zone); + const { asset: image } = await store.ingestAsset({ + siteId: site.id, + bytes: makeJpeg({ width: 8, height: 8 }), + filename: "p.jpg", + uploadedBy: person, + source: { kind: "upload" }, + }); + const world: World = { ...proto, site, image: image.id }; + await populate(world, g); + let matched = 0; + for (let round = 0; round < ROUNDS; round++) { + for (let i = 0; i < MUTATIONS; i++) await mutate(world, g).catch(() => undefined); + if (round > 0) await reshape(world, g); + await expectReady(site.id); + world.site = await fresh(site.id); + const entries = await store.listContentEntries(site.id); + for (let i = 0; i < QUERIES; i++) + matched += await check(world, randomQuery(g, world, entries)); + } + expect(matched).toBeGreaterThan(0); + }, + SEED_TIMEOUT, + ); + + it("falls back while a collection is not ready, and when a read races a write", async () => { + const fields: [FieldDefinition, ...FieldDefinition[]] = [ + { key: "n", type: "number", labels: { en: { label: "N" } }, localized: false }, + ]; + const site = await setup({ events: { fields, listFields: ["n"] } }, "UTC"); + const g = mulberry32(5); + const world: World = { seed: 5, site, slugs: 0, image: "", now: NOW }; + const add = () => + store.createContentEntry(draft(world, g, { collection: "events", fields: {} }) as never); + for (let i = 0; i < 6; i++) await add(); + const where: FieldCondition[] = [{ field: "n", op: "unset" }]; + // The three statement shapes: count then page (the window reaches its segment's end), the + // matches' ids then the window's bodies, and one statement for an unpaged answer. + const paged = { where, orderBy: { field: "n", direction: "asc" }, limit: 10 } as const; + const byIds = { where, limit: 4 }; + const unpaged = { where }; + const shapes = [paged, byIds, unpaged]; + const run = (driver: SqlDriver, shape: Omit) => { + const q = { collection: "events", ...shape, now: NOW }; + const parsed = parseFieldQuery(site, "events", q); + if (!parsed.ok) throw new Error(JSON.stringify(parsed.issues)); + return queryListValues(driver, site, { ...q, query: parsed.query }); + }; + const answers = async (driver: SqlDriver) => { + const out = []; + for (const shape of shapes) out.push(await run(driver, shape)); + return out; + }; + for (const a of await answers(sql)) expect(a).toBeDefined(); + // A write between the count and the page, inside the window: the count no longer holds. + expect(await run(beforeRead(sql, isPage, add), paged)).toBeUndefined(); + // An entry of the window stops matching between the ids and the bodies. + const stops = async () => { + const [first] = await store.listContentEntries(site.id, "events"); + await store.updateContentEntry({ ...(first as ContentEntry), fields: { n: 1 } }); + }; + expect(await run(beforeRead(sql, isBodies, stops), byIds)).toBeUndefined(); + // Readiness lost between the statements, or before the only one. + const reset = () => + sql.run("UPDATE entry_list_state SET ready = 0 WHERE site_id = ?", [site.id]); + expect(await run(beforeRead(sql, isPage, reset), paged)).toBeUndefined(); + await rebuildListValues(sql, { siteId: site.id }); + expect(await run(beforeRead(sql, isBodies, reset), byIds)).toBeUndefined(); + for (const a of await answers(sql)) expect(a).toBeUndefined(); + await rebuildListValues(sql, { siteId: site.id }); + for (const a of await answers(sql)) expect(a).toBeDefined(); + await sql.run("UPDATE entry_list_state SET fingerprint = 'x' WHERE site_id = ?", [site.id]); + for (const a of await answers(sql)) expect(a).toBeUndefined(); + for (const shape of shapes) { + const q = { collection: "events", ...shape, now: NOW }; + const parsed = parseFieldQuery(site, "events", q); + if (!parsed.ok) throw new Error(JSON.stringify(parsed.issues)); + const oracle = await scanFieldQuery(store, site, parsed.query, q); + expect(oracle.total).toBeGreaterThan(0); + expect(await queryContentEntries(store, site.id, q)).toEqual(oracle); + } + }); + + it("a term filter, and an ill-formed text operand, run on the evaluator", async () => { + const where: FieldCondition[] = [{ field: "t", op: "eq", value: "\uD800x" }]; + const orderBy: FieldOrder = { field: "n", direction: "asc" }; + const fields: [FieldDefinition, ...FieldDefinition[]] = [ + { key: "t", type: "text", labels: { en: { label: "T" } }, localized: false }, + { key: "n", type: "number", labels: { en: { label: "N" } }, localized: false }, + ]; + const site = await setup({ events: { fields, listFields: ["t", "n"] } }, "UTC"); + const g = mulberry32(9); + const world: World = { seed: 9, site, slugs: 0, image: "", now: NOW }; + for (const t of ["\uD800x", "a", "\uD800x"]) + await store.createContentEntry( + draft(world, g, { collection: "events", fields: { t, n: world.slugs } }) as never, + ); + const q = { collection: "events", where, orderBy, now: NOW }; + const parsed = parseFieldQuery(site, "events", q); + if (!parsed.ok) throw new Error(JSON.stringify(parsed.issues)); + expect(await queryListValues(sql, site, { ...q, query: parsed.query })).toBeUndefined(); + const r = await queryContentEntries(store, site.id, q); + expect(r.total).toBe(2); + expect(r).toEqual(await scanFieldQuery(store, site, parsed.query, q)); + // A term filter: the store's scan compiles it; the table is not read. + const reads: string[] = []; + const watched = new SiteGraphStore({ + sql: watching(sql, reads), + blobs, + gates: GATES_ENFORCED, + }); + const filtered = { + ...q, + where: [{ field: "n", op: "gte", value: "1" }], + filter: "collection:events", + } as const; + const fp = parseFieldQuery(site, "events", filtered); + if (!fp.ok) throw new Error(JSON.stringify(fp.issues)); + const viaFilter = await queryContentEntries(watched, site.id, filtered); + expect(viaFilter.total).toBe(2); + expect(viaFilter).toEqual(await scanFieldQuery(store, site, fp.query, filtered)); + expect(reads.some((x) => x.includes("entry_list_values"))).toBe(false); + }); + }); +} diff --git a/packages/core-ts/src/testing-list-values-random.ts b/packages/core-ts/src/testing-list-values-random.ts new file mode 100644 index 00000000..c03ec6c9 --- /dev/null +++ b/packages/core-ts/src/testing-list-values-random.ts @@ -0,0 +1,309 @@ +/** + * RFC-0042 §4, #110: the random sites, values and queries of the list-value property suite + * (`testing-list-values-query.ts`). Seeded (`mulberry32`); values aim at the encodings' edges: + * -0, subnormal and extreme doubles, `0.1 + 0.2`, days around today in the site's zone, instants + * around `now` (with milliseconds and offsets), empty and whitespace strings, combining marks, + * astral and private-use characters, a lone surrogate, strings of 100 and 101 units, duplicates + * (so `eq` and ties hit), and empty multiple selects. + */ +import type { + CollectionSettings, + ContentEntry, + FieldCondition, + FieldDefinition, + FieldOption, + FieldOrder, + Site, +} from "@cloudhal/core-contracts"; +import type { FieldEntryListQuery } from "./entry-query.ts"; +import { dayIn } from "./fields/conversions.ts"; +import { canOrderBy, FIELD_QUERY_OPS } from "./fields/index.ts"; + +export type Gen = () => number; + +/** The suite's state for one seed. */ +export interface World { + readonly seed: number; + site: Site; + slugs: number; + readonly image: string; + readonly now: number; +} + +/** A seeded uniform generator in [0, 1) (as `packages/schedules/test/oracle.test.ts`). */ +export function mulberry32(seed: number): Gen { + let a = seed >>> 0; + return () => { + a = (a + 0x6d2b79f5) >>> 0; + let t = a; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +export const pick = (g: Gen, xs: readonly T[]): T => xs[Math.floor(g() * xs.length)] as T; + +function shuffle(g: Gen, xs: readonly T[]): T[] { + const out = [...xs]; + for (let i = out.length - 1; i > 0; i--) { + const j = Math.floor(g() * (i + 1)); + [out[i], out[j]] = [out[j] as T, out[i] as T]; + } + return out; +} + +const L = (label: string) => ({ en: { label } }); +const OPTIONS: [FieldOption, ...FieldOption[]] = [ + { value: "a", labels: { en: "A" } }, + { value: "b", labels: { en: "B" } }, + { value: "c", labels: { en: "C" } }, + { value: "d", labels: { en: "D" } }, +]; +const NUMBERS = [ + 0, + -0, + 1, + -1, + 2.5, + 0.1 + 0.2, + 0.3, + 1e-300, + 5e-324, + 1.7976931348623157e308, + -1.7976931348623157e308, + 10, + 10, + 10, + 42, + 2 ** 53, + 1e20, + 7, + 7, +]; +const TEXTS = [ + "", + " ", + "a", + "A", + "é", + "é", + "😀", + "𝒜", + "Ａ", + "Hall", + "hall", + "Hall", + "", + "￿", + "\uD800x", + "x".repeat(100), + "x".repeat(101), +]; +const URLS = [ + "https://a.example/x", + "https://a.example/X", + "/path", + "#frag", + "mailto:a@b.example", + `https://a.example/${"y".repeat(100)}`, +]; +const RICH = ["", " ", "**bold**", "Text"]; +const ZONES = ["Europe/Istanbul", "UTC", "Asia/Tokyo"]; +const DECIMAL = /^-?(?:\d+(?:\.\d+)?|\.\d+)$/; +const HOUR = 3_600_000; + +/** Every type once (two selects), each localized or not at random. */ +function pool(g: Gen): FieldDefinition[] { + const loc = () => g() < 0.5; + return [ + { key: "n", type: "number", labels: L("N"), localized: loc() }, + { key: "d", type: "date", labels: L("D"), localized: loc() }, + { key: "dt", type: "datetime", labels: L("Dt"), localized: loc() }, + { key: "b", type: "boolean", labels: L("B"), localized: loc() }, + { key: "s", type: "select", labels: L("S"), localized: loc(), options: OPTIONS.slice(0, 3) }, + { + key: "m", + type: "select", + labels: L("M"), + localized: loc(), + options: OPTIONS, + multiple: true, + }, + { key: "t", type: "text", labels: L("T"), localized: loc() }, + { key: "u", type: "url", labels: L("U"), localized: loc() }, + { key: "r", type: "richText", labels: L("R"), localized: loc() }, + { key: "i", type: "image", labels: L("I"), localized: loc() }, + ] as FieldDefinition[]; +} + +/** A later settings change: bounds that make stored values unreadable, or lift them again. */ +function reshapeDef(g: Gen, def: FieldDefinition): FieldDefinition { + if (g() < 0.5) return def; + if (def.type === "number") { + const { maximum, ...rest } = def; + return maximum === undefined ? { ...def, maximum: 10 } : rest; + } + if (def.type === "text") { + const { maxLength, ...rest } = def; + return maxLength === undefined ? { ...def, maxLength: 5 } : rest; + } + return def; +} + +/** `events` and `talks`: four random list fields and a random sort (new, or reshaped). */ +export function randomCollections( + g: Gen, + _world: World, + existing: Readonly>, +): Record { + const out: Record = {}; + for (const c of ["events", "talks"]) { + const prev = existing[c]; + const fields = prev?.fields ? prev.fields.map((d) => reshapeDef(g, d)) : pool(g); + const keep = prev?.listFields && g() < 0.5; + const listFields = keep + ? prev.listFields + : shuffle( + g, + fields.map((f) => f.key), + ).slice(0, 4); + const sort = pick(g, ["newest", "oldest", "title", undefined] as const); + out[c] = { fields, listFields, ...(sort ? { sort } : {}) } as CollectionSettings; + } + return out; +} + +const addDays = (day: string, n: number) => + new Date(Date.parse(`${day}T00:00:00Z`) + n * 24 * HOUR).toISOString().slice(0, 10); + +const today = (world: World) => + dayIn(new Date(world.now).toISOString(), world.site.timeZone ?? "UTC"); + +function randomValue(g: Gen, world: World, def: FieldDefinition): unknown { + switch (def.type) { + case "number": + return pick( + g, + NUMBERS.filter((x) => def.maximum === undefined || x <= def.maximum), + ); + case "date": + return g() < 0.15 ? "2026-01-01" : addDays(today(world), Math.floor(g() * 5) - 2); + case "datetime": { + const ms = world.now + (Math.floor(g() * 9) - 4) * (HOUR / 2) + (g() < 0.2 ? 1 : 0); + return { at: new Date(ms).toISOString(), timeZone: pick(g, ZONES) }; + } + case "boolean": + return g() < 0.5; + case "select": + if (!def.multiple) return pick(g, def.options).value; + return def.options.filter(() => g() < 0.4).map((o) => o.value); + case "text": + return pick( + g, + TEXTS.filter((t) => def.maxLength === undefined || t.length <= def.maxLength), + ); + case "url": + return pick(g, URLS); + case "richText": + return pick(g, RICH); + default: + return world.image; + } +} + +/** + * Random values for a collection's fields, each absent with probability `absent`; a translation + * holds only localized ones (the others are its original's). + */ +export function randomFields( + g: Gen, + world: World, + collection: string, + options: { absent?: number; translation?: boolean } = {}, +): Record { + const out: Record = {}; + for (const def of world.site.collections?.[collection]?.fields ?? []) { + if (g() < (options.absent ?? 0.25) || (def.type === "image" && !world.image)) continue; + if (options.translation && !def.localized) continue; + out[def.key] = randomValue(g, world, def); + } + return out; +} + +/** Strings a condition on `def` may compare with: stored values (so `eq` hits) and edges. */ +function operands(g: Gen, world: World, def: FieldDefinition, seen: unknown[]): string[] { + const strings = seen.filter((v): v is string => typeof v === "string"); + switch (def.type) { + case "number": + return [ + ...seen.filter((v) => typeof v === "number").map(String), + ...["0", "-0", "1", "-1", "2.5", "0.3", "0.30000000000000004", "7", "10", "42"], + ...["9007199254740992", "100000000000000000000"], + ].filter((s) => DECIMAL.test(s)); + case "date": + return [...strings, "now", addDays(today(world), 1), addDays(today(world), -1)]; + case "datetime": { + const ats = seen.map((v) => (v as { at?: string }).at).filter((s) => s !== undefined); + const shifted = ats.map((at) => + new Date(Date.parse(at) + 3 * HOUR).toISOString().replace("Z", "+03:00"), + ); + return [...ats, ...shifted, "now"]; + } + case "boolean": + return ["true", "false"]; + case "select": + return def.options.map((o) => o.value); + default: + return [...strings, "a", "Hall", "x".repeat(100), "\uD800x", pick(g, URLS)].filter( + (s) => s.length >= 1 && s.length <= 100, + ); + } +} + +function condition(g: Gen, world: World, def: FieldDefinition, seen: unknown[]): FieldCondition { + const op = pick(g, FIELD_QUERY_OPS[def.type]); + if (op === "set" || op === "unset") return { field: def.key, op }; + const choices = operands(g, world, def, seen); + if (op === "in") { + const values = [...new Set(shuffle(g, choices).slice(0, 1 + Math.floor(g() * 3)))]; + return { field: def.key, op, values } as FieldCondition; + } + return { field: def.key, op, value: pick(g, choices) } as FieldCondition; +} + +const PAGE = [1, 2, 3, 5, 10, 20, 100]; + +/** A random valid query on one of the collections (`entries`: the site's, for operands). */ +export function randomQuery( + g: Gen, + world: World, + entries: readonly ContentEntry[], +): FieldEntryListQuery & { collection: string } { + const collection = pick(g, ["events", "talks"]); + const settings = world.site.collections?.[collection]; + const defs = new Map((settings?.fields ?? []).map((d) => [d.key, d])); + const keys = (settings?.listFields ?? []).filter((k) => defs.has(k)); + const seenOf = (k: string) => entries.map((e) => e.fields?.[k]).filter((v) => v !== undefined); + const count = pick(g, [0, 1, 1, 1, 2, 2, 3, 4]); + const where = Array.from({ length: count }, () => { + const def = defs.get(pick(g, keys)) as FieldDefinition; + return condition(g, world, def, seenOf(def.key)); + }); + const orderable = keys.filter((k) => canOrderBy((defs.get(k) as FieldDefinition).type)); + const orderBy: FieldOrder | undefined = + orderable.length > 0 && g() < 0.6 + ? { field: pick(g, orderable), direction: pick(g, ["asc", "desc"] as const) } + : undefined; + const status = g() < 0.25 ? (["draft", "review", "published", "archived"] as const) : undefined; + return { + collection, + now: world.now, + ...(count > 0 || !orderBy || g() < 0.5 ? { where } : {}), + ...(orderBy ? { orderBy } : {}), + ...(status ? { status: status.filter(() => g() < 0.6) } : {}), + ...(g() < 0.25 ? { locale: pick(g, ["en", "tr", "de"]) } : {}), + ...(g() < 0.6 ? { limit: pick(g, PAGE) } : {}), + ...(g() < 0.4 ? { offset: Math.floor(g() * (entries.length / 2 + 5)) } : {}), + }; +} diff --git a/packages/core-ts/src/testing-list-values-repair.ts b/packages/core-ts/src/testing-list-values-repair.ts new file mode 100644 index 00000000..eaa8ae18 --- /dev/null +++ b/packages/core-ts/src/testing-list-values-repair.ts @@ -0,0 +1,249 @@ +/** + * RFC-0042 §4, #110 behavioural suite, the review's repairs: the admin job's tidy never removes + * rows of a collection listed while it ran; restores that bring back a whole earlier body (a + * field operation's revert, `revertTagsToTerms`) never leave rows of the newer one behind; and + * the backfill keeps to its statement budget, discovery included. Each ends with `expectExact` + * and, where a list is involved, the pushdown against the in-memory oracle. Runs on + * better-sqlite3, Miniflare D1 and Durable Object SQLite (`test/list-values.test.ts`). + */ +import type { FieldDefinition, Site } from "@cloudhal/core-contracts"; +import type { SqlDriver } from "@cloudhal/storage"; +import { describe, expect, it } from "vitest"; +import { type FieldEntryListQuery, scanFieldQuery } from "./entry-query.ts"; +import { parseFieldQuery } from "./fields/index.ts"; +import { + continueFieldOperation, + revertFieldOperation, + runFieldOperation, +} from "./fields-operations/index.ts"; +import { rebuildListValues } from "./list-values/backfill.ts"; +import { queryListValues } from "./list-values/query.ts"; +import type { SiteGraphStore } from "./store.ts"; +import { + EVENTS, + FIELDS, + type ListValuesFactory, + listValuesKit, + person, + TALKS, +} from "./testing-list-values-kit.ts"; +import { racingDriver } from "./testing-taxonomy.ts"; + +const NOW = Date.parse("2026-11-05T20:00:00Z"); +const number = (key: string): FieldDefinition => ({ + key, + type: "number", + labels: { en: { label: key } }, + localized: false, +}); +const NUMBERS: [FieldDefinition, ...FieldDefinition[]] = [ + number("a"), + number("b"), + number("c"), + number("d"), +]; + +/** `sql`, counting every statement it runs (a batch counts each), as a tick meter does. */ +function counting(sql: SqlDriver): { sql: SqlDriver; count: () => number; reset: () => void } { + let n = 0; + const driver: SqlDriver = { + name: `${sql.name}+count`, + get: async (q, p) => { + n++; + return sql.get(q, p); + }, + all: async (q, p) => { + n++; + return sql.all(q, p); + }, + run: async (q, p) => { + n++; + return sql.run(q, p); + }, + batch: async (s) => { + n += s.length; + return sql.batch(s); + }, + exec: (q) => sql.exec(q), + close: async () => {}, + }; + return { + sql: driver, + count: () => n, + reset: () => { + n = 0; + }, + }; +} + +export function listValuesRepairSuite(label: string, factory: ListValuesFactory): void { + describe(`RFC-0042 §4 list values, repairs: ${label}`, () => { + const k = listValuesKit(factory); + const { setup, entry, fresh, get, state, expectExact } = k; + + /** The pushdown answers `q` and equals the evaluator over the store's scan. */ + async function expectOracle(site: Site, q: FieldEntryListQuery & { collection: string }) { + const now = await fresh(site); + const parsed = parseFieldQuery(now, q.collection, q); + if (!parsed.ok) throw new Error(JSON.stringify(parsed.issues)); + const pushed = await queryListValues(k.sql, now, { ...q, query: parsed.query }); + expect(pushed, JSON.stringify(q)).toBeDefined(); + expect(pushed).toEqual(await scanFieldQuery(k.store, now, parsed.query, q)); + } + + const rowsOf = async (siteId: string, collection: string) => + ( + await k.sql.all("SELECT 1 FROM entry_list_values WHERE site_id = ? AND collection = ?", [ + siteId, + collection, + ]) + ).length; + + it("tidy keeps the rows of a collection listed and rebuilt while the job ran", async () => { + const site = await setup({ events: EVENTS, posts: { fields: FIELDS }, gone: TALKS }); + for (let i = 0; i < 3; i++) await entry(site, `p${i}`, { price: i }, { collection: "posts" }); + for (let i = 0; i < 2; i++) await entry(site, `g${i}`, { price: i }, { collection: "gone" }); + const before = await fresh(site); + await k.store.updateSite({ + ...before, + collections: { ...before.collections, gone: { fields: FIELDS } }, + }); + expect(await rowsOf(site.id, "gone")).toBe(6); + // The job read the site with `posts` unlisted; a Studio save lists it (and settles it) + // just before the tidy statement runs. + const racing = racingDriver( + k.sql, + async () => { + const s = await fresh(site); + await k.store.updateSite({ + ...s, + collections: { ...s.collections, posts: { fields: FIELDS, listFields: ["price"] } }, + }); + }, + (q) => /DELETE FROM entry_list_values WHERE rowid IN/.test(q), + ); + await rebuildListValues(racing, { tidy: true }); + expect(await state(site.id, "posts")).toMatchObject({ ready: 1 }); + expect(await rowsOf(site.id, "posts")).toBe(3); + const again = await rebuildListValues(k.sql, { tidy: true }); + expect(again.tidied).toBe(6); + await expectExact(site.id); + await expectOracle(site, { + collection: "posts", + where: [{ field: "price", op: "gte", value: "1" }], + now: NOW, + }); + }); + + it("a field operation's revert that restores an edited listed value rebuilds its rows", async () => { + const site = await setup(); + for (let i = 0; i < 3; i++) await entry(site, `e${i}`, { price: i, day: "2026-11-05" }); + // An operation on an unlisted field, cut short by its budget. + const half = await runFieldOperation( + k.store, + site.id, + { op: "rename_field", collection: "events", from: "day", to: "date" }, + { actor: person, budget: { entries: 1, ms: 60_000 } }, + ); + expect(half.done).toBe(false); + // Meanwhile an editor changes a listed value of an entry the operation has not reached. + const last = (await k.store.listContentEntries(site.id, "events")).find( + (e) => e.slug === "e2", + ); + const edited = await get(last?.id as string); + await k.store.updateContentEntry({ ...edited, fields: { ...edited.fields, price: 20 } }); + const done = await continueFieldOperation(k.store, site.id, half.operationId, { + actor: person, + }); + expect(done.done).toBe(true); + await expectExact(site.id); + // The revert puts the earlier body back, with its earlier `price`. + await revertFieldOperation(k.store, site.id, done.operationId, { actor: person }); + expect((await get(edited.id)).fields?.price).toBe(2); + await expectExact(site.id); + for (const value of ["2", "20"]) + await expectOracle(site, { + collection: "events", + where: [{ field: "price", op: "eq", value }], + now: NOW, + }); + }); + + it("a tags revert leaves an entry edited during the migration as edited", async () => { + const site = await setup(); + await entry(site, "a", { price: 1 }, { tags: ["x"] }); + const b = await entry(site, "b", { price: 2 }, { tags: ["y"] }); + // The edit lands after the before-snapshot, before the migration's entry batch. + const racing: SiteGraphStore = k.open( + racingDriver( + k.sql, + async () => { + await k.store.updateContentEntry({ ...(await get(b.id)), fields: { price: 9 } }); + }, + (q) => /^UPDATE nodes/.test(q), + ), + ); + const run = await racing.migrateTagsToTerms(site.id, person); + expect(run.skipped.map((s) => s.entryId)).toEqual([b.id]); + const rv = await k.store.revertTagsToTerms(site.id, run.migrationId, { actor: person }); + expect(rv).toMatchObject({ restored: 1, edited: [b.id] }); + expect((await get(b.id)).fields?.price).toBe(9); + await expectExact(site.id); + await expectOracle(site, { + collection: "events", + where: [{ field: "price", op: "gte", value: "2" }], + now: NOW, + }); + }); + + it("the backfill keeps to its statement budget; discovery is two statements", async () => { + const site = await setup({ + events: EVENTS, + talks: TALKS, + nums: { + fields: NUMBERS, + listFields: ["a", "b", "c", "d"], + }, + }); + const other = await setup({ events: EVENTS, talks: TALKS, more: TALKS }); + // Four decimals an entry: 50 entries need three INSERTs. + for (let i = 0; i < 50; i++) { + const x = i + 0.5; + await entry(site, `n${i}`, { a: x, b: x / 3, c: -x, d: x * 1.1 }, { collection: "nums" }); + } + await entry(other, "o", { price: 1.5 }); + const counted = counting(k.sql); + const idle = await rebuildListValues(counted.sql, { + budget: { entries: 200, ms: 60_000, statements: 7 }, + }); + expect(counted.count()).toBe(2); + expect(idle.collections.every((c) => c.ready)).toBe(true); + expect(idle.collections).toHaveLength(6); + + await k.sql.run("DELETE FROM entry_list_state"); + await k.sql.run("DELETE FROM entry_list_values"); + const written: number[] = []; + for (let run = 0; run < 30; run++) { + counted.reset(); + const r = await rebuildListValues(counted.sql, { + budget: { entries: 200, ms: 60_000, statements: 7 }, + start: run, + }); + expect(counted.count(), `run ${run}`).toBeLessThanOrEqual(7); + written.push(...r.collections.filter((c) => c.collection === "nums").map((c) => c.written)); + if (!r.exhausted && r.collections.every((c) => c.ready)) break; + } + // A chunk of 50 needs five statements with its advance; four were left: it was cut at 40. + expect(written).toContain(40); + await expectExact(site.id); + await expectExact(other.id); + await expectOracle(site, { + collection: "nums", + where: [{ field: "a", op: "gte", value: "10" }], + orderBy: { field: "b", direction: "desc" }, + limit: 5, + now: NOW, + }); + }); + }); +} diff --git a/packages/core-ts/src/testing-list-values.ts b/packages/core-ts/src/testing-list-values.ts new file mode 100644 index 00000000..61dbb268 --- /dev/null +++ b/packages/core-ts/src/testing-list-values.ts @@ -0,0 +1,398 @@ +/** + * RFC-0042 §4, #110 behavioural suite: `entry_list_values` stays exact through every write path + * (entry create, update, moves and delete; translations and their originals; site settings; + * field operations; import; site delete), races degrade readiness instead of leaving stale rows, + * and the backfill resumes, serializes and starts over as its state row says. `expectExact` + * (`testing-list-values-kit.ts`) recomputes every entry's rows from the nodes and compares them + * with the table. Runs on better-sqlite3, Miniflare D1 and Durable Object SQLite + * (`test/list-values.test.ts`); the repairs of #110's review are `testing-list-values-repair.ts`. + */ +import { describe, expect, it } from "vitest"; +import { + continueFieldOperation, + revertFieldOperation, + runFieldOperation, +} from "./fields-operations/index.ts"; +import { rebuildListValues } from "./list-values/backfill.ts"; +import { deleteSiteData } from "./site-delete.ts"; +import { exportSite, importSite } from "./sync.ts"; +import { + at, + EVENTS, + FIELDS, + type ListValuesFactory, + listValuesKit, + person, + recording, + TALKS, +} from "./testing-list-values-kit.ts"; +import { racingDriver } from "./testing-taxonomy.ts"; + +export type { ListValuesFactory } from "./testing-list-values-kit.ts"; + +export function listValuesSuite(label: string, factory: ListValuesFactory): void { + describe(`RFC-0042 §4 list values: ${label}`, () => { + const k = listValuesKit(factory); + const { open, setup, entry, translate, fresh, get, hashOf, state, expectExact } = k; + + it("a new site's listed collections are ready, and entry writes keep their rows", async () => { + const site = await setup({ events: EVENTS, talks: TALKS, posts: { fields: FIELDS } }); + expect(await state(site.id, "events")).toMatchObject({ ready: 1, cursor: null }); + expect(await state(site.id, "posts")).toBeUndefined(); + const a = await entry(site, "a", { + startsAt: at("2026-11-05T16:30:00Z"), + price: 10, + kind: ["talk", "social"], + venue: "Hall", + }); + const b = await entry( + site, + "b", + { price: 2.5, venue: "x".repeat(101) }, + { status: "published" }, + ); + await expectExact(site.id); + const a2 = await k.store.updateContentEntry({ + ...a, + fields: { ...a.fields, price: 11, kind: [] }, + status: "published", + }); + await k.store.updateContentEntry({ + ...b, + title: "B!", + publishedAt: "2026-01-01T00:00:00.000Z", + }); + await expectExact(site.id); + const moved = await k.store.updateContentEntry({ + ...a2, + collection: "talks", + fields: { price: 4, free: true, day: "2026-11-05" }, + }); + await expectExact(site.id); + const unlisted = await k.store.updateContentEntry({ ...moved, collection: "posts" }); + await expectExact(site.id); + await k.store.updateContentEntry({ ...unlisted, collection: "events", locale: "de" }); + await expectExact(site.id); + await k.store.deleteContentEntry(await get(b.id)); + await expectExact(site.id); + expect( + (await k.sql.all("SELECT 1 FROM entry_list_values WHERE node_id = ?", [b.id])).length, + ).toBe(0); + }); + + it("an original's values reach its translations' rows, in its collection and another", async () => { + const site = await setup(); + const o = await entry(site, "o", { price: 10, free: true, venue: "Hall", day: "2026-11-05" }); + const tr = await translate(site, o, { fields: { venue: "Salon" } }); + const de = await translate(site, o, { locale: "de", collection: "talks", title: "Zeta" }); + await expectExact(site.id); + const rowOf = async (id: string, field: string) => + k.sql.get<{ num: number | null }>( + "SELECT num FROM entry_list_values WHERE node_id = ? AND field = ? AND has < 2", + [id, field], + ); + expect(await rowOf(tr.id, "price")).toEqual({ num: 10 }); + const o2 = await k.store.updateContentEntry({ + ...(await get(o.id)), + fields: { ...o.fields, price: 12 }, + }); + expect(await rowOf(tr.id, "price")).toEqual({ num: 12 }); + expect(await rowOf(de.id, "price")).toEqual({ num: 12 }); + await k.store.updateContentEntry({ ...(await get(de.id)), fields: { day: "2027-01-01" } }); + await k.store.updateContentEntry({ ...(await get(tr.id)), fields: { venue: "Sahne" } }); + await expectExact(site.id); + await k.store.deleteContentEntry(await get(o2.id)); + expect(await rowOf(tr.id, "price")).toEqual({ num: null }); + await expectExact(site.id); + }); + + it("settings changes reset readiness and the inline backfill makes it exact", async () => { + const site = await setup({ events: { fields: FIELDS }, talks: TALKS, gone: TALKS }); + for (let i = 0; i < 5; i++) + await entry( + site, + `e${i}`, + { price: i, kind: ["workshop"], venue: `v${i}` }, + { title: `t${4 - i}` }, + ); + await expectExact(site.id); + const listed = await k.store.updateSite({ + ...(await fresh(site)), + collections: { events: EVENTS, talks: TALKS, gone: TALKS }, + }); + await expectExact(site.id); + const before = await state(site.id, "events"); + await k.store.updateSite({ + ...listed, + collections: { + ...listed.collections, + events: { ...EVENTS, listFields: ["venue", "kind", "price", "startsAt"] }, + }, + }); + expect(await state(site.id, "events")).toEqual(before); + const sorted = await k.store.updateSite({ + ...(await fresh(site)), + collections: { ...(await fresh(site)).collections, events: { ...EVENTS, sort: "title" } }, + }); + expect((await state(site.id, "events"))?.epoch).not.toBe(before?.epoch); + await expectExact(site.id); + const { gone: _gone, ...rest } = sorted.collections ?? {}; + const unlisted = { ...rest, events: { fields: FIELDS } }; + await k.store.updateSite({ ...sorted, collections: unlisted }); + expect(await state(site.id, "events")).toBeUndefined(); + expect(await state(site.id, "gone")).toBeUndefined(); + expect(await state(site.id, "talks")).toMatchObject({ ready: 1 }); + }); + + it("the backfill resumes within its budget, from no state row (a site from before 0029)", async () => { + const site = await setup(); + for (let i = 0; i < 23; i++) + await entry(site, `e${String(i).padStart(2, "0")}`, { price: i + 0.25, kind: ["talk"] }); + await k.sql.run("DELETE FROM entry_list_state WHERE site_id = ?", [site.id]); + await k.sql.run("UPDATE entry_list_values SET num = 99 WHERE site_id = ?", [site.id]); + let calls = 0; + for (;;) { + calls++; + const r = await rebuildListValues(k.sql, { + siteId: site.id, + budget: { entries: 7, ms: 60_000 }, + }); + if (!r.exhausted) break; + expect((await state(site.id, "events"))?.ready).toBe(0); + } + expect(calls).toBeGreaterThanOrEqual(4); + await expectExact(site.id); + const again = await rebuildListValues(k.sql, { siteId: site.id }); + expect(again.collections.every((c) => c.ready && c.written === 0)).toBe(true); + }); + + it("binds decimals exactly, over a chunk's parameter limit", async () => { + const site = await setup({ events: EVENTS }); + const odd = [1e-300, 5e-324, 0.1 + 0.2, 1.7976931348623157e308, -1.7976931348623157e308, -0]; + for (const [i, price] of odd.entries()) await entry(site, `odd-${i}`, { price }); + for (let i = 0; i < 90; i++) + await entry(site, `d-${String(i).padStart(2, "0")}`, { price: i / 7 }); + await expectExact(site.id); + const back = await k.sql.all<{ num: number }>( + "SELECT num FROM entry_list_values e JOIN nodes n ON n.id = e.node_id WHERE e.field = 'price' AND n.key LIKE 'events:en:odd-%' ORDER BY n.key", + ); + expect(back.map((r) => r.num)).toEqual(odd.map((x) => (x === 0 ? 0 : x))); + await k.sql.run("DELETE FROM entry_list_state WHERE site_id = ?", [site.id]); + await rebuildListValues(k.sql, { siteId: site.id }); + await expectExact(site.id); + }); + + it("a field operation on a listed field resets the collection; done, it is ready again", async () => { + const site = await setup(); + const o = await entry(site, "o", { price: 1, venue: "Hall" }); + await translate(site, o, { fields: { venue: "Salon" } }); + // In another collection, reading the original's `price` under its own definition. + const de = await translate(site, o, { locale: "de", collection: "talks", title: "Zeta" }); + for (let i = 0; i < 3; i++) await entry(site, `e${i}`, { price: i, venue: `v${i}` }); + const quiet = await state(site.id, "events"); + const talks = await state(site.id, "talks"); + const price = async () => + k.sql.get<{ has: number }>( + "SELECT has FROM entry_list_values WHERE node_id = ? AND field = 'price'", + [de.id], + ); + expect(await price()).toEqual({ has: 1 }); + await runFieldOperation( + k.store, + site.id, + { op: "rename_field", collection: "events", from: "notes", to: "remarks" }, + { actor: person }, + ); + expect(await state(site.id, "events")).toEqual(quiet); + expect(await state(site.id, "talks")).toEqual(talks); + const half = await runFieldOperation( + k.store, + site.id, + { op: "rename_field", collection: "events", from: "price", to: "cost" }, + { actor: person, budget: { entries: 1, ms: 60_000 } }, + ); + expect(half.done).toBe(false); + expect((await state(site.id, "events"))?.ready).toBe(0); + const skipped = await rebuildListValues(k.sql, { siteId: site.id }); + expect(skipped.collections.find((c) => c.collection === "events")?.reason).toBe( + "operation_running", + ); + const done = await continueFieldOperation(k.store, site.id, half.operationId, { + actor: person, + }); + expect(done.done).toBe(true); + expect((await fresh(site)).collections?.events?.listFields).toContain("cost"); + // Phase B moved the original's `price` away: the pass reset `talks`, the settle rebuilt it. + expect((await state(site.id, "talks"))?.epoch).not.toBe(talks?.epoch); + expect(await price()).toEqual({ has: 0 }); + await expectExact(site.id); + await revertFieldOperation(k.store, site.id, done.operationId, { actor: person }); + expect(await price()).toEqual({ has: 1 }); + await expectExact(site.id); + await runFieldOperation( + k.store, + site.id, + { op: "set_field_localized", collection: "events", key: "venue", localized: false }, + { actor: person }, + ); + await expectExact(site.id); + }); + + it("a translation's write racing its original's degrades, and the backfill repairs", async () => { + const site = await setup(); + const o = await entry(site, "o", { price: 1, venue: "Hall" }); + const tr = await translate(site, o, { fields: { venue: "Salon" } }); + const original = open( + racingDriver(k.sql, async () => { + await k.store.updateContentEntry({ ...(await get(tr.id)), title: "moved first" }); + }), + ); + await original.updateContentEntry({ + ...(await get(o.id)), + fields: { price: 2, venue: "Hall" }, + }); + expect((await state(site.id, "events"))?.ready).toBe(0); + await rebuildListValues(k.sql, { siteId: site.id }); + await expectExact(site.id); + const translation = open( + racingDriver(k.sql, async () => { + await k.store.updateContentEntry({ + ...(await get(o.id)), + fields: { price: 3, venue: "Hall" }, + }); + }), + ); + await translation.updateContentEntry({ ...(await get(tr.id)), fields: { venue: "Sahne" } }); + expect((await state(site.id, "events"))?.ready).toBe(0); + await rebuildListValues(k.sql, { siteId: site.id }); + await expectExact(site.id); + }); + + it("a write planned before listFields were added degrades the collection", async () => { + const site = await setup({ events: { fields: FIELDS }, talks: TALKS }); + const late = open( + racingDriver(k.sql, async () => { + await k.store.updateSite({ + ...(await fresh(site)), + collections: { events: EVENTS, talks: TALKS }, + }); + }), + ); + await entry(site, "late", { price: 5 }, {}, late); + expect((await state(site.id, "events"))?.ready).toBe(0); + await rebuildListValues(k.sql, { siteId: site.id }); + await expectExact(site.id); + }); + + it("source-forward on an original's write keeps the collection ready", async () => { + const site = await setup(); + const o = await entry(site, "o", { price: 1, venue: "Hall" }); + const tr = await translate(site, o, { fields: { venue: "Salon" } }); + const before = await state(site.id, "events"); + await k.store.updateContentEntry({ + ...(await get(o.id)), + fields: { price: 7, venue: "Hall" }, + }); + expect((await get(tr.id)).translation?.source?.hash).toBe(await hashOf(o.id)); + expect(await state(site.id, "events")).toMatchObject({ epoch: before?.epoch, ready: 1 }); + await expectExact(site.id); + }); + + describe("the backfill against concurrent work", () => { + const listWrite = (q: string) => /^\s*DELETE FROM entry_list_values WHERE node_id IN/.test(q); + + async function stale(count: number) { + const site = await setup(); + const ids: string[] = []; + for (let i = 0; i < count; i++) + ids.push((await entry(site, `e${String(i).padStart(3, "0")}`, { price: i })).id); + await k.sql.run("DELETE FROM entry_list_state WHERE site_id = ?", [site.id]); + return { site, ids }; + } + + it("rereads a chunk whose entry a writer changed", async () => { + const { site, ids } = await stale(5); + const racing = racingDriver( + k.sql, + async () => { + await k.store.updateContentEntry({ + ...(await get(ids[1] as string)), + fields: { price: 50 }, + }); + }, + listWrite, + ); + const r = await rebuildListValues(racing, { siteId: site.id }); + expect(r.collections.find((c) => c.collection === "events")).toMatchObject({ + ready: true, + written: 5, + }); + await expectExact(site.id); + }); + + it("lets one of two runners write", async () => { + const { site } = await stale(4); + const racing = racingDriver( + k.sql, + async () => { + await rebuildListValues(k.sql, { siteId: site.id }); + }, + listWrite, + ); + const r = await rebuildListValues(racing, { siteId: site.id }); + expect(r.collections.find((c) => c.collection === "events")).toMatchObject({ + ready: true, + written: 0, + }); + await expectExact(site.id); + }); + + it("starts over when the definitions change mid-run", async () => { + const { site } = await stale(4); + const racing = racingDriver( + k.sql, + async () => { + const s = await fresh(site); + await k.store.updateSite({ + ...s, + collections: { ...s.collections, events: { ...EVENTS, sort: "oldest" } }, + }); + }, + listWrite, + ); + const r = await rebuildListValues(racing, { siteId: site.id }); + expect(r.collections.find((c) => c.collection === "events")?.reason).toBe("changed"); + // The site write's own settle rebuilt `events`; the run's site is stale for `talks`. + expect((await state(site.id, "events"))?.ready).toBe(1); + await rebuildListValues(k.sql, { siteId: site.id }); + await expectExact(site.id); + }); + }); + + it("import resets and rebuilds the rows; an unchanged import sends none", async () => { + const site = await setup(); + const o = await entry(site, "o", { price: 1, kind: ["talk"], venue: "Hall" }); + await translate(site, o, { fields: { venue: "Salon" } }); + await entry(site, "p", { price: 2 }, { collection: "talks", title: "P" }); + const bundle = await exportSite({ sql: k.sql, blobs: k.blobs }, site.id); + await deleteSiteData({ sql: k.sql, blobs: k.blobs }, site.id); + expect( + (await k.sql.all("SELECT 1 FROM entry_list_values WHERE site_id = ?", [site.id])).length, + ).toBe(0); + expect( + (await k.sql.all("SELECT 1 FROM entry_list_state WHERE site_id = ?", [site.id])).length, + ).toBe(0); + await importSite({ sql: k.sql, blobs: k.blobs }, bundle); + await expectExact(site.id); + const seen: string[] = []; + await importSite({ sql: recording(k.sql, seen), blobs: k.blobs }, bundle); + expect(seen.some((q) => q.includes("entry_list"))).toBe(false); + await k.store.updateContentEntry({ + ...(await get(o.id)), + fields: { price: 9, kind: [], venue: "Hall" }, + }); + await importSite({ sql: k.sql, blobs: k.blobs }, bundle, { mode: "replace" }); + await expectExact(site.id); + }); + }); +} diff --git a/packages/core-ts/test/list-values-query.test.ts b/packages/core-ts/test/list-values-query.test.ts new file mode 100644 index 00000000..5d2c05db --- /dev/null +++ b/packages/core-ts/test/list-values-query.test.ts @@ -0,0 +1,41 @@ +/** + * RFC-0042 §4, #110: list-value queries in SQL equal the in-memory evaluator on random data and + * queries, on each storage backend (`src/testing-list-values-query.ts`). + */ +import { MemoryBlobStore, SqlBlobStore } from "@cloudhal/storage"; +import { D1Driver } from "@cloudhal/storage/d1"; +import { BetterSqlite3Driver } from "@cloudhal/storage/sqlite"; +import { createMiniflareD1, createMiniflareDoSqlite } from "@cloudhal/storage/testing"; +import { listValuesQuerySuite } from "../src/testing-list-values-query.ts"; + +listValuesQuerySuite("better-sqlite3 + memory blobs", { + create: async () => ({ + sql: await BetterSqlite3Driver.open(":memory:"), + blobs: new MemoryBlobStore(), + }), +}); + +let d1: Awaited> | undefined; +listValuesQuerySuite("Cloudflare D1 (Miniflare) + memory blobs", { + async create() { + d1 ??= await createMiniflareD1(); + return { sql: new D1Driver(d1.db), blobs: new MemoryBlobStore() }; + }, + async dispose() { + await d1?.dispose(); + d1 = undefined; + }, +}); + +let doSqlite: Awaited> | undefined; +listValuesQuerySuite("Durable Object SQLite (Miniflare) + sql blobs", { + async create() { + doSqlite ??= await createMiniflareDoSqlite(); + const sql = doSqlite.driver("store"); + return { sql, blobs: new SqlBlobStore(sql) }; + }, + async dispose() { + await doSqlite?.dispose(); + doSqlite = undefined; + }, +}); diff --git a/packages/core-ts/test/list-values-rows.test.ts b/packages/core-ts/test/list-values-rows.test.ts new file mode 100644 index 00000000..339678b7 --- /dev/null +++ b/packages/core-ts/test/list-values-rows.test.ts @@ -0,0 +1,236 @@ +/** + * RFC-0042 §4, #110: the pure parts of `entry_list_values` — per-type encodings, `utf16Key` + * (SQLite BINARY order equals JavaScript's `<`), text values that are too long or not well + * formed, the safe-integer / bound-number split, and the list fingerprint (labels excluded, + * column order ignored, every FieldDefinition property either fingerprinted or excluded). + */ +import { readFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import type { FieldDefinition, Site } from "@cloudhal/core-contracts"; +import { BetterSqlite3Driver } from "@cloudhal/storage/sqlite"; +import { describe, expect, it } from "vitest"; +import { EXCLUDED, listFingerprint, NO_LIST } from "../src/list-values/fingerprint.ts"; +import { dayNumber, fieldRows, listValueRows, textKey, utf16Key } from "../src/list-values/rows.ts"; +import { nodeRowsStatements } from "../src/list-values/statements.ts"; + +const here = path.dirname(fileURLToPath(import.meta.url)); +const L = { en: { label: "x" } }; +const def = (d: Partial & { type: FieldDefinition["type"] }) => + ({ key: "f", labels: L, localized: false, ...d }) as FieldDefinition; + +function mulberry32(seed: number): () => number { + let a = seed >>> 0; + return () => { + a = (a + 0x6d2b79f5) >>> 0; + let t = a; + t = Math.imul(t ^ (t >>> 15), t | 1); + t ^= t + Math.imul(t ^ (t >>> 7), t | 61); + return ((t ^ (t >>> 14)) >>> 0) / 4294967296; + }; +} + +describe("list value rows (RFC-0042 §4)", () => { + it("encodes each type as the evaluator reads it", () => { + expect(fieldRows("f", def({ type: "number" }), -0)).toEqual([ + { field: "f", has: 1, num: 0, text: null }, + ]); + expect(Object.is(fieldRows("f", def({ type: "number" }), -0)[0]?.num, 0)).toBe(true); + expect(fieldRows("f", def({ type: "number" }), undefined)).toEqual([ + { field: "f", has: 0, num: null, text: null }, + ]); + expect(fieldRows("f", def({ type: "date" }), "2026-11-05")[0]?.num).toBe(20261105); + expect(dayNumber("0001-01-31")).toBe(10131); + expect( + fieldRows("f", def({ type: "datetime" }), { + at: "2026-11-05T20:00:00.5Z", + timeZone: "UTC", + })[0]?.num, + ).toBe(Date.parse("2026-11-05T20:00:00.5Z")); + expect(fieldRows("f", def({ type: "boolean" }), true)[0]).toMatchObject({ has: 1, num: 1 }); + expect(fieldRows("f", def({ type: "boolean" }), false)[0]).toMatchObject({ has: 1, num: 0 }); + expect(fieldRows("f", def({ type: "boolean" }), undefined)[0]).toMatchObject({ + has: 0, + num: 0, + }); + const options = [ + { value: "a", labels: { en: "A" } }, + { value: "b", labels: { en: "B" } }, + ]; + expect(fieldRows("f", def({ type: "select", options } as never), "a")).toEqual([ + { field: "f", has: 1, num: null, text: "a" }, + ]); + expect( + fieldRows("f", def({ type: "select", options, multiple: true } as never), ["b", "a"]), + ).toEqual([ + { field: "f", has: 1, num: null, text: null }, + { field: "f", has: 2, num: null, text: "b" }, + { field: "f", has: 2, num: null, text: "a" }, + ]); + expect(fieldRows("f", def({ type: "select", options, multiple: true } as never), [])).toEqual([ + { field: "f", has: 0, num: null, text: null }, + ]); + expect(fieldRows("f", def({ type: "text" }), " ")[0]).toMatchObject({ has: 0, text: " " }); + expect(fieldRows("f", def({ type: "richText" }), "Hi")[0]).toMatchObject({ + has: 1, + text: null, + }); + expect(fieldRows("f", undefined, "x")).toEqual([{ field: "f", has: 0, num: null, text: null }]); + }); + + it("stores text only when it compares exactly, never truncated", () => { + expect(textKey("a".repeat(100))).toBe("a".repeat(100)); + expect(textKey("a".repeat(101))).toBeNull(); + expect(textKey("x\ud800y")).toBeNull(); + expect(textKey("𝒜")).toBe("𝒜"); + }); + + it("utf16Key sorts in SQLite exactly as JavaScript's < does", async () => { + const rnd = mulberry32(110); + const pool = [0x41, 0x61, 0xe9, 0xd7ff, 0xd835, 0xdc9c, 0xe000, 0xff21, 0xfffd, 0xffff]; + const strings = new Set(); + while (strings.size < 3000) { + let s = ""; + const n = Math.floor(rnd() * 6); + for (let i = 0; i < n; i++) { + const r = rnd(); + if (r < 0.15) s += String.fromCodePoint(0x1d49c + Math.floor(rnd() * 4)); + else s += String.fromCharCode(pool[Math.floor(rnd() * pool.length)] as number); + } + strings.add(s); + } + const sql = await BetterSqlite3Driver.open(":memory:"); + await sql.exec("CREATE TABLE t (k TEXT NOT NULL, i INTEGER NOT NULL)"); + const list = [...strings]; + await sql.run( + "INSERT INTO t (k, i) SELECT json_extract(value, '$[0]'), json_extract(value, '$[1]') FROM json_each(?)", + [JSON.stringify(list.map((s, i) => [utf16Key(s), i]))], + ); + const order = (await sql.all<{ i: number }>("SELECT i FROM t ORDER BY k")).map( + (r) => list[r.i], + ); + expect(order).toEqual([...list].sort((a, b) => (a < b ? -1 : a > b ? 1 : 0))); + await sql.close(); + }); + + it("binds numbers that are not safe integers, and carries the rest in JSON", () => { + const values = { + nodeId: "ent_a", + collection: "c", + locale: "en", + status: "draft", + tie: "t", + rows: [ + { field: "a", has: 1 as const, num: 0.1 + 0.2, text: null }, + { field: "b", has: 1 as const, num: 1_700_000_000_000, text: null }, + { field: "c", has: 1 as const, num: 1e300, text: null }, + ], + }; + const [, insert] = nodeRowsStatements("site_a", [{ id: "ent_a", hash: "h", values }]); + expect(insert?.params?.slice(0, 2)).toEqual([0.1 + 0.2, 1e300]); + expect(String(insert?.params?.[2])).toContain("1700000000000"); + // 81 decimals do not fit one statement's parameters. + const many = Array.from({ length: 81 }, (_, i) => ({ + id: `ent_${i}`, + hash: "h", + values: { + ...values, + nodeId: `ent_${i}`, + rows: [{ field: "a", has: 1 as const, num: i + 0.5, text: null }], + }, + })); + const statements = nodeRowsStatements("site_a", many); + expect(statements).toHaveLength(3); + for (const s of statements) expect(s.params?.length).toBeLessThanOrEqual(100); + }); + + it("rows exist only for listed fields of a listed collection", () => { + const site = { + collections: { + events: { + fields: [def({ key: "price", type: "number" }), def({ key: "city", type: "text" })], + listFields: ["price"], + sort: "title", + }, + posts: { fields: [def({ key: "price", type: "number" })] }, + }, + } as unknown as Site; + const entry = { + id: "ent_1", + collection: "events", + locale: "en", + status: "published", + title: "B￿", + createdAt: "2026-01-01T00:00:00.000Z", + fields: { price: 3, city: "Izmir" }, + } as never; + expect(listValueRows(site, entry)).toEqual({ + nodeId: "ent_1", + collection: "events", + locale: "en", + status: "published", + tie: utf16Key("B￿"), + rows: [{ field: "price", has: 1, num: 3, text: null }], + }); + expect(listValueRows(site, { ...(entry as object), collection: "posts" } as never)).toBe( + undefined, + ); + }); +}); + +describe("list fingerprint (RFC-0042 §4)", () => { + const site = (fields: FieldDefinition[], listFields: string[], sort?: string) => + ({ collections: { c: { fields, listFields, ...(sort ? { sort } : {}) } } }) as unknown as Site; + const a = def({ key: "a", type: "number" }); + const b = def({ key: "b", type: "text" }); + + it("ignores labels and column order, not definitions or the sort", async () => { + const base = await listFingerprint(site([a, b], ["a", "b"]), "c"); + expect(await listFingerprint(site([a, b], ["b", "a"]), "c")).toBe(base); + expect( + await listFingerprint(site([{ ...a, labels: { tr: { label: "y" } } }, b], ["a", "b"]), "c"), + ).toBe(base); + expect(await listFingerprint(site([a, b], ["a", "b"], "title"), "c")).not.toBe(base); + expect(await listFingerprint(site([a, b], ["a"]), "c")).not.toBe(base); + expect( + await listFingerprint(site([{ ...a, maximum: 3 } as never, b], ["a", "b"]), "c"), + ).not.toBe(base); + expect(await listFingerprint({ collections: { c: { fields: [a] } } } as never, "c")).toBe( + NO_LIST, + ); + }); + + it("fingerprints every FieldDefinition property but the excluded ones", async () => { + const schema = JSON.parse( + readFileSync(path.resolve(here, "../../core-contracts/schemas/site.schema.json"), "utf8"), + ) as { + $defs: { FieldDefinition: { oneOf: { properties: Record }[] } }; + }; + const variants = schema.$defs.FieldDefinition.oneOf; + expect(variants.length).toBe(9); + const sentinel = (name: string, type: string | undefined): unknown => { + if (name === "migration") return { id: "op_1", fromLocalized: true }; + if (name === "options") return [{ value: "z", labels: { en: "Z" } }]; + if (type === "boolean") return true; + if (type === "integer" || type === "number") return 7; + return name === "labels" ? { tr: { label: "y" } } : "other"; + }; + for (const v of variants) { + const props = v.properties; + const type = (props.type as unknown as { enum: [FieldDefinition["type"]] }).enum[0]; + const base = def({ + key: "a", + type, + ...(type === "select" ? { options: [{ value: "q", labels: { en: "Q" } }] } : {}), + } as never); + const before = await listFingerprint(site([base], ["a"]), "c"); + for (const [name, p] of Object.entries(props)) { + if (name === "key" || name === "type") continue; + const changed = { ...base, [name]: sentinel(name, p.type) } as FieldDefinition; + const after = await listFingerprint(site([changed], ["a"]), "c"); + if (EXCLUDED.includes(name)) expect(after, `${type}.${name}`).toBe(before); + else expect(after, `${type}.${name}`).not.toBe(before); + } + } + }); +}); diff --git a/packages/core-ts/test/list-values-sql.test.ts b/packages/core-ts/test/list-values-sql.test.ts new file mode 100644 index 00000000..a5666f6f --- /dev/null +++ b/packages/core-ts/test/list-values-sql.test.ts @@ -0,0 +1,317 @@ +/** + * RFC-0042 §4, #110: the list-value query compiler (`list-values/query-sql.ts`) — what it refuses, + * how pages split into segments, and, on better-sqlite3, that every statement seeks + * `entry_list_values_query` (the driver) and `entry_list_values_node` (the probes and the bodies + * by id) and reads `nodes` by id: no scan of either table, whatever the operator, type or order. + * The backfill's and the write paths' statements seek too, never walking a site's entries. + */ +import type { + Actor, + FieldCondition, + FieldDefinition, + FieldOrder, + Site, +} from "@cloudhal/core-contracts"; +import { MemoryBlobStore } from "@cloudhal/storage"; +import { BetterSqlite3Driver } from "@cloudhal/storage/sqlite"; +import { beforeAll, describe, expect, it } from "vitest"; +import { parseFieldQuery } from "../src/fields/index.ts"; +import { CHUNK_SQL, ORIGINALS_SQL } from "../src/list-values/backfill.ts"; +import { listFingerprint } from "../src/list-values/fingerprint.ts"; +import { pageWindows } from "../src/list-values/query.ts"; +import { type CompiledListQuery, compileListQuery } from "../src/list-values/query-sql.ts"; +import { + advanceStatement, + passResetStatement, + siteStateStatements, + staleStateStatement, + stateCheckStatement, + stateIs, +} from "../src/list-values/state-sql.ts"; +import { type NodeValues, nodeRowsStatements } from "../src/list-values/statements.ts"; +import { tidyStatement } from "../src/list-values/tidy.ts"; +import { hashIs } from "../src/node-sql.ts"; +import { SiteGraphStore } from "../src/store.ts"; + +const person: Actor = { kind: "human", id: "user_01HZX8K3Q9CD", name: "Halil" }; +const NOW = Date.parse("2026-11-05T20:00:00Z"); +const L = (label: string) => ({ en: { label } }); +const FIELDS: [FieldDefinition, ...FieldDefinition[]] = [ + { key: "startsAt", type: "datetime", labels: L("Starts"), localized: false }, + { key: "price", type: "number", labels: L("Price"), localized: false }, + { + key: "kind", + type: "select", + labels: L("Kind"), + localized: false, + multiple: true, + options: [ + { value: "talk", labels: { en: "Talk" } }, + { value: "social", labels: { en: "Social" } }, + ], + }, + { key: "venue", type: "text", labels: L("Venue"), localized: true }, + { key: "free", type: "boolean", labels: L("Free"), localized: false }, + { key: "day", type: "date", labels: L("Day"), localized: false }, +]; + +describe("RFC-0042 §4 list-value query compiler", () => { + let sql: BetterSqlite3Driver; + let site: Site; + + beforeAll(async () => { + sql = await BetterSqlite3Driver.open(":memory:"); + const store = new SiteGraphStore({ sql, blobs: new MemoryBlobStore() }); + await store.init(); + const project = await store.createProject({ + name: "Plans", + slug: "plans", + ownerId: person.id, + siteIds: [], + }); + site = await store.createSite({ + projectId: project.id, + name: "Plans", + slug: "plans", + defaultLocale: "en", + locales: ["en", "tr"], + domains: [], + pageIds: [], + theme: { name: "blog" }, + timeZone: "Europe/Istanbul", + collections: { + events: { fields: FIELDS, listFields: ["startsAt", "price", "kind", "venue"] }, + talks: { fields: FIELDS, listFields: ["free", "day", "price"], sort: "title" }, + }, + }); + for (let i = 0; i < 30; i++) + await store.createContentEntry({ + siteId: site.id, + collection: i % 2 ? "events" : "talks", + slug: `e-${i}`, + locale: "en", + status: "published", + title: `E ${i}`, + body: { format: "markdown", value: "Body" }, + fields: { + startsAt: { at: new Date(NOW + i * 3_600_000).toISOString(), timeZone: "UTC" }, + price: i % 7, + kind: i % 3 ? ["talk"] : [], + venue: `Hall ${i % 4}`, + free: i % 2 === 0, + day: "2026-11-05", + }, + tags: [], + author: person, + } as never); + }); + + async function compile( + collection: string, + where: FieldCondition[], + orderBy?: FieldOrder, + extra: { locale?: string; status?: string[] } = {}, + ): Promise { + const parsed = parseFieldQuery(site, collection, { where, ...(orderBy ? { orderBy } : {}) }); + if (!parsed.ok) throw new Error(JSON.stringify(parsed.issues)); + const settings = site.collections?.[collection]; + return compileListQuery({ + siteId: site.id, + collection, + fingerprint: await listFingerprint(site, collection), + listFields: settings?.listFields ?? [], + query: parsed.query, + sort: settings?.sort ?? "newest", + now: NOW, + ...extra, + }); + } + + async function plan(statement: { sql: string; params?: readonly unknown[] }) { + const rows = await sql.all<{ detail: string }>( + `EXPLAIN QUERY PLAN ${statement.sql}`, + statement.params as never, + ); + return rows.map((r) => r.detail); + } + + const probesOf = (all: string) => + all.match(/SEARCH p\d+ (EXISTS )?USING COVERING INDEX entry_list_values_node/g) ?? []; + + /** Every statement of the query: no table scan, the driver and probes on their indexes. */ + async function expectSeeks(compiled: CompiledListQuery | string, probes = 0) { + if (typeof compiled === "string") throw new Error(compiled); + const statements = [ + compiled.count, + compiled.all, + ...(compiled.matches ? [compiled.matches] : []), + ...compiled.segments.map((s) => compiled.page(s, { limit: 20, offset: 0 })), + ...compiled.segments.map((s) => compiled.page(s, { limit: 20, offset: 0, reverse: true })), + ]; + for (const st of statements) { + const details = await plan(st); + const all = details.join("\n"); + expect(all).not.toMatch(/SCAN (d|n|p\d+|nodes|entry_list_values)\b/); + expect(all).toMatch(/SEARCH d USING COVERING INDEX entry_list_values_query/); + expect(probesOf(all)).toHaveLength(probes); + if (st !== compiled.count && st !== compiled.matches) + expect(all).toMatch(/SEARCH n USING INDEX sqlite_autoindex_nodes/); + } + // The window's bodies by id: one seek of the node index per id, never the query index. + const bodies = (await plan(compiled.bodies(["a", "b"]))).join("\n"); + expect(bodies).not.toMatch(/SCAN (d|n|p\d+|nodes|entry_list_values)\b|entry_list_values_query/); + expect(bodies).toMatch(/SEARCH d USING INDEX entry_list_values_node \(node_id=\? AND field=\?/); + expect(bodies).toMatch(/SEARCH n USING INDEX sqlite_autoindex_nodes/); + expect(probesOf(bodies)).toHaveLength(probes); + } + + it("seeks the index for the bench query: a range on the field it is ordered by", async () => { + const q = await compile("events", [{ field: "startsAt", op: "gte", value: "now" }], { + field: "startsAt", + direction: "asc", + }); + await expectSeeks(q); + if (typeof q === "string") return; + const count = (await plan(q.count)).join("\n"); + expect(count).toMatch( + /SEARCH d USING COVERING INDEX entry_list_values_query \(site_id=\? AND collection=\? AND field=\? AND has=\? AND num>\?\)/, + ); + }); + + it("seeks for text eq, select in, ne, an order alone, and probes the others", async () => { + await expectSeeks(await compile("events", [{ field: "venue", op: "eq", value: "Hall 1" }])); + await expectSeeks(await compile("events", [{ field: "kind", op: "eq", value: "talk" }])); + await expectSeeks( + await compile("events", [ + { field: "kind", op: "in", values: ["talk", "social"] }, + { field: "venue", op: "eq", value: "Hall 1" }, + ]), + 1, + ); + await expectSeeks(await compile("events", [{ field: "price", op: "ne", value: "3" }])); + await expectSeeks(await compile("events", [], { field: "price", direction: "desc" })); + await expectSeeks( + await compile( + "talks", + [ + { field: "free", op: "eq", value: "false" }, + { field: "day", op: "lte", value: "now" }, + ], + { field: "price", direction: "asc" }, + { locale: "en", status: ["published"] }, + ), + 2, + ); + await expectSeeks(await compile("events", [{ field: "kind", op: "ne", value: "talk" }]), 1); + }); + + it("orders a segment by the index when the directions agree (no sort but the outer page's)", async () => { + const q = await compile("talks", [], { field: "price", direction: "asc" }); + if (typeof q === "string") throw new Error(q); + const inner = (await plan(q.page("valued", { limit: 20, offset: 0 }))).join("\n"); + expect(inner).not.toMatch(/USE TEMP B-TREE FOR (ORDER BY|LAST|RIGHT)/); + }); + + it("reads an unordered page's ids in one statement, unsorted; an unpaged answer in one", async () => { + const q = await compile("events", [{ field: "price", op: "gte", value: "3" }]); + if (typeof q === "string") throw new Error(q); + expect((await plan(q.matches as never)).join("\n")).not.toMatch(/TEMP B-TREE/); + expect(q.all.sql).not.toMatch(/LIMIT/); + const ordered = await compile("events", [], { field: "price", direction: "asc" }); + if (typeof ordered === "string") throw new Error(ordered); + expect(ordered.matches).toBeUndefined(); + expect(await sql.all(q.ready.sql, q.ready.params as never)).toEqual([{ ok: 1 }]); + }); + + it("the backfill's and the write paths' statements seek, never walking the site's entries", async () => { + const view = { siteId: site.id, collection: "events", fingerprint: "f", epoch: "e" }; + const rows = [{ field: "price", has: 1 as const, num: 1.5, text: null }]; + const values: NodeValues[] = [ + { + id: "a", + hash: "h", + original: { id: "o", hash: "oh" }, + values: { + nodeId: "a", + collection: "events", + locale: "en", + status: "draft", + tie: "t", + rows, + }, + }, + ]; + const anchor = hashIs("a", "h"); + const statements = { + chunk: { sql: CHUNK_SQL, params: [site.id, "events:", "events;", 100] }, + originals: { sql: ORIGINALS_SQL, params: ['["o"]', site.id] }, + stale: staleStateStatement(view, { id: site.id, hash: "h" }, "t"), + advance: advanceStatement({ ...view, cursor: null }, values, { + cursor: "k", + ready: false, + at: "t", + }), + check: stateCheckStatement({ + siteId: site.id, + seen: new Map([["events", "f"]]), + own: { collection: "events", original: { id: "o", hash: "oh" } }, + translations: { group: "a", read: [{ id: "t1", hash: "th", collection: "talks" }] }, + anchor, + at: "t", + }), + pass: passResetStatement({ + siteId: site.id, + collection: "events", + ids: ["a"], + keys: ["price"], + at: "t", + }), + tidy: tidyStatement([[site.id, "events", "h"]]), + }; + const lists = [ + ...nodeRowsStatements(site.id, values, stateIs({ ...view, cursor: null })), + ...siteStateStatements( + site.id, + new Map([ + ["events", "f"], + ["posts", "none"], + ]), + { anchor, at: "t" }, + ), + ]; + for (const [name, st] of [...Object.entries(statements), ...lists.map((x, i) => [`${i}`, x])]) { + const all = (await plan(st as { sql: string; params?: readonly unknown[] })).join("\n"); + expect(all, `${name}\n${all}`).not.toMatch( + /SCAN (nodes|n|o|s|t|v|entry_list_values|entry_list_state)\b|nodes_site_kind(_created)?\b/, + ); + } + expect((await plan(statements.originals)).join("\n")).toMatch( + /SCAN j VIRTUAL TABLE[^\n]*\nSEARCH n USING INDEX sqlite_autoindex_nodes_1 \(id=\?\)/, + ); + }); + + it("leaves an ill-formed text operand to the evaluator", async () => { + expect(await compile("events", [{ field: "venue", op: "eq", value: "\uD800" }])).toBe( + "a string operand that is not well formed", + ); + }); + + it("splits a window into the valued and missing segments", () => { + const sizes = [ + { segment: "valued" as const, size: 5 }, + { segment: "missing" as const, size: 3 }, + ]; + expect(pageWindows(sizes, 0, 2)).toEqual([ + { segment: "valued", size: 5, offset: 0, limit: 2, end: false }, + ]); + expect(pageWindows(sizes, 4, 3)).toEqual([ + { segment: "valued", size: 5, offset: 4, limit: 1, end: true }, + { segment: "missing", size: 3, offset: 0, limit: 2, end: false }, + ]); + expect(pageWindows(sizes, 6, undefined)).toEqual([ + { segment: "missing", size: 3, offset: 1, limit: 2, end: true }, + ]); + expect(pageWindows(sizes, 8, 5)).toEqual([]); + expect(pageWindows([{ segment: "all", size: 0 }], 0, 5)).toEqual([]); + }); +}); diff --git a/packages/core-ts/test/list-values.test.ts b/packages/core-ts/test/list-values.test.ts new file mode 100644 index 00000000..142b09eb --- /dev/null +++ b/packages/core-ts/test/list-values.test.ts @@ -0,0 +1,48 @@ +/** + * RFC-0042 §4, #110: `entry_list_values` through every write path, races and the backfill, on + * each storage backend (`src/testing-list-values.ts`, and the review's repairs in + * `src/testing-list-values-repair.ts`). + */ +import { MemoryBlobStore, SqlBlobStore } from "@cloudhal/storage"; +import { D1Driver } from "@cloudhal/storage/d1"; +import { BetterSqlite3Driver } from "@cloudhal/storage/sqlite"; +import { createMiniflareD1, createMiniflareDoSqlite } from "@cloudhal/storage/testing"; +import { type ListValuesFactory, listValuesSuite } from "../src/testing-list-values.ts"; +import { listValuesRepairSuite } from "../src/testing-list-values-repair.ts"; + +const both = (label: string, factory: ListValuesFactory) => { + listValuesSuite(label, factory); + listValuesRepairSuite(label, factory); +}; + +both("better-sqlite3 + memory blobs", { + create: async () => ({ + sql: await BetterSqlite3Driver.open(":memory:"), + blobs: new MemoryBlobStore(), + }), +}); + +let d1: Awaited> | undefined; +both("Cloudflare D1 (Miniflare) + memory blobs", { + async create() { + d1 ??= await createMiniflareD1(); + return { sql: new D1Driver(d1.db), blobs: new MemoryBlobStore() }; + }, + async dispose() { + await d1?.dispose(); + d1 = undefined; + }, +}); + +let doSqlite: Awaited> | undefined; +both("Durable Object SQLite (Miniflare) + sql blobs", { + async create() { + doSqlite ??= await createMiniflareDoSqlite(); + const sql = doSqlite.driver("store"); + return { sql, blobs: new SqlBlobStore(sql) }; + }, + async dispose() { + await doSqlite?.dispose(); + doSqlite = undefined; + }, +}); diff --git a/packages/mcp-server/src/entry-list.ts b/packages/mcp-server/src/entry-list.ts index 5c35d19e..7c23e816 100644 --- a/packages/mcp-server/src/entry-list.ts +++ b/packages/mcp-server/src/entry-list.ts @@ -1,12 +1,18 @@ /** * RFC-0042 §4 "API and MCP" for `list_entries`: `where` and `orderBy` over a collection's list - * fields, run by core-ts's shared evaluator (`queryContentEntries`), and `limit`/`cursor` paging. - * Without `orderBy` the list keeps its order (last updated first); without `limit` it is unpaged, + * fields, run by core-ts's shared evaluator (`queryContentEntries`, in SQL while the collection's + * list values are ready), and `limit`/`cursor` paging: with `orderBy` the store pages, without it + * the list keeps its order (last updated first) and is paged here; without `limit` it is unpaged, * as before. Problems are refused as `invalid_query` with their paths (`where/0/value`). */ import type { ContentEntry, FieldCondition, FieldOrder } from "@cloudhal/core-contracts"; import { SiteSchema } from "@cloudhal/core-contracts"; -import { type EntryQuery, InvariantError, queryContentEntries } from "@cloudhal/core-ts"; +import { + ENTRY_PAGE_LIMIT, + type EntryQuery, + InvariantError, + queryContentEntries, +} from "@cloudhal/core-ts"; import type { JsonSchema, ToolContext } from "./common.ts"; const DEFS = SiteSchema.$defs as Record; @@ -83,19 +89,28 @@ export async function listEntries( if (base.collection === undefined && (where !== undefined || orderBy !== undefined)) refuse("collection", "where and orderBy need the collection whose list fields they use"); const offset = a.cursor === undefined ? 0 : decodeCursor(a.cursor as string); - const { entries } = await queryContentEntries(ctx.store, siteId, { + // Ordered by a field, the store pages (in SQL once the collection's list values are ready); + // otherwise the list keeps its last-updated order, sorted here. + const paged = + orderBy !== undefined && + Number.isInteger(limit) && + (limit as number) >= 1 && + (limit as number) <= ENTRY_PAGE_LIMIT; + const result = await queryContentEntries(ctx.store, siteId, { ...base, ...(where ? { where } : {}), ...(orderBy ? { orderBy } : {}), + ...(paged ? { limit, offset } : {}), now: (ctx.now?.() ?? new Date()).getTime(), }); - const ordered = orderBy ? entries : entries.sort(byUpdated); + const ordered = orderBy ? result.entries : result.entries.sort(byUpdated); if (limit === undefined) return { entries: ordered }; - const page = ordered.slice(offset, offset + limit); + const page = paged ? ordered : ordered.slice(offset, offset + limit); + const total = paged ? result.total : ordered.length; const next = offset + page.length; return { entries: page, - total: ordered.length, - ...(next < ordered.length ? { nextCursor: encodeCursor(next) } : {}), + total, + ...(next < total ? { nextCursor: encodeCursor(next) } : {}), }; } diff --git a/packages/storage/migrations/0029_entry_list_values.ts b/packages/storage/migrations/0029_entry_list_values.ts new file mode 100644 index 00000000..ffdf1662 --- /dev/null +++ b/packages/storage/migrations/0029_entry_list_values.ts @@ -0,0 +1,57 @@ +import type { Migration } from "../src/migrate.ts"; + +/** + * RFC-0042 §4, #110: an index of list-field values, so the store, the API and MCP filter and sort + * a collection without reading every entry. Rows are written by core-ts in each entry's own batch + * (AGENTS.md rule 4), only for a collection's `listFields`; the store reads them only while the + * collection's `entry_list_state` row says they are complete. DDL only: nothing is backfilled + * here (the admin Worker migrates on its first request); `rebuildListValues` does that. + * + * `entry_list_values`, one row per entry and listed field, plus one per chosen option of a + * multiple select: + * - `field`: the listed field's key; + * - `has`: 1 when the resolved value counts as a value (`hasFieldValue`), else 0; 2 marks an option + * row of a multiple select; + * - `num`: a number, a datetime in epoch ms, a date as the integer YYYYMMDD, a boolean as 1/0; + * - `text`: a select option, or a text or url value of at most 100 UTF-16 units that is well + * formed (never truncated; NULL otherwise); + * - `tie`: the collection sort's key (`publishedAt ?? createdAt`, or the title mapped so that + * BINARY order is JavaScript's); + * - `created`: the node's `created_at` (the store's own order); + * - `locale`, `status`: the entry's, for the prefilter. + * + * `entry_list_state`, one row per listed collection: the `fingerprint` of the definitions the rows + * were built with, `ready` once every entry has its rows, `epoch` (a token every reset changes, + * which guards the backfill's chunks) and `cursor` (the last `nodes.key` the backfill wrote; NULL + * means from the start). + */ +export const migration0029EntryListValues: Migration = { + name: "0029_entry_list_values", + sql: ` +CREATE TABLE entry_list_values ( + site_id TEXT NOT NULL, + collection TEXT NOT NULL, + field TEXT NOT NULL, + has INTEGER NOT NULL, + num REAL, + text TEXT, + tie TEXT NOT NULL, + node_id TEXT NOT NULL, + created TEXT NOT NULL, + locale TEXT NOT NULL, + status TEXT NOT NULL +); +CREATE INDEX entry_list_values_query ON entry_list_values (site_id, collection, field, has, num, text, tie, node_id, created, locale, status); +CREATE INDEX entry_list_values_node ON entry_list_values (node_id, field, has, num, text); +CREATE TABLE entry_list_state ( + site_id TEXT NOT NULL, + collection TEXT NOT NULL, + fingerprint TEXT NOT NULL, + epoch TEXT NOT NULL, + ready INTEGER NOT NULL, + cursor TEXT, + updated_at TEXT NOT NULL, + PRIMARY KEY (site_id, collection) +); +`, +}; diff --git a/packages/storage/migrations/index.ts b/packages/storage/migrations/index.ts index 5aaec3fa..0104e059 100644 --- a/packages/storage/migrations/index.ts +++ b/packages/storage/migrations/index.ts @@ -27,6 +27,7 @@ import { migration0025TranslationGroups } from "./0025_translation_groups.ts"; import { migration0026SearchCjk } from "./0026_search_cjk.ts"; import { migration0027MultiSite } from "./0027_multi_site.ts"; import { migration0028SiteMembers } from "./0028_site_members.ts"; +import { migration0029EntryListValues } from "./0029_entry_list_values.ts"; /** All Cloudhal storage migrations, in order. */ export const migrations: readonly Migration[] = [ @@ -58,4 +59,5 @@ export const migrations: readonly Migration[] = [ migration0026SearchCjk, migration0027MultiSite, migration0028SiteMembers, + migration0029EntryListValues, ]; diff --git a/packages/storage/test/migrate.test.ts b/packages/storage/test/migrate.test.ts index 11b85ea9..e83454e7 100644 --- a/packages/storage/test/migrate.test.ts +++ b/packages/storage/test/migrate.test.ts @@ -46,6 +46,7 @@ describe("migrate", () => { "0026_search_cjk", "0027_multi_site", "0028_site_members", + "0029_entry_list_values", ]); expect((await migrate(driver, migrations)).applied).toEqual([]); expect(await appliedMigrations(driver)).toEqual([ @@ -77,6 +78,7 @@ describe("migrate", () => { "0026_search_cjk", "0027_multi_site", "0028_site_members", + "0029_entry_list_values", ]); const tables = await driver.all<{ name: string }>( "SELECT name FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%' ORDER BY name", @@ -110,6 +112,8 @@ describe("migrate", () => { "comment_threads", "commenter_emails", "comments", + "entry_list_state", + "entry_list_values", "invitation", "media_blobs", "media_uploads", @@ -185,6 +189,7 @@ describe("migrate", () => { "0026_search_cjk", "0027_multi_site", "0028_site_members", + "0029_entry_list_values", ]); expect(await appliedMigrations(driver)).toEqual([ "0001_init", @@ -215,6 +220,7 @@ describe("migrate", () => { "0026_search_cjk", "0027_multi_site", "0028_site_members", + "0029_entry_list_values", ]); await h.dispose(); }); @@ -252,6 +258,7 @@ describe("migrate", () => { "0026_search_cjk", "0027_multi_site", "0028_site_members", + "0029_entry_list_values", ]); const ok = await driver.get("SELECT name FROM sqlite_master WHERE name = 'ok'"); expect(ok).toBeUndefined(); diff --git a/scripts/size-baseline.json b/scripts/size-baseline.json index 561642ea..b9e092f8 100644 --- a/scripts/size-baseline.json +++ b/scripts/size-baseline.json @@ -21,7 +21,7 @@ "packages/core-ts/src/automations.ts": 1003, "packages/core-ts/src/menu.ts": 928, "packages/core-ts/src/releases.ts": 996, - "packages/core-ts/src/store.ts": 1221, + "packages/core-ts/src/store.ts": 1212, "packages/core-ts/src/surface-interop.ts": 1235, "packages/core-ts/src/taxonomy.ts": 1718, "packages/mcp-server/src/menu-tools.ts": 1058,