feat(core-ts): reference consumers, gated off (RFC-0045 step 2b, #114)

.changeset/reference-consumers.md

@@ -0,0 +1,17 @@
+---
+"@cloudhal/core-ts": minor
+"@cloudhal/api": minor
+"@cloudhal/cli": patch
+"@cloudhal/cloudflare": patch
+"@cloudhal/mcp-server": patch
+---
+
+The consumers of reference fields (RFC-0045 §2, rollout step 2b, #114). Both new field types stay refused on every write path in production, so none of this is reachable before rollout step 7. The shared suites run it through `SiteGraphStoreOptions.fieldTypes`.
+
+- **Assets inside repeater items.** `fieldAssetRefs` walks items and records `fields.<key>.<sub>` with each use's item path. `fieldAssetIssues` checks sub-field types (a PDF in an image sub-field is refused) and reports at every item that uses the file. Skips match issue paths by prefix, in the store and in the MCP precheck alike, so top-level behaviour is unchanged.
+- **Field-operation rewrites** write their entries' `entry_refs` rows. A rewrite that changes a reference value is guarded on its target, and every rewrite in a site with reference fields is guarded on the collection's definitions.
+- **Releases** no longer treat reference values as dependencies, so a referenced entry is neither locked nor required live.
+- **Language editions** (`forEdition`) keep only what the edition's own write accepts. A reference needs a member of its group in an allowed collection that meets the field's `where`. A single-target field keeps one id, and a multiple field keeps at most `maxItems`. An entry holds at most 100 references, and a repeater at most `maxItems` items. Values are read through a running rename's old key. Everything not carried, including repeater items left empty, is reported with its reason through `onDropped`. A re-save keeps the values the edition entry already holds, as its own write would.
+- **CLI.** `cloudhal import --dry-run` warns about references that will show nothing, and about originals relinked into another group.
+- **Cloudflare.** `siteFieldTypeSupport` reads the site Worker's `fields` feature. `FIELDS_FEATURE_VERSION` and `FIELDS_SUPPORTED` are exported from `@cloudhal/core-ts/fields`. Nothing passes them to a store before step 7.
+- **API.** `GET /sites/:siteId/entries/pick` is the entry picker. Its search folds case and diacritics on both sides with `foldName`, Turkish İ and ı included, and reads at most 1,000 entries per request. When that runs out before 20 matches, it answers `truncated: true` with a `cursor` to continue from. `GET /sites/:siteId/entries/:entryId/references` lists an entry's referrers, each with the language versions that show it. Both need `content:read`.

docs/rfcs/0045-collections-stage-4-references-and-repeaters.md

@@ -642,7 +642,11 @@ All new UI follows `kumo-design`, `studio-ui` and DESIGN.md "Fields", imports on
   - **repeater:** a nested sub-field list (rows like `FieldRow`, with up/down and ⋯), the maximum and the item label;
   - the ⋯ menu's operations gain the sub-field forms and the new `change_field_type` inputs (§7).
 - **API** (site-scoped, under the `/sites/:siteId` membership check):
-  - `GET /sites/:siteId/entries/pick?collections=a,b&q=&locale=&limit=` answers at most 20 `{groupId, id, title, collection, status, locale}` per page, never archived ones. It needs `content:read`. `q` is at most 100 characters, matched as a case-folded substring on titles and slugs (`includes`, no regex: rule 13), under a per-member window of the existing `rate_limits` counters (`packages/api/src/rate-limits.ts`, migration 0030).
+  - `GET /sites/:siteId/entries/pick?collections=a,b&q=&locale=&limit=&cursor=` (amended in 2b, PR #206). It needs `content:read`.
+    - The answer is `{entries, cursor?, truncated?}`: at most 20 `{groupId, id, title, collection, status, locale}`, never archived ones, in key order (collection, locale, slug).
+    - Each allowed collection (and locale) is a key range of `nodes_site_kind_key`, read with a keyset cursor. A request reads at most 1,000 rows. When the budget ends before `limit` matches, the answer says `truncated: true` and carries the `cursor` to continue from, so any match can be found.
+    - `q` is at most 100 characters. Both it and each row's title and slug are folded with `foldName` (case, diacritics, Turkish İ/ı) and matched as a substring with no pattern (rule 13). A `q` that folds to nothing matches nothing.
+    - The existing `rate_limits` counters (`packages/api/src/rate-limits.ts`, migration 0030) give a per-member window.
   - `GET /sites/:siteId/entries/:id/references` answers the referrers from `entry_refs`, expanded to members, paged.
 - **Accessibility:**
   - Each repeater item is a `fieldset` whose `legend` is its label ("Speaker 2: Ada Lovelace"), an ICU `selectordinal`/`plural` message, never concatenated.

packages/api/src/entry-pick.ts

@@ -0,0 +1,116 @@
+/**
+ * RFC-0045 §5: the entry picker's search, bounded per request and complete across requests.
+ *
+ * - **Where it reads.** An entry's node key is `<collection>:<locale>:<slug>` (`NODE_SPEC`), and
+ *   `nodes_site_kind_key (site_id, kind, key)` is a unique index. So each allowed collection, or
+ *   collection and locale, is one index range, read in key order with a keyset cursor, and every
+ *   row read is a row returned: the `WHERE` names only indexed columns.
+ * - **The budget.** At most `PICK_ROWS` rows per request, across collections. When the budget
+ *   ends before `limit` matches, the answer says `truncated: true` and carries a `cursor`. Asking
+ *   again with it continues where this request stopped, so any matching entry can be found.
+ * - **The match.** `q` and each row's title and slug are folded with the same `foldName` (case,
+ *   diacritics, Turkish İ/ı), and matched as a substring, with no pattern over the input (rule 13).
+ *   A `q` with nothing left once folded (only punctuation) matches nothing.
+ *
+ * Archived entries are skipped (they count toward the budget). Results come in key order:
+ * collection, locale, slug.
+ */
+import type { ContentEntry } from "@cloudhal/core-contracts";
+import { foldName } from "@cloudhal/core-ts/names";
+import type { SqlDriver } from "@cloudhal/storage";
+
+/** Rows one picker request reads at most. */
+export const PICK_ROWS = 1000;
+
+export interface PickQuery {
+  q: string;
+  collections: readonly string[];
+  locale?: string | undefined;
+  limit: number;
+  /** The `cursor` of an earlier answer: the last node key it read. */
+  cursor?: string | undefined;
+  /** Rows to read at most (default `PICK_ROWS`; tests pass less). */
+  budget?: number | undefined;
+}
+
+export interface PickCard {
+  groupId: string;
+  id: string;
+  title: string;
+  collection: string;
+  status: ContentEntry["status"];
+  locale: string;
+}
+
+export interface PickPage {
+  entries: PickCard[];
+  /** Where to continue; absent once every allowed collection was read to its end. */
+  cursor?: string;
+  /** The row budget ended before `limit` matches were found: ask again with `cursor`. */
+  truncated?: true;
+}
+
+interface Row extends Record<string, unknown> {
+  key: string;
+  id: string;
+  title: string | null;
+  status: ContentEntry["status"];
+  grp: string | null;
+}
+
+/** The SQL of one collection range (exported for the query-plan test). */
+export const PICK_RANGE_SQL = `SELECT key, id, json_extract(body, '$.title') AS title,
+    json_extract(body, '$.status') AS status, json_extract(body, '$.translation.group') AS grp
+  FROM nodes WHERE site_id = ? AND kind = 'contentEntry' AND key >= ? AND key > ? AND key < ?
+  ORDER BY key LIMIT ?`;
+
+/** The key range of a collection (and locale): `<prefix>` up to the next character after `:`. */
+function rangeOf(collection: string, locale: string | undefined) {
+  const prefix = locale === undefined ? `${collection}:` : `${collection}:${locale}:`;
+  return { prefix, upper: `${prefix.slice(0, -1)};` };
+}
+
+/** One picker answer. */
+export async function pickEntries(
+  sql: SqlDriver,
+  siteId: string,
+  query: PickQuery,
+): Promise<PickPage> {
+  const raw = query.q.trim();
+  const q = foldName(raw);
+  if (raw !== "" && q === "") return { entries: [] };
+  const hits = (text: string | null) => text !== null && foldName(text).includes(q);
+  const budget = query.budget ?? PICK_ROWS;
+  const entries: PickCard[] = [];
+  let read = 0;
+  let last = query.cursor;
+  // Ranges in the order of their keys, not of the names: "post-x:" and "news2:" sort before
+  // "post:" and "news:" ('-' and digits sort below ':'), and the cursor walks key order.
+  const ranges = [...new Set(query.collections)]
+    .map((collection) => ({ collection, ...rangeOf(collection, query.locale) }))
+    .sort((a, b) => (a.prefix < b.prefix ? -1 : a.prefix > b.prefix ? 1 : 0));
+  for (const { collection, prefix, upper } of ranges) {
+    if (last !== undefined && last >= upper) continue;
+    // One read per range, for the rest of the budget: the range ends first, or the budget does.
+    const want = budget - read;
+    const rows = await sql.all<Row>(PICK_RANGE_SQL, [siteId, prefix, last ?? "", upper, want]);
+    for (const r of rows) {
+      read++;
+      last = r.key;
+      if (r.status === "archived") continue;
+      const cut = r.key.indexOf(":", collection.length + 1);
+      if (q !== "" && !hits(r.title) && !hits(r.key.slice(cut + 1))) continue;
+      entries.push({
+        groupId: r.grp ?? r.id,
+        id: r.id,
+        title: r.title ?? "",
+        collection,
+        status: r.status,
+        locale: r.key.slice(collection.length + 1, cut),
+      });
+      if (entries.length >= query.limit) return { entries, cursor: r.key };
+    }
+    if (read >= budget) return { entries, ...(last ? { cursor: last } : {}), truncated: true };
+  }
+  return { entries };
+}

packages/api/src/playground.ts

@@ -144,6 +144,9 @@ export const PLAYGROUND_ALLOW: ReadonlySet<string> = new Set([
   `POST ${S}/entries/:entryId/preview`,
   `GET ${S}/entries/:entryId/structured-data`,
   `POST ${S}/entries/preview-fragment`,
+  // RFC-0045 §5: the entry picker and an entry's referrers, read-only (content:read).
+  `GET ${S}/entries/pick`,
+  `GET ${S}/entries/:entryId/references`,
   `GET ${S}/entries/:entryId`,
   `PUT ${S}/entries/:entryId`,
   `DELETE ${S}/entries/:entryId`,

packages/api/src/routes/entries.ts

@@ -15,6 +15,7 @@ import { editsLiveCorrections, entryLimit, personOf } from "../own.ts";
 import { param } from "../params.ts";
 import { type DraftPreview, previewResponse } from "../preview.ts";
 import { previewStructuredData } from "../structured-data.ts";
+import { entryReferences } from "./entry-references.ts";
 
 const STATUSES = new Set<PublishStatus>(["draft", "review", "published", "archived"]);
 
@@ -56,6 +57,8 @@ export const entries: Hono<Env> = new Hono<Env>()
     // `filter` is the RFC-0002 §5 language; a FilterError maps to 400 with suggestions.
     return c.json(await c.get("store").listContentEntries(param(c, "siteId"), base));
   })
+  // RFC-0045 §5: the entry picker and an entry's referrers (before `/:entryId`).
+  .route("/", entryReferences)
   .post("/", async (c) => {
     const siteId = param(c, "siteId");
     if (!(await c.get("store").getSite(siteId))) throw notFound("site not found");

packages/api/src/routes/entry-references.ts

@@ -0,0 +1,158 @@
+/**
+ * RFC-0045 §5 "API": the entry picker and an entry's referrers, site-scoped under the
+ * `/sites/:siteId` membership check, both `content:read`.
+ *
+ * - `GET …/entries/pick?collections=a,b&q=&locale=&limit=&cursor=`: at most 20 `{groupId, id,
+ *   title, collection, status, locale}` of the allowed collections, never archived ones, with a
+ *   `cursor` and `truncated` when the row budget ended first (`entry-pick.ts`). `q` is at most
+ *   100 characters and refused before it is read further; a per-member window of the
+ *   `rate_limits` counters applies.
+ * - `GET …/entries/:entryId/references?limit=&offset=`: the referrers of the entry's translation
+ *   group from `entry_refs`, published holders first, each holder with the same-site members that
+ *   show its value (`shared` 1, or 2 without their own value), paged.
+ */
+import type { ContentEntry } from "@cloudhal/core-contracts";
+import { memberGroup } from "@cloudhal/core-ts/fields";
+import type { SqlDriver } from "@cloudhal/storage";
+import type { Context } from "hono";
+import { Hono } from "hono";
+import type { Env } from "../context.ts";
+import { pickEntries } from "../entry-pick.ts";
+import { apiError, badRequest, notFound } from "../errors.ts";
+import { param } from "../params.ts";
+import { countHit } from "../rate-limits.ts";
+
+const SLUG = /^[a-z0-9]+(?:-[a-z0-9]+)*$/;
+const LOCALE = /^[A-Za-z]{2,3}(?:-[A-Za-z0-9]{2,8}){0,3}$/;
+const MAX_Q = 100;
+/** A node key: collection, locale and slug (each well under 100 characters). */
+const MAX_CURSOR = 400;
+const PICK_LIMIT = 20;
+/** Picker searches per member and site per minute. */
+const PICK_WINDOW = { hits: 120, ms: 60_000 };
+const REFERRERS_LIMIT = 100;
+
+function intParam(raw: string | undefined, max: number, fallback: number): number {
+  if (raw === undefined || raw === "") return fallback;
+  const n = Number(raw);
+  if (!Number.isInteger(n) || n < 0) throw badRequest("limit and offset are non-negative integers");
+  return Math.min(n, max);
+}
+
+/** The picker's query, checked before any read. */
+function pickQuery(c: Context<Env>) {
+  const q = c.req.query("q") ?? "";
+  if (q.length > MAX_Q) throw badRequest(`q is at most ${MAX_Q} characters`);
+  const collections = (c.req.query("collections") ?? "").split(",").filter((s) => s !== "");
+  if (collections.length === 0 || collections.length > 5 || !collections.every((s) => SLUG.test(s)))
+    throw badRequest("collections lists one to five collection keys");
+  const locale = c.req.query("locale") || undefined;
+  if (locale !== undefined && !LOCALE.test(locale)) throw badRequest("locale is a language tag");
+  const limit = Math.max(1, intParam(c.req.query("limit"), PICK_LIMIT, PICK_LIMIT));
+  const cursor = c.req.query("cursor") || undefined;
+  if (cursor !== undefined && cursor.length > MAX_CURSOR)
+    throw badRequest("cursor is not one this route gave");
+  return { q, collections, locale, limit, cursor };
+}
+
+async function rateLimited(c: Context<Env>, siteId: string): Promise<boolean> {
+  const who = c.get("actor")?.id ?? "anonymous";
+  const count = await countHit(c.get("store").sql, {
+    key: `entries-pick:${siteId}:${who}`,
+    windowMs: PICK_WINDOW.ms,
+    now: c.get("now")().getTime(),
+  });
+  return count > PICK_WINDOW.hits;
+}
+
+interface Card {
+  groupId: string;
+  id: string;
+  title: string;
+  collection: string;
+  status: ContentEntry["status"];
+  locale: string;
+}
+
+const cardOf = (e: ContentEntry): Card => ({
+  groupId: memberGroup(e),
+  id: e.id,
+  title: e.title,
+  collection: e.collection,
+  status: e.status,
+  locale: e.locale,
+});
+
+interface RefRow {
+  holder: string;
+  field: string;
+  shared: number;
+  root_key: string;
+  source_status: string;
+  body: string;
+}
+
+/** One page of the group's rows, each holder with the members that show its value. */
+async function referrers(
+  sql: SqlDriver,
+  siteId: string,
+  page: { group: string; limit: number; offset: number },
+) {
+  const rows = await sql.all<RefRow & Record<string, unknown>>(
+    `SELECT r.source_id AS holder, r.field AS field, r.shared AS shared, r.root_key AS root_key,
+        r.source_status AS source_status, n.body AS body
+      FROM entry_refs r JOIN nodes n ON n.id = r.source_id AND n.site_id = r.site_id
+      WHERE r.site_id = ? AND r.target_id = ?
+      ORDER BY r.source_status = 'published' DESC, r.source_id, r.field LIMIT ? OFFSET ?`,
+    [siteId, page.group, page.limit + 1, page.offset],
+  );
+  const shown = rows.slice(0, page.limit);
+  const holders = [...new Set(shown.map((r) => r.holder))];
+  const members = holders.length
+    ? await sql.all<{ body: string }>(
+        `SELECT body FROM nodes WHERE site_id = ? AND kind = 'contentEntry'
+          AND json_extract(body, '$.translation.group') IN (SELECT value FROM json_each(?))
+          AND json_extract(body, '$.translation.source.siteId') IS NULL`,
+        [siteId, JSON.stringify(holders)],
+      )
+    : [];
+  const inheriting = members.map((m) => JSON.parse(m.body) as ContentEntry);
+  const items = shown.map((r) => ({
+    ...cardOf(JSON.parse(r.body) as ContentEntry),
+    field: r.field,
+    shownBy: inheriting
+      .filter(
+        (m) =>
+          m.translation?.group === r.holder &&
+          // Only the row's own holder is left out: another holder on this page may still show it.
+          m.id !== r.holder &&
+          (r.shared === 1 || !Object.hasOwn(m.fields ?? {}, r.root_key)),
+      )
+      .map(cardOf),
+  }));
+  return {
+    referrers: items,
+    ...(rows.length > page.limit ? { next: page.offset + page.limit } : {}),
+  };
+}
+
+export const entryReferences: Hono<Env> = new Hono<Env>()
+  .get("/pick", async (c) => {
+    const siteId = param(c, "siteId");
+    const query = pickQuery(c);
+    if (!(await c.get("store").getSite(siteId))) throw notFound("site not found");
+    if (await rateLimited(c, siteId))
+      throw apiError(429, "rate_limited", "too many picker searches; wait a minute");
+    return c.json(await pickEntries(c.get("store").sql, siteId, query));
+  })
+  .get("/:entryId/references", async (c) => {
+    const store = c.get("store");
+    const entry = await store.getContentEntry(param(c, "entryId"));
+    if (!entry || entry.siteId !== param(c, "siteId")) throw notFound("entry not found");
+    const page = {
+      group: memberGroup(entry),
+      limit: Math.max(1, intParam(c.req.query("limit"), REFERRERS_LIMIT, 20)),
+      offset: intParam(c.req.query("offset"), Number.MAX_SAFE_INTEGER, 0),
+    };
+    return c.json({ groupId: page.group, ...(await referrers(store.sql, entry.siteId, page)) });
+  });

packages/api/src/scopes.ts

@@ -172,6 +172,9 @@ export const ROUTE_SCOPES: readonly (readonly [Method, string, RouteRequirement]
   ["POST", `${S}/entries/:entryId/preview`, "content:read"],
   // the body editor's inline preview of one ```surface (writes nothing)
   ["POST", `${S}/entries/preview-fragment`, "content:read"],
+  // RFC-0045 §5: the entry picker and an entry's referrers
+  ["GET", `${S}/entries/pick`, "content:read"],
+  ["GET", `${S}/entries/:entryId/references`, "content:read"],
   ["GET", `${S}/entries/:entryId`, "content:read"],
   ["PUT", `${S}/entries/:entryId`, "content:draft:own"],
   ["DELETE", `${S}/entries/:entryId`, "content:draft:own"],