feat: store and API checks for typed fields (RFC-0041 step 2)

.changeset/typed-fields-store.md

@@ -0,0 +1,12 @@
+---
+"@cloudhal/core-ts": minor
+"@cloudhal/api": minor
+---
+
+Typed custom fields, step 2 (RFC-0041 / ADR-0043): the store and the API enforce field definitions.
+
+- **Site writes** (`updateSite`, `createSite`, `PUT /api/sites/:id/collections/:key`, dry run included) check definitions: reserved (`field_key_reserved`) and duplicate keys or options, `minimum` above `maximum`, 30 fields per collection and 200 per site (`too_many_fields`), 256 KB of collection settings (`too_large`). A plain edit never loses a value: changing a field's `type`, `localized` or `multiple`, or dropping a key or a select option that an entry still holds, is refused with `409 field_change_needs_operation`, naming the field operation to use. Other site writes do not re-run the checks.
+- **Entry writes** check only the field values that changed: `422 invalid_fields`, `unknown_field` (a new key without a definition, once the collection has definitions; stored unknown keys are kept), `field_not_localized` (a translation's own copy of a non-localized value) and `required_field` (review or published without a `requiredToPublish` value; drafts save freely). `fields` is capped at 256 KB per entry in every collection when it changes (`too_large`). Image values must be image assets of the site, and image and rich-text asset references are recorded in `asset_refs` under `fields.<key>`, so a used file cannot be deleted. Every refusal lists its problems with paths in `details.issues` (`fields/startsAt/at`).
+- **API inputs** remove a key with `null` in a collection that has definitions. Collections without definitions keep `fields` as any JSON, exactly as before.
+- **Search**: `text` and `richText` fields marked `searchable` add their words to the entry's search text at the next publish (off by default; sites without searchable fields index byte for byte as before).
+- `refuseFieldIssues` and `fieldRefusalCode` are exported from `@cloudhal/core-ts/fields`.

docs/guides/search.md

@@ -14,6 +14,7 @@ The design and its reasons are in [RFC-0029](../rfcs/0029-search-providers-and-s
 - **Published content only.**
   - Found: articles, pages, categories and tags, and authors.
   - Never found: drafts, pages marked `noindex`, embargoed or expired content outside its time window.
+- **Custom fields** (RFC-0041) are not searched unless a text or rich-text field is marked *searchable* in its collection's settings. Then its words count as part of the article's text from the next publish on. A translation without its own value uses the original's.
 - **Categories, tags and authors** are found only while at least one article readers can see carries them. This is the same rule that decides whether their archive page exists.
 - **Matching.** Every word must start a word of the title, summary or text, and title matches come first. Accents and Turkish letters are folded, so "cay" finds "Çay".
 - **Language.** Results come in the site's default language.

packages/api/README.md

@@ -12,12 +12,26 @@ The core routes are below; the feature routes (taxonomy, menus, media, comments,
 | `GET/PUT /api/sites[/:id]` | sites (PUT = full document, CAS on `rev`) |
 | `GET /api/sites/:id/pending` | what a publish would carry, as counts (Studio's badge beside Publish) |
 | `/api/sites/:id/pages[/:pageId]` | pages (POST creates and appends to `Site.pageIds` atomically; DELETE needs `?rev=`) |
-| `/api/sites/:id/entries[/:entryId]` | content entries (`?collection=` filter) |
+| `/api/sites/:id/entries[/:entryId]` | content entries (`?collection=` filter); `fields` is checked against the collection's typed fields (RFC-0041), and `null` removes a key there |
+| `/api/sites/:id/collections[/:key]` | the collection registry; `PUT /:key {settings, rev}` (`?dryRun=1`) declares a collection or changes its settings, typed `fields` included; `DELETE /:key?rev=` |
 | `/api/sites/:id/snapshots` | create (`{label, publish}`), list, `:id/publish`, `:id/rollback`, `:a/diff/:b` |
 | `/api/agent-keys` | API keys for agents: `POST {name, scopes}` (key shown once), `GET`, `DELETE /:id` — signed-in humans only |
 | `/mcp` | the Cloudhal MCP server (ADR-0011); session cookie → human, `x-api-key` or `Authorization: Bearer <key>` → agent limited to the key's scopes (REST answers agent keys with a pointer to `/mcp`) |
 
-Errors: `400 validation` (contract errors with ajv details), `409 conflict` (`expectedRev`/`actualRev`), `409 unique`, `404 not_found`, `401 unauthorized`.
+Errors: `400 validation` (contract errors with ajv details), `409 conflict` (`expectedRev`/`actualRev`), `409 unique`, `404 not_found`, `401 unauthorized`. Store invariants answer with their code (`src/errors.ts`, `INVARIANT_STATUS`).
+
+Typed fields (RFC-0041) answer with every problem and its path in `details.issues` (`[{ path: "fields/startsAt/at", code, message }]`):
+
+| Code | Status | When |
+|---|---|---|
+| `invalid_fields` | 422 | values that do not fit their definitions (or several kinds of problem at once) |
+| `unknown_field` | 422 | a new key without a definition, in a collection that has definitions |
+| `field_not_localized` | 422 | a translation's own copy of a non-localized value (set it on the original) |
+| `required_field` | 422 | an entry in review or published without a `requiredToPublish` value |
+| `too_large` | 409 | `fields` over 256 KB, or collection settings over 256 KB |
+| `field_key_reserved`, `field_key_duplicate`, `field_option_duplicate`, `field_bounds` | 400 | definitions that can never be saved as written |
+| `too_many_fields` | 422 | over 30 fields per collection, 200 per site or 50 options per select |
+| `field_change_needs_operation` | 409 | changing a field's type, localization or `multiple`, or dropping a key or option that entries still hold |
 
 ```ts
 import { createApi } from "@cloudhal/api";

packages/api/src/entry-fields.ts

@@ -0,0 +1,29 @@
+/**
+ * RFC-0041 §2: `null` is never stored. In a collection with typed fields an API input uses `null`
+ * to remove a key, so the entry routes drop such keys before the store checks the values
+ * (`invalid_fields`, `unknown_field`, … with each issue's path in `details.issues`). Collections
+ * without definitions keep `fields` as "any JSON", nulls included, as before.
+ */
+
+import type { SiteGraphStore } from "@cloudhal/core-ts";
+import { fieldDefinitions } from "@cloudhal/core-ts/fields";
+
+const isRecord = (v: unknown): v is Record<string, unknown> =>
+  typeof v === "object" && v !== null && !Array.isArray(v);
+
+/** `input` with `null` field values removed when its collection defines fields. */
+export async function withoutNullFields<T extends { collection?: unknown; fields?: unknown }>(
+  store: SiteGraphStore,
+  siteId: string,
+  input: T,
+): Promise<T> {
+  const fields = input.fields;
+  if (!isRecord(fields) || !Object.values(fields).includes(null)) return input;
+  if (typeof input.collection !== "string") return input;
+  const site = await store.getSite(siteId);
+  if (!site || fieldDefinitions(site, input.collection).length === 0) return input;
+  return {
+    ...input,
+    fields: Object.fromEntries(Object.entries(fields).filter(([, v]) => v !== null)),
+  };
+}

packages/api/src/errors.ts

@@ -69,6 +69,18 @@ export const INVARIANT_STATUS: Record<InvariantCode, 400 | 409 | 410 | 415 | 422
   collection_undeclared: 422,
   collection_in_use: 409,
   collection_exists: 409,
+  // RFC-0041 (typed custom fields): values and definitions the data rules refuse are 422, rules
+  // a request can never meet as written 400, and a change that would lose stored values 409.
+  invalid_fields: 422,
+  unknown_field: 422,
+  field_not_localized: 422,
+  required_field: 422,
+  too_many_fields: 422,
+  field_key_reserved: 400,
+  field_key_duplicate: 400,
+  field_option_duplicate: 400,
+  field_bounds: 400,
+  field_change_needs_operation: 409,
   reserved_key: 400,
   immutable_field: 400,
   primary_not_assigned: 422,

packages/api/src/routes/entries.ts

@@ -7,6 +7,7 @@ import { Hono } from "hono";
 import { patchBody } from "../body.ts";
 import type { Env } from "../context.ts";
 import { entryDraft } from "../drafts.ts";
+import { withoutNullFields } from "../entry-fields.ts";
 import { badRequest, notFound } from "../errors.ts";
 import { API_ACTOR, writerOf } from "../governance.ts";
 import { editsLiveCorrections, entryLimit, personOf } from "../own.ts";
@@ -41,10 +42,15 @@ export const entries: Hono<Env> = new Hono<Env>()
   .post("/", async (c) => {
     const siteId = param(c, "siteId");
     if (!(await c.get("store").getSite(siteId))) throw notFound("site not found");
-    const input = (await c.req.json()) as Omit<
-      ContentEntry,
-      "id" | "rev" | "createdAt" | "updatedAt" | "schemaVersion"
-    >;
+    // RFC-0041: in a collection with typed fields, `null` removes a key.
+    const input = await withoutNullFields(
+      c.get("store"),
+      siteId,
+      (await c.req.json()) as Omit<
+        ContentEntry,
+        "id" | "rev" | "createdAt" | "updatedAt" | "schemaVersion"
+      >,
+    );
     // RFC-0040 §5: a new collection is declared first (Studio's New collection).
     const registry = await c.get("store").listCollections(siteId);
     if (typeof input.collection === "string" && !isRegistered(registry, input.collection))
@@ -104,9 +110,10 @@ export const entries: Hono<Env> = new Hono<Env>()
     return c.json(entry);
   })
   .put("/:entryId", async (c) => {
-    const body = (await c.req.json()) as ContentEntry;
-    if (body.id !== param(c, "entryId") || body.siteId !== param(c, "siteId"))
+    const raw = (await c.req.json()) as ContentEntry;
+    if (raw.id !== param(c, "entryId") || raw.siteId !== param(c, "siteId"))
       throw badRequest("ids must match the URL");
+    const body = await withoutNullFields(c.get("store"), raw.siteId, raw);
     const limit = entryLimit(c.get("principal"));
     if (limit) {
       const stored = await c.get("store").getContentEntry(body.id);

packages/api/src/testing-fields.ts

@@ -0,0 +1,164 @@
+/**
+ * Shared API suite for RFC-0041 step 2: entry routes validate typed field values through the store
+ * and answer with each issue's path (`details.issues`); `null` removes a key in a collection with
+ * definitions; `PUT /collections/:key` takes definitions with the §6 guard (dry run included).
+ * Runs against any SiteGraphStore backend, like `testing.ts`.
+ */
+import type { FieldDefinition, Site } from "@cloudhal/core-contracts";
+import { type SiteGraphStore, seedStarterSite } from "@cloudhal/core-ts";
+import { afterAll, beforeEach, describe, expect, it } from "vitest";
+import { createApi } from "./app.ts";
+import type { ApiStoreFactory } from "./testing.ts";
+
+const human = { kind: "human" as const, id: "user_apitest00001", name: "Tester" };
+
+type Json = Record<string, unknown>;
+
+const FIELDS: FieldDefinition[] = [
+  {
+    key: "venue",
+    type: "text",
+    labels: { en: { label: "Venue" } },
+    localized: true,
+    requiredToPublish: true,
+  },
+  {
+    key: "seats",
+    type: "number",
+    labels: { en: { label: "Seats" } },
+    localized: false,
+    integer: true,
+  },
+  {
+    key: "level",
+    type: "select",
+    labels: { en: { label: "Level" } },
+    localized: false,
+    options: [
+      { value: "beginner", labels: { en: "Beginner" } },
+      { value: "expert", labels: { en: "Expert" } },
+    ],
+  },
+];
+const EVENTS = { labels: { en: { singular: "Event", plural: "Events" } }, fields: FIELDS };
+
+export function describeFieldsApi(label: string, factory: ApiStoreFactory): void {
+  describe(`API typed fields: ${label}`, () => {
+    let store: SiteGraphStore;
+    let siteId: string;
+    let app: ReturnType<typeof createApi>;
+    const headers = { "content-type": "application/json", "x-test-user": "1" };
+    const call = async (path: string, init: RequestInit = {}) => {
+      const res = await app.request(path, { headers, ...init });
+      return { status: res.status, body: (await res.json()) as Json };
+    };
+    const site = async () => (await call(`/api/sites/${siteId}`)).body as unknown as Site;
+    const put = async (settings: unknown, query = "") =>
+      call(`/api/sites/${siteId}/collections/events${query}`, {
+        method: "PUT",
+        body: JSON.stringify({ settings, rev: (await site()).rev }),
+      });
+    const create = (fields: Json, status = "draft") =>
+      call(`/api/sites/${siteId}/entries`, {
+        method: "POST",
+        body: JSON.stringify({
+          collection: "events",
+          slug: `e-${Math.random().toString(36).slice(2, 8)}`,
+          locale: "en",
+          status,
+          title: "Launch",
+          body: { format: "markdown", value: "Body" },
+          fields,
+          tags: [],
+        }),
+      });
+    const issues = (body: Json) =>
+      ((body.details as { issues?: { path: string; code: string }[] }).issues ?? []).map(
+        (i) => `${i.path} ${i.code}`,
+      );
+
+    beforeEach(async () => {
+      store = await factory.create();
+      siteId = (await seedStarterSite(store, { name: "Fields", slug: "fields" })).id;
+      app = createApi({
+        store,
+        resolveActor: async (c) => (c.req.header("x-test-user") ? human : undefined),
+      });
+    });
+
+    afterAll(async () => {
+      await factory.dispose?.();
+    });
+
+    it("declares fields; a dry run checks them and writes nothing", async () => {
+      const dry = await put(EVENTS, "?dryRun=1");
+      expect(dry.status, JSON.stringify(dry.body)).toBe(200);
+      expect((await site()).collections?.events).toBeUndefined();
+      const bad = await put({ fields: [{ ...FIELDS[0], key: "title" }] }, "?dryRun=1");
+      expect(bad.status).toBe(400);
+      expect(bad.body.error).toBe("field_key_reserved");
+      expect(issues(bad.body)).toEqual(["collections/events/fields/0/key field_key_reserved"]);
+      const saved = await put(EVENTS);
+      expect(saved.status, JSON.stringify(saved.body)).toBe(200);
+      expect((await site()).collections?.events?.fields).toEqual(FIELDS);
+    });
+
+    it("refuses a definition change that would lose stored values (409)", async () => {
+      expect((await put(EVENTS)).status).toBe(200);
+      expect((await create({ level: "expert" })).status).toBe(201);
+      const dropped = await put({ ...EVENTS, fields: FIELDS.filter((f) => f.key !== "level") });
+      expect(dropped.status).toBe(409);
+      expect(dropped.body.error).toBe("field_change_needs_operation");
+      expect(dropped.body.message).toContain("remove_field");
+      const retyped = await put({
+        ...EVENTS,
+        fields: FIELDS.map((f) =>
+          f.key === "seats"
+            ? { key: "seats", type: "text", labels: f.labels, localized: false }
+            : f,
+        ),
+      });
+      expect(retyped.status).toBe(409);
+      expect(retyped.body.error).toBe("field_change_needs_operation");
+    });
+
+    it("entry routes answer value problems with paths (422)", async () => {
+      expect((await put(EVENTS)).status).toBe(200);
+      const bad = await create({ seats: 1.5, level: "master" });
+      expect(bad.status).toBe(422);
+      expect(bad.body.error).toBe("invalid_fields");
+      expect(issues(bad.body).sort()).toEqual([
+        "fields/level invalid_value",
+        "fields/seats invalid_value",
+      ]);
+      const unknown = await create({ colour: "red" });
+      expect(unknown.status).toBe(422);
+      expect(unknown.body.error).toBe("unknown_field");
+      const required = await create({ seats: 10 }, "review");
+      expect(required.status).toBe(422);
+      expect(required.body.error).toBe("required_field");
+      expect(issues(required.body)).toEqual(["fields/venue required_field"]);
+    });
+
+    it("null removes a key in a collection with fields, on create and update", async () => {
+      expect((await put(EVENTS)).status).toBe(200);
+      const made = await create({ venue: "Hall", seats: 10, level: null });
+      expect(made.status, JSON.stringify(made.body)).toBe(201);
+      expect(made.body.fields).toEqual({ venue: "Hall", seats: 10 });
+      const updated = await call(`/api/sites/${siteId}/entries/${made.body.id}`, {
+        method: "PUT",
+        body: JSON.stringify({ ...made.body, fields: { venue: "Hall", seats: null } }),
+      });
+      expect(updated.status, JSON.stringify(updated.body)).toBe(200);
+      expect(updated.body.fields).toEqual({ venue: "Hall" });
+    });
+
+    it("collections without fields keep any JSON, nulls included", async () => {
+      expect((await put({ labels: EVENTS.labels })).status).toBe(200);
+      const loose = { anything: [1, null], gone: null };
+      const made = await create(loose);
+      expect(made.status, JSON.stringify(made.body)).toBe(201);
+      expect(made.body.fields).toEqual(loose);
+    });
+  });
+}

packages/api/test/fields-api.test.ts

@@ -0,0 +1,32 @@
+import { SiteGraphStore } from "@cloudhal/core-ts";
+import { MemoryBlobStore } from "@cloudhal/storage";
+import { D1Driver } from "@cloudhal/storage/d1";
+import { BetterSqlite3Driver } from "@cloudhal/storage/sqlite";
+import { createMiniflareD1, createTableReset } from "@cloudhal/storage/testing";
+import { describeFieldsApi } from "../src/testing-fields.ts";
+
+describeFieldsApi("better-sqlite3", {
+  async create() {
+    const store = new SiteGraphStore({
+      sql: await BetterSqlite3Driver.open(":memory:"),
+      blobs: new MemoryBlobStore(),
+    });
+    await store.init();
+    return store;
+  },
+});
+
+let d1: Awaited<ReturnType<typeof createMiniflareD1>> | undefined;
+const reset = createTableReset();
+describeFieldsApi("Cloudflare D1 (Miniflare)", {
+  async create() {
+    d1 ??= await createMiniflareD1();
+    const sql = new D1Driver(d1.db);
+    const store = new SiteGraphStore({ sql, blobs: new MemoryBlobStore() });
+    await reset(sql, () => store.init());
+    return store;
+  },
+  async dispose() {
+    await d1?.dispose();
+  },
+});

packages/core-ts/src/assets.ts

@@ -31,6 +31,7 @@ import { ID_PREFIX } from "./ids.ts";
 import { RASTER_TYPES } from "./media/image.ts";
 import { type InspectOptions, inspectUpload } from "./media/ingest.ts";
 import { type AssetRefUse, assetRefsOf, canonicalImageRefs } from "./media/refs.ts";
+import { refsRebuild } from "./media/refs-rebuild.ts";
 import { sniff } from "./media/sniff.ts";
 import {
   assetKindFor,
@@ -1021,7 +1022,7 @@ export class AssetEngine {
   /** Rebuilds a site's `asset_refs` from its current nodes (sync import, repair). */
   async rebuildRefsStatements(siteId: string): Promise<SqlStatement[]> {
     const rows = await this.#sql.all<{ id: string; kind: string; body: string }>(
-      "SELECT id, kind, body FROM nodes WHERE site_id = ? AND kind IN ('page', 'contentEntry', 'term')",
+      "SELECT id, kind, body FROM nodes WHERE site_id = ? AND kind IN ('page', 'contentEntry', 'term', 'site')",
       [siteId],
     );
     return refsRebuild(siteId, rows);
@@ -1354,27 +1355,6 @@ function refsKey(refs: readonly AssetRefUse[]): string {
   return refs.map((r) => `${r.assetId}\t${r.field}`).join("\n");
 }
 
-/** `asset_refs` for a whole site from node rows (DELETE + one INSERT). */
-export function refsRebuild(
-  siteId: string,
-  rows: readonly { id: string; kind: string; body: string }[],
-): SqlStatement[] {
-  const all: { a: string; n: string; k: string; f: string }[] = [];
-  for (const r of rows)
-    for (const ref of assetRefsOf(r.kind, JSON.parse(r.body)))
-      all.push({ a: ref.assetId, n: r.id, k: r.kind, f: ref.field });
-  const out: SqlStatement[] = [
-    { sql: "DELETE FROM asset_refs WHERE site_id = ?", params: [siteId] },
-  ];
-  if (all.length > 0)
-    out.push({
-      sql: `INSERT OR IGNORE INTO asset_refs (site_id, asset_id, node_id, node_kind, field)
-            SELECT ?, json_extract(value, '$.a'), json_extract(value, '$.n'), json_extract(value, '$.k'), json_extract(value, '$.f') FROM json_each(?)`,
-      params: [siteId, json(all)],
-    });
-  return out;
-}
-
 function encodeCursor(at: string, id: string): string {
   return btoa(`${at}|${id}`).replace(/=+$/, "");
 }