From cf2c3315f4df5b3c8d3a4ece5883c746f9f7e44a Mon Sep 17 00:00:00 2001 From: Madraka <8898640+Madraka@users.noreply.github.com> Date: Tue, 10 Nov 2026 14:47:46 +0300 Subject: [PATCH 1/8] docs(rfc): RFC-0045 collections stage 4, references and repeaters (#114) Draft RFC for reference fields and repeaters: the contract diff (ReferenceField, RepeaterField, SubFieldDefinition, FieldMigration.sub), the entry_refs reverse index with hash-anchored guards, the locale rule (translation groups), render depth and budgets, Studio blocks, strict structured-output schemas, field operations, rollout, tests and threat model. Docs only; generators not run (phase 1 runs contract-change). Co-Authored-By: Claude Opus 5.5 Signed-off-by: Madraka <8898640+Madraka@users.noreply.github.com> --- ...ctions-stage-4-references-and-repeaters.md | 577 ++++++++++++++++++ 1 file changed, 577 insertions(+) create mode 100644 docs/rfcs/0045-collections-stage-4-references-and-repeaters.md diff --git a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md new file mode 100644 index 00000000..5bad34f6 --- /dev/null +++ b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md @@ -0,0 +1,577 @@ +# RFC-0045: Collections stage 4: reference fields and repeaters + +- Status: Draft +- Author: Halil (Madraka), drafted with Claude · Date: 2026-11-10 +- Related: + - [#114](https://github.com/cloudhal-dev/cloudhal/issues/114) (this stage); #71 (the collections plan). + - RFC-0040 / ADR-0042 (the registry), RFC-0041 / ADR-0043 (typed fields, field operations), RFC-0042 / ADR-0044 (§2 presets, §4 listings and `entry_list_values`, §6 "out: stage 4"). + - RFC-0004 §4.4 (`asset_refs`, the model for the reverse index), RFC-0008 §6.2 and §7 (holds, gates, the gate index), RFC-0010 (snapshot bundles), RFC-0015 / ADR-0030 (the props subset), RFC-0033 / ADR-0035 (one node per locale, translation groups, `localization.fallback`), RFC-0034 (language editions), RFC-0039 / ADR-0041 (archives). + - Prior art (checked 2026-11-10; "unverified" where a primary page could not be read): + - [Sanity reference type](https://www.sanity.io/docs/studio/reference-type): a strong reference (the default) blocks deleting its target and blocks publishing until the target is published; `weak: true` may point at missing or draft documents and dereferences to `null`. `to` is a static list of types; `filter` narrows the picker only and "does not guarantee" the target keeps matching. + - [Payload relationship field](https://payloadcms.com/docs/fields/relationship): `relationTo` one or several collections (polymorphic values become `{relationTo, value}`), `hasMany`, `minRows`/`maxRows`, `filterOptions`, per-field `maxDepth` for population; reverse lookups need a separate Join field. + - Contentful links: resolution up to 10 `include` levels and unresolvable links reported separately (unverified: the concepts page answered 429). + - Strict structured output: [OpenAI](https://developers.openai.com/api/docs/guides/structured-outputs) (types plus `anyOf`, no `allOf`/`not`/`if`; "Root objects must not be `anyOf`"; every property required, optional as a `null` union; up to 5,000 properties, 10 levels of nesting, 1,000 enum values; recursion allowed; `oneOf` not mentioned). [Anthropic](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) (`anyOf`, `$defs`/`$ref` without recursion; no `minimum`/`maximum`/`minLength`/`maxLength`; `minItems` only 0 or 1; across a request at most 20 strict tools, 24 optional parameters and 16 parameters with union types, `anyOf` or type arrays). + +## Summary + +Two new field types join RFC-0041's nine. A `reference` field points to entries of one to five declared collections of the same site, one or up to fifty, and stores bare entry ids, like an `image` field stores asset ids. A `repeater` field is an ordered list of up to fifty items of up to ten flat sub-fields. A repeater cannot contain a repeater, so every agent-facing schema has a fixed depth and no recursion. + +References are strong. A reverse index, `entry_refs`, is written in each entry's own batch, anchored on its new content hash like `asset_refs`. It refuses to delete a referenced entry, and refuses to move one out of the collections its referrers allow. Archiving or unpublishing a target is allowed: the renderer resolves references only against what the published snapshot shows, so a dangling reference renders as nothing and publish warns but never fails. A reference names the target's translation group and resolves to the version in the page's language. Resolution is breadth-first over in-memory snapshot indexes, at most two levels deep, under a budget per page. Studio gets an entry picker and nested repeater forms from new `@cloudhal/ui` blocks, and the four field operations learn the new types. + +## Motivation + +- **Today:** + - An event's venue, a talk's speakers or a product's brand are typed as plain text (RFC-0042 §6: `organizer`, `performer` and `brand` "map from text"). The same venue is retyped on every event. Its address, map and page are not linked, and a change to the venue's name does not reach the events. + - A recipe's ingredients, an agenda's sessions or a job's benefits are multiline text, one item per line (RFC-0042 §2's `recipeIngredient`). There is no second column: an ingredient's quantity, a session's speaker. + - Agents write these as free text, which no theme, listing or JSON-LD preset can read back. +- **Who:** the collections #71 names (events with venues and speakers, team and projects, products and brands, docs and related docs, locations), the themes that render them, and the agents that fill them. +- **Why now:** stages 1–3 shipped (RFC-0040 to RFC-0042, released through 0.7.0). Every competitor in RFC-0041's prior art has references and lists of groups. RFC-0042's preset table already names the paths references fill (`location` as a `Place`, `performer`, `organizer`). + +## Proposal + +### 1. Contract + +All additions are optional variants and `$defs` in `site.schema.json`. `content-entry.schema.json` changes only its `fields` description. No `schemaVersion` bump, no new root type. + +```jsonc +// site.schema.json $defs.FieldDefinition.oneOf (line 274): two variants appended after SelectField (line 542) +{ + "title": "ReferenceField", + "type": "object", + "description": "RFC-0045: entries of other (or this) collections of the same site, by the id of their translation group's original (entry_…).", + "additionalProperties": false, + "required": ["key", "type", "labels", "localized", "collections"], + "properties": { + "key": { "$ref": "#/$defs/FieldKey" }, + "type": { "type": "string", "enum": ["reference"] }, + "labels": { "$ref": "#/$defs/FieldLabels" }, + "localized": { "type": "boolean", "description": "true: each language version picks its own targets; false: only the translation group's original holds them, and every version resolves them in its own language." }, + "requiredToPublish": { "type": "boolean", "description": "An entry in review or published needs at least one target. Absent: false." }, + "migration": { "$ref": "#/$defs/FieldMigration" }, + "collections": { + "type": "array", + "description": "The collections a target may belong to: declared collections of this site (checked in code).", + "minItems": 1, "maxItems": 5, "uniqueItems": true, + "items": { "$ref": "common.schema.json#/$defs/Slug" } + }, + "multiple": { "type": "boolean", "description": "The value is an ordered list of targets. Absent: false." }, + "maxItems": { "type": "integer", "minimum": 1, "maximum": 50, "description": "With multiple: at most this many targets. Absent: 20." }, + "where": { + "type": "array", + "description": "Only with one target collection: conditions on its list fields that a newly chosen target must meet (no `now`; checked in code). Narrows the picker; not re-checked when the target changes later.", + "minItems": 1, "maxItems": 4, + "items": { "$ref": "common.schema.json#/$defs/FieldCondition" } + } + } +}, +{ + "title": "RepeaterField", + "type": "object", + "description": "RFC-0045: an ordered list of items, each a flat object of sub-fields. A repeater cannot contain a repeater.", + "additionalProperties": false, + "required": ["key", "type", "labels", "localized", "fields"], + "properties": { + "key": { "$ref": "#/$defs/FieldKey" }, + "type": { "type": "string", "enum": ["repeater"] }, + "labels": { "$ref": "#/$defs/FieldLabels" }, + "localized": { "type": "boolean", "description": "true: each language version has its own list (missing reads the original's); false: only the original holds the list." }, + "requiredToPublish": { "type": "boolean", "description": "An entry in review or published needs at least one item. Absent: false." }, + "migration": { "$ref": "#/$defs/FieldMigration" }, + "fields": { + "type": "array", + "description": "The sub-fields of every item, in form order. Unique keys (checked in code).", + "minItems": 1, "maxItems": 10, + "items": { "$ref": "#/$defs/SubFieldDefinition" } + }, + "maxItems": { "type": "integer", "minimum": 1, "maximum": 50, "description": "At most this many items. Absent: 20." }, + "itemLabel": { "$ref": "#/$defs/FieldKey", "description": "The text, select or reference sub-field that names an item in Studio and approval prompts. Absent: the first such sub-field." } + } +} + +// site.schema.json $defs (new). Every variant: key, type, labels, then its own; no localized, no migration (the repeater's apply). +"SubFieldDefinition": { + "description": "RFC-0045: one sub-field of a repeater item, discriminated by `type`. Only types whose value is a scalar: no datetime, no multiple select or reference, no repeater.", + "oneOf": [ + { "title": "TextSubField", "type": "object", "additionalProperties": false, "required": ["key", "type", "labels"], + "properties": { "key": { "$ref": "#/$defs/FieldKey" }, "type": { "type": "string", "enum": ["text"] }, "labels": { "$ref": "#/$defs/FieldLabels" }, + "requiredToPublish": { "type": "boolean" }, "maxLength": { "type": "integer", "minimum": 1, "maximum": 5000 }, "multiline": { "type": "boolean" } } }, + { "title": "RichTextSubField", "type": "object", "additionalProperties": false, "required": ["key", "type", "labels"], + "properties": { "key": { "$ref": "#/$defs/FieldKey" }, "type": { "type": "string", "enum": ["richText"] }, "labels": { "$ref": "#/$defs/FieldLabels" }, + "requiredToPublish": { "type": "boolean" } } }, + { "title": "NumberSubField", "type": "object", "additionalProperties": false, "required": ["key", "type", "labels"], + "properties": { "key": { "$ref": "#/$defs/FieldKey" }, "type": { "type": "string", "enum": ["number"] }, "labels": { "$ref": "#/$defs/FieldLabels" }, + "requiredToPublish": { "type": "boolean" }, "integer": { "type": "boolean" }, "minimum": { "type": "number" }, "maximum": { "type": "number" } } }, + { "title": "BooleanSubField", "type": "object", "additionalProperties": false, "required": ["key", "type", "labels"], + "properties": { "key": { "$ref": "#/$defs/FieldKey" }, "type": { "type": "string", "enum": ["boolean"] }, "labels": { "$ref": "#/$defs/FieldLabels" } } }, + { "title": "DateSubField", "type": "object", "additionalProperties": false, "required": ["key", "type", "labels"], + "properties": { "key": { "$ref": "#/$defs/FieldKey" }, "type": { "type": "string", "enum": ["date"] }, "labels": { "$ref": "#/$defs/FieldLabels" }, + "requiredToPublish": { "type": "boolean" } } }, + { "title": "UrlSubField", "type": "object", "additionalProperties": false, "required": ["key", "type", "labels"], + "properties": { "key": { "$ref": "#/$defs/FieldKey" }, "type": { "type": "string", "enum": ["url"] }, "labels": { "$ref": "#/$defs/FieldLabels" }, + "requiredToPublish": { "type": "boolean" } } }, + { "title": "ImageSubField", "type": "object", "additionalProperties": false, "required": ["key", "type", "labels"], + "properties": { "key": { "$ref": "#/$defs/FieldKey" }, "type": { "type": "string", "enum": ["image"] }, "labels": { "$ref": "#/$defs/FieldLabels" }, + "requiredToPublish": { "type": "boolean" } } }, + { "title": "SelectSubField", "type": "object", "additionalProperties": false, "required": ["key", "type", "labels", "options"], + "properties": { "key": { "$ref": "#/$defs/FieldKey" }, "type": { "type": "string", "enum": ["select"] }, "labels": { "$ref": "#/$defs/FieldLabels" }, + "requiredToPublish": { "type": "boolean" }, + "options": { "type": "array", "minItems": 1, "maxItems": 50, "items": { "$ref": "#/$defs/FieldOption" } } } }, + { "title": "ReferenceSubField", "type": "object", "additionalProperties": false, "required": ["key", "type", "labels", "collections"], + "properties": { "key": { "$ref": "#/$defs/FieldKey" }, "type": { "type": "string", "enum": ["reference"] }, "labels": { "$ref": "#/$defs/FieldLabels" }, + "requiredToPublish": { "type": "boolean" }, + "collections": { "type": "array", "minItems": 1, "maxItems": 5, "uniqueItems": true, "items": { "$ref": "common.schema.json#/$defs/Slug" } } } } + ] +} + +// site.schema.json $defs.FieldMigration (line 641), changed +"properties": { + "id": { … unchanged }, + "fromKey": { … unchanged }, + "fromType": { "type": "string", "enum": ["text", "richText", "number", "boolean", "date", "datetime", "url", "image", "select", +- ] }, ++ "reference", "repeater"] }, + "fromLocalized": { … unchanged }, ++ "sub": { "$ref": "#/$defs/FieldKey", "description": "RFC-0045: the repeater's sub-field the operation rewrites (its key once the operation is done); fromKey and fromType then describe the sub-field." } +} + +// content-entry.schema.json properties.fields.description (line 38), changed +- "… otherwise any JSON. An absent key means no value." ++ "… otherwise any JSON. An absent key means no value. A reference value is an entry id or a list of them, a repeater value a list of flat objects (RFC-0045)." +``` + +- **Values:** + + | Type | Value | Checked on write | + |---|---|---| + | `reference` | `"entry_…"`, or with `multiple` a list of them, no repeats, ≤ `maxItems` | each id is an entry of this site in one of `collections`, is its translation group's original in this site (a same-site translation's id is canonicalized to the original's, as `canonicalImageRefs` does for assets), is not the entry itself, and meets `where`; checked for changed keys only (RFC-0041 §3) | + | `repeater` | a list of ≤ `maxItems` objects; each holds only sub-field keys, with RFC-0041 §2's values for its type; no empty item | each item against the sub-fields; `requiredToPublish` sub-fields need a value in every item of an entry in review or published | + +- **Why these shapes.** + - A bare id (not `{collection, id}`, as Payload's polymorphic values) keeps a reference what an image field already is. The id names its node and so its collection, and the store re-checks the collection on every write. + - Repeater items are flat objects of scalars, which is exactly ADR-0030's `ArrayPropSchema` with `FlatObjectPropSchema` items (`packages/core-ts/src/blocks.ts:55-68`). `fieldPropSchema` compiles a repeater into the subset, and `validateProps` checks it unchanged, eval-free (rule 6). + - Excluded from sub-fields: `datetime` (an object) and multiple selects and references (arrays). These would need a second nesting level in the subset (open question 3). + - `maxItems` ≤ 50 is `BLOCK_LIMITS.arrayItems` (`blocks.ts:113`). +- **Code rules beyond the schema** (`checkFieldDefinitions`, `definitions.ts:177`): + - every `collections` entry is a declared or inferred collection (`reference_target_unknown`); + - `where` only with one target collection, on its `listFields`, without `now` (`invalid_query`); + - `maxItems` only with `multiple`; + - sub-field keys unique in their repeater and not reserved, with the same list as fields; + - `itemLabel` names a text, select or reference sub-field; + - at most 100 *leaves* per collection (fields plus sub-fields); + - sub-fields count toward the per-site 200 (`FIELD_LIMITS`, `types.ts:32`); + - at most 100 reference values per entry across fields and items (`too_many_references`). +- **Adding a reference or repeater field is a free edit**, except over a key some entry already holds (`field_key_in_use`, one `json_type` count). This guarantees that every reference value in the working graph was written through §2's checks and has its index rows. +- **Rule 2 holds:** + - `oneOf` with inline variants and the one-value `type` discriminator (ADR-0043); + - no `default`; + - `additionalProperties: false` everywhere; + - cross-file `$ref` only to `common.schema.json#/$defs/*`; + - no recursion: `SubFieldDefinition` has no repeater variant, so depth is fixed by the contract. +- **Generation impact:** + - **TypeScript:** `FieldDefinition` gains `ReferenceField | RepeaterField`, and a new `SubFieldDefinition` union of nine `*SubField` interfaces. `FieldType` (`types.ts:7`) is derived from the contract, so every total dispatch table over it stops type-checking until it handles the two types (`COMPILE`, `props.ts:45`; `EMIT`, `json-schema.ts:49`). The compiler lists the work. + - **Rust:** cargo-typify emits `#[serde(tag = "type", deny_unknown_fields)] enum FieldDefinition` with two more variants, a second tagged enum `SubFieldDefinition`, and the extended `fromType` enum. Values stay `serde_json::Value`. + - **Naming:** the sub-field variants carry their own titles (`TextSubField`, not `TextField`). Both generators name types from `title`, and RFC-0042 §1 found that cargo-typify 0.8.0 keeps only the first of two equal names. + - **Not run on this branch.** This is a docs-only PR, written while the owner's machine ran a test-heavy session. Phase 1 (§9) runs `contract-change` first, with these fixtures: + - `valid/site/collection-stage4.json`: an `events` collection with `venue` (reference to `places`), `speakers` (repeater with a reference, a select and a text sub-field) and a migration marker with `sub`; + - `invalid/site/`: + - reference without `collections`; + - six target collections; + - `maxItems` 51; + - a repeater sub-field of type `repeater`; + - one of type `datetime`; + - a sub-field with `localized`; + - eleven sub-fields; + - `fromType` `blocks`. + +### 2. Cross-node integrity: the reverse index + +```sql +-- packages/storage/migrations/0033_entry_refs.ts (new; DDL only, nothing to backfill: no reference values exist yet) +CREATE TABLE entry_refs ( + site_id TEXT NOT NULL, + target_id TEXT NOT NULL, -- the referenced entry: its group's original in this site + source_id TEXT NOT NULL, -- the referring entry node (any locale) + source_collection TEXT NOT NULL, + field TEXT NOT NULL, -- "venue", or "speakers.person" inside a repeater + source_status TEXT NOT NULL, -- the referrer's status, for "used by published entries" + PRIMARY KEY (source_id, field, target_id) +); +CREATE INDEX entry_refs_target ON entry_refs (site_id, target_id, source_status); +``` + +Each statement below follows a pattern that already exists. The rows of an entry are `asset_refs`'s fields rows (`store/entry-fields.ts:119-144`) with another table. + +1. **Writing a referrer** (`EntryEngine.write`, `store/entries.ts:30`). `planEntryFields` returns two more parts. + - A **guard**, ANDed into the entry's CAS (`entries.ts:104`): every target exists, in this site, is a `contentEntry`, and is in the field's allowed collections at write time. It is one count over a JSON list of `{t: id, c: [collections]}` pairs, like the asset guard (`entry-fields.ts:163`): + ```sql + (SELECT count(*) FROM json_each(?) p JOIN nodes n ON n.id = json_extract(p.value, '$.t') + WHERE n.site_id = ? AND n.kind = 'contentEntry' + AND json_extract(n.body, '$.collection') IN (SELECT value FROM json_each(json_extract(p.value, '$.c')))) = ? + ``` + - **Statements** anchored on the entry's new hash: `DELETE FROM entry_refs WHERE source_id = ? AND EXISTS (SELECT 1 FROM nodes WHERE id = ? AND hash = ?)`, then one `INSERT … SELECT … FROM json_each(?) WHERE `. They run only if the entry's own write did (rule 4), and are appended to `own` in `entries.ts:110`. + + A target deleted or moved between the check and the batch fails the guard; `diagnose()` then names it (`unknown_entry`, `reference_target_moved`). One bound parameter carries the whole list, so D1's 100-parameter limit does not apply. 100 ids of at most 48 characters stay far below its 100 KB per statement. +2. **Deleting a target** (`EntryEngine.delete`, `entries.ts:162`). The node's `DELETE` gains `AND NOT EXISTS (SELECT 1 FROM entry_refs WHERE target_id = ? AND source_id <> ?)`, as an asset's delete guards on `asset_refs` (`assets.ts:867`). A refused delete answers `409 entry_referenced`, with up to 20 referrers (id, title, collection, field) and the total. + - Only a group's original can be a target, so deleting a translation is never blocked. + - The referrer's own rows go with its delete: `DELETE FROM entry_refs WHERE source_id = ? AND NOT EXISTS (SELECT 1 FROM nodes WHERE id = ?)`, as `dropRefsStatement` (`assets.ts:1015`). +3. **Moving a target to another collection** (`entersCollection`, `entries.ts:101`). The write computes, from the site node it read, every `"."` pair whose allowed collections include the new one. It is guarded with: + - `NOT EXISTS (SELECT 1 FROM entry_refs WHERE target_id = ? AND source_collection || '.' || field NOT IN (SELECT value FROM json_each(?)))`; + - the site's hash as read (`EXISTS (SELECT 1 FROM nodes WHERE id = ? AND hash = ?)`), so a concurrent change to the definitions cannot slip between. + + Refused: `409 reference_target_moved`, naming the referring fields. During a field operation, the pairs include the marker's `fromKey` as well as the new key. +4. **Archiving and unpublishing a target** are allowed and write nothing to `entry_refs`. An archived target stays pickable for existing values only. The picker hides archived entries, and the write check refuses a *newly* chosen archived target (`retired`-asset rule, `assets.ts` `planRefs`). +5. **Rebuilds** follow `refsRebuild` (`media/refs-rebuild.ts:12`) in the same batch as the node set: `importSite`/`pull`/`push` (`sync-plan.ts:308`), `cloudhal import --replace` and `--revert`, and snapshot restores. `entryRefsRebuild(siteId, rows)` chunks its `INSERT` so each statement stays below 90 KB, still inside the one atomic batch. `asset_refs`'s rebuild needs the same chunking, which is a separate fix (follow-ups). +6. **Field definition changes** (`fieldDefinitionChanges`, `definitions.ts:135`) add these refusals: + - removing a collection from `collections` while a value points into it (an `entry_refs` join count); + - turning `multiple` off; + - removing or retyping a sub-field that holds values. + + Each refusal names the operation to use (§7). + +**Invariant.** The store never *creates* a dangling reference: deleting a target, moving it, and writing a reference to a missing target are all guarded in the same batch. Dangling references can still *exist*: +- in archives and bundles written by hand or by another version; +- in publish snapshots composed with holds, which leave out a held node that is not live (`store/snapshots.ts:29-37`); +- after a release applies `nodes(L) − ids + targets`. + +Every reader therefore treats a missing target as unresolved and never throws. + +**Snapshots, publish and rollback:** + +- **A snapshot is the whole working copy at one read** (`snapshots.ts:23`), so referrers and targets are taken together. Rollback moves the pointer to an ancestor snapshot (`store.ts:1127`), which is just as consistent. Restores and `import --revert` rebuild the index (point 5). +- **Visibility decides, not existence.** A snapshot holds drafts and archived entries too, so the renderer resolves only against entries the snapshot *shows* at that instant: `Loaded.reachable` (`astro-renderer/src/source.ts:147`): published, gates open, not `hide`-expired. + - An `unlist`ed target (RFC-0042 §3) still resolves: its page exists. + - A draft, archived, gated or missing target resolves to nothing. `EntryFields.astro` then shows the field only if other targets remain. +- **Publish warns, never fails.** `referenceIssues(snapshotNodes)` (pure, `@cloudhal/core-ts/fields`) lists published entries whose references will not resolve: target missing, draft, archived, gated, or no version in a needed locale. It appears in: + - Studio's publish dialog ("3 published entries point to entries that are not published"); + - the MCP `publish` tool's approval text; + - the output of `cloudhal snapshot --publish`. + + A build or publish never fails on it, for the same reason as RFC-0042 §4: "a published snapshot must keep rendering". Sanity's strong references block publishing per document, but Cloudhal publishes a site snapshot. A warning keeps a person in charge without making an unrelated fix wait for a venue page. +- **The gate index already covers targets.** It holds every instant at which a published entry's gate changes (`source.ts` `Loaded.gates`), so a referrer's page expires from the edge cache when its target's embargo opens or its `hide` expiry passes, with no new mechanism. +- **Purges.** A referrer's response adds `cloudhal-node-` for every target it rendered, as it already does for "the related entries an article links to" (`astro-renderer/src/cache-tags.ts:6-8`). A publish that changes a target's title therefore purges its referrers' pages. The per-page budget (§4) keeps the tags far below the 8 KB header cap. + +**Archives (RFC-0039).** +- An archive holds every node of the site, so references stay complete. Import rebuilds the index in its node batch. +- Dangling references are reported, not refused, as RFC-0041 open question 8 decided for invalid values: `cloudhal import --dry-run` adds `referenceIssues` per collection, and Studio lists them on the collection screen. +- Language editions (RFC-0034) never share ids across sites (§3). + +### 3. The locale rule + +**A reference names a translation group, not an entry-locale.** The stored id is the group's original in this site (§1). Resolution then picks the group member in the page's language: + +1. the member of group `G` with `locale === page locale` that the snapshot shows; +2. otherwise, when `Site.localization.fallback` is `default-with-notice` (RFC-0033 §2), the member in the site's default locale, flagged `fromFallbackLocale`, so themes can set `lang` on it; +3. otherwise, nothing. + +The same rule serves all three readers: the renderer, Studio's preview of a translation, and `referenceIssues`, which reports "Main hall has no Turkish version". + +- **`localized: false` is the default** that Studio pre-sets for references. The original holds the ids, and every translation resolves them in its own language (RFC-0041 §5). A translator never re-picks targets, and a venue's Turkish version appears on the Turkish event without any write. +- **`localized: true`** is for real per-language differences, such as a regional partner. Each version holds its own ids, still group ids, and an empty value falls back to the original's (`fromOriginal`). +- **Repeaters are localized as a whole.** With `localized: true`, each version has its own list, and the original's is read while its own is absent. With `localized: false`, the list lives on the original only. Text sub-fields then show the original's words on every version, and Studio says so ("Not translated: this list is shared by every language"). + - Per-sub-field localization inside a shared list would need stable item keys. It is open question 2. +- **Machine translation:** + - `get_translation_source` returns a localized repeater's `text`/`richText` sub-values at their paths (`speakers/1/talk`); + - `save_translation` writes them back only if the list has the original's length (`translation_shape_mismatch` otherwise), copying the other sub-values; + - reference values are never sent for translation. +- **Language editions** (an original in another site of the project). Non-localized values are copied when the edition is made (RFC-0041 §5). Each referenced id is mapped to the edition site's member of the same translation group. Ids with no such member are dropped, and the result reports them ("2 references not carried: Main hall has no edition in site B yet"). A reference never crosses sites. + +### 4. Render cost + +`resolveFields` (`fields/resolve.ts:68`) stays pure. A reference resolves to its id or ids, and a repeater to its items, each sub-value checked as a field is (an invalid one is `undefined` with an issue). Loading targets is the renderer's job, in `entryFields` (`astro-renderer/src/fields.ts:172`). + +```ts +// @cloudhal/astro-renderer (additive) +export interface ReferencedEntry { + id: string; groupId: string; collection: string; locale: string; slug: string; + title: string; summary?: string; url: string; image?: FieldImage; // the target's cover image + fromFallbackLocale: boolean; + /** Present only when depth 2 was asked for: the target's own fields, its references as cards without fields. */ + fields?: RenderedFields; +} +interface RenderedField { // existing interface, two optional additions + references?: ReferencedEntry[]; // reference: resolved targets, in stored order (unresolved ones left out) + items?: RenderedFields[]; // repeater: each item's sub-fields, rendered like fields +} +interface EntryFieldsOptions { /* existing */ depth?: 1 | 2; } // default 1 +interface GraphSource { + /** Visible entries by group id in one locale (one call per resolution level). */ + entriesByGroups?(groupIds: readonly string[], locale: string): Promise>; + /** "Who references me": visible referrers of a group, newest first, at most 100. */ + referencedBy?(groupId: string, q?: { collection?: string; field?: string; locale?: string; limit?: number }): Promise; +} +``` + +- **Depth: 1 by default, 2 at most.** Depth 1 yields cards: title, summary, URL and cover image, which is what a venue link or a speaker list draws. Depth 2 adds each target's own fields, with *its* references as cards without fields. Nothing goes deeper, so cycles (A→B→A) cost nothing: resolution is breadth-first over a visited set of group ids. Payload's per-field `maxDepth` and Contentful's `include` levels are the same idea. +- **Fan-out:** + - by contract: 50 per field and 50 items per repeater; + - in code: 100 reference values per entry (§1); + - per render: a page budget of **200 resolved targets**, shared by every `entryFields` call of one request (the `MediaContext.budget` object already carries per-request budgets). + + Over budget, remaining targets resolve to nothing with one `console.warn`, never an error. A listing of 20 events at depth 1 with one venue each costs 20. +- **Batching:** + - **Snapshot source:** the loaded snapshot gains two indexes, built once per snapshot and isolate beside `assetIndex` (`source.ts:147`): `byGroup: Map>` over `reachable`, and a lazy reverse map for `referencedBy`. Every lookup is in memory, with no reads. Cards are cached per `(groupId, locale)` beside `Loaded.resolved` (the snapshot is immutable). Bundles (RFC-0010) need nothing new, because entries are already in them. + - **Live source** (`cloudhal dev`, previews): one store read per level, `getContentEntriesByGroups(siteId, ids)` with a `json_each` IN list, plus one `entry_refs` probe for `referencedBy`. A depth-2 page therefore costs at most two reads, whatever its fan-out. +- **Listings** (RFC-0042 §4): + - **Filter by a reference: yes.** A `reference` field may be a list field, with ops `eq`, `ne`, `in`, `set` and `unset` on target ids. It serves "events at this venue" and "talks by this speaker". Its `entry_list_values` rows put the id in `text` (an id is at most 48 characters, under the 100-unit limit). A multiple reference writes one row per id with `has = 2`, exactly like a multiple select's options (migration 0029's doc comment). `checkFieldQuery` and the property suite gain the type. + - **Sort by a reference, or filter by a referenced entry's field** ("events whose venue is in Istanbul"): **no.** Either needs the target's values denormalized into the referrer's rows, which every target edit would invalidate across collections. Agents and themes do it in two steps: `queryEntries` on `places` for `city eq istanbul`, then `venue in [...]` (≤ 20 values). + - **Repeaters** cannot be list fields (`list_field_type`). + - **Theme-side "used by"** is `referencedBy`, not a query on the referring collection, so no list field is needed for it. +- **Structured data** (RFC-0042 §2), phase 7: a fifth source kind, `{ "kind": "reference", "field": "venue", "property": "title" | "url" | "summary" }`. It fills `location.name`, `performer.name`, `organizer.name`/`url` and `brand.name` from a depth-1 card. It is a new `StructuredDataSource` variant with its own named `property` enum, `ReferenceValueProperty`, as §1 of that RFC requires for Rust. + +### 5. Studio + +All new UI follows `kumo-design`, `studio-ui` and DESIGN.md "Fields", imports only from `@cloudhal/ui`, and puts its words in the lazy `fields` namespace. + +**New blocks in `packages/ui/src/blocks`:** + +| Block | Built from (Kumo) | What it does | +|---|---|---| +| `EntryPicker` | `Dialog`, `Input` (search), `Combobox` list semantics, `Badge`, `Checkbox`, `Empty`, `Loader` | Search a site's entries in the allowed collections: title, collection badge, status badge, locale. Single or multiple choice, at most the remaining `maxItems`. Takes a `search(q)` prop, so the block makes no API calls. | +| `ReferenceList` | `InsetPanel` rows (existing block), `Button`, `Badge`, `DropdownMenu`, `Link` | The chosen targets in order: open, remove, move up/down. A "Draft" or "Archived" badge with "Will not show until published", and "Not found" for dangling ids. | +| `RepeaterField` | `Collapsible`, `Button`, `DropdownMenu`, `Text`, `Empty` | The list of items with "Add item", a count ("3 of 20") and per-item move/duplicate/remove. Each item renders the sub-field controls through a `renderControl` prop. | +| `FieldControl` | (extracted) | One control per type, taken out of `EntryFieldsForm.tsx`, which is already over the soft 400-line limit at 424 lines. `EntryFieldsForm` and `RepeaterField` then share it, and neither file grows. | +| `UsedByList` | `Table` or the `resource-list` Kumo block, `Badge`, `Link` | The entry editor's "Used by" card and the delete dialog's list of referrers. | + +- **Entry editor.** + - `EntryFieldsForm` gains a `renderReference?: (slot: FieldReferenceSlot) => ReactNode`, the same pattern as `renderImage` (`EntryFieldsForm.tsx:61-69, 97`). Studio's `FieldReference` (in `apps/studio/src/components/fields/`) composes `ReferenceList` and `EntryPicker` around `api.entries.pick`. + - Repeaters render inside the form with `RepeaterField`. + - On a translation, a non-localized reference or repeater is read-only, with "From the original" and a link (RFC-0041 §7). A localized one shows the original's value as a placeholder until edited. + - A new "Used by" `SectionCard` lists referrers (`GET …/entries/:id/references`). + - The delete dialog lists them too and disables Delete while any exist ("Remove it from these 4 entries first"). +- **Fields builder** (`FieldDialog.tsx`): + - **reference:** target collections (multi-select of declared collections), "Several" (`multiple`), the maximum, and optional conditions when one collection is chosen; + - **repeater:** a nested sub-field list (rows like `FieldRow` with up/down and ⋯), the maximum and the item label; + - the ⋯ menu's operations gain the sub-field forms (§7). +- **API** (site-scoped, under the existing `/sites/:siteId` membership check): + - `GET /sites/:siteId/entries/pick?collections=a,b&q=&locale=&limit=` returns at most 20 `{id, title, collection, status, locale}` per page, never archived ones. Scope `content:read`. `q` is at most 100 characters, matched as a case-folded substring (`includes`, no regex: rule 13) on titles and slugs. + - `GET /sites/:siteId/entries/:id/references` returns the referrers from `entry_refs`, paged. +- **Accessibility:** + - Each repeater item is a `fieldset` whose `legend` is its item label ("Speaker 2: Ada Lovelace"), as an ICU `selectordinal`/`plural` message, never concatenated. + - Moving is done with buttons ("Move Speaker 2 up"), never drag only. + - After "Add item", focus moves to the new item's first control. After "Remove", it moves to the next item's legend, or to "Add item" when none is left. A removal shows an undo toast. + - `EntryPicker` labels its search field, announces the result count in a polite live region, supports arrow keys and Enter in the list, and returns focus to the trigger on close. + - Errors map to their controls by path (`fields/speakers/2/person`) and are listed above the form when they sit in collapsed items. Those items open on submit. +- **Mobile (390 px):** + - item actions fold into one `DropdownMenu` (⋯); + - sub-fields stack one per row; + - `EntryPicker` takes the full width; + - `ReferenceList` rows wrap the badges under the title. + + `pnpm design:shots` at 1440 and 390 must show no sideways scroll, and `design:lint` (including `rtl-logical`) must pass. +- **Bundle:** the blocks load with `EntryFieldsForm`, which loads only for collections with fields and is already lazy. The initial bundle stays at +0 kB (`bundle-budget.ts`). + +### 6. MCP and the JSON Schema agents receive + +- **Tools.** + - `list_collections` returns the new definitions and their schemas. + - `create_entry`/`update_entry` precheck the reference and repeater rules with the store's checker, so a call the store would refuse never reaches a person. + - Approval prompts name targets readably: `venue → "Main hall" (entry_…, Places)`, `speakers: 2 items → 3 items (added "Ada Lovelace", Speaker)`, values quoted and truncated (`quote()`). + - `delete_entry` refuses in precheck with `entry_referenced` and the referrers. + - New read-only `list_references { siteId, entryId, direction: "to" | "from" }` (`content:read`): "who references me" and "what I reference", for an agent about to archive or delete. + - `list_entries` keeps working as the agent's picker, with `where: [{field: "venue", op: "eq", value: "entry_…"}]` once the field is listed. +- **`update_collection`'s input** embeds the contract's `FieldDefinition` `oneOf` inside `fields.items` (`mcp-server/src/collection-fields.ts:13-31`). OpenAI's strict mode does not document `oneOf`, so the MCP input schema emits the same variants as **`anyOf`**. Every variant has a distinct one-value `type` enum, so at most one can match and `anyOf` means exactly what `oneOf` does. The contract itself keeps `oneOf` (rule 2). +- **`fieldsSchema`** (`fieldsJsonSchema`, `json-schema.ts:145`) emits: + - `reference` as `{type: "string", pattern: "^entry_[A-Za-z0-9]{6,32}$"}`, or an array of them, with the target collections and `maxItems` in words; + - `repeater` as `{type: "array", items: {type: "object", properties: …, additionalProperties: false}}`. + + The constraints of the emitted schemas: + - the root is always `type: "object"`, never `oneOf` or `anyOf`; + - no `$ref`, `$defs` or recursion: depth is at most 4 (fields → list → item → value), against OpenAI's 10; + - no unions in the default output; + - limits are in `description`, as today, because Anthropic drops `minimum`/`maxLength` and accepts `minItems` only as 0 or 1. + + `JSON_SCHEMA_KEYWORDS` (`json-schema.ts:29`) is unchanged, and the test holding the emitter to it stays. +- **A strict profile.** OpenAI's strict mode needs every property `required`, with optional ones as a `null` union. Today's output leaves properties optional, which suits Anthropic and non-strict MCP calls. `fieldsJsonSchema(defs, locale, defaultLocale, { strict: true })` therefore emits every property as required with `type: [T, "null"]`, plus `enum` with `null` for selects. `null` already means "leave unset or remove" in `update_entry`'s merge patch (`mcp-server/src/entry-fields.ts:45`), so the two profiles mean the same thing. + - `list_collections` returns `fieldsSchemaStrict` and `strictCost: { properties, depth, enumValues, unions }` beside it, counted by `strictSchemaCost(schema)`. + - Anthropic allows 16 union-typed parameters per request, so a collection with more than 16 leaves cannot be written in one strict Anthropic call, today included. The caller splits `update_entry` by keys, as RFC-0041 open question 3 decided. MCP tool calls themselves stay non-strict. +- **Example.** An `events` collection with `startsAt` (datetime), `venue` (a reference to `places`) and `speakers` (a repeater of `person` → `people`, `role` select, `talk` text): + + ```json + { + "type": "object", + "description": "Typed custom fields of this collection. Every field is optional; leave a key out to leave it unset.", + "properties": { + "startsAt": { + "type": "object", + "properties": { + "at": { "type": "string", "format": "date-time", "description": "The instant in UTC, RFC 3339 ending in Z, e.g. 2026-11-05T16:30:00Z." }, + "timeZone": { "type": "string", "description": "The IANA time zone it happens in, e.g. Europe/Istanbul." } + }, + "required": ["at", "timeZone"], "additionalProperties": false, + "title": "Starts", "description": "A date and time with its time zone. Set on the original only; translations read it from there." + }, + "venue": { + "type": "string", "pattern": "^entry_[A-Za-z0-9]{6,32}$", + "title": "Venue", + "description": "The id of an entry in Places (places) of this site, as list_entries returns it; a translation's id is replaced by its original's. Set on the original only; every language version shows the venue in its own language." + }, + "speakers": { + "type": "array", + "title": "Speakers", + "description": "An ordered list of at most 20 items. Set on the original only.", + "items": { + "type": "object", + "properties": { + "person": { "type": "string", "pattern": "^entry_[A-Za-z0-9]{6,32}$", "title": "Person", "description": "The id of an entry in People (people). Needed to publish." }, + "role": { "type": "string", "enum": ["host", "speaker", "panelist"], "title": "Role", "description": "One of: host (Host), speaker (Speaker), panelist (Panelist)." }, + "talk": { "type": "string", "title": "Talk", "description": "Plain text, at most 200 characters. One line." } + }, + "additionalProperties": false + } + } + }, + "additionalProperties": false + } + ``` + + In the strict profile, each property above is listed in its object's `required`, and `venue`, `role` and `talk` read `"type": ["string", "null"]` (`role`'s `enum` gains `null`). `startsAt` reads `["object", "null"]`, and `speakers` reads `["array", "null"]`. `strictCost` is `{ properties: 8, depth: 4, enumValues: 4, unions: 6 }`, within both providers' limits. + +### 7. Field operations (`fields-operations/`) + +The four operations, their phases, their budgets and their revert stay as RFC-0041 §6 built them (`fields-operations/types.ts:22-50`). Entry rewrites carry their `entry_refs` rows, anchored on the new hash, so the index follows every phase B batch with no separate pass. A repeater's sub-field is addressed with a new optional `sub` parameter (`rename_field {from: "speakers", sub: "name", to: "fullName"}`), and the marker records it (`FieldMigration.sub`, §1). + +| Operation | Reference | Repeater | +|---|---|---| +| `rename_field` | moves the key; rows' `field` follow each rewritten entry; list fields and sections follow as today (`fields-operations/references.ts`) | moves the key, or with `sub` the sub-key in every item; rows `speakers.person` follow | +| `change_field_type` | see the table below | see the table below; with `sub`, a sub-field converts by RFC-0041's table among sub-field types | +| `set_field_localized` | true→false keeps the original's ids and deletes translations' copies (and their rows); false→true writes nothing | the same, for whole lists | +| `remove_field` | deletes values and their rows; refused while list fields or presets use it (`list_field_in_use`) | deletes the list, or with `sub` the sub-value in every item (an item left empty is dropped) | + +New rows of the closed conversion table (`conversions.ts:38`, `FIELD_CONVERSIONS`): + +| from → to | conversion | unconvertible | +|---|---|---| +| reference single ↔ multiple | as select: one id ↔ a list of one; an empty list is no value | multiple → single with more than one id | +| text → reference | the one entry of the allowed collections whose slug, or default-locale title, equals the text exactly (case-folded) | no match, or more than one | +| reference → text | the target original's title in the site's default locale (a list: comma separated) | a missing target | +| text (multiline) → repeater | one item per non-empty line into the repeater's single `text` sub-field (RFC-0042's ingredient lines) | a line over the sub-field's `maxLength`; more lines than `maxItems` | +| repeater → text | a repeater of one `text`/`richText` sub-field: items joined by line breaks (multiline) | other sub-field shapes | +| reference (multiple) ↔ repeater | a repeater of one `reference` sub-field: one item per id, and back | a repeater with other sub-fields | + +Anything else is refused as today. Without `onInvalid: "clear"`, unconvertible values refuse the run, and the dry run counts and samples them first. + +- **Revert** (`fields-operations/revert.ts`) restores each touched entry's earlier values. A restored reference whose target was deleted meanwhile is dropped from the restored value, and its id is reported in a new `FieldRevertResult.dropped: { entryId, field, targetId }[]`. The restore statement carries §2's target guard, so a target deleted during the revert fails that statement and is retried on the next pass. A remove can make a target deletable, and that deletion must not make the revert write a dangling id. +- **Free definition edits:** + - add a sub-field; + - reorder sub-fields; + - change labels, help, `itemLabel` or `where`; + - widen `maxItems` or `collections`; + - narrow `maxItems`: existing values are checked on their next write, as RFC-0041 §6 already does for narrowed limits. + +### 8. Threat model + +| Threat | Mitigation | Test | +|---|---|---| +| **A draft, archived, gated or hidden target leaks through a published referrer** (the snapshot holds drafts, §2) | Resolution reads only `Loaded.reachable`. A card carries only the target's public title, summary, URL and cover image, and the image goes through the existing asset gates. Unresolved values render nothing: no id, no `data-` attribute, no empty link. JSON-LD's `reference` source (phase 7) uses the same cards. | RFC-0008's route sweep (extended by RFC-0042 for `hide`) is extended again: a published referrer pointing at a draft, an archived, an embargoed and a `hide`-expired target. No id, slug, title or asset of those targets may appear in HTML, JSON-LD, feeds, the sitemap, search, `llms.txt`, status pages or cache tags. | +| **IDOR in the entry picker and on write** | Pick and references routes run under the `/sites/:siteId` membership check and query by `site_id` only. The write guard requires `n.site_id = ?` (§2.1). An id from another site or tenant gets the same `unknown_entry` as an id that does not exist, so the answer reveals nothing. Language editions map ids and never read across sites (§3). | A shared-suite case: another site's entry id in `create_entry`, `update_entry`, the API and `push` is refused identically. The pick route with another site's id in the path answers 404. | +| **Fan-out DoS on render** | Contract caps (50 per field, 50 items, 10 sub-fields), the code cap (100 reference values per entry), depth ≤ 2, the visited set, and 200 targets per page. Live reads are ≤ 2 per page, snapshot reads 0. `referencedBy` is capped at 100. | A property test: random graphs with cycles and 10× the caps resolve within the budget, never throw, and cost at most `depth` store reads on the live source. | +| **Write amplification and slow guards** | Each guard is one indexed probe or one `json_each` count (`entry_refs_target`). Delete and move cost one probe. Rebuilds are chunked statements inside one batch (§2.5). | The benchmark of RFC-0042 #105, extended to 10,000 events × 3 references on Miniflare D1: write, delete and move guard costs are reported. | +| **Picker search abuse** | `q` ≤ 100 characters, a substring match with no regex (rule 13), at most 20 results per page, and a per-member window on the pick route through the existing `rate_limits` counters (`packages/api/src/rate-limits.ts`, migration 0030). | A 100k-character `q` is refused before it is read. | +| **Prompt injection through target titles** | Titles are data. Approval prompts quote and truncate them (`quote()`, 60 characters), and `list_references` marks them as content, never instructions. | Existing approval-prompt tests gain reference cases. | +| **A broken archive or push slips dangling ids in** | Import validates nodes (`assertValidRows`) and rebuilds the index. Dangling ids are reported, never resolved and never thrown on. | The archive round-trip test with a hand-edited dangling id. | +| **An older site Worker crashes on the new types** (`COMPILE[def.type]`, `props.ts:45`, has no entry) | `GET /__cloudhal/features` adds `{ "fields": 2 }`. Writers refuse reference and repeater definitions with `409 fields_unsupported` until the site Worker reports it, as `gates: 2` guards `expiry` (`availability.ts:243`, `cloudflare/src/features.ts`). Phase 1 also makes `resolveFields` skip unknown types, so later types are safe on older renderers. | The site Worker features test; a renderer test with an unknown field type. | + +No new scopes. Field definitions remain site settings (`site:settings`; agents use `site:write` plus a person's approval, `person`-classed, as today, `field-tools.ts:264-267`). Values are content (`content:write`, with the `:own` limits). Reading targets in the picker and in `list_references` needs `content:read`, which every role already holds (`agent-runtime/src/roles.ts` `READ`). Referencing an entry never writes it. + +### 9. Rollout (one PR each) + +1. **Contract and pure core:** + - the §1 diff, fixtures and `pnpm gen`, through the `contract-change` skill; + - definition checks, `fieldPropSchema` for both types, value checks, `resolveFields` (and skipping unknown types); + - `fieldsJsonSchema` with the strict profile and `strictSchemaCost`; + - `referenceIssues`. + + No behaviour change. +2. **Storage and store:** + - migration `0033_entry_refs`; + - the write guard and rows, and the delete and move guards; + - the definition-change refusals; + - `entryRefsRebuild` in import, sync and restore (chunked), and `referenceIssues` in `import --dry-run`; + - the `fields: 2` writer guard; + - API: entry routes, `pick` and `references`. +3. **Renderer:** + - `entryFields` cards, depth and budget; + - `entriesByGroups`/`referencedBy` on both sources, and the snapshot indexes; + - `EntryFields.astro` (links for references, a nested `
` per item); + - cache tags, and features `fields: 2`; + - the route sweep extension; + - a frozen theme copy for `old-theme-compat.test.ts`. +4. **Field operations:** `sub`, the conversion rows, revert `dropped`, and the CLI flags (`cloudhal fields rename|retype|localize|remove --sub`). +5. **MCP:** + - `list_collections` (definitions, `fieldsSchema`, `fieldsSchemaStrict`, `strictCost`); + - the entry prechecks, `delete_entry`'s refusal and `list_references`; + - `anyOf` in `update_collection`'s input; + - the operation tools' `sub`; + - regenerated tool docs (`node scripts/mcp-tools-doc.ts`). +6. **Studio:** the five blocks, `FieldReference`, the builder, "Used by", the delete dialog and the publish warnings; `design:shots`; the `fields` namespace and `pnpm i18n:import`. +7. **Listings and structured data:** reference list fields (rows, `checkFieldQuery`, the property suite), the `reference` source kind for presets, and `docs/guides/fields.md`. + +Phase 3 ships before phases 5 and 6, so no agent or person can define a type that the live site cannot render; the `fields: 2` guard enforces it anyway. Phase 7 is independent of 4 to 6. + +**Test plan.** Every store-facing behaviour goes into the shared suites (`packages/core-ts/src/testing-*.ts`), which run on SQLite, Miniflare D1 and Durable Object SQLite: +- `testing-references.ts` (new): + - writing, canonicalizing a translation's id to the original's; + - refusing a missing, cross-site, wrong-collection, self or archived new target; + - delete and move refusals, and the races of each guard against a concurrent write (two batches interleaved); + - rows after every write, delete, import, restore and operation phase. +- `testing-fields.ts`: repeater values, `requiredToPublish` per item, the leaf and reference caps. +- `testing-field-operations.ts`: every new conversion row, `sub` operations, resume, and revert with a deleted target. +- `testing-list-values*.ts`: reference list fields, including the seeded random comparison of SQL and evaluator. +- **Pure tests:** the emitter against `JSON_SCHEMA_KEYWORDS`, depth ≤ 4, no recursion, root `object`, `strictCost`; `referenceIssues`; the locale rule; resolution budget and cycles (property-based). +- **Renderer:** snapshot and live sources give identical cards; the route sweep (§8); cache tags. +- **Studio:** block tests (keyboard, focus after add and remove, live region), `design:shots` at both widths, `design:lint`. + +## Alternatives considered + +| Option | Pros | Cons | Why not | +|---|---|---|---| +| Weak references by default (deletes allowed, dangling ids kept) | No refusals for people | Silent breakage; "who uses this" still needs the index; Sanity made strong the default | Strong by default; readers tolerate dangling anyway (§2) | +| Cascade or "unlink and delete" on target delete | One click | Rewrites other people's entries as a side effect of a delete; needs an operation, approval and a revert of its own | Refuse and list referrers; open question 1 | +| Block publish (or fail the build) on unresolved references | Strict | Publish is a site snapshot; one draft venue would hold back every unrelated fix; "a published snapshot must keep rendering" | Warn in Studio, MCP and CLI (§2) | +| Reference an entry-locale (a specific node) | Exact; no resolution step | Every translation re-picks targets; a new translation of the target never appears; non-localized references become impossible | Translation groups, resolved per page locale (§3) | +| Polymorphic `{collection, id}` values (Payload) | Explicit collection | The id already names its node; two values to keep consistent; larger strict schemas | Bare ids, collection checked on write | +| Reverse lookups by scanning `json_extract` over `nodes` | No table | Every delete and move scans the site's entries; repeater paths need `json_each` over every item | `entry_refs`, the `asset_refs` pattern | +| `blocks` (a heterogeneous list: each item picks one of several shapes) | ACF flexible content, Payload blocks | Each item needs a discriminated union: `anyOf` per item counts against Anthropic's 16 unions; a second Studio form engine; overlaps Cloudhal's own body directives, surfaces and page sections, which already compose varied content | Repeaters only; varied layouts stay in bodies and sections | +| Nested repeaters (depth 2+) | Richer models | Recursion or a second subset level; strict depth and property counts multiply; nested Studio forms on 390 px | Depth fixed at 1 by the contract | +| Per-sub-field localization in a shared list | Shared structure, translated words | Needs stable item keys and a merge by key on every read; two value shapes per field | Repeater-level `localized`; open question 2 | +| Sort or filter by a referenced entry's field | "Events by venue city" in one query | Denormalized values that every target edit invalidates across collections | Two-step queries (§4) | +| Unbounded resolution with a request timeout | Simple themes | Cost depends on content, not code; cycles | Depth ≤ 2, breadth-first, 200 per page | +| `oneOf` in the MCP input as in the contract | One source | OpenAI's strict mode does not document `oneOf` | `anyOf` over disjoint variants, identical meaning (§6) | + +## Compatibility & migration + +- **Contract:** two new variants, a new `$defs` and one optional marker property. Existing sites, snapshots, bundles and archives stay valid. No `schemaVersion` bump. + - An older admin Worker refuses the new definitions (`additionalProperties: false` inside `oneOf`), so servers upgrade first, as with every addition. + - An older site Worker is covered by `fields: 2` (§8). +- **Data:** none to migrate. Migration 0033 is DDL only, because no reference value can exist before phase 2: `field_key_in_use` refuses definitions over keys that already hold values. +- **Themes:** additive. `entry.fields` keeps meaning what is stored: ids and item objects. Old theme copies render the new types as nothing, since `EntryFields.astro` ships in the renderer package and shows them once updated. Themes opt into cards, depth and `referencedBy`. +- **Studio:** additive types in the builder and form. +- **Rollback:** + - A snapshot restores the site node and entries, and the index is rebuilt in its batch. + - A field operation reverts through its `before` snapshot, dropping deleted targets (§7). + - Removing a reference or repeater field with `remove_field` removes its values and rows. + - Downgrading below this RFC restores a snapshot from before the first reference or repeater definition, because the old validator refuses them. The `entry_refs` table is then unused and harmless. + +## Security & policy impact + +§8 is the threat model. In short: +- **No new scopes and no new approval points:** + - definitions stay `site:settings` and person-classed; + - values stay content; + - `list_references` and the picker need `content:read`. +- **Tenant isolation:** every statement filters on `site_id`, every guard re-checks it, and nothing resolves across sites. +- **New refusals visible to agents:** `entry_referenced`, `reference_target_moved`, `unknown_entry`, `too_many_references`, `field_key_in_use`, `fields_unsupported` and `translation_shape_mismatch`, each with a hint in `field-hints.ts`. + +## Open questions (each with a recommended answer) + +1. **Deleting a referenced entry:** refuse and list the referrers; no cascade and no "unlink and delete" in stage 4. *Recommended.* If people ask for it, add an explicit `unlink_entry` operation (dry run, approval, snapshot, revert) rather than a delete flag. +2. **Per-sub-field localization in a non-localized repeater:** not in stage 4. Repeaters are localized as a whole, and Studio marks shared text "not translated". *Recommended.* Revisit with stable item keys (`_key`, which cannot collide with a `FieldKey`) when a real case needs shared structure with translated words. +3. **`datetime`, multiple selects and multiple references inside repeaters** (an agenda's session times): not in stage 4, because the props subset's array items are flat scalars (ADR-0030). *Recommended:* the first follow-up, by allowing ADR-0030's flat object one level below an array item for `datetime` only (an ADR-0030 amendment), measured against strict depth (5 of 10). +4. **The locale rule with `fallback: none`:** a target with no version in the page's language resolves to nothing, and `referenceIssues` reports it. *Recommended*, because it matches how pages behave under `none`. The alternative, linking the original in its own language with `lang`, could be a per-field option later. +5. **Strong references to archived targets:** an existing value keeps pointing at an archived target, which renders nothing; a *new* choice is refused. *Recommended.* The alternative is to refuse archiving while referenced, which makes retiring old venues painful. +6. **`where` on reference fields:** checked when a value is newly chosen only, never re-checked (Sanity's `filter` model). *Recommended.* +7. **Default depth for themes:** 1, with 2 as the ceiling. *Recommended.* Payload populates two levels by default, but a card is what themes draw. +8. **`fieldsSchemaStrict` beside `fieldsSchema`, or a `strict` argument on `list_collections`:** beside it, with `strictCost`. *Recommended:* it is small, and agents need not know their provider's mode in advance. Splitting a collection into strict-sized groups automatically is left out. +9. **Publish warnings as a gate:** never; a warning only. *Recommended.* A future site setting could turn them into a person-approval point for agents' publishes, like RFC-0042 §3.5's expiry rule. +10. **Structured data from references** (phase 7): the `reference` source kind in this RFC rather than a later one. *Recommended*, because RFC-0042 §6 named it as stage 4's starting point. +11. **The `asset_refs` rebuild chunking** (§2.5): fix it in its own small PR before phase 2, since import of large sites can hit D1's 100 KB statement limit today. *Recommended.* From fe8ac929ee198ab800a321d694a667627dc3b782 Mon Sep 17 00:00:00 2001 From: Madraka <8898640+Madraka@users.noreply.github.com> Date: Tue, 10 Nov 2026 15:15:42 +0300 Subject: [PATCH 2/8] docs(rfc): fold the PR #193 review into RFC-0045 Every upheld and split finding gets a decision, listed in a review log: - references store the translation group id, resolved and guarded per site - entry writes anchor on the collection's definitions - only published referrers lock deletes and moves - resolution is gated per view, and cache tags purge by group - repeater assets are recorded in asset_refs - release closures exclude references - strict output gets its own input (fieldsStrict, clearFields) - every guard filters site_id, and rebuilds join same-site members - the types stay refused until a final enabling step - the update_collection $defs are fixed, and the anyOf plan is dropped - Event addresses come from a Place via targetField Co-Authored-By: Claude Opus 5.5 Signed-off-by: Madraka <8898640+Madraka@users.noreply.github.com> --- ...ctions-stage-4-references-and-repeaters.md | 703 +++++++++++------- 1 file changed, 436 insertions(+), 267 deletions(-) diff --git a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md index 5bad34f6..4127431a 100644 --- a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md +++ b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md @@ -1,44 +1,58 @@ # RFC-0045: Collections stage 4: reference fields and repeaters -- Status: Draft +- Status: Draft (revised 2026-11-10 after an independent review of PR #193; the review log is at the end) - Author: Halil (Madraka), drafted with Claude · Date: 2026-11-10 - Related: - [#114](https://github.com/cloudhal-dev/cloudhal/issues/114) (this stage); #71 (the collections plan). - RFC-0040 / ADR-0042 (the registry), RFC-0041 / ADR-0043 (typed fields, field operations), RFC-0042 / ADR-0044 (§2 presets, §4 listings and `entry_list_values`, §6 "out: stage 4"). - - RFC-0004 §4.4 (`asset_refs`, the model for the reverse index), RFC-0008 §6.2 and §7 (holds, gates, the gate index), RFC-0010 (snapshot bundles), RFC-0015 / ADR-0030 (the props subset), RFC-0033 / ADR-0035 (one node per locale, translation groups, `localization.fallback`), RFC-0034 (language editions), RFC-0039 / ADR-0041 (archives). + - RFC-0004 §4.4 (`asset_refs`, the model for the reverse index), RFC-0008 §6 and §7 (releases and their dependency closure, holds, gates, the gate index), RFC-0010 (snapshot bundles), RFC-0015 / ADR-0030 (the props subset), RFC-0033 / ADR-0035 (one node per locale, translation groups, `localization.fallback`), RFC-0034 (language editions), RFC-0039 / ADR-0041 (archives). - Prior art (checked 2026-11-10; "unverified" where a primary page could not be read): - - [Sanity reference type](https://www.sanity.io/docs/studio/reference-type): a strong reference (the default) blocks deleting its target and blocks publishing until the target is published; `weak: true` may point at missing or draft documents and dereferences to `null`. `to` is a static list of types; `filter` narrows the picker only and "does not guarantee" the target keeps matching. - - [Payload relationship field](https://payloadcms.com/docs/fields/relationship): `relationTo` one or several collections (polymorphic values become `{relationTo, value}`), `hasMany`, `minRows`/`maxRows`, `filterOptions`, per-field `maxDepth` for population; reverse lookups need a separate Join field. - - Contentful links: resolution up to 10 `include` levels and unresolvable links reported separately (unverified: the concepts page answered 429). - - Strict structured output: [OpenAI](https://developers.openai.com/api/docs/guides/structured-outputs) (types plus `anyOf`, no `allOf`/`not`/`if`; "Root objects must not be `anyOf`"; every property required, optional as a `null` union; up to 5,000 properties, 10 levels of nesting, 1,000 enum values; recursion allowed; `oneOf` not mentioned). [Anthropic](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) (`anyOf`, `$defs`/`$ref` without recursion; no `minimum`/`maximum`/`minLength`/`maxLength`; `minItems` only 0 or 1; across a request at most 20 strict tools, 24 optional parameters and 16 parameters with union types, `anyOf` or type arrays). + - [Sanity reference type](https://www.sanity.io/docs/studio/reference-type): + - a strong reference (the default) blocks deleting its target, and blocks publishing until the target is published; + - `weak: true` may point at missing or draft documents and dereferences to `null`; + - `to` is a static list of types, and `filter` narrows the picker only: it "does not guarantee" the target keeps matching. + - [Payload relationship field](https://payloadcms.com/docs/fields/relationship): + - `relationTo` names one or several collections; polymorphic values become `{relationTo, value}`; + - `hasMany`, `minRows`/`maxRows`, `filterOptions`, and a per-field `maxDepth` for population; + - reverse lookups need a separate Join field. + - Contentful links: resolution up to 10 `include` levels, with unresolvable links reported separately (unverified: the concepts page answered 429). + - Strict structured output: + - [OpenAI](https://developers.openai.com/api/docs/guides/structured-outputs): types plus `anyOf`; no `allOf`/`not`/`if`; "Root objects must not be `anyOf`"; every property required, optional ones as a `null` union; up to 5,000 properties, 10 levels of nesting and 1,000 enum values; recursion allowed; `oneOf` not mentioned. + - [Anthropic](https://platform.claude.com/docs/en/build-with-claude/structured-outputs): `anyOf`, and `$defs`/`$ref` without recursion; no `minimum`/`maximum`/`minLength`/`maxLength`; `minItems` only 0 or 1; per request at most 20 strict tools, 24 optional parameters, and 16 parameters with union types (`anyOf` or type arrays). ## Summary -Two new field types join RFC-0041's nine. A `reference` field points to entries of one to five declared collections of the same site, one or up to fifty, and stores bare entry ids, like an `image` field stores asset ids. A `repeater` field is an ordered list of up to fifty items of up to ten flat sub-fields. A repeater cannot contain a repeater, so every agent-facing schema has a fixed depth and no recursion. +Two new field types join RFC-0041's nine: +- A **`reference`** field points to entries of one to five declared collections of the same site, one target or up to fifty. It stores the target's **translation group id**, the same id in every site of a project. It always resolves to that group's member in this site and in the page's language. +- A **`repeater`** field is an ordered list of up to fifty items, each of up to ten flat sub-fields. A repeater cannot contain a repeater, so every agent-facing schema has a fixed depth and no recursion. -References are strong. A reverse index, `entry_refs`, is written in each entry's own batch, anchored on its new content hash like `asset_refs`. It refuses to delete a referenced entry, and refuses to move one out of the collections its referrers allow. Archiving or unpublishing a target is allowed: the renderer resolves references only against what the published snapshot shows, so a dangling reference renders as nothing and publish warns but never fails. A reference names the target's translation group and resolves to the version in the page's language. Resolution is breadth-first over in-memory snapshot indexes, at most two levels deep, under a budget per page. Studio gets an entry picker and nested repeater forms from new `@cloudhal/ui` blocks, and the four field operations learn the new types. +References are strong where it matters: a target cannot be deleted or moved while a published entry shows it. A reverse index, `entry_refs`, is written in each entry's own batch, anchored on the entry's new content hash and on its collection's definitions as read, scoped by site like every other guard. References from drafts never lock anything. Archiving or unpublishing a target is always allowed: the renderer resolves only what the published snapshot shows *at that instant*, gates included, so an unresolved reference renders as nothing and publish warns but never fails. Assets used inside repeaters count in `asset_refs`, like any field's assets. + +Resolution is breadth-first, at most two levels deep, under a budget per page. Referrer pages are purged whenever a target group changes, and a target that becomes visible counts as a change. Studio gets an entry picker and nested repeater forms from new `@cloudhal/ui` blocks. The four field operations learn the new types. + +Nothing goes live until the index, the guards, the renderer, the operations and Studio have all shipped: the two types stay refused until a final enabling step. ## Motivation - **Today:** - - An event's venue, a talk's speakers or a product's brand are typed as plain text (RFC-0042 §6: `organizer`, `performer` and `brand` "map from text"). The same venue is retyped on every event. Its address, map and page are not linked, and a change to the venue's name does not reach the events. - - A recipe's ingredients, an agenda's sessions or a job's benefits are multiline text, one item per line (RFC-0042 §2's `recipeIngredient`). There is no second column: an ingredient's quantity, a session's speaker. + - An event's venue, a talk's speakers or a product's brand are typed as plain text (RFC-0042 §6: `organizer`, `performer` and `brand` "map from text"). The same venue, address included, is retyped on every event. Its page is not linked, and a change to its name does not reach the events. + - A recipe's ingredients, an agenda's sessions or a job's benefits are multiline text, one item per line (RFC-0042 §2's `recipeIngredient`). There is no room for a second value: an ingredient's quantity, a session's speaker. - Agents write these as free text, which no theme, listing or JSON-LD preset can read back. - **Who:** the collections #71 names (events with venues and speakers, team and projects, products and brands, docs and related docs, locations), the themes that render them, and the agents that fill them. -- **Why now:** stages 1–3 shipped (RFC-0040 to RFC-0042, released through 0.7.0). Every competitor in RFC-0041's prior art has references and lists of groups. RFC-0042's preset table already names the paths references fill (`location` as a `Place`, `performer`, `organizer`). +- **Why now:** stages 1 to 3 shipped (RFC-0040 to RFC-0042, released through 0.7.0). Every competitor in RFC-0041's prior art has references and lists of groups. RFC-0042's preset table already names the paths references fill: `location` as a `Place`, `performer`, `organizer`. §4 lets a preset read the venue's own address fields. ## Proposal ### 1. Contract -All additions are optional variants and `$defs` in `site.schema.json`. `content-entry.schema.json` changes only its `fields` description. No `schemaVersion` bump, no new root type. +All additions are optional variants and `$defs` in `site.schema.json`. `content-entry.schema.json` changes only its `fields` description. There is no `schemaVersion` bump and no new root type. ```jsonc // site.schema.json $defs.FieldDefinition.oneOf (line 274): two variants appended after SelectField (line 542) { "title": "ReferenceField", "type": "object", - "description": "RFC-0045: entries of other (or this) collections of the same site, by the id of their translation group's original (entry_…).", + "description": "RFC-0045: entries of other (or this) collections of the same site, by translation group id (entry_…): the id of the group's original, which every language version and edition carries in translation.group.", "additionalProperties": false, "required": ["key", "type", "labels", "localized", "collections"], "properties": { @@ -136,255 +150,340 @@ All additions are optional variants and `$defs` in `site.schema.json`. `content- // content-entry.schema.json properties.fields.description (line 38), changed - "… otherwise any JSON. An absent key means no value." -+ "… otherwise any JSON. An absent key means no value. A reference value is an entry id or a list of them, a repeater value a list of flat objects (RFC-0045)." ++ "… otherwise any JSON. An absent key means no value. A reference value is a translation group id or a list of them, a repeater value a list of flat objects (RFC-0045)." ``` - **Values:** | Type | Value | Checked on write | |---|---|---| - | `reference` | `"entry_…"`, or with `multiple` a list of them, no repeats, ≤ `maxItems` | each id is an entry of this site in one of `collections`, is its translation group's original in this site (a same-site translation's id is canonicalized to the original's, as `canonicalImageRefs` does for assets), is not the entry itself, and meets `where`; checked for changed keys only (RFC-0041 §3) | - | `repeater` | a list of ≤ `maxItems` objects; each holds only sub-field keys, with RFC-0041 §2's values for its type; no empty item | each item against the sub-fields; `requiredToPublish` sub-fields need a value in every item of an entry in review or published | - -- **Why these shapes.** - - A bare id (not `{collection, id}`, as Payload's polymorphic values) keeps a reference what an image field already is. The id names its node and so its collection, and the store re-checks the collection on every write. - - Repeater items are flat objects of scalars, which is exactly ADR-0030's `ArrayPropSchema` with `FlatObjectPropSchema` items (`packages/core-ts/src/blocks.ts:55-68`). `fieldPropSchema` compiles a repeater into the subset, and `validateProps` checks it unchanged, eval-free (rule 6). - - Excluded from sub-fields: `datetime` (an object) and multiple selects and references (arrays). These would need a second nesting level in the subset (open question 3). + | `reference` | a group id `"entry_…"`, or with `multiple` a list of them, no repeats, at most `maxItems` | see §2.1: the group has a member in this site in one of `collections`; it is not the entry's own group; it meets `where`. Checked for changed keys, and for every own value when the entry becomes published (§2.2). A member's id is canonicalized to its group id (`translation.group ?? id`), as `canonicalImageRefs` canonicalizes asset references. | + | `repeater` | a list of at most `maxItems` objects. Each holds only sub-field keys, with RFC-0041 §2's values for its type. No empty item. | each item against the sub-fields. `requiredToPublish` sub-fields need a value in every item of an entry in review or published. Image and rich-text sub-values are checked and recorded as assets (§2.4). | + +- **Why a group id.** `Translation.group` is "the original node's id … May cross sites in one project" (`common.schema.json:121-124`). + - A language edition's entries carry their source site's group id (`editions.ts:121-125`; `checkTranslationLinks` requires a member's group to be its source's, `store/translations.ts:131`). + - The group id is therefore the one identity that is the same for every language version in every site. "The original in this site" is undefined in an edition site, where no member is the group's node. + - The group id is used as a *key*: it is never read as a node of another site. A reference resolves, and is guarded, only through members whose `site_id` is this site (§2, §3). + - It is a bare id, not `{collection, id}` as in Payload's polymorphic values: it keeps a reference what an image field already is, and the store checks the collection on every write. +- **Why flat items.** Repeater items are flat objects of scalars, which is exactly ADR-0030's `ArrayPropSchema` with `FlatObjectPropSchema` items (`blocks.ts:55-68`). + - `fieldPropSchema` compiles a repeater into the subset, and `validateProps` checks it unchanged, eval-free (rule 6). + - Excluded: `datetime` (an object), multiple selects and references (arrays). These would need a second level in the subset (open question 3). - `maxItems` ≤ 50 is `BLOCK_LIMITS.arrayItems` (`blocks.ts:113`). - **Code rules beyond the schema** (`checkFieldDefinitions`, `definitions.ts:177`): - every `collections` entry is a declared or inferred collection (`reference_target_unknown`); - `where` only with one target collection, on its `listFields`, without `now` (`invalid_query`); - `maxItems` only with `multiple`; - - sub-field keys unique in their repeater and not reserved, with the same list as fields; + - sub-field keys unique in their repeater and not reserved (the same list as fields); - `itemLabel` names a text, select or reference sub-field; - at most 100 *leaves* per collection (fields plus sub-fields); - - sub-fields count toward the per-site 200 (`FIELD_LIMITS`, `types.ts:32`); - - at most 100 reference values per entry across fields and items (`too_many_references`). -- **Adding a reference or repeater field is a free edit**, except over a key some entry already holds (`field_key_in_use`, one `json_type` count). This guarantees that every reference value in the working graph was written through §2's checks and has its index rows. + - sub-fields count toward the per-site cap of 200 (`FIELD_LIMITS`, `types.ts:32`); + - at most 100 reference values per entry, across fields and items (`too_many_references`). + + **Until the enabling step (§9 step 7), `checkFieldDefinitions` refuses both types** (`fields_unsupported`). The contract can thus ship first without anyone being able to create a value that has no index rows. +- **Adding a reference or repeater field is a free edit**, except over a key some entry already holds (`field_key_in_use`, one `json_type` count). The count races with entry writes in that collection; the definitions anchor of §2.1 closes the race. - **Rule 2 holds:** - `oneOf` with inline variants and the one-value `type` discriminator (ADR-0043); - no `default`; - - `additionalProperties: false` everywhere; + - `additionalProperties: false`; - cross-file `$ref` only to `common.schema.json#/$defs/*`; - - no recursion: `SubFieldDefinition` has no repeater variant, so depth is fixed by the contract. + - no recursion: `SubFieldDefinition` has no repeater variant. - **Generation impact:** - - **TypeScript:** `FieldDefinition` gains `ReferenceField | RepeaterField`, and a new `SubFieldDefinition` union of nine `*SubField` interfaces. `FieldType` (`types.ts:7`) is derived from the contract, so every total dispatch table over it stops type-checking until it handles the two types (`COMPILE`, `props.ts:45`; `EMIT`, `json-schema.ts:49`). The compiler lists the work. + - **TypeScript:** `FieldDefinition` gains `ReferenceField | RepeaterField`, and a new `SubFieldDefinition` union of nine `*SubField` interfaces. `FieldType` (`types.ts:7`) derives from the contract, so every total dispatch table over it stops type-checking until it handles the two types. Phase 1 gives each an explicit entry: an "unsupported" stub where its real code comes later. + - `COMPILE` (`props.ts:46`) and `EMIT` (`json-schema.ts:49`): real entries in phase 1. + - `CONTROLS` (`EntryFieldsForm.tsx:329`): a read-only "needs a newer Studio" stub until phase 6. + - `FIELD_QUERY_OPS` (`fields/query.ts:53`): `[]` until phase 8, so the type is not listable. + - `FIELD` (`structured-data/values.ts:154`): "no value" until phase 8. + - `archive-reassemble.ts`'s `FIELD_CHECKS`/`FIELD_WORDS` key the archive manifest's own types, not field types, and are unaffected. + - `TYPE_RULES` and `EXTRA` are `Partial` and need no change. The compiler lists anything else. - **Rust:** cargo-typify emits `#[serde(tag = "type", deny_unknown_fields)] enum FieldDefinition` with two more variants, a second tagged enum `SubFieldDefinition`, and the extended `fromType` enum. Values stay `serde_json::Value`. - - **Naming:** the sub-field variants carry their own titles (`TextSubField`, not `TextField`). Both generators name types from `title`, and RFC-0042 §1 found that cargo-typify 0.8.0 keeps only the first of two equal names. - - **Not run on this branch.** This is a docs-only PR, written while the owner's machine ran a test-heavy session. Phase 1 (§9) runs `contract-change` first, with these fixtures: - - `valid/site/collection-stage4.json`: an `events` collection with `venue` (reference to `places`), `speakers` (repeater with a reference, a select and a text sub-field) and a migration marker with `sub`; + - **Names:** the sub-field variants carry their own titles (`TextSubField`, not `TextField`). Both generators name types from `title`, and RFC-0042 §1 found that cargo-typify 0.8.0 keeps only the first of two equal names. + - **Not run on this branch:** this is a docs-only PR, written while the owner's machine ran a test-heavy session. Phase 1 runs `contract-change` first, with these fixtures: + - `valid/site/collection-stage4.json`: an `events` collection with `venue` (a reference to `places`), `speakers` (a repeater with reference, select, image and text sub-fields) and a migration marker with `sub`; - `invalid/site/`: - - reference without `collections`; + - a reference without `collections`; - six target collections; - `maxItems` 51; - a repeater sub-field of type `repeater`; - - one of type `datetime`; + - a sub-field of type `datetime`; - a sub-field with `localized`; - eleven sub-fields; - `fromType` `blocks`. -### 2. Cross-node integrity: the reverse index +### 2. Cross-node integrity ```sql --- packages/storage/migrations/0033_entry_refs.ts (new; DDL only, nothing to backfill: no reference values exist yet) +-- packages/storage/migrations/0033_entry_refs.ts (new; DDL only: no value can exist before §9 step 7) CREATE TABLE entry_refs ( site_id TEXT NOT NULL, - target_id TEXT NOT NULL, -- the referenced entry: its group's original in this site - source_id TEXT NOT NULL, -- the referring entry node (any locale) + target_id TEXT NOT NULL, -- the referenced translation group id + source_id TEXT NOT NULL, -- the node holding the value (an original for localized: false) + source_group TEXT NOT NULL, -- that node's group id (translation.group ?? id), to reach members that inherit the value source_collection TEXT NOT NULL, field TEXT NOT NULL, -- "venue", or "speakers.person" inside a repeater - source_status TEXT NOT NULL, -- the referrer's status, for "used by published entries" + source_status TEXT NOT NULL, -- the holding node's own status PRIMARY KEY (source_id, field, target_id) ); CREATE INDEX entry_refs_target ON entry_refs (site_id, target_id, source_status); ``` -Each statement below follows a pattern that already exists. The rows of an entry are `asset_refs`'s fields rows (`store/entry-fields.ts:119-144`) with another table. - -1. **Writing a referrer** (`EntryEngine.write`, `store/entries.ts:30`). `planEntryFields` returns two more parts. - - A **guard**, ANDed into the entry's CAS (`entries.ts:104`): every target exists, in this site, is a `contentEntry`, and is in the field's allowed collections at write time. It is one count over a JSON list of `{t: id, c: [collections]}` pairs, like the asset guard (`entry-fields.ts:163`): - ```sql - (SELECT count(*) FROM json_each(?) p JOIN nodes n ON n.id = json_extract(p.value, '$.t') - WHERE n.site_id = ? AND n.kind = 'contentEntry' - AND json_extract(n.body, '$.collection') IN (SELECT value FROM json_each(json_extract(p.value, '$.c')))) = ? - ``` - - **Statements** anchored on the entry's new hash: `DELETE FROM entry_refs WHERE source_id = ? AND EXISTS (SELECT 1 FROM nodes WHERE id = ? AND hash = ?)`, then one `INSERT … SELECT … FROM json_each(?) WHERE `. They run only if the entry's own write did (rule 4), and are appended to `own` in `entries.ts:110`. - - A target deleted or moved between the check and the batch fails the guard; `diagnose()` then names it (`unknown_entry`, `reference_target_moved`). One bound parameter carries the whole list, so D1's 100-parameter limit does not apply. 100 ids of at most 48 characters stay far below its 100 KB per statement. -2. **Deleting a target** (`EntryEngine.delete`, `entries.ts:162`). The node's `DELETE` gains `AND NOT EXISTS (SELECT 1 FROM entry_refs WHERE target_id = ? AND source_id <> ?)`, as an asset's delete guards on `asset_refs` (`assets.ts:867`). A refused delete answers `409 entry_referenced`, with up to 20 referrers (id, title, collection, field) and the total. - - Only a group's original can be a target, so deleting a translation is never blocked. - - The referrer's own rows go with its delete: `DELETE FROM entry_refs WHERE source_id = ? AND NOT EXISTS (SELECT 1 FROM nodes WHERE id = ?)`, as `dropRefsStatement` (`assets.ts:1015`). -3. **Moving a target to another collection** (`entersCollection`, `entries.ts:101`). The write computes, from the site node it read, every `"."` pair whose allowed collections include the new one. It is guarded with: - - `NOT EXISTS (SELECT 1 FROM entry_refs WHERE target_id = ? AND source_collection || '.' || field NOT IN (SELECT value FROM json_each(?)))`; - - the site's hash as read (`EXISTS (SELECT 1 FROM nodes WHERE id = ? AND hash = ?)`), so a concurrent change to the definitions cannot slip between. - - Refused: `409 reference_target_moved`, naming the referring fields. During a field operation, the pairs include the marker's `fromKey` as well as the new key. -4. **Archiving and unpublishing a target** are allowed and write nothing to `entry_refs`. An archived target stays pickable for existing values only. The picker hides archived entries, and the write check refuses a *newly* chosen archived target (`retired`-asset rule, `assets.ts` `planRefs`). -5. **Rebuilds** follow `refsRebuild` (`media/refs-rebuild.ts:12`) in the same batch as the node set: `importSite`/`pull`/`push` (`sync-plan.ts:308`), `cloudhal import --replace` and `--revert`, and snapshot restores. `entryRefsRebuild(siteId, rows)` chunks its `INSERT` so each statement stays below 90 KB, still inside the one atomic batch. `asset_refs`'s rebuild needs the same chunking, which is a separate fix (follow-ups). -6. **Field definition changes** (`fieldDefinitionChanges`, `definitions.ts:135`) add these refusals: - - removing a collection from `collections` while a value points into it (an `entry_refs` join count); - - turning `multiple` off; - - removing or retyping a sub-field that holds values. - - Each refusal names the operation to use (§7). - -**Invariant.** The store never *creates* a dangling reference: deleting a target, moving it, and writing a reference to a missing target are all guarded in the same batch. Dangling references can still *exist*: -- in archives and bundles written by hand or by another version; +**Invariant (the index):** `entry_refs` holds exactly one row per `(node, field path, target group)` whose value is stored on that node and whose target group has a member in the node's site. +- A target named in two repeater items (a host who also gives a talk) is one row: the writer deduplicates the pairs, and the `INSERT` is `INSERT OR IGNORE`. +- Every statement below filters on `site_id`, so it uses `entry_refs_target` and stays inside its tenant. + +**1. Writing a referrer** (`EntryEngine.write`, `store/entries.ts:30`). `planEntryFields` returns three more parts: +- **The target guard**, ANDed into the entry's CAS (`entries.ts:104`). Every target group has a member in this site in one of the field's allowed collections. It is one count over a JSON list of `{t: group, c: [collections]}` pairs, like the asset guard (`entry-fields.ts:163`). The members are found through `nodes_translation_locale` (migration 0025: `site_id, kind, json_extract(body, '$.translation.group')`), or by id for an untranslated original: + ```sql + (SELECT count(*) FROM json_each(?) p WHERE EXISTS ( + SELECT 1 FROM nodes n WHERE n.site_id = ? AND n.kind = 'contentEntry' + AND (n.id = json_extract(p.value, '$.t') OR json_extract(n.body, '$.translation.group') = json_extract(p.value, '$.t')) + AND json_extract(n.body, '$.collection') IN (SELECT value FROM json_each(json_extract(p.value, '$.c'))))) = ? + ``` +- **The definitions anchor.** The allowed collections and the set of reference keys come from the site row the writer read, and a site-side refusal (`field_key_in_use`, removing an allowed collection) only counts entries already committed. Every entry write that changes `fields` or `collection` therefore also guards on its collection's definitions as read: + ```sql + (SELECT json_extract(body, ?) FROM nodes WHERE id = ? AND kind = 'site') IS ? + ``` + Here `?` is `$.collections."".fields`, and the text is read with the same expression while planning: one primary-key read, `NULL` for a collection without definitions. It conflicts only when that collection's definitions change, never on other site edits, unlike the site hash. A writer that loses retries under the new definitions, and `field_key_in_use` then sees its value. This is the same idea as RFC-0042's list-value rows, which are valid only "under the fingerprint they were built with" (`list-values/fingerprint.ts`). +- **The rows**, anchored on the entry's new hash: `DELETE FROM entry_refs WHERE source_id = ? AND `, then one `INSERT OR IGNORE … SELECT … FROM json_each(?) WHERE `, appended to `own` (`entries.ts:110`). + +A target deleted or moved between check and batch fails the guard, and `diagnose()` names it (`unknown_entry`, `reference_target_moved`). One bound parameter carries the whole list, so D1's 100-parameter limit does not apply, and 100 group ids stay far below its 100 KB per statement. + +**2. Deleting and moving a target: who can lock what.** +- A reference **blocks** only while a *published* entry shows it: the holding node is published, or a published member of its group inherits the value (`localized: false`). +- Drafts, entries in review and archived entries never lock anything. A contributor (`content:draft:own`, `roles.ts`) can therefore never pin someone else's entry. An author pins only through their own published entries, which the live site really depends on, and an editor can unpublish them. +- The holding node's status is in the row. A member that inherits the value is reached through `source_group`, using the same translation index: + ```sql + -- "blocking rows" for group G in site S + SELECT 1 FROM entry_refs r WHERE r.site_id = ? AND r.target_id = ? + AND (r.source_status = 'published' OR EXISTS ( + SELECT 1 FROM nodes m WHERE m.site_id = r.site_id AND m.kind = 'contentEntry' + AND json_extract(m.body, '$.translation.group') = r.source_group + AND json_extract(m.body, '$.status') = 'published')) + ``` +- **Delete** (`EntryEngine.delete`, `entries.ts:162`). The node's `DELETE`, already scoped by `site_id` (`entries.ts:168`), gains `AND (NOT EXISTS OR EXISTS ?>)`. Only removing a group's *last* member in this site can strand a reference. Deleting a translation, or an original that still has translations, never blocks, and references keep resolving to the remaining members. + - A refusal answers `409 entry_referenced`, with up to 20 published referrers (id, title, collection, field) and the total. + - When the last member does go, its remaining non-blocking rows are deleted in the same batch, guarded on the node being gone, as `dropRefsStatement` (`assets.ts:1015`). The draft referrers keep a dangling id, which their editor shows as "Not found". Their own rows go with their delete. +- **Publishing a referrer.** A write that makes an entry published re-checks *every* own reference value, not just changed keys. A dangling id is refused (`unknown_entry`: remove or replace it), so a draft that lost its target cannot go live pointing at nothing. + - Inherited values are not the translation's to fix. A translation that goes live while its original holds a dangling id gets a warning, not a refusal, and the reference renders as nothing. +- **Move** (`entersCollection`, `entries.ts:101`). Moving *any* member of group G from C to D is refused while a blocking row comes from a field that allows C but not D: `NOT EXISTS ( AND source_collection || '.' || field NOT IN (SELECT value FROM json_each(?)))`. The JSON list holds the `"."` pairs that allow D, computed from the site as read, and the write carries the definitions anchor of its own collection. + - Refused: `409 reference_target_moved`, naming the fields. + - During a field operation, the pairs include the marker's `fromKey`. + - Members of a group may still sit in different collections, as today (`checkTranslationLinks` has no collection rule). Resolution and `referenceIssues` filter members by the field's `collections` (§3), so a stray member never renders under a field that does not allow it. +- **Archiving and unpublishing** a target are allowed and write nothing to `entry_refs`. The picker hides archived entries, and a *newly* chosen group needs a member that is not archived (the `retired` rule of `planRefs`). + +**3. Rebuilds** run in the same batch as the node set: `importSite`/`pull`/`push` (`sync-plan.ts:308`), `cloudhal import --replace` and `--revert`, and snapshot restores. `entryRefsRebuild(siteId, rows)` builds the rows from the bodies, then `INSERT`s only rows whose target group has a member *in the same site*: it joins `nodes` with `site_id = ?`. +- An archive or push naming another tenant's entry id therefore pins nothing anywhere. The id stays in the value, never resolves, and is reported (`referenceIssues`). +- The `INSERT` is chunked so each statement stays below 90 KB, inside the one atomic batch. + +**4. Assets inside repeaters.** `fieldAssetRefs` (`media/field-refs.ts:17-36`) reads only top-level string values today (`if (typeof value !== "string") continue;`), so it would skip a repeater's items. +- It gains a walk over each item's `image` and `richText` sub-values, recorded under `fields..`. +- The same function feeds all of these, so each follows without further change: + - the write guard (`refsPlan`) and `fieldAssetIssues`: the asset exists in this site, is an image, and is not newly retired; + - field-operation rewrites (`fields-operations/rewrite.ts:150-158`); + - `refsRebuild` (`media/refs-rebuild.ts:22-24`); + - the purge planner's asset rule. +- An asset used only in a speaker's photo can therefore not be deleted or collected while used. Editions copy such assets already (`cross-site.ts` `assetIdsOf` scans the whole node). + +**5. Field operations** (`fields-operations/rewrite.ts:150`). From phase 2, `statementsOf` emits each rewritten entry's `entry_refs` rows next to its asset rows. It ANDs the target guard into the entry's CAS whenever the entry's reference values change, so a target deleted between plan and batch skips the entry for the next pass, like a concurrent edit. Operations that *name* a reference or repeater field are refused (`field_operation_unsupported`) until phase 4 implements them (§7). + +**6. Releases** (RFC-0008 §6). Today's dependency extractor treats every string matching `NODE_REF` at any depth as a dependency (`schedules/src/release.ts:268-289`), and core-ts passes it as `depsOf` (`core-ts/src/releases.ts:566`). Group ids in field values would therefore be pulled into a release's closure and locked (`assertUnlocked`). An edition's group id names a node of another site, so it would also fail as `dependency_missing` (`release.ts:416`). +- **Decision: references are not release dependencies.** Phase 2 replaces `depsOf` with a kind-aware extractor that drops reference values (top-level and inside items, by the site's definitions as pinned) before calling `referencedNodeIds`. +- A scheduled event whose venue is not live renders without it, exactly as after a publish, and the release's approval text carries `referenceIssues`. +- Rationale: closure dependencies are hard (they lock nodes and fail runs); references are soft at render. Holding every language version of a venue for an event's release would surprise its editors. + +**7. Definition changes** (`fieldDefinitionChanges`, `definitions.ts:135`) add these refusals: +- removing a collection from `collections` while a value points into it (an `entry_refs` join count); +- turning `multiple` off; +- removing or retyping a sub-field that holds values. + +Each refusal names the operation to use (§7). + +**The invariant restated.** The store never lets a *published* entry show a reference whose target group has no member in its site, and never leaves the index out of step with stored values. + +Dangling ids can still exist, and every reader treats them as unresolved without throwing: +- in drafts whose target was deleted; +- in hand-made or foreign archives; - in publish snapshots composed with holds, which leave out a held node that is not live (`store/snapshots.ts:29-37`); -- after a release applies `nodes(L) − ids + targets`. - -Every reader therefore treats a missing target as unresolved and never throws. +- after a release removes a target. **Snapshots, publish and rollback:** - -- **A snapshot is the whole working copy at one read** (`snapshots.ts:23`), so referrers and targets are taken together. Rollback moves the pointer to an ancestor snapshot (`store.ts:1127`), which is just as consistent. Restores and `import --revert` rebuild the index (point 5). -- **Visibility decides, not existence.** A snapshot holds drafts and archived entries too, so the renderer resolves only against entries the snapshot *shows* at that instant: `Loaded.reachable` (`astro-renderer/src/source.ts:147`): published, gates open, not `hide`-expired. - - An `unlist`ed target (RFC-0042 §3) still resolves: its page exists. - - A draft, archived, gated or missing target resolves to nothing. `EntryFields.astro` then shows the field only if other targets remain. -- **Publish warns, never fails.** `referenceIssues(snapshotNodes)` (pure, `@cloudhal/core-ts/fields`) lists published entries whose references will not resolve: target missing, draft, archived, gated, or no version in a needed locale. It appears in: - - Studio's publish dialog ("3 published entries point to entries that are not published"); - - the MCP `publish` tool's approval text; - - the output of `cloudhal snapshot --publish`. - - A build or publish never fails on it, for the same reason as RFC-0042 §4: "a published snapshot must keep rendering". Sanity's strong references block publishing per document, but Cloudhal publishes a site snapshot. A warning keeps a person in charge without making an unrelated fix wait for a venue page. -- **The gate index already covers targets.** It holds every instant at which a published entry's gate changes (`source.ts` `Loaded.gates`), so a referrer's page expires from the edge cache when its target's embargo opens or its `hide` expiry passes, with no new mechanism. -- **Purges.** A referrer's response adds `cloudhal-node-` for every target it rendered, as it already does for "the related entries an article links to" (`astro-renderer/src/cache-tags.ts:6-8`). A publish that changes a target's title therefore purges its referrers' pages. The per-page budget (§4) keeps the tags far below the 8 KB header cap. - -**Archives (RFC-0039).** -- An archive holds every node of the site, so references stay complete. Import rebuilds the index in its node batch. -- Dangling references are reported, not refused, as RFC-0041 open question 8 decided for invalid values: `cloudhal import --dry-run` adds `referenceIssues` per collection, and Studio lists them on the collection screen. -- Language editions (RFC-0034) never share ids across sites (§3). +- A snapshot is the whole working copy at one read (`snapshots.ts:23`). Rollback moves the pointer to an ancestor (`store.ts:1127`), which is just as consistent, and restores rebuild the index (point 3). +- **Visibility decides.** A snapshot holds drafts and archived entries. `Loaded.reachable` is the published entries *before gates* (`astro-renderer/src/source.ts:147-155`), and the gated set exists only per view instant (`#current()`, `source.ts:409-431`, `splitEntries` at `now`). The renderer therefore resolves through the view (§4). + - A target resolves only when its member is published and its gates are open at that instant. An `unlist`ed target (RFC-0042 §3) still resolves, because its page exists. + - A draft, archived, embargoed, `hide`-expired or missing target resolves to nothing. +- **Publish warns, never fails.** `referenceIssues(snapshot, now)`, pure and in `@cloudhal/core-ts/fields`, lists published entries whose references will not resolve: target missing, draft, archived, gated, in a disallowed collection, or with no version in a needed locale. It appears in Studio's publish dialog, in the MCP `publish` tool's approval text, and in `cloudhal snapshot --publish` output. + - A publish never fails on it: "a published snapshot must keep rendering" (RFC-0042 §4). + - Sanity's strong references block publishing per document, but Cloudhal publishes a whole site snapshot, and one draft venue must not hold back an unrelated fix. +- **Gates** of targets are in the snapshot's gate index already, since it holds every published entry's instants (`Loaded.gates`). A referrer's cached page therefore expires when its target's embargo opens or its `hide` expiry passes. + +**Purges** (`cache-tags.ts`, `cache-purge.ts`): +- **Tags.** Every reference value a response *read* adds `cloudhal-ref-`, resolved or not. These tags come before optional tags, up to 4 KB (about 100 groups at 41 bytes each). Past that, the response carries one `cloudhal-refs-` tag instead. +- **Purging.** `planCachePurge` purges `cloudhal-ref-` for every entry it already purges (changed, added or removed), by its group (`translation.group ?? id`), plus `cloudhal-refs-` whenever any entry changed. + - So a retitled, unpublished or archived venue purges its referrers, in every language. + - So does a venue that *becomes* visible: published, a new Turkish member, an embargo that opens at the next publish. +- **Why this is safe.** The tags carry opaque group ids, never titles. Cache-Tag is an edge-only header that Cloudflare strips before responses reach visitors (Cloudflare purge-by-tag documentation; unverified on this branch). Node builds set no tags. +- **The §8 rule, revised:** no unresolved target may appear in HTML, JSON-LD, feeds, the sitemap, search or `llms.txt`. Cache tags are excluded from that rule. + +**Archives (RFC-0039).** An archive holds every node of the site, so references stay complete, and import rebuilds the index in its node batch (point 3). Dangling or foreign ids are reported, not refused, as RFC-0041 open question 8 decided for invalid values: `cloudhal import --dry-run` adds `referenceIssues`. ### 3. The locale rule -**A reference names a translation group, not an entry-locale.** The stored id is the group's original in this site (§1). Resolution then picks the group member in the page's language: +A reference names a translation group (§1). In this site, the candidates are the group's members (`id = G` or `translation.group = G`) whose collection is in the field's `collections`. Resolution picks, among those the view shows: -1. the member of group `G` with `locale === page locale` that the snapshot shows; -2. otherwise, when `Site.localization.fallback` is `default-with-notice` (RFC-0033 §2), the member in the site's default locale, flagged `fromFallbackLocale`, so themes can set `lang` on it; +1. the member whose `locale` is the page's locale; +2. otherwise, when `Site.localization.fallback` is `default-with-notice` (RFC-0033 §2), the member in the site's default locale, flagged `fromFallbackLocale` so themes set `lang` on it; 3. otherwise, nothing. -The same rule serves all three readers: the renderer, Studio's preview of a translation, and `referenceIssues`, which reports "Main hall has no Turkish version". +The renderer, Studio's preview of a translation and `referenceIssues` ("Main hall has no Turkish version") share this one function, `pickMember`. - **`localized: false` is the default** that Studio pre-sets for references. The original holds the ids, and every translation resolves them in its own language (RFC-0041 §5). A translator never re-picks targets, and a venue's Turkish version appears on the Turkish event without any write. -- **`localized: true`** is for real per-language differences, such as a regional partner. Each version holds its own ids, still group ids, and an empty value falls back to the original's (`fromOriginal`). -- **Repeaters are localized as a whole.** With `localized: true`, each version has its own list, and the original's is read while its own is absent. With `localized: false`, the list lives on the original only. Text sub-fields then show the original's words on every version, and Studio says so ("Not translated: this list is shared by every language"). - - Per-sub-field localization inside a shared list would need stable item keys. It is open question 2. -- **Machine translation:** +- **`localized: true`** is for real differences between languages, such as a regional partner. Each version holds its own group ids, and an empty value falls back to the original's (`fromOriginal`). +- **Language editions** (RFC-0034; an original in another site of the project). Non-localized values are copied when the edition is made (RFC-0041 §5, through `forEdition`, `cross-site.ts:115`). Group ids are shared across sites, so a copied id stays the same id. + - `forEdition` keeps an id when the edition site has a member of that group in a collection the edition's own field allows. Otherwise the id is dropped and reported ("2 references not carried: Main hall has no edition in site B yet"). + - A reference never resolves across sites: it reads only members with this site's `site_id`. + - This lands in phase 2 with the guard, because without it making an edition would be refused (`unknown_entry`) as soon as references exist. +- **Repeaters are localized as a whole.** + - With `localized: true`, each version has its own list, and while its own is absent it reads the original's. + - With `localized: false`, the list lives on the original only, and text sub-fields show the original's words on every version. Studio says so: "Not translated: this list is shared by every language". Per-sub-field localization is open question 2. +- **Machine translation** (phase 5): - `get_translation_source` returns a localized repeater's `text`/`richText` sub-values at their paths (`speakers/1/talk`); - - `save_translation` writes them back only if the list has the original's length (`translation_shape_mismatch` otherwise), copying the other sub-values; + - `save_translation` writes them back only into a list of the original's length (`translation_shape_mismatch` otherwise), copying the other sub-values; - reference values are never sent for translation. -- **Language editions** (an original in another site of the project). Non-localized values are copied when the edition is made (RFC-0041 §5). Each referenced id is mapped to the edition site's member of the same translation group. Ids with no such member are dropped, and the result reports them ("2 references not carried: Main hall has no edition in site B yet"). A reference never crosses sites. + + Until phase 5, a localized repeater is translated by hand. The tools ignore it, as they ignore every non-text field today. ### 4. Render cost -`resolveFields` (`fields/resolve.ts:68`) stays pure. A reference resolves to its id or ids, and a repeater to its items, each sub-value checked as a field is (an invalid one is `undefined` with an issue). Loading targets is the renderer's job, in `entryFields` (`astro-renderer/src/fields.ts:172`). +`resolveFields` (`fields/resolve.ts:68`) stays pure. A reference resolves to its group ids, and a repeater to its items, each sub-value checked as a field is. It also skips definitions of a type it does not know, so a future type is safe on an older renderer. Loading targets is the renderer's job, in `entryFields` (`astro-renderer/src/fields.ts:172`). ```ts // @cloudhal/astro-renderer (additive) export interface ReferencedEntry { - id: string; groupId: string; collection: string; locale: string; slug: string; - title: string; summary?: string; url: string; image?: FieldImage; // the target's cover image + groupId: string; id: string; collection: string; locale: string; slug: string; + title: string; summary?: string; url: string; image?: FieldImage; // the member's cover image fromFallbackLocale: boolean; - /** Present only when depth 2 was asked for: the target's own fields, its references as cards without fields. */ + /** Present only when depth 2 was asked for: the member's own fields, its references as cards without fields. */ fields?: RenderedFields; } interface RenderedField { // existing interface, two optional additions - references?: ReferencedEntry[]; // reference: resolved targets, in stored order (unresolved ones left out) + references?: ReferencedEntry[]; // reference: resolved members, in stored order (unresolved ones left out) items?: RenderedFields[]; // repeater: each item's sub-fields, rendered like fields } interface EntryFieldsOptions { /* existing */ depth?: 1 | 2; } // default 1 interface GraphSource { - /** Visible entries by group id in one locale (one call per resolution level). */ - entriesByGroups?(groupIds: readonly string[], locale: string): Promise>; - /** "Who references me": visible referrers of a group, newest first, at most 100. */ + /** The members of these groups that this view shows, in this site (one call per resolution level). */ + groupMembers?(groupIds: readonly string[]): Promise>; + /** "Who references me": entries this view shows whose resolved value names the group, newest first, at most 100. */ referencedBy?(groupId: string, q?: { collection?: string; field?: string; locale?: string; limit?: number }): Promise; } ``` -- **Depth: 1 by default, 2 at most.** Depth 1 yields cards: title, summary, URL and cover image, which is what a venue link or a speaker list draws. Depth 2 adds each target's own fields, with *its* references as cards without fields. Nothing goes deeper, so cycles (A→B→A) cost nothing: resolution is breadth-first over a visited set of group ids. Payload's per-field `maxDepth` and Contentful's `include` levels are the same idea. +- **Gated, per view.** The snapshot source builds two gate-independent indexes once per snapshot and isolate, beside `assetIndex`: + - `members: Map` over every published entry; + - a lazy reverse map from resolved values. + + Every lookup then filters through the *view's* gate state at its instant: the `isAvailable` test and derived gates that `splitEntries` applies (`source.ts:423`), and the view's `assetIndex` for cover images. The snapshot caches only gate-independent card parts: title, slug, URL path and cover asset id per member. The member choice, the fallback and the picture are decided per view, so a Turkish member that opens replaces the fallback at its instant, and one that hides disappears. +- **Depth: 1 by default, 2 at most.** Depth 1 yields cards (title, summary, URL, cover image), which is what a venue link or a speaker list draws. Depth 2 adds each member's own fields, with *its* references as cards. Cycles (A→B→A) cost nothing: resolution is breadth-first over a visited set of group ids. Payload's per-field `maxDepth` and Contentful's `include` levels are the same idea. - **Fan-out:** - - by contract: 50 per field and 50 items per repeater; - - in code: 100 reference values per entry (§1); - - per render: a page budget of **200 resolved targets**, shared by every `entryFields` call of one request (the `MediaContext.budget` object already carries per-request budgets). - - Over budget, remaining targets resolve to nothing with one `console.warn`, never an error. A listing of 20 events at depth 1 with one venue each costs 20. -- **Batching:** - - **Snapshot source:** the loaded snapshot gains two indexes, built once per snapshot and isolate beside `assetIndex` (`source.ts:147`): `byGroup: Map>` over `reachable`, and a lazy reverse map for `referencedBy`. Every lookup is in memory, with no reads. Cards are cached per `(groupId, locale)` beside `Loaded.resolved` (the snapshot is immutable). Bundles (RFC-0010) need nothing new, because entries are already in them. - - **Live source** (`cloudhal dev`, previews): one store read per level, `getContentEntriesByGroups(siteId, ids)` with a `json_each` IN list, plus one `entry_refs` probe for `referencedBy`. A depth-2 page therefore costs at most two reads, whatever its fan-out. + - per field and per repeater: 50 (contract); + - per entry: 100 reference values (code, §1); + - per render: **200 resolved targets**, shared by every `entryFields` call of one request (`MediaContext.budget` already carries per-request budgets). + + Over budget, the remaining targets resolve to nothing with one `console.warn`, never an error. Their `cloudhal-ref-` tags are still added within the tag budget (§2). A listing of 20 events at depth 1 costs 20. +- **Batching.** + - The snapshot source does no store reads: every lookup is in memory, and bundles (RFC-0010) need nothing new. + - The live source (`cloudhal dev`, previews) makes one store read per level: `getGroupMembers(siteId, ids)`, a `json_each` IN list over `nodes_translation_locale`, plus the original by id. +- **`referencedBy` covers translations.** A row sits on the node holding the value, which is the original for `localized: false`. + - The live source reads `entry_refs` by `(site_id, target_id)`, expands each `source_group` to its members in the requested locale, and filters them by the view. + - The snapshot's reverse map is built from resolved values, so it includes inheriting members by construction. + - The test plan compares the two sources. - **Listings** (RFC-0042 §4): - - **Filter by a reference: yes.** A `reference` field may be a list field, with ops `eq`, `ne`, `in`, `set` and `unset` on target ids. It serves "events at this venue" and "talks by this speaker". Its `entry_list_values` rows put the id in `text` (an id is at most 48 characters, under the 100-unit limit). A multiple reference writes one row per id with `has = 2`, exactly like a multiple select's options (migration 0029's doc comment). `checkFieldQuery` and the property suite gain the type. - - **Sort by a reference, or filter by a referenced entry's field** ("events whose venue is in Istanbul"): **no.** Either needs the target's values denormalized into the referrer's rows, which every target edit would invalidate across collections. Agents and themes do it in two steps: `queryEntries` on `places` for `city eq istanbul`, then `venue in [...]` (≤ 20 values). + - **Filter by a reference: yes.** A `reference` field may be a list field, with ops `eq`, `ne`, `in`, `set` and `unset` on group ids: "events at this venue", "talks by this speaker". Its `entry_list_values` rows hold the group id in `text` (≤ 48 characters, below the 100-unit limit). A multiple reference writes one row per id with `has = 2`, like a multiple select (migration 0029). + - **Sort by a reference, or filter by a referenced entry's field** ("events whose venue is in Istanbul"): **no.** Either needs denormalized target values that every target edit would invalidate. Agents and themes do it in two steps: `places` by `city eq istanbul`, then `venue in [...]` (at most 20 values). - **Repeaters** cannot be list fields (`list_field_type`). - - **Theme-side "used by"** is `referencedBy`, not a query on the referring collection, so no list field is needed for it. -- **Structured data** (RFC-0042 §2), phase 7: a fifth source kind, `{ "kind": "reference", "field": "venue", "property": "title" | "url" | "summary" }`. It fills `location.name`, `performer.name`, `organizer.name`/`url` and `brand.name` from a depth-1 card. It is a new `StructuredDataSource` variant with its own named `property` enum, `ReferenceValueProperty`, as §1 of that RFC requires for Rust. +- **Structured data** (RFC-0042 §2), phase 8. A fifth source kind, `ReferenceValueSource`: + ```jsonc + { "title": "ReferenceValueSource", "type": "object", "additionalProperties": false, "required": ["kind", "field"], + "properties": { "kind": { "type": "string", "enum": ["reference"] }, + "field": { "$ref": "#/$defs/FieldKey" }, // a single reference field with one target collection + "property": { "$ref": "#/$defs/ReferenceValueProperty" }, // title | url | summary + "targetField": { "$ref": "#/$defs/FieldKey" } } } // or a field of the target collection + ``` + - Exactly one of `property` and `targetField` is given (checked in code). + - `targetField` is checked against the target collection's definitions with the preset table's type rules. So `location.address.streetAddress` can come from the venue's `street` field, and the motivation's "address retyped on every event" is solved, not just `location.name`. + - The value is read from the depth-1 member's resolved fields. + - `ReferenceValueProperty` is a named `$defs` enum, as RFC-0042 §1 requires for Rust. ### 5. Studio All new UI follows `kumo-design`, `studio-ui` and DESIGN.md "Fields", imports only from `@cloudhal/ui`, and puts its words in the lazy `fields` namespace. -**New blocks in `packages/ui/src/blocks`:** - -| Block | Built from (Kumo) | What it does | +| New block (`packages/ui/src/blocks`) | Built from (Kumo) | What it does | |---|---|---| | `EntryPicker` | `Dialog`, `Input` (search), `Combobox` list semantics, `Badge`, `Checkbox`, `Empty`, `Loader` | Search a site's entries in the allowed collections: title, collection badge, status badge, locale. Single or multiple choice, at most the remaining `maxItems`. Takes a `search(q)` prop, so the block makes no API calls. | -| `ReferenceList` | `InsetPanel` rows (existing block), `Button`, `Badge`, `DropdownMenu`, `Link` | The chosen targets in order: open, remove, move up/down. A "Draft" or "Archived" badge with "Will not show until published", and "Not found" for dangling ids. | -| `RepeaterField` | `Collapsible`, `Button`, `DropdownMenu`, `Text`, `Empty` | The list of items with "Add item", a count ("3 of 20") and per-item move/duplicate/remove. Each item renders the sub-field controls through a `renderControl` prop. | -| `FieldControl` | (extracted) | One control per type, taken out of `EntryFieldsForm.tsx`, which is already over the soft 400-line limit at 424 lines. `EntryFieldsForm` and `RepeaterField` then share it, and neither file grows. | -| `UsedByList` | `Table` or the `resource-list` Kumo block, `Badge`, `Link` | The entry editor's "Used by" card and the delete dialog's list of referrers. | +| `ReferenceList` | `InsetPanel` rows (existing block), `Button`, `Badge`, `DropdownMenu`, `Link` | The chosen targets in order: open, remove, move up/down. A "Draft" or "Archived" badge with "Will not show until published"; "Not found" for dangling ids. | +| `RepeaterField` | `Collapsible`, `Button`, `DropdownMenu`, `Text`, `Empty` | Items with "Add item", a count ("3 of 20"), and move/duplicate/remove per item. Sub-field controls come through a `renderControl` prop. | +| `FieldControl` | (extracted) | One control per type, taken out of `EntryFieldsForm.tsx` (already 424 lines, over the soft 400). `EntryFieldsForm` and `RepeaterField` both use it, so neither file grows. | +| `UsedByList` | `Table` / the `resource-list` Kumo block, `Badge`, `Link` | The entry editor's "Used by" card and the delete dialog's referrers. | - **Entry editor.** - - `EntryFieldsForm` gains a `renderReference?: (slot: FieldReferenceSlot) => ReactNode`, the same pattern as `renderImage` (`EntryFieldsForm.tsx:61-69, 97`). Studio's `FieldReference` (in `apps/studio/src/components/fields/`) composes `ReferenceList` and `EntryPicker` around `api.entries.pick`. - - Repeaters render inside the form with `RepeaterField`. - - On a translation, a non-localized reference or repeater is read-only, with "From the original" and a link (RFC-0041 §7). A localized one shows the original's value as a placeholder until edited. - - A new "Used by" `SectionCard` lists referrers (`GET …/entries/:id/references`). - - The delete dialog lists them too and disables Delete while any exist ("Remove it from these 4 entries first"). + - `EntryFieldsForm` gains `renderReference?: (slot: FieldReferenceSlot) => ReactNode`, the pattern of `renderImage` (`EntryFieldsForm.tsx:61-69, 97`). Studio's `FieldReference` (`apps/studio/src/components/fields/`) composes `ReferenceList` and `EntryPicker` around `api.entries.pick`. + - Repeaters render inside the form with `RepeaterField`. Image sub-fields reuse `FieldImage`. + - On a translation, a non-localized reference or repeater is read-only, with "From the original" and a link. A localized one shows the original's value as a placeholder. + - **"Used by":** a `SectionCard` with every referrer (`GET …/entries/:id/references`), published ones first. + - **The delete dialog:** it disables Delete only while a *published* referrer exists and this is the group's last member ("Unpublish or edit these 2 published entries first"). It lists draft referrers as "Will show nothing for this field" and lets the delete go ahead. - **Fields builder** (`FieldDialog.tsx`): - - **reference:** target collections (multi-select of declared collections), "Several" (`multiple`), the maximum, and optional conditions when one collection is chosen; - - **repeater:** a nested sub-field list (rows like `FieldRow` with up/down and ⋯), the maximum and the item label; - - the ⋯ menu's operations gain the sub-field forms (§7). -- **API** (site-scoped, under the existing `/sites/:siteId` membership check): - - `GET /sites/:siteId/entries/pick?collections=a,b&q=&locale=&limit=` returns at most 20 `{id, title, collection, status, locale}` per page, never archived ones. Scope `content:read`. `q` is at most 100 characters, matched as a case-folded substring (`includes`, no regex: rule 13) on titles and slugs. - - `GET /sites/:siteId/entries/:id/references` returns the referrers from `entry_refs`, paged. + - **reference:** target collections (a multi-select of declared collections), "Several" (`multiple`), the maximum, and conditions when one collection is chosen; + - **repeater:** a nested sub-field list (rows like `FieldRow`, with up/down and ⋯), the maximum and the item label; + - the ⋯ menu's operations gain the sub-field forms and the new `change_field_type` inputs (§7). +- **API** (site-scoped, under the `/sites/:siteId` membership check): + - `GET /sites/:siteId/entries/pick?collections=a,b&q=&locale=&limit=` answers at most 20 `{groupId, id, title, collection, status, locale}` per page, never archived ones. It needs `content:read`. `q` is at most 100 characters, matched as a case-folded substring on titles and slugs (`includes`, no regex: rule 13), under a per-member window of the existing `rate_limits` counters (`packages/api/src/rate-limits.ts`, migration 0030). + - `GET /sites/:siteId/entries/:id/references` answers the referrers from `entry_refs`, expanded to members, paged. - **Accessibility:** - - Each repeater item is a `fieldset` whose `legend` is its item label ("Speaker 2: Ada Lovelace"), as an ICU `selectordinal`/`plural` message, never concatenated. - - Moving is done with buttons ("Move Speaker 2 up"), never drag only. - - After "Add item", focus moves to the new item's first control. After "Remove", it moves to the next item's legend, or to "Add item" when none is left. A removal shows an undo toast. - - `EntryPicker` labels its search field, announces the result count in a polite live region, supports arrow keys and Enter in the list, and returns focus to the trigger on close. - - Errors map to their controls by path (`fields/speakers/2/person`) and are listed above the form when they sit in collapsed items. Those items open on submit. + - Each repeater item is a `fieldset` whose `legend` is its label ("Speaker 2: Ada Lovelace"), an ICU `selectordinal`/`plural` message, never concatenated. + - Moving is done with buttons ("Move Speaker 2 up"), never by dragging only. + - After "Add item", focus moves to the new item's first control. After "Remove", it moves to the next item's legend, or to "Add item"; an undo toast follows. + - `EntryPicker` labels its search field, announces the result count in a polite live region, supports arrow keys and Enter in the list, and returns focus to its trigger. + - Errors map to their controls by path (`fields/speakers/2/person`) and are listed above the form when they sit in collapsed items, which open on submit. - **Mobile (390 px):** - item actions fold into one `DropdownMenu` (⋯); - - sub-fields stack one per row; + - sub-fields stack; - `EntryPicker` takes the full width; - - `ReferenceList` rows wrap the badges under the title. + - `ReferenceList` rows wrap their badges under the title. - `pnpm design:shots` at 1440 and 390 must show no sideways scroll, and `design:lint` (including `rtl-logical`) must pass. -- **Bundle:** the blocks load with `EntryFieldsForm`, which loads only for collections with fields and is already lazy. The initial bundle stays at +0 kB (`bundle-budget.ts`). + `pnpm design:shots` at 1440 and 390 shows no sideways scroll, and `design:lint` passes, `rtl-logical` included. +- **Bundle:** the blocks load with `EntryFieldsForm`, which loads only for collections with fields and is already lazy. The initial bundle grows by +0 kB (`bundle-budget.ts`). ### 6. MCP and the JSON Schema agents receive - **Tools.** - `list_collections` returns the new definitions and their schemas. - - `create_entry`/`update_entry` precheck the reference and repeater rules with the store's checker, so a call the store would refuse never reaches a person. - - Approval prompts name targets readably: `venue → "Main hall" (entry_…, Places)`, `speakers: 2 items → 3 items (added "Ada Lovelace", Speaker)`, values quoted and truncated (`quote()`). - - `delete_entry` refuses in precheck with `entry_referenced` and the referrers. - - New read-only `list_references { siteId, entryId, direction: "to" | "from" }` (`content:read`): "who references me" and "what I reference", for an agent about to archive or delete. - - `list_entries` keeps working as the agent's picker, with `where: [{field: "venue", op: "eq", value: "entry_…"}]` once the field is listed. -- **`update_collection`'s input** embeds the contract's `FieldDefinition` `oneOf` inside `fields.items` (`mcp-server/src/collection-fields.ts:13-31`). OpenAI's strict mode does not document `oneOf`, so the MCP input schema emits the same variants as **`anyOf`**. Every variant has a distinct one-value `type` enum, so at most one can match and `anyOf` means exactly what `oneOf` does. The contract itself keeps `oneOf` (rule 2). -- **`fieldsSchema`** (`fieldsJsonSchema`, `json-schema.ts:145`) emits: - - `reference` as `{type: "string", pattern: "^entry_[A-Za-z0-9]{6,32}$"}`, or an array of them, with the target collections and `maxItems` in words; - - `repeater` as `{type: "array", items: {type: "object", properties: …, additionalProperties: false}}`. - - The constraints of the emitted schemas: + - `create_entry`/`update_entry` precheck the reference and repeater rules with the store's checker. + - Approval prompts name targets readably: `venue → "Main hall" (entry_…, Places)`, `speakers: 2 items → 3 items (added "Ada Lovelace", Speaker)`, with values quoted and truncated (`quote()`). + - `delete_entry` refuses in precheck with `entry_referenced` and the published referrers. + - New, read-only: `list_references { siteId, entryId, direction: "to" | "from" }` (`content:read`). + - `list_entries` with `where: [{field: "venue", op: "eq", value: "entry_…"}]` works once the field is listed. +- **`update_collection`'s input.** + - MCP tool calls stay non-strict, as RFC-0041 open question 3 decided, so the input keeps the contract's `oneOf`. The first draft proposed rewriting it to `anyOf` for strict modes. That is dropped: the variants' optional properties, `uniqueItems` and numeric bounds fail strict modes anyway. + - The real bug fixed here: `FIELD_DEFINITIONS_INPUT` hard-codes its `$defs` (`mcp-server/src/collection-fields.ts:22-39`), and the new variants `$ref` `SubFieldDefinition` and `FieldCondition`. Phase 1 replaces the list with `defsClosure(SiteSchema, "FieldDefinition")`, a transitive walk over `$ref`s (`hoistDefs` in `schema-transforms.ts:10` only hoists nested `$defs`, and finds no missing ones). + - A test asserts that every `$ref` in every tool's input and output schema resolves. + - The input lists only the *enabled* types, so agents see the new variants from the enabling step on, not before. +- **`fieldsSchema`** (`fieldsJsonSchema`, `json-schema.ts:145`), the default and open profile: + - `reference` is `{type: "string", pattern: "^entry_[A-Za-z0-9]{6,32}$"}`, or an array of them; + - `repeater` is `{type: "array", items: {type: "object", properties: …, additionalProperties: false}}`; - the root is always `type: "object"`, never `oneOf` or `anyOf`; - - no `$ref`, `$defs` or recursion: depth is at most 4 (fields → list → item → value), against OpenAI's 10; - - no unions in the default output; - - limits are in `description`, as today, because Anthropic drops `minimum`/`maxLength` and accepts `minItems` only as 0 or 1. - - `JSON_SCHEMA_KEYWORDS` (`json-schema.ts:29`) is unchanged, and the test holding the emitter to it stays. -- **A strict profile.** OpenAI's strict mode needs every property `required`, with optional ones as a `null` union. Today's output leaves properties optional, which suits Anthropic and non-strict MCP calls. `fieldsJsonSchema(defs, locale, defaultLocale, { strict: true })` therefore emits every property as required with `type: [T, "null"]`, plus `enum` with `null` for selects. `null` already means "leave unset or remove" in `update_entry`'s merge patch (`mcp-server/src/entry-fields.ts:45`), so the two profiles mean the same thing. - - `list_collections` returns `fieldsSchemaStrict` and `strictCost: { properties, depth, enumValues, unions }` beside it, counted by `strictSchemaCost(schema)`. - - Anthropic allows 16 union-typed parameters per request, so a collection with more than 16 leaves cannot be written in one strict Anthropic call, today included. The caller splits `update_entry` by keys, as RFC-0041 open question 3 decided. MCP tool calls themselves stay non-strict. -- **Example.** An `events` collection with `startsAt` (datetime), `venue` (a reference to `places`) and `speakers` (a repeater of `person` → `people`, `role` select, `talk` text): + - there is no `$ref`, `$defs` or recursion, and depth is at most 4 (fields → list → item → value) against OpenAI's 10; + - limits are written into `description`, because Anthropic drops `minimum`/`maxLength` and takes `minItems` only as 0 or 1. + + `JSON_SCHEMA_KEYWORDS` (`json-schema.ts:29`) is unchanged. +- **The strict profile and what `null` means.** OpenAI's strict mode needs every property required, with optional ones as a `null` union. Today's tools give `null` other meanings: `update_entry`'s merge patch deletes the key (`mergeFields`, `mcp-server/src/entry-fields.ts:45-53`), and `create_entry`'s checker refuses `null` as a value (`value-checks.ts:137-138`). Fed into `fields`, strict output would therefore delete untouched fields on update and be refused on create. So strict output gets its own input: + - `create_entry` and `update_entry` gain **`fieldsStrict`**, which excludes `fields` (`invalid` when both are given). In it, `null` means **"leave as it is"**: on create that is no value, and on update the stored value stays. + - A `null` sub-value inside a repeater item is stripped before checking: that sub-field is unset in that item. + - A given reference or repeater value replaces the whole value. + - Removals go in a separate **`clearFields: FieldKey[]`**, which works with either input. + - `fieldsJsonSchema(defs, locale, defaultLocale, { strict: true })` emits that meaning: every property required, `type: [T, "null"]`, and `enum` with `null` for selects. `list_collections` returns it as `fieldsSchemaStrict`, with `strictCost: { properties, depth, enumValues, unions }` counted by `strictSchemaCost`. + - **Over a provider's limit** (Anthropic: 16 unions per request), a caller builds a smaller schema from a subset of `fieldsSchemaStrict.properties`. Each property's schema stands alone, with no `$ref`, and splitting is safe precisely because `null` leaves other keys alone. + - Phase 5 tests the round trip: strict output that changes only `venue` leaves `startsAt` and `speakers` untouched; a create with nulls stores no value; nulls inside items are stripped. +- **Example.** An `events` collection with `startsAt` (datetime), `venue` (a reference to `places`) and `speakers` (a repeater: `person` → `people`, `role` select, `talk` text): ```json { @@ -403,7 +502,7 @@ All new UI follows `kumo-design`, `studio-ui` and DESIGN.md "Fields", imports on "venue": { "type": "string", "pattern": "^entry_[A-Za-z0-9]{6,32}$", "title": "Venue", - "description": "The id of an entry in Places (places) of this site, as list_entries returns it; a translation's id is replaced by its original's. Set on the original only; every language version shows the venue in its own language." + "description": "An entry in Places (places) of this site, by the groupId list_entries returns (an entry's own id is replaced by its group's). Set on the original only; every language version shows the venue in its own language." }, "speakers": { "type": "array", @@ -412,7 +511,7 @@ All new UI follows `kumo-design`, `studio-ui` and DESIGN.md "Fields", imports on "items": { "type": "object", "properties": { - "person": { "type": "string", "pattern": "^entry_[A-Za-z0-9]{6,32}$", "title": "Person", "description": "The id of an entry in People (people). Needed to publish." }, + "person": { "type": "string", "pattern": "^entry_[A-Za-z0-9]{6,32}$", "title": "Person", "description": "An entry in People (people), by groupId. Needed to publish." }, "role": { "type": "string", "enum": ["host", "speaker", "panelist"], "title": "Role", "description": "One of: host (Host), speaker (Speaker), panelist (Panelist)." }, "talk": { "type": "string", "title": "Talk", "description": "Plain text, at most 200 characters. One line." } }, @@ -424,154 +523,224 @@ All new UI follows `kumo-design`, `studio-ui` and DESIGN.md "Fields", imports on } ``` - In the strict profile, each property above is listed in its object's `required`, and `venue`, `role` and `talk` read `"type": ["string", "null"]` (`role`'s `enum` gains `null`). `startsAt` reads `["object", "null"]`, and `speakers` reads `["array", "null"]`. `strictCost` is `{ properties: 8, depth: 4, enumValues: 4, unions: 6 }`, within both providers' limits. + In the strict profile, every object lists all its properties in `required`. + - `venue`, `person`, `role` and `talk` read `"type": ["string", "null"]`, and `role`'s `enum` gains `null`. + - `startsAt` reads `["object", "null"]`, and `speakers` reads `["array", "null"]`. + - `strictCost` is `{ properties: 8, depth: 4, enumValues: 4, unions: 6 }` (`startsAt`, `venue`, `speakers`, `person`, `role`, `talk`), within both providers' limits. + - `list_entries` and the pick route return `groupId` for this reason. ### 7. Field operations (`fields-operations/`) -The four operations, their phases, their budgets and their revert stay as RFC-0041 §6 built them (`fields-operations/types.ts:22-50`). Entry rewrites carry their `entry_refs` rows, anchored on the new hash, so the index follows every phase B batch with no separate pass. A repeater's sub-field is addressed with a new optional `sub` parameter (`rename_field {from: "speakers", sub: "name", to: "fullName"}`), and the marker records it (`FieldMigration.sub`, §1). +The four operations, their phases, budgets and revert stay as RFC-0041 §6 built them (`fields-operations/types.ts:22-50`). Phase 2 already makes every rewrite carry its `entry_refs` rows and the target guard (§2.5). Phase 4 adds the new types: + +- **`sub`.** A repeater's sub-field is addressed with a new optional `sub` parameter (`rename_field {from: "speakers", sub: "name", to: "fullName"}`), and the marker records it (`FieldMigration.sub`, §1). +- **`change_field_type` gains the inputs the new definitions require** (`FieldOperation` in `types.ts:36-46`, the MCP tool and `cloudhal fields retype`): + - `collections?: string[]` (required to a reference); + - `maxItems?: number`; + - `subFields?: SubFieldDefinition[]` and `itemLabel?: FieldKey` (required to a repeater); + - `options` and `multiple` as today. + + The CLI takes `--collections a,b`, `--max-items 20` and `--sub-fields `. The dry run refuses a target definition that would not pass `checkFieldDefinitions`. | Operation | Reference | Repeater | |---|---|---| -| `rename_field` | moves the key; rows' `field` follow each rewritten entry; list fields and sections follow as today (`fields-operations/references.ts`) | moves the key, or with `sub` the sub-key in every item; rows `speakers.person` follow | -| `change_field_type` | see the table below | see the table below; with `sub`, a sub-field converts by RFC-0041's table among sub-field types | -| `set_field_localized` | true→false keeps the original's ids and deletes translations' copies (and their rows); false→true writes nothing | the same, for whole lists | +| `rename_field` | moves the key; rows' `field` follow each rewritten entry; list fields and sections follow as today (`fields-operations/references.ts`) | moves the key, or with `sub` the sub-key in every item; rows (`speakers.person`) and asset rows (`fields.speakers.photo`) follow | +| `change_field_type` | see below | see below; with `sub`, a sub-field converts by RFC-0041's table among sub-field types | +| `set_field_localized` | true→false keeps the original's ids and deletes translations' copies and their rows; false→true writes nothing | the same, for whole lists | | `remove_field` | deletes values and their rows; refused while list fields or presets use it (`list_field_in_use`) | deletes the list, or with `sub` the sub-value in every item (an item left empty is dropped) | -New rows of the closed conversion table (`conversions.ts:38`, `FIELD_CONVERSIONS`): +New rows of the closed table (`conversions.ts:38`): | from → to | conversion | unconvertible | |---|---|---| | reference single ↔ multiple | as select: one id ↔ a list of one; an empty list is no value | multiple → single with more than one id | -| text → reference | the one entry of the allowed collections whose slug, or default-locale title, equals the text exactly (case-folded) | no match, or more than one | -| reference → text | the target original's title in the site's default locale (a list: comma separated) | a missing target | +| text → reference | the one group among the allowed collections with a member whose slug, or default-locale title, equals the text exactly (case-folded) | no match, or more than one | +| reference → text | the group's default-locale member's title (a list: comma separated) | a group with no member in the site | | text (multiline) → repeater | one item per non-empty line into the repeater's single `text` sub-field (RFC-0042's ingredient lines) | a line over the sub-field's `maxLength`; more lines than `maxItems` | -| repeater → text | a repeater of one `text`/`richText` sub-field: items joined by line breaks (multiline) | other sub-field shapes | -| reference (multiple) ↔ repeater | a repeater of one `reference` sub-field: one item per id, and back | a repeater with other sub-fields | +| repeater → text | a repeater with a single `text`/`richText` sub-field: items joined by line breaks (multiline) | other sub-field shapes | +| reference (multiple) ↔ repeater | a repeater with a single `reference` sub-field: one item per id, and back | a repeater with other sub-fields | -Anything else is refused as today. Without `onInvalid: "clear"`, unconvertible values refuse the run, and the dry run counts and samples them first. +- **Where the graph reads happen.** The table is pure and synchronous (`conversions.ts:1-3`, `ConvertContext` at `:26-31`). The two text ↔ reference rows need entries, so `ConvertContext` gains an optional, prefetched `entries?: { byText: ReadonlyMap; titles: ReadonlyMap }` (case-folded slug or title → group ids; group id → default-locale title). Who builds it: + - each operation pass (dry run, phase B pass, revert pass), from one read of the target collections, before the pass; + - the store's `EntryEngine.write`, which is async, before it calls `normalizeEntryFields` (`entry-fields.ts:69-78`), only while a marker with such a conversion is active; + - the snapshot source, from its in-memory entries, lazily and only while the snapshot carries such a marker. -- **Revert** (`fields-operations/revert.ts`) restores each touched entry's earlier values. A restored reference whose target was deleted meanwhile is dropped from the restored value, and its id is reported in a new `FieldRevertResult.dropped: { entryId, field, targetId }[]`. The restore statement carries §2's target guard, so a target deleted during the revert fails that statement and is retried on the next pass. A remove can make a target deletable, and that deletion must not make the revert write a dangling id. + A caller without it (the live source) resolves an unconverted value to `undefined` with an issue, which renders nothing and is never wrongly typed. Readers, writers and phase B therefore convert identically, and RFC-0041 §6's "a person's edit and the operation converge" holds. +- **Phase B writes references only under the target guard** (§2.5). A missed guard skips the entry for the next pass, so a conversion can never write a deleted target. +- **Revert** (`fields-operations/revert.ts`) restores earlier values under the same guard. A restored reference whose group has no member left in the site is dropped and reported in a new `FieldRevertResult.dropped: { entryId, field, targetId }[]`. - **Free definition edits:** - - add a sub-field; - - reorder sub-fields; + - add or reorder sub-fields; - change labels, help, `itemLabel` or `where`; - widen `maxItems` or `collections`; - - narrow `maxItems`: existing values are checked on their next write, as RFC-0041 §6 already does for narrowed limits. + - narrow `maxItems`: values are checked on their next write (RFC-0041 §6). ### 8. Threat model | Threat | Mitigation | Test | |---|---|---| -| **A draft, archived, gated or hidden target leaks through a published referrer** (the snapshot holds drafts, §2) | Resolution reads only `Loaded.reachable`. A card carries only the target's public title, summary, URL and cover image, and the image goes through the existing asset gates. Unresolved values render nothing: no id, no `data-` attribute, no empty link. JSON-LD's `reference` source (phase 7) uses the same cards. | RFC-0008's route sweep (extended by RFC-0042 for `hide`) is extended again: a published referrer pointing at a draft, an archived, an embargoed and a `hide`-expired target. No id, slug, title or asset of those targets may appear in HTML, JSON-LD, feeds, the sitemap, search, `llms.txt`, status pages or cache tags. | -| **IDOR in the entry picker and on write** | Pick and references routes run under the `/sites/:siteId` membership check and query by `site_id` only. The write guard requires `n.site_id = ?` (§2.1). An id from another site or tenant gets the same `unknown_entry` as an id that does not exist, so the answer reveals nothing. Language editions map ids and never read across sites (§3). | A shared-suite case: another site's entry id in `create_entry`, `update_entry`, the API and `push` is refused identically. The pick route with another site's id in the path answers 404. | -| **Fan-out DoS on render** | Contract caps (50 per field, 50 items, 10 sub-fields), the code cap (100 reference values per entry), depth ≤ 2, the visited set, and 200 targets per page. Live reads are ≤ 2 per page, snapshot reads 0. `referencedBy` is capped at 100. | A property test: random graphs with cycles and 10× the caps resolve within the budget, never throw, and cost at most `depth` store reads on the live source. | -| **Write amplification and slow guards** | Each guard is one indexed probe or one `json_each` count (`entry_refs_target`). Delete and move cost one probe. Rebuilds are chunked statements inside one batch (§2.5). | The benchmark of RFC-0042 #105, extended to 10,000 events × 3 references on Miniflare D1: write, delete and move guard costs are reported. | -| **Picker search abuse** | `q` ≤ 100 characters, a substring match with no regex (rule 13), at most 20 results per page, and a per-member window on the pick route through the existing `rate_limits` counters (`packages/api/src/rate-limits.ts`, migration 0030). | A 100k-character `q` is refused before it is read. | -| **Prompt injection through target titles** | Titles are data. Approval prompts quote and truncate them (`quote()`, 60 characters), and `list_references` marks them as content, never instructions. | Existing approval-prompt tests gain reference cases. | -| **A broken archive or push slips dangling ids in** | Import validates nodes (`assertValidRows`) and rebuilds the index. Dangling ids are reported, never resolved and never thrown on. | The archive round-trip test with a hand-edited dangling id. | -| **An older site Worker crashes on the new types** (`COMPILE[def.type]`, `props.ts:45`, has no entry) | `GET /__cloudhal/features` adds `{ "fields": 2 }`. Writers refuse reference and repeater definitions with `409 fields_unsupported` until the site Worker reports it, as `gates: 2` guards `expiry` (`availability.ts:243`, `cloudflare/src/features.ts`). Phase 1 also makes `resolveFields` skip unknown types, so later types are safe on older renderers. | The site Worker features test; a renderer test with an unknown field type. | - -No new scopes. Field definitions remain site settings (`site:settings`; agents use `site:write` plus a person's approval, `person`-classed, as today, `field-tools.ts:264-267`). Values are content (`content:write`, with the `:own` limits). Reading targets in the picker and in `list_references` needs `content:read`, which every role already holds (`agent-runtime/src/roles.ts` `READ`). Referencing an entry never writes it. - -### 9. Rollout (one PR each) +| **A draft, archived, embargoed or hidden target leaks through a published referrer** (the snapshot holds drafts, and `Loaded` holds published entries before gates) | Resolution filters through the view's gates at its instant (§4); only gate-independent card parts are cached. A card carries only the member's public title, summary, URL and cover image, the image through the view's asset gates. Unresolved values render nothing: no id, no `data-` attribute, no empty link. JSON-LD's `reference` source uses the same cards. | RFC-0008's route sweep (extended by RFC-0042) gains a published referrer pointing at a draft, an archived, an embargoed and a `hide`-expired target, before and after their instants. No id, slug, title or asset of a hidden target may appear in HTML, JSON-LD, feeds, the sitemap, search, `llms.txt` or status pages. Cache tags may carry group ids (§2). | +| **IDOR in the picker and on write** | Pick and references routes run under the `/sites/:siteId` membership check and query by `site_id`. The target guard requires a member with `n.site_id = ?` (§2.1). An id of another site gets the same `unknown_entry` as an id that does not exist. Editions copy only ids the edition site can resolve (§3). | A shared-suite case: another site's entry or group id in `create_entry`, `update_entry`, the API and `push` is refused identically; the pick route with another site's id answers 404. | +| **An imported or pushed archive pins another tenant's entries** | The rebuild inserts only rows whose target has a member in the same site, and every guard and referrer query filters `site_id` (§2.3). | An archive referencing site Y's entry, imported into site X, must not block Y's delete or move, and must not list X's titles to Y. | +| **Lock-out by low-privilege actors** (`content:draft:own`, `content:write:own`) | Only published referrers block delete and move (§2.2). Contributors cannot publish, and drafts, review and archived entries never lock. | Policy-suite cases: a contributor's draft and an author's review entry referencing another author's entry do not block its delete; an author's published entry does. | +| **Fan-out DoS on render** | Caps of 50 per field, 50 items and 10 sub-fields (contract); 100 reference values per entry (code); depth ≤ 2 with a visited set; 200 targets per page; ≤ 2 live reads per page, 0 for snapshots; `referencedBy` ≤ 100. | A property test: random graphs with cycles and 10× the caps resolve within budget, never throw, and cost at most `depth` live reads. | +| **Write amplification and slow guards** | Each guard is one indexed probe on `(site_id, target_id)` or one `json_each` count; members are found through `nodes_translation_locale`. Delete and move cost one probe each. Rebuilds are chunked inside one batch. | RFC-0042's benchmark (#105), extended to 10,000 events × 3 references on Miniflare D1, reports the write, delete and move guard costs. | +| **Picker search abuse** | `q` ≤ 100 characters, a substring match with no regex (rule 13), ≤ 20 results, and a per-member rate window. | A 100k-character `q` is refused before it is read. | +| **Prompt injection through target titles** | Titles are data, quoted and truncated in approval prompts (`quote()`, 60 characters); `list_references` marks them as content. | Approval-prompt tests gain reference cases. | +| **An older site Worker crashes on the new types** (`COMPILE[def.type]`, `props.ts:46`, has no entry) | `GET /__cloudhal/features` adds `{ "fields": 2 }`. Writers refuse the new types with `409 fields_unsupported` until the site Worker reports it, as `gates: 2` guards `expiry` (`availability.ts:243`, `cloudflare/src/features.ts`). `resolveFields` skips unknown types from phase 1. | The site Worker features test; a renderer test with an unknown field type. | +| **Assets in repeaters deleted while used** | `fieldAssetRefs` walks items (§2.4), so the guard, rebuild and purge cover them. | An asset used only in a repeater item cannot be deleted; the rebuild keeps its row; a retired or foreign asset id in an item is refused. | + +**Policy.** No new scopes. +- Field definitions stay site settings: `site:settings`; for agents, `site:write` plus a person's approval, person-classed as today (`field-tools.ts:264-267`). +- Values are content: `content:write`, with the `:own` limits. +- Reading targets in the picker and `list_references` needs `content:read`, which every role already holds (`roles.ts` `READ`). +- Referencing an entry never writes it, and only a published referrer can stop its delete or move. + +### 9. Rollout (one PR each; the types stay refused until step 7) 1. **Contract and pure core:** - - the §1 diff, fixtures and `pnpm gen`, through the `contract-change` skill; - - definition checks, `fieldPropSchema` for both types, value checks, `resolveFields` (and skipping unknown types); - - `fieldsJsonSchema` with the strict profile and `strictSchemaCost`; - - `referenceIssues`. + - the §1 diff, fixtures and `pnpm gen` through `contract-change`; + - `checkFieldDefinitions` with the new rules, **refusing both types** (`fields_unsupported`); + - `fieldPropSchema`, value checks, `resolveFields` (skipping unknown types); + - `fieldsJsonSchema` (open and strict) with `strictSchemaCost`, `referenceIssues`, `pickMember`; + - the dispatch-table entries of §1; + - `defsClosure` and the resolving-`$ref`s test for every tool schema, with the new variants filtered out of `update_collection`. No behaviour change. -2. **Storage and store:** - - migration `0033_entry_refs`; - - the write guard and rows, and the delete and move guards; - - the definition-change refusals; - - `entryRefsRebuild` in import, sync and restore (chunked), and `referenceIssues` in `import --dry-run`; +2. **Store and integrity:** + - migration 0033; + - the target guard, the definitions anchor and the rows; + - the delete, move and publish rules with the policy (§2.2); + - site-checked, chunked `entryRefsRebuild` in import, sync and restore, and `referenceIssues` in `import --dry-run`; + - `fieldAssetRefs` walking items; + - rows and the target guard in field-operation rewrites, with operations on the new types refused; + - the kind-aware release `depsOf`; + - edition id mapping in `forEdition`; - the `fields: 2` writer guard; - API: entry routes, `pick` and `references`. + + The shared suites enable the types with a store option (`SiteGraphStoreOptions.fieldTypes`); production keeps refusing them. 3. **Renderer:** - - `entryFields` cards, depth and budget; - - `entriesByGroups`/`referencedBy` on both sources, and the snapshot indexes; + - cards, depth and budget, gated per view; + - `groupMembers`/`referencedBy` on both sources; - `EntryFields.astro` (links for references, a nested `
` per item); - - cache tags, and features `fields: 2`; + - `cloudhal-ref-` tags and the purge planner rules; + - features `fields: 2`; - the route sweep extension; - a frozen theme copy for `old-theme-compat.test.ts`. -4. **Field operations:** `sub`, the conversion rows, revert `dropped`, and the CLI flags (`cloudhal fields rename|retype|localize|remove --sub`). +4. **Field operations:** `sub`, the new `change_field_type` inputs, the conversion rows with the prefetched context, revert `dropped`, and the CLI flags. 5. **MCP:** - - `list_collections` (definitions, `fieldsSchema`, `fieldsSchemaStrict`, `strictCost`); - - the entry prechecks, `delete_entry`'s refusal and `list_references`; - - `anyOf` in `update_collection`'s input; - - the operation tools' `sub`; + - `list_collections` (`fieldsSchema`, `fieldsSchemaStrict`, `strictCost`); + - `fieldsStrict` and `clearFields`; + - prechecks, `delete_entry`'s refusal and `list_references`; + - the translation tools' repeater paths; + - the operation tools' new inputs; - regenerated tool docs (`node scripts/mcp-tools-doc.ts`). 6. **Studio:** the five blocks, `FieldReference`, the builder, "Used by", the delete dialog and the publish warnings; `design:shots`; the `fields` namespace and `pnpm i18n:import`. -7. **Listings and structured data:** reference list fields (rows, `checkFieldQuery`, the property suite), the `reference` source kind for presets, and `docs/guides/fields.md`. - -Phase 3 ships before phases 5 and 6, so no agent or person can define a type that the live site cannot render; the `fields: 2` guard enforces it anyway. Phase 7 is independent of 4 to 6. +7. **Enable** (small): `checkFieldDefinitions` accepts the two types where the writer guard confirms `fields: 2`; `update_collection`'s input lists them; `docs/guides/fields.md`. It needs steps 2 to 6 released. +8. **Listings and structured data:** reference list fields (rows, `checkFieldQuery`, the property suite) and `ReferenceValueSource` with `targetField`. -**Test plan.** Every store-facing behaviour goes into the shared suites (`packages/core-ts/src/testing-*.ts`), which run on SQLite, Miniflare D1 and Durable Object SQLite: +**Test plan.** Store behaviour goes into the shared suites (`packages/core-ts/src/testing-*.ts`), on SQLite, Miniflare D1 and Durable Object SQLite. - `testing-references.ts` (new): - - writing, canonicalizing a translation's id to the original's; - - refusing a missing, cross-site, wrong-collection, self or archived new target; - - delete and move refusals, and the races of each guard against a concurrent write (two batches interleaved); - - rows after every write, delete, import, restore and operation phase. -- `testing-fields.ts`: repeater values, `requiredToPublish` per item, the leaf and reference caps. -- `testing-field-operations.ts`: every new conversion row, `sub` operations, resume, and revert with a deleted target. -- `testing-list-values*.ts`: reference list fields, including the seeded random comparison of SQL and evaluator. -- **Pure tests:** the emitter against `JSON_SCHEMA_KEYWORDS`, depth ≤ 4, no recursion, root `object`, `strictCost`; `referenceIssues`; the locale rule; resolution budget and cycles (property-based). -- **Renderer:** snapshot and live sources give identical cards; the route sweep (§8); cache tags. -- **Studio:** block tests (keyboard, focus after add and remove, live region), `design:shots` at both widths, `design:lint`. + - writes, and canonicalizing a member's id to its group; + - refusing a missing, cross-site, wrong-collection, own-group or archived new target; + - the definitions-anchor races: a reference field added, or an allowed collection removed, between plan and batch; + - delete: the last member versus a remaining member, and published versus draft referrers; + - move, of an original and of a translation; + - publish re-checks; + - the same target in two items; + - an edition site referencing a cross-site group; + - rows after every write, delete, import, restore and operation phase; + - a foreign-tenant archive. +- `testing-fields.ts`: repeater values, `requiredToPublish` per item, the leaf and reference caps, assets in items. +- `testing-assets.ts`: an asset used only in a repeater is not deletable, and its row survives a rebuild. +- `testing-field-operations.ts`: every new conversion row, `sub`, a target deleted during phase B, resume, revert with a deleted target. +- `testing-releases.ts` (today on SQLite and D1): a scheduled referrer of a non-live target is not held and renders without it; an edition's group id never fails the run. +- `testing-translations.ts`: edition mapping, and drops reported. +- `testing-list-values*.ts`: reference list fields, the seeded SQL-versus-evaluator comparison included. +- Pure tests: the emitter's keywords and depth, root `object`, no recursion, `strictCost`; the strict round trip (phase 5); `referenceIssues`; `pickMember`; budget and cycles, property-based. +- Renderer: identical cards on the snapshot and live sources, gate instants, `referencedBy` through translations, cache tags and purges, including a target that becomes visible. +- Studio: block tests (keyboard, focus after add and remove, live region), `design:shots` at both widths, `design:lint`. ## Alternatives considered | Option | Pros | Cons | Why not | |---|---|---|---| -| Weak references by default (deletes allowed, dangling ids kept) | No refusals for people | Silent breakage; "who uses this" still needs the index; Sanity made strong the default | Strong by default; readers tolerate dangling anyway (§2) | -| Cascade or "unlink and delete" on target delete | One click | Rewrites other people's entries as a side effect of a delete; needs an operation, approval and a revert of its own | Refuse and list referrers; open question 1 | -| Block publish (or fail the build) on unresolved references | Strict | Publish is a site snapshot; one draft venue would hold back every unrelated fix; "a published snapshot must keep rendering" | Warn in Studio, MCP and CLI (§2) | -| Reference an entry-locale (a specific node) | Exact; no resolution step | Every translation re-picks targets; a new translation of the target never appears; non-localized references become impossible | Translation groups, resolved per page locale (§3) | -| Polymorphic `{collection, id}` values (Payload) | Explicit collection | The id already names its node; two values to keep consistent; larger strict schemas | Bare ids, collection checked on write | -| Reverse lookups by scanning `json_extract` over `nodes` | No table | Every delete and move scans the site's entries; repeater paths need `json_each` over every item | `entry_refs`, the `asset_refs` pattern | -| `blocks` (a heterogeneous list: each item picks one of several shapes) | ACF flexible content, Payload blocks | Each item needs a discriminated union: `anyOf` per item counts against Anthropic's 16 unions; a second Studio form engine; overlaps Cloudhal's own body directives, surfaces and page sections, which already compose varied content | Repeaters only; varied layouts stay in bodies and sections | -| Nested repeaters (depth 2+) | Richer models | Recursion or a second subset level; strict depth and property counts multiply; nested Studio forms on 390 px | Depth fixed at 1 by the contract | -| Per-sub-field localization in a shared list | Shared structure, translated words | Needs stable item keys and a merge by key on every read; two value shapes per field | Repeater-level `localized`; open question 2 | -| Sort or filter by a referenced entry's field | "Events by venue city" in one query | Denormalized values that every target edit invalidates across collections | Two-step queries (§4) | -| Unbounded resolution with a request timeout | Simple themes | Cost depends on content, not code; cycles | Depth ≤ 2, breadth-first, 200 per page | -| `oneOf` in the MCP input as in the contract | One source | OpenAI's strict mode does not document `oneOf` | `anyOf` over disjoint variants, identical meaning (§6) | +| Store a per-site anchor member's id instead of the group id | No foreign ids in values | Editions need an id rewrite per site; the anchor changes when the original is deleted; every lookup maps member → anchor | The group id is project-stable and already indexed (§1) | +| Anchor every entry write on the site hash | Simple | Any site edit (theme, menus) conflicts every concurrent entry write | Anchor on the collection's definitions only (§2.1) | +| Every reference locks, drafts included (Sanity's strong default) | Strictest | Contributors pin others' entries; abandoned drafts pin forever | Only published referrers lock (§2.2) | +| Weak references (no locking at all) | No refusals | Live pages silently lose venues on delete; "used by" still needs the index | Lock where the live site depends | +| Cascade or "unlink and delete" | One click | Rewrites other people's entries as a side effect | Refuse and list; open question 1 | +| Block publish or fail the build on unresolved references | Strict | Publish is a site snapshot; one draft venue blocks every fix | Warn (§2) | +| References as release dependencies (today's generic extractor) | A scheduled event takes its venue live with it | Locks every member of the target until the run; edition group ids fail runs; hard, where render is soft | Kind-aware `depsOf` drops them (§2.6) | +| Tag only rendered targets; forbid group ids in tags | Fewer tags | Newly visible targets never purge their referrers; per-member purges miss group-keyed referrers | `cloudhal-ref-` for every value read; tags are edge-only | +| `null` in strict output means "remove", as in `fields` | One input | Following the schema deletes every untouched field | `fieldsStrict` with `null` = unchanged, plus `clearFields` | +| `anyOf` in `update_collection`'s input for strict modes | Looks strict-friendly | Optional properties and bounds still fail strict modes | Inputs stay non-strict; `$defs` fixed instead | +| Reverse lookups by scanning `json_extract` over `nodes` | No table | Every delete and move scans the site | `entry_refs`, the `asset_refs` pattern | +| `blocks` (a heterogeneous list) | ACF flexible content, Payload blocks | A union per item (Anthropic: 16 per request); a second form engine; overlaps body directives, surfaces and sections | Repeaters only | +| Nested repeaters | Richer models | Recursion or a second subset level; strict limits multiply | Depth fixed at 1 | +| Per-sub-field localization in a shared list | Shared structure, translated words | Stable item keys and a merge by key on every read | Repeater-level `localized`; open question 2 | +| Sort or filter by a referenced entry's field | One query | Denormalization invalidated by every target edit | Two-step queries (§4) | +| Drop the text ↔ reference conversions | A purely pure table | Loses the main migration path (organizer text → a reference) | A prefetched conversion context (§7) | ## Compatibility & migration -- **Contract:** two new variants, a new `$defs` and one optional marker property. Existing sites, snapshots, bundles and archives stay valid. No `schemaVersion` bump. - - An older admin Worker refuses the new definitions (`additionalProperties: false` inside `oneOf`), so servers upgrade first, as with every addition. - - An older site Worker is covered by `fields: 2` (§8). -- **Data:** none to migrate. Migration 0033 is DDL only, because no reference value can exist before phase 2: `field_key_in_use` refuses definitions over keys that already hold values. -- **Themes:** additive. `entry.fields` keeps meaning what is stored: ids and item objects. Old theme copies render the new types as nothing, since `EntryFields.astro` ships in the renderer package and shows them once updated. Themes opt into cards, depth and `referencedBy`. -- **Studio:** additive types in the builder and form. +- **Contract:** two new variants, a new `$defs` and one optional marker property. Existing sites, snapshots, bundles and archives stay valid, and there is no `schemaVersion` bump. + - An older admin Worker refuses the new definitions, so servers upgrade first. + - An older site Worker is covered by `fields: 2`. +- **Data:** none to migrate. The types are refused until step 7, so migration 0033 is DDL only, and the definitions anchor keeps `field_key_in_use` honest. +- **Themes:** additive. `entry.fields` keeps meaning what is stored: group ids and item objects. Old theme copies render the new types as nothing. `EntryFields.astro` ships in the renderer package. +- **Releases:** the kind-aware `depsOf` changes nothing for existing content, because no field value can be a group id before step 7. - **Rollback:** - A snapshot restores the site node and entries, and the index is rebuilt in its batch. - - A field operation reverts through its `before` snapshot, dropping deleted targets (§7). - - Removing a reference or repeater field with `remove_field` removes its values and rows. - - Downgrading below this RFC restores a snapshot from before the first reference or repeater definition, because the old validator refuses them. The `entry_refs` table is then unused and harmless. + - An operation reverts through its `before` snapshot, dropping deleted targets. + - `remove_field` removes values and rows. + - Downgrading below this RFC restores a snapshot from before the first reference or repeater definition; `entry_refs` is then unused. ## Security & policy impact -§8 is the threat model. In short: -- **No new scopes and no new approval points:** - - definitions stay `site:settings` and person-classed; - - values stay content; - - `list_references` and the picker need `content:read`. -- **Tenant isolation:** every statement filters on `site_id`, every guard re-checks it, and nothing resolves across sites. -- **New refusals visible to agents:** `entry_referenced`, `reference_target_moved`, `unknown_entry`, `too_many_references`, `field_key_in_use`, `fields_unsupported` and `translation_shape_mismatch`, each with a hint in `field-hints.ts`. +§8 is the threat model and the policy. +- No new scopes and no new approval points. +- Every statement filters on `site_id`, every guard re-checks it, and nothing resolves across sites. +- New refusals, each with a hint in `field-hints.ts`: `entry_referenced`, `reference_target_moved`, `unknown_entry`, `too_many_references`, `field_key_in_use`, `fields_unsupported`, `field_operation_unsupported`, `translation_shape_mismatch`. ## Open questions (each with a recommended answer) -1. **Deleting a referenced entry:** refuse and list the referrers; no cascade and no "unlink and delete" in stage 4. *Recommended.* If people ask for it, add an explicit `unlink_entry` operation (dry run, approval, snapshot, revert) rather than a delete flag. -2. **Per-sub-field localization in a non-localized repeater:** not in stage 4. Repeaters are localized as a whole, and Studio marks shared text "not translated". *Recommended.* Revisit with stable item keys (`_key`, which cannot collide with a `FieldKey`) when a real case needs shared structure with translated words. -3. **`datetime`, multiple selects and multiple references inside repeaters** (an agenda's session times): not in stage 4, because the props subset's array items are flat scalars (ADR-0030). *Recommended:* the first follow-up, by allowing ADR-0030's flat object one level below an array item for `datetime` only (an ADR-0030 amendment), measured against strict depth (5 of 10). -4. **The locale rule with `fallback: none`:** a target with no version in the page's language resolves to nothing, and `referenceIssues` reports it. *Recommended*, because it matches how pages behave under `none`. The alternative, linking the original in its own language with `lang`, could be a per-field option later. -5. **Strong references to archived targets:** an existing value keeps pointing at an archived target, which renders nothing; a *new* choice is refused. *Recommended.* The alternative is to refuse archiving while referenced, which makes retiring old venues painful. -6. **`where` on reference fields:** checked when a value is newly chosen only, never re-checked (Sanity's `filter` model). *Recommended.* -7. **Default depth for themes:** 1, with 2 as the ceiling. *Recommended.* Payload populates two levels by default, but a card is what themes draw. -8. **`fieldsSchemaStrict` beside `fieldsSchema`, or a `strict` argument on `list_collections`:** beside it, with `strictCost`. *Recommended:* it is small, and agents need not know their provider's mode in advance. Splitting a collection into strict-sized groups automatically is left out. -9. **Publish warnings as a gate:** never; a warning only. *Recommended.* A future site setting could turn them into a person-approval point for agents' publishes, like RFC-0042 §3.5's expiry rule. -10. **Structured data from references** (phase 7): the `reference` source kind in this RFC rather than a later one. *Recommended*, because RFC-0042 §6 named it as stage 4's starting point. -11. **The `asset_refs` rebuild chunking** (§2.5): fix it in its own small PR before phase 2, since import of large sites can hit D1's 100 KB statement limit today. *Recommended.* +1. **Deleting a referenced entry:** refuse while a published referrer exists and this is the group's last member in the site; no cascade. *Recommended.* Add an explicit `unlink_entry` operation (dry run, approval, snapshot, revert) if people ask for it. +2. **Per-sub-field localization in a shared repeater:** not in stage 4. *Recommended.* Revisit with stable item keys (`_key`, which cannot collide with a `FieldKey`). +3. **`datetime`, multiple selects and references inside repeaters** (an agenda's session times): not in stage 4, because ADR-0030's array items are flat scalars. *Recommended:* the first follow-up, allowing a flat object one level below an item for `datetime` only (an ADR-0030 amendment), at a strict depth of 5 of 10. +4. **The locale rule under `fallback: none`:** no version in the page's language resolves to nothing, and `referenceIssues` reports it. *Recommended*, because it is how pages behave under `none`. +5. **Archived targets:** existing values keep pointing at an archived group, which renders nothing; a *new* choice needs a member that is not archived. *Recommended.* +6. **`where` on references:** checked when a value is newly chosen only (Sanity's `filter` model). *Recommended.* +7. **Default render depth:** 1, with 2 as the ceiling. *Recommended.* +8. **`fieldsSchemaStrict` beside `fieldsSchema`**, with `strictCost`, and no automatic splitting into groups. *Recommended.* +9. **Publish warnings as a gate:** never; warnings only. *Recommended.* A site setting could later make them a person-approval point for agents' publishes, like RFC-0042 §3.5's expiry rule. +10. **Review referrers:** they do not lock, like drafts. *Recommended*, so that a contributor's review request never pins an entry. The alternative, locking from review, protects content about to go live, but it reopens the contributor lock-out. +11. **`asset_refs` follow-ups found here:** `refsRebuild` puts a whole site in one `INSERT`, which can exceed D1's 100 KB statement on large imports, and `deleteAsset`'s guard (`assets.ts:867`) and the rebuild have no `site_id` filter, the same cross-tenant pin as §2.3. *Recommended:* one small PR before phase 2, outside this RFC. + +## Review log (2026-11-10, PR #193) + +An independent multi-agent review upheld 28 findings and split two. Each one, with the decision taken: + +| # | Finding | Decision | +|---|---|---| +| U1, U18 | Repeater image and rich-text sub-fields are missing from `asset_refs` | `fieldAssetRefs` walks items under `fields..` (§2.4, phase 2, tests) | +| U2, U14, U20 | "The original in this site" is undefined for editions; lookup keys mismatch | Values store the group id; members are resolved and guarded in-site (§1, §2.1, §3, §4) | +| U3 | Entry writes are not anchored on the definitions | Definitions anchor on every entry write that changes fields (§2.1); races in the test plan | +| U4 | Group members may sit in other collections | Resolution and `referenceIssues` filter by `collections`; the move guard covers any member (§2.2, §3) | +| U5 | Reference conversions need reads; the table is pure | A prefetched `ConvertContext.entries`; who builds it is stated (§7) | +| U6, U27 | Phase B writes references unguarded | Target guard in every rewrite that changes references (§2.5, §7) | +| U7 | `referencedBy` misses inheriting translations | `source_group` column and expansion; the snapshot map is built from resolved values (§2, §4) | +| U8, U28 | Guards cannot use the index | `site_id` in every guard (§2) | +| U9, U15 | Resolution on the gate-free `Loaded` | Gated per view; only gate-independent parts cached (§2, §4, §8) | +| U10, U19 | Strict `null` contradicts the tools | `fieldsStrict` (`null` = unchanged) plus `clearFields`; round-trip tests (§6) | +| U11, U23 | Tag arithmetic; no purge on newly visible targets | `cloudhal-ref-` for every value read, a 4 KB budget with a site-level fallback tag, group purges (§2) | +| U12 | Rebuilds pin foreign tenants | Rebuild joins same-site members; `site_id` filters; `asset_refs` follow-up (§2.3, OQ 11) | +| U13 | Lock-out by `:own` actors | Only published referrers lock (§2.2, §8, OQ 10) | +| U16 | Phase 1 accepts values before the index exists | Types refused until step 7; the store option for suites; the dispatch tables listed (§1, §9). `archive-reassemble.ts`'s tables key manifest types and are not affected. | +| U17, S1 | Dangling `$defs`; `anyOf` is not strict anyway | `defsClosure` plus a `$ref` test in phase 1; the `anyOf` plan is dropped; the input stays non-strict (§6) | +| U21 | Edition mapping and repeater MT in no phase | Edition mapping in phase 2, repeater MT in phase 5 (§3, §9) | +| U22 | Operations leave `entry_refs` stale until phase 4 | Rows in rewrites from phase 2; operations on new types refused until phase 4 (§2.5) | +| U24 | Primary-key collision across items | Pairs deduplicated, `INSERT OR IGNORE` (§2) | +| U25 | The Event address cannot come from a Place | `ReferenceValueSource.targetField` (§4) | +| U26 | `change_field_type` lacks inputs | `collections`, `maxItems`, `subFields`, `itemLabel`, and CLI flags (§7) | +| S2 | The release closure treats ids as dependencies | Kind-aware `depsOf` drops references; a `testing-releases` case (§2.6) | From 9e457d12e45a03fa4064a7841ce303abc6926869 Mon Sep 17 00:00:00 2001 From: Madraka <8898640+Madraka@users.noreply.github.com> Date: Tue, 10 Nov 2026 15:36:09 +0300 Subject: [PATCH 3/8] docs(rfc): RFC-0045 second revision after the verification pass Folds in every unresolved finding and new issue from the verification of fe8ac929, extending the review log: - group changes are guarded like deletes, with the planned site's hash - checkBundle gates the new types on import, sync, push, pull and restore, with a one-time per-site rebuild at enable - entry_refs rows carry shared/root_key mirroring resolveFields; one "shown by a published entry" fragment serves guards, live referencedBy and purges - the definitions anchor binds the planned Site's definitions - rows are kept for every stored value; entry_refs joins SITE_TABLES - cloudhal-ref tags are dropped: the purge planner adds referrers' node tags through entry_refs (no new prefixes, no header ids) - repeater asset checks and the purge asset rule are specified - the asset_refs site scoping becomes rollout step 0 - races R1-R11 are listed with their guards - edition chains, emptied items, the conversion context, the fieldsStrict edge cases and the Event address data path are settled Co-Authored-By: Claude Opus 5.5 Signed-off-by: Madraka <8898640+Madraka@users.noreply.github.com> --- ...ctions-stage-4-references-and-repeaters.md | 428 ++++++++++++------ 1 file changed, 291 insertions(+), 137 deletions(-) diff --git a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md index 4127431a..2fc47379 100644 --- a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md +++ b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md @@ -1,6 +1,6 @@ # RFC-0045: Collections stage 4: reference fields and repeaters -- Status: Draft (revised 2026-11-10 after an independent review of PR #193; the review log is at the end) +- Status: Draft (revised twice on 2026-11-10, after an independent review of PR #193 and a verification pass; the review log is at the end) - Author: Halil (Madraka), drafted with Claude · Date: 2026-11-10 - Related: - [#114](https://github.com/cloudhal-dev/cloudhal/issues/114) (this stage); #71 (the collections plan). @@ -26,11 +26,17 @@ Two new field types join RFC-0041's nine: - A **`reference`** field points to entries of one to five declared collections of the same site, one target or up to fifty. It stores the target's **translation group id**, the same id in every site of a project. It always resolves to that group's member in this site and in the page's language. - A **`repeater`** field is an ordered list of up to fifty items, each of up to ten flat sub-fields. A repeater cannot contain a repeater, so every agent-facing schema has a fixed depth and no recursion. -References are strong where it matters: a target cannot be deleted or moved while a published entry shows it. A reverse index, `entry_refs`, is written in each entry's own batch, anchored on the entry's new content hash and on its collection's definitions as read, scoped by site like every other guard. References from drafts never lock anything. Archiving or unpublishing a target is always allowed: the renderer resolves only what the published snapshot shows *at that instant*, gates included, so an unresolved reference renders as nothing and publish warns but never fails. Assets used inside repeaters count in `asset_refs`, like any field's assets. +References are strong where it matters. A target cannot be deleted, moved, or taken out of its translation group while a published entry shows it, exactly as `resolveFields` would show it. References held by drafts, by entries in review, or by archived entries never lock anything. -Resolution is breadth-first, at most two levels deep, under a budget per page. Referrer pages are purged whenever a target group changes, and a target that becomes visible counts as a change. Studio gets an entry picker and nested repeater forms from new `@cloudhal/ui` blocks. The four field operations learn the new types. +A reverse index, `entry_refs`, holds one row per stored value, whatever the target's state. It is written in each entry's own batch, anchored on the entry's new content hash and on its collection's definitions *as planned*. Every guard filters it by site. -Nothing goes live until the index, the guards, the renderer, the operations and Studio have all shipped: the two types stay refused until a final enabling step. +Archiving or unpublishing a target is always allowed. The renderer resolves only what the published snapshot shows *at that instant*, gates included, so an unresolved reference renders as nothing, and publish warns but never fails. Assets used inside repeaters count in `asset_refs`, like any field's assets. + +Resolution is breadth-first, at most two levels deep, under a budget per page. A publish purges referrers' own pages, found through `entry_refs`, whenever a target group changes, becomes visible, or changes its cover. This needs no new cache tags and no theme code. + +Studio gets an entry picker and nested repeater forms from new `@cloudhal/ui` blocks. The four field operations learn the new types. + +Nothing goes live until the index, the guards, the renderer, the operations and Studio have all shipped. The two types stay refused on every write path (interactive writes, import, sync, push, pull and restore) until a final enabling step. ## Motivation @@ -47,6 +53,8 @@ Nothing goes live until the index, the guards, the renderer, the operations and All additions are optional variants and `$defs` in `site.schema.json`. `content-entry.schema.json` changes only its `fields` description. There is no `schemaVersion` bump and no new root type. +Phase 8 adds one more variant, `ReferenceValueSource`, and the named enum `ReferenceValueProperty` to `StructuredDataSource`. That is a contract change of its own, listed with its diff in §4. + ```jsonc // site.schema.json $defs.FieldDefinition.oneOf (line 274): two variants appended after SelectField (line 542) { @@ -158,12 +166,13 @@ All additions are optional variants and `$defs` in `site.schema.json`. `content- | Type | Value | Checked on write | |---|---|---| | `reference` | a group id `"entry_…"`, or with `multiple` a list of them, no repeats, at most `maxItems` | see §2.1: the group has a member in this site in one of `collections`; it is not the entry's own group; it meets `where`. Checked for changed keys, and for every own value when the entry becomes published (§2.2). A member's id is canonicalized to its group id (`translation.group ?? id`), as `canonicalImageRefs` canonicalizes asset references. | - | `repeater` | a list of at most `maxItems` objects. Each holds only sub-field keys, with RFC-0041 §2's values for its type. No empty item. | each item against the sub-fields. `requiredToPublish` sub-fields need a value in every item of an entry in review or published. Image and rich-text sub-values are checked and recorded as assets (§2.4). | + | `repeater` | a list of at most `maxItems` objects. Each holds only sub-field keys, with RFC-0041 §2's values for its type. No empty item. | each item against the sub-fields. `requiredToPublish` sub-fields need a value in every item of an entry in review or published. Image and rich-text sub-values are checked and recorded as assets (§2.5). | - **Why a group id.** `Translation.group` is "the original node's id … May cross sites in one project" (`common.schema.json:121-124`). - A language edition's entries carry their source site's group id (`editions.ts:121-125`; `checkTranslationLinks` requires a member's group to be its source's, `store/translations.ts:131`). - The group id is therefore the one identity that is the same for every language version in every site. "The original in this site" is undefined in an edition site, where no member is the group's node. - The group id is used as a *key*: it is never read as a node of another site. A reference resolves, and is guarded, only through members whose `site_id` is this site (§2, §3). + - **A node's group can change.** `checkTranslationLinks` (`store/translations.ts:107-145`) accepts a new `translation.source`, and with it a new group, and `PUT …/entries/:id` (`api/routes/entries.ts:138-175`) passes the whole entry. A stored id therefore stays valid only because §2.2 guards group changes exactly like deletes: leaving a referenced group is leaving it. - It is a bare id, not `{collection, id}` as in Payload's polymorphic values: it keeps a reference what an image field already is, and the store checks the collection on every write. - **Why flat items.** Repeater items are flat objects of scalars, which is exactly ADR-0030's `ArrayPropSchema` with `FlatObjectPropSchema` items (`blocks.ts:55-68`). - `fieldPropSchema` compiles a repeater into the subset, and `validateProps` checks it unchanged, eval-free (rule 6). @@ -179,7 +188,15 @@ All additions are optional variants and `$defs` in `site.schema.json`. `content- - sub-fields count toward the per-site cap of 200 (`FIELD_LIMITS`, `types.ts:32`); - at most 100 reference values per entry, across fields and items (`too_many_references`). - **Until the enabling step (§9 step 7), `checkFieldDefinitions` refuses both types** (`fields_unsupported`). The contract can thus ship first without anyone being able to create a value that has no index rows. +- **The gate.** Until the enabling step (§9 step 7), both types are refused (`fields_unsupported`) on every path that writes a site node: + - **Interactive writes.** `checkFieldDefinitions`, called from `store/site-fields.ts:87` and `fields-operations/state.ts:94`. + - **Rows.** `checkBundle` (`sync-checks.ts:254-271`) gains `assertFieldTypes(siteId, bundle, support)` beside `assertGates`. That one function covers `importSite`, `pull`, `push`, `cloudhal import --replace` and `--revert`, and snapshot restores, all of which go through `importSite` (`sync-restore.ts:4`). Today `checkBundle` validates the schema only (`assertValidRows`, `:130-158`), which accepts the new variants as soon as phase 1 runs `pnpm gen`. + + `support` is the same capability object for both: + - the shared suites' store option (`SiteGraphStoreOptions.fieldTypes`); + - after step 7, the site Worker's `fields: 2`, through the `GateSupport` pattern (`availability.ts:235-237`). + + A server at phases 1 to 6 therefore refuses the types from Studio, MCP, the API and every bundle alike, so no value can exist without index rows. - **Adding a reference or repeater field is a free edit**, except over a key some entry already holds (`field_key_in_use`, one `json_type` count). The count races with entry writes in that collection; the definitions anchor of §2.1 closes the race. - **Rule 2 holds:** - `oneOf` with inline variants and the one-value `type` discriminator (ADR-0043); @@ -212,118 +229,171 @@ All additions are optional variants and `$defs` in `site.schema.json`. `content- ### 2. Cross-node integrity ```sql --- packages/storage/migrations/0033_entry_refs.ts (new; DDL only: no value can exist before §9 step 7) +-- packages/storage/migrations/0033_entry_refs.ts (new; DDL only: no value can exist before §9 step 7, on any path, §1) CREATE TABLE entry_refs ( site_id TEXT NOT NULL, - target_id TEXT NOT NULL, -- the referenced translation group id - source_id TEXT NOT NULL, -- the node holding the value (an original for localized: false) - source_group TEXT NOT NULL, -- that node's group id (translation.group ?? id), to reach members that inherit the value + target_id TEXT NOT NULL, -- the referenced translation group id, as stored + source_id TEXT NOT NULL, -- the node holding the value source_collection TEXT NOT NULL, field TEXT NOT NULL, -- "venue", or "speakers.person" inside a repeater + root_key TEXT NOT NULL, -- the top-level field key ("speakers") + shared INTEGER NOT NULL, -- who else shows the value (below): 0 nobody, 1 every inheriting member, 2 inheriting members without their own value source_status TEXT NOT NULL, -- the holding node's own status PRIMARY KEY (source_id, field, target_id) ); CREATE INDEX entry_refs_target ON entry_refs (site_id, target_id, source_status); ``` -**Invariant (the index):** `entry_refs` holds exactly one row per `(node, field path, target group)` whose value is stored on that node and whose target group has a member in the node's site. -- A target named in two repeater items (a host who also gives a talk) is one row: the writer deduplicates the pairs, and the `INSERT` is `INSERT OR IGNORE`. -- Every statement below filters on `site_id`, so it uses `entry_refs_target` and stays inside its tenant. +**Invariant (the index).** `entry_refs` holds exactly one row per `(node, field path, stored group id)`, whatever the target's state: present, missing, archived or foreign. Rows depend only on the holder's own body, so nothing re-creates or deletes them except a write of the holder, its deletion, or a rebuild. +- **No cross-tenant pinning.** Every guard and lookup filters `r.site_id` (the holder's site) and `n.site_id` (the member's site). A row naming another tenant's id pins nothing there and never resolves. +- **Repeats.** A target named in two repeater items (a host who also gives a talk) is one row: the writer deduplicates `(field, target)` pairs, and the `INSERT` is `INSERT OR IGNORE`. +- **`shared`** mirrors `resolveFields` (`fields/resolve.ts:37-65`). Only the group's same-site original, the node whose `id` is the group id, passes values on, and only to members of that group in the same site whose own `translation.source` is not in another site (`readsOriginal`). Hence: + - `shared = 1`: the holder is that original and the field is `localized: false`; + - `shared = 2`: the holder is that original and the field is `localized: true` (members without their own value fall back to it, `fromOriginal`); + - `shared = 0`: everything else. That covers a translation's own value, and any node of an edition site whose group lives in another site: `originalOf` returns nothing there (`store/entry-fields.ts:54-62`), so no edition member inherits (open question 12). + +**"Shown by a published entry"** is one SQL fragment. The guards, the live `referencedBy` and the purge planner all use it, so they mean the same as the snapshot's resolved values: + +```sql +-- rows of site S naming group G that a published entry shows +SELECT r.* FROM entry_refs r WHERE r.site_id = ? AND r.target_id = ? + AND (r.source_status = 'published' OR (r.shared > 0 AND EXISTS ( + SELECT 1 FROM nodes m WHERE m.site_id = r.site_id AND m.kind = 'contentEntry' + AND json_extract(m.body, '$.translation.group') = r.source_id AND m.id <> r.source_id + AND json_extract(m.body, '$.translation.source.siteId') IS NULL + AND json_extract(m.body, '$.status') = 'published' + AND (r.shared = 1 OR json_type(m.body, '$.fields.' || r.root_key) IS NULL)))) +``` + +The member probe uses `nodes_translation_locale` (migration 0025: `site_id, kind, json_extract(body, '$.translation.group')`; its partial `WHERE … IS NOT NULL` is implied by the equality). **1. Writing a referrer** (`EntryEngine.write`, `store/entries.ts:30`). `planEntryFields` returns three more parts: -- **The target guard**, ANDed into the entry's CAS (`entries.ts:104`). Every target group has a member in this site in one of the field's allowed collections. It is one count over a JSON list of `{t: group, c: [collections]}` pairs, like the asset guard (`entry-fields.ts:163`). The members are found through `nodes_translation_locale` (migration 0025: `site_id, kind, json_extract(body, '$.translation.group')`), or by id for an untranslated original: +- **The target guard**, ANDed into the entry's CAS (`entries.ts:104`), for changed reference keys, and for every own reference value when the entry becomes published (point 2). Every named group must have a member in this site, in one of the field's allowed collections, that is not archived (for a newly chosen group). It is one count over a JSON list of `{t: group, c: [collections]}` pairs, like the asset guard (`entry-fields.ts:163`): ```sql (SELECT count(*) FROM json_each(?) p WHERE EXISTS ( SELECT 1 FROM nodes n WHERE n.site_id = ? AND n.kind = 'contentEntry' AND (n.id = json_extract(p.value, '$.t') OR json_extract(n.body, '$.translation.group') = json_extract(p.value, '$.t')) AND json_extract(n.body, '$.collection') IN (SELECT value FROM json_each(json_extract(p.value, '$.c'))))) = ? ``` -- **The definitions anchor.** The allowed collections and the set of reference keys come from the site row the writer read, and a site-side refusal (`field_key_in_use`, removing an allowed collection) only counts entries already committed. Every entry write that changes `fields` or `collection` therefore also guards on its collection's definitions as read: - ```sql - (SELECT json_extract(body, ?) FROM nodes WHERE id = ? AND kind = 'site') IS ? - ``` - Here `?` is `$.collections."".fields`, and the text is read with the same expression while planning: one primary-key read, `NULL` for a collection without definitions. It conflicts only when that collection's definitions change, never on other site edits, unlike the site hash. A writer that loses retries under the new definitions, and `field_key_in_use` then sees its value. This is the same idea as RFC-0042's list-value rows, which are valid only "under the fingerprint they were built with" (`list-values/fingerprint.ts`). -- **The rows**, anchored on the entry's new hash: `DELETE FROM entry_refs WHERE source_id = ? AND `, then one `INSERT OR IGNORE … SELECT … FROM json_each(?) WHERE `, appended to `own` (`entries.ts:110`). - -A target deleted or moved between check and batch fails the guard, and `diagnose()` names it (`unknown_entry`, `reference_target_moved`). One bound parameter carries the whole list, so D1's 100-parameter limit does not apply, and 100 group ids stay far below its 100 KB per statement. - -**2. Deleting and moving a target: who can lock what.** -- A reference **blocks** only while a *published* entry shows it: the holding node is published, or a published member of its group inherits the value (`localized: false`). -- Drafts, entries in review and archived entries never lock anything. A contributor (`content:draft:own`, `roles.ts`) can therefore never pin someone else's entry. An author pins only through their own published entries, which the live site really depends on, and an editor can unpublish them. -- The holding node's status is in the row. A member that inherits the value is reached through `source_group`, using the same translation index: +- **The definitions anchor**, on every entry write that changes `fields` or `collection`. The writer binds its collection's definitions *as planned*: the canonical JSON of `site.collections[c].fields` from the very `Site` object it planned with (`entries.ts:39-41`), so there is no second read and no gap. ```sql - -- "blocking rows" for group G in site S - SELECT 1 FROM entry_refs r WHERE r.site_id = ? AND r.target_id = ? - AND (r.source_status = 'published' OR EXISTS ( - SELECT 1 FROM nodes m WHERE m.site_id = r.site_id AND m.kind = 'contentEntry' - AND json_extract(m.body, '$.translation.group') = r.source_group - AND json_extract(m.body, '$.status') = 'published')) + json((SELECT json_extract(body, ?) FROM nodes WHERE id = ? AND kind = 'site')) IS json(?) ``` -- **Delete** (`EntryEngine.delete`, `entries.ts:162`). The node's `DELETE`, already scoped by `site_id` (`entries.ts:168`), gains `AND (NOT EXISTS OR EXISTS ?>)`. Only removing a group's *last* member in this site can strand a reference. Deleting a translation, or an original that still has translations, never blocks, and references keep resolving to the remaining members. - - A refusal answers `409 entry_referenced`, with up to 20 published referrers (id, title, collection, field) and the total. - - When the last member does go, its remaining non-blocking rows are deleted in the same batch, guarded on the node being gone, as `dropRefsStatement` (`assets.ts:1015`). The draft referrers keep a dangling id, which their editor shows as "Not found". Their own rows go with their delete. -- **Publishing a referrer.** A write that makes an entry published re-checks *every* own reference value, not just changed keys. A dangling id is refused (`unknown_entry`: remove or replace it), so a draft that lost its target cannot go live pointing at nothing. - - Inherited values are not the translation's to fix. A translation that goes live while its original holds a dangling id gets a warning, not a refusal, and the reference renders as nothing. -- **Move** (`entersCollection`, `entries.ts:101`). Moving *any* member of group G from C to D is refused while a blocking row comes from a field that allows C but not D: `NOT EXISTS ( AND source_collection || '.' || field NOT IN (SELECT value FROM json_each(?)))`. The JSON list holds the `"."` pairs that allow D, computed from the site as read, and the write carries the definitions anchor of its own collection. - - Refused: `409 reference_target_moved`, naming the fields. - - During a field operation, the pairs include the marker's `fromKey`. - - Members of a group may still sit in different collections, as today (`checkTranslationLinks` has no collection rule). Resolution and `referenceIssues` filter members by the field's `collections` (§3), so a stray member never renders under a field that does not allow it. -- **Archiving and unpublishing** a target are allowed and write nothing to `entry_refs`. The picker hides archived entries, and a *newly* chosen group needs a member that is not archived (the `retired` rule of `planRefs`). - -**3. Rebuilds** run in the same batch as the node set: `importSite`/`pull`/`push` (`sync-plan.ts:308`), `cloudhal import --replace` and `--revert`, and snapshot restores. `entryRefsRebuild(siteId, rows)` builds the rows from the bodies, then `INSERT`s only rows whose target group has a member *in the same site*: it joins `nodes` with `site_id = ?`. -- An archive or push naming another tenant's entry id therefore pins nothing anywhere. The id stays in the value, never resolves, and is reported (`referenceIssues`). + The path is `$.collections."".fields`, and both sides are normalized by `json()`. `NULL IS NULL` holds for a collection without definitions. A unit test round-trips every fixture definition through the stored form (fractional bounds included), so text equality is exact. + + The anchor conflicts only when that collection's definitions change, never on other site edits. The loser retries under the new definitions, where `field_key_in_use` then sees its value. This is RFC-0042's rule that list-value rows are valid only "under the fingerprint they were built with" (`list-values/fingerprint.ts`), applied to writes. +- **The rows**, anchored on the entry's new hash: `DELETE FROM entry_refs WHERE source_id = ? AND `, then one `INSERT OR IGNORE … SELECT … FROM json_each(?) WHERE `, appended to `own` (`entries.ts:110`). Every write of the holder rewrites all its rows, so `source_status` and `shared` stay current. + +A target deleted or moved between check and batch fails the guard, and `diagnose()` names it (`unknown_entry`, `reference_target_moved`). One bound parameter carries the whole list (D1 allows 100 parameters), and 100 group ids stay far below 100 KB per statement. + +**2. Who can lock what: delete, move, leaving a group, publishing.** +- **The policy.** A reference blocks only while a *published* entry shows it, as `resolveFields` would show it: its own value, or one inherited from its same-site original (the fragment above). + - Drafts, entries in review and archived entries never lock anything. + - A contributor (`content:draft:own`, `roles.ts`) cannot publish, so nothing a contributor writes locks on its own. A value it holds locks only once someone with publish rights publishes an entry that shows it, such as a translation inheriting it. That person's publish is the decision. + - An author pins only through what published entries show, which the live site really depends on. + - An editor can always unpublish to release a lock. +- **Writes that make a group lose a member in this site.** That is a delete, a move to a collection no blocking field allows, or a change of `translation.group` (including dropping `translation`, `store/translations.ts:107-145`). Each such write of member X of group G carries: + - `NOT EXISTS () OR EXISTS ( X, not archived, in a collection every blocking row's field allows>)`. The allowed set is the intersection of those fields' `collections`, computed from the site as planned. + - **The site anchor**, `EXISTS (SELECT 1 FROM nodes WHERE id = ? AND kind = 'site' AND hash = ?)`, with the hash of the `Site` object the write planned with. The allowed collections come from *referrer* collections' definitions, so anchoring only the target's own collection (the first draft) left a race (R4). + + These writes are rare, so whole-site anchoring costs nothing in practice. It is added only when the plan saw blocking rows. Without them, the write carries only `NOT EXISTS ()`, which is safe on its own: at worst it over-refuses. + - A refusal answers `409 entry_referenced`, with up to 20 published referrers (id, title, collection, field) and the total. For a group change, it also names the group. + - Deleting a translation, or an original that still has another usable member, never blocks. References keep resolving to the remaining members (§3). +- **Moving a member** from C to D additionally refuses while a blocking row's field allows C but not D, and no other usable member remains (`409 reference_target_moved`). During a field operation, the pairs include the marker's `fromKey`. +- **Publishing a referrer.** A write that makes an entry published runs the target guard over *every* own reference value. A dangling id is refused (`unknown_entry`: remove or replace it), so a draft that lost its target cannot go live pointing at nothing. + - Inherited values are not the translation's to fix. A translation that goes live while its original holds a dangling id gets a warning, and the reference renders as nothing (R8). +- **Archiving and unpublishing** a target are allowed, and write nothing to `entry_refs`. The picker hides archived entries. +- Members of a group may sit in different collections, as today (`checkTranslationLinks` has no collection rule). Resolution, the guards and `referenceIssues` count only members in the field's `collections`. + +**3. Races.** Every interleaving that could leave a published reference stranded, and the guard that catches it. "Second" is whichever batch commits later; it fails and retries or refuses. + +| # | Interleaving | Caught by | +|---|---|---| +| R1 | referrer write ↔ deleting the target's last usable member | the referrer's target guard (members at batch time), or the delete's `NOT EXISTS ` (rows at batch time) | +| R2 | referrer write ↔ moving the target to a disallowed collection | the target guard (collection at batch time), or the move's blocking-row check | +| R3 | referrer write ↔ adding a reference field over its key, or removing an allowed collection | the definitions anchor, bound as planned (point 1). `field_key_in_use` counts committed values. The collection refusal counts values whose group has a member in the removed collection. | +| R4 | moving a member ↔ narrowing a referrer field's `collections` | the move's site anchor. The narrowing's count sees the member once it has moved. | +| R5 | delete ↔ any change to the definitions a blocking row's field depends on | the delete's site anchor | +| R6 | changing a member's group ↔ referrer write | the group change carries the delete guard and site anchor for its old group; the referrer's target guard checks membership at batch time | +| R7 | a referrer becoming published ↔ deleting its target | the publish re-checks all own values with the target guard. The delete sees the referrer's new `source_status`, because its rows are rewritten in the same batch. | +| R8 | publishing a translation that *inherits* the value ↔ deleting the target | when the publish is first, the delete's fragment reads member status at batch time. When the delete is first, nothing published showed the value at that moment. **Accepted:** the translation goes live with a warning and renders nothing; no guard can refuse a write the translator cannot fix. | +| R9 | field-operation phase B ↔ deleting a target | the target guard ANDed into each rewritten entry's CAS (point 6) | +| R10 | import, restore or sync ↔ anything | one atomic batch with the index rebuilt in it (point 4) | +| R11 | a member reappearing (a new translation or edition member) | rows are kept whatever the membership (the invariant), so nothing needs re-creating | + +**4. Rebuilds** run in the same batch as the node set: `importSite`/`pull`/`push` (`sync-plan.ts:308`), `cloudhal import --replace` and `--revert`, and snapshot restores. +- `entryRefsRebuild(siteId, rows)` builds the rows from the bodies, with the site's definitions from the site row, as `refsRebuild` does (`media/refs-rebuild.ts:12-37`), and `DELETE`s and `INSERT`s them under `site_id = ?`. +- There is no membership join: the invariant keeps rows for every stored value, and the site filters keep them harmless. - The `INSERT` is chunked so each statement stays below 90 KB, inside the one atomic batch. - -**4. Assets inside repeaters.** `fieldAssetRefs` (`media/field-refs.ts:17-36`) reads only top-level string values today (`if (typeof value !== "string") continue;`), so it would skip a repeater's items. -- It gains a walk over each item's `image` and `richText` sub-values, recorded under `fields..`. -- The same function feeds all of these, so each follows without further change: - - the write guard (`refsPlan`) and `fieldAssetIssues`: the asset exists in this site, is an image, and is not newly retired; - - field-operation rewrites (`fields-operations/rewrite.ts:150-158`); - - `refsRebuild` (`media/refs-rebuild.ts:22-24`); - - the purge planner's asset rule. -- An asset used only in a speaker's photo can therefore not be deleted or collected while used. Editions copy such assets already (`cross-site.ts` `assetIdsOf` scans the whole node). - -**5. Field operations** (`fields-operations/rewrite.ts:150`). From phase 2, `statementsOf` emits each rewritten entry's `entry_refs` rows next to its asset rows. It ANDs the target guard into the entry's CAS whenever the entry's reference values change, so a target deleted between plan and batch skips the entry for the next pass, like a concurrent edit. Operations that *name* a reference or repeater field are refused (`field_operation_unsupported`) until phase 4 implements them (§7). - -**6. Releases** (RFC-0008 §6). Today's dependency extractor treats every string matching `NODE_REF` at any depth as a dependency (`schedules/src/release.ts:268-289`), and core-ts passes it as `depsOf` (`core-ts/src/releases.ts:566`). Group ids in field values would therefore be pulled into a release's closure and locked (`assertUnlocked`). An edition's group id names a node of another site, so it would also fail as `dependency_missing` (`release.ts:416`). -- **Decision: references are not release dependencies.** Phase 2 replaces `depsOf` with a kind-aware extractor that drops reference values (top-level and inside items, by the site's definitions as pinned) before calling `referencedNodeIds`. -- A scheduled event whose venue is not live renders without it, exactly as after a publish, and the release's approval text carries `referenceIssues`. -- Rationale: closure dependencies are hard (they lock nodes and fail runs); references are soft at render. Holding every language version of a venue for an event's release would surprise its editors. - -**7. Definition changes** (`fieldDefinitionChanges`, `definitions.ts:135`) add these refusals: -- removing a collection from `collections` while a value points into it (an `entry_refs` join count); +- **Step 7** also runs a one-time, budgeted, resumable `entryRefsRebuild` per site as an admin job, the pattern of `lists.backfill`. It is idempotent, and is a safety net: the gate (§1) means it should find nothing. + +**5. Assets inside repeaters.** Each of these needs its own change, because none follows from the others: +- **`fieldAssetRefs`** (`media/field-refs.ts:17-36`) reads only top-level string values (`if (typeof value !== "string") continue;`). It gains a walk over items. It records `fields..` in `asset_refs.field` (no index: the row is per asset and field), and returns each use's item path (`fields///`) beside it. +- **`fieldAssetIssues`** (`field-refs.ts:87-91`) maps definitions by top-level key (`types = new Map(defs.map(d => [d.key, d.type]))`). It gains a lookup of `.` to the sub-field's type, so the image-kind check applies (a PDF in an image sub-field is refused). It reports issues at the item path, so Studio maps them to controls (§5), and its `skip` set uses item paths too. +- **`refsRebuild`** and field-operation rewrites (`fields-operations/rewrite.ts:150-158`) call `fieldAssetRefs`, so they follow the walk. +- **The purge planner's asset rule** (`cache-purge.ts:205-232`) uses `assetRefsOf` (`media/refs.ts:113`), which knows neither field images nor item images. **This is an existing gap for top-level image fields too.** It gains `fieldAssetRefs` with the target snapshot's site definitions. Entries it reaches also bring their referrers through the referrer rule (Purges, below), so a target's changed cover purges the cards that show it. +- Editions copy item assets already: `assetIdsOf` (`cross-site.ts:17`) scans the whole node. +- **A prerequisite, decided, not optional:** before phase 2 widens what feeds `asset_refs`, a small PR scopes it by site. + - `deleteAsset`'s guard (`assets.ts:867`) becomes `NOT EXISTS (SELECT 1 FROM asset_refs WHERE site_id = ? AND asset_id = ?)`. + - `refsRebuild` inserts only rows whose asset is a node of the same site, via a join. + - The rebuild's `INSERT` is chunked below 90 KB. + - Its tests: an archive in site X naming site Y's asset does not block Y's delete; a large rebuild stays under the statement limit. + +**6. Field operations** (`fields-operations/rewrite.ts:150`). From phase 2, `statementsOf` emits each rewritten entry's `entry_refs` rows next to its asset rows. It ANDs the target guard into the entry's CAS whenever the entry's reference values change, so a target deleted between plan and batch skips the entry for the next pass, like a concurrent edit (R9). Operations that *name* a reference or repeater field are refused (`field_operation_unsupported`) until phase 4 (§7). + +**7. Releases** (RFC-0008 §6). Today's dependency extractor treats every string matching `NODE_REF` at any depth as a dependency (`schedules/src/release.ts:268-289`), and core-ts passes it as `depsOf` (`core-ts/src/releases.ts:566`). +- Without a change, group ids in field values would be pulled into a release's closure and locked (`assertUnlocked`). An edition's foreign group id would also fail the run as `dependency_missing` (`release.ts:416`). +- **Decision: references are not release dependencies.** Phase 2 replaces `depsOf` with a kind-aware extractor that drops reference values (top-level and inside items, by the site's definitions as pinned) before `referencedNodeIds`. +- A scheduled event whose venue is not live renders without it, as after a publish, and the release's approval text carries `referenceIssues`. +- Rationale: closure dependencies are hard (they lock and fail), while references are soft at render. + +**8. Definition changes** (`fieldDefinitionChanges`, `definitions.ts:135`) add these refusals: +- removing a collection from `collections` while a value's group has a member in it (an `entry_refs` join on members); - turning `multiple` off; - removing or retyping a sub-field that holds values. Each refusal names the operation to use (§7). -**The invariant restated.** The store never lets a *published* entry show a reference whose target group has no member in its site, and never leaves the index out of step with stored values. +**9. Site deletion.** `entry_refs` joins `SITE_TABLES` (`site-delete.ts:16`) in phase 2. The existing test fails on a `site_id` table missing from that list. + +**The invariant restated.** +- No write lets a group lose its last usable member in a site while a published entry there shows a reference to it. +- No write stores a new own reference to a group without such a member. +- The index always matches the stored values. -Dangling ids can still exist, and every reader treats them as unresolved without throwing: -- in drafts whose target was deleted; +R8 is the one accepted exception, and it is warned. Dangling ids can still exist, and every reader treats them as unresolved without throwing: +- in drafts whose target went; - in hand-made or foreign archives; - in publish snapshots composed with holds, which leave out a held node that is not live (`store/snapshots.ts:29-37`); - after a release removes a target. **Snapshots, publish and rollback:** -- A snapshot is the whole working copy at one read (`snapshots.ts:23`). Rollback moves the pointer to an ancestor (`store.ts:1127`), which is just as consistent, and restores rebuild the index (point 3). -- **Visibility decides.** A snapshot holds drafts and archived entries. `Loaded.reachable` is the published entries *before gates* (`astro-renderer/src/source.ts:147-155`), and the gated set exists only per view instant (`#current()`, `source.ts:409-431`, `splitEntries` at `now`). The renderer therefore resolves through the view (§4). - - A target resolves only when its member is published and its gates are open at that instant. An `unlist`ed target (RFC-0042 §3) still resolves, because its page exists. +- A snapshot is the whole working copy at one read (`snapshots.ts:23`). Rollback moves the pointer to an ancestor (`store.ts:1127`), which is just as consistent, and restores rebuild the index (point 4). +- **Visibility decides.** `Loaded.reachable` is the published entries *before gates* (`astro-renderer/src/source.ts:147-155`); the gated set exists only per view instant (`#current()`, `source.ts:409-431`). The renderer resolves through the view (§4). + - A target resolves only when its member is published and its gates are open at that instant. An `unlist`ed target (RFC-0042 §3) still resolves. - A draft, archived, embargoed, `hide`-expired or missing target resolves to nothing. -- **Publish warns, never fails.** `referenceIssues(snapshot, now)`, pure and in `@cloudhal/core-ts/fields`, lists published entries whose references will not resolve: target missing, draft, archived, gated, in a disallowed collection, or with no version in a needed locale. It appears in Studio's publish dialog, in the MCP `publish` tool's approval text, and in `cloudhal snapshot --publish` output. - - A publish never fails on it: "a published snapshot must keep rendering" (RFC-0042 §4). - - Sanity's strong references block publishing per document, but Cloudhal publishes a whole site snapshot, and one draft venue must not hold back an unrelated fix. -- **Gates** of targets are in the snapshot's gate index already, since it holds every published entry's instants (`Loaded.gates`). A referrer's cached page therefore expires when its target's embargo opens or its `hide` expiry passes. +- **Publish warns, never fails.** `referenceIssues(snapshot, now)`, pure and in `@cloudhal/core-ts/fields`, lists published entries whose references will not resolve. It appears in Studio's publish dialog, the MCP `publish` tool's approval text and `cloudhal snapshot --publish` output. A publish never fails on it: "a published snapshot must keep rendering" (RFC-0042 §4). +- **Gates.** Every published entry's instants are in the snapshot's gate index already (`Loaded.gates`), so a referrer's cached page expires when its target's embargo opens or its `hide` expiry passes. -**Purges** (`cache-tags.ts`, `cache-purge.ts`): -- **Tags.** Every reference value a response *read* adds `cloudhal-ref-`, resolved or not. These tags come before optional tags, up to 4 KB (about 100 groups at 41 bytes each). Past that, the response carries one `cloudhal-refs-` tag instead. -- **Purging.** `planCachePurge` purges `cloudhal-ref-` for every entry it already purges (changed, added or removed), by its group (`translation.group ?? id`), plus `cloudhal-refs-` whenever any entry changed. - - So a retitled, unpublished or archived venue purges its referrers, in every language. - - So does a venue that *becomes* visible: published, a new Turkish member, an embargo that opens at the next publish. -- **Why this is safe.** The tags carry opaque group ids, never titles. Cache-Tag is an edge-only header that Cloudflare strips before responses reach visitors (Cloudflare purge-by-tag documentation; unverified on this branch). Node builds set no tags. -- **The §8 rule, revised:** no unresolved target may appear in HTML, JSON-LD, feeds, the sitemap, search or `llms.txt`. Cache tags are excluded from that rule. +**Purges** (`cache-purge.ts`). There are no new tags. Referrer pages already carry their own node tag (`cloudhal-node-`), set by the theme for the entry a response renders and for the entries an archive lists (`cache-tags.ts:6-8`). The planner reaches them through the index. +- **The referrer rule.** Groups touched by the diff are collected from the before and after bodies (`translation.group ?? id`): + - every changed, added or removed entry; + - the entries the asset rule reaches; + - a changed group, which counts both its old and its new group. -**Archives (RFC-0039).** An archive holds every node of the site, so references stay complete, and import rebuilds the index in its node batch (point 3). Dangling or foreign ids are reported, not refused, as RFC-0041 open question 8 decided for invalid values: `cloudhal import --dry-run` adds `referenceIssues`. + For those groups, the planner adds the node tags of: + - their referrers in the index that the fragment above shows as published, plus the inheriting members it names; + - the release's held referrers, read from the live snapshot, which the index may not match. + + Lookup is one `entry_refs` probe per group, run through a new `SnapshotReader.referrersOf(siteId, groups)`. The store implements it, and so does the release planner's virtual reader (`releases.ts:607`), on the same index. +- **What it covers.** Retitled, unpublished, archived and *newly visible* targets: a draft venue published, a new Turkish member, an embargo opening at a publish. Also changed covers, in every language and with any theme. +- **Costs and limits.** + - A group referenced by thousands of pages exceeds `maxTags` (300) and falls back to purging everything, as any large diff does today. + - `isCloudhalCacheTag` (`cache-tags.ts:48`) and the site Worker's purge endpoint (`cloudflare-entry.ts:312-325`) are unchanged, so an older site Worker accepts every purge. + - No ids of unresolved targets reach any header. +- The first draft's `cloudhal-ref-` tags are dropped. Existing themes build their own tag lists (`starters/blog/src/pages/[...path].astro:261-321`), so they would never have emitted them. The new prefixes would also have made older site Workers answer 400 and fall back to a full purge on every publish. And their privacy relied on unverified header stripping. + +**Archives (RFC-0039).** An archive holds every node of the site, so references stay complete, and import rebuilds the index in its node batch (point 4). Dangling or foreign ids are reported, not refused (RFC-0041 open question 8): `cloudhal import --dry-run` adds `referenceIssues`. Until step 7, an archive *with* reference or repeater definitions is refused (`fields_unsupported`, §1). ### 3. The locale rule @@ -335,10 +405,12 @@ A reference names a translation group (§1). In this site, the candidates are th The renderer, Studio's preview of a translation and `referenceIssues` ("Main hall has no Turkish version") share this one function, `pickMember`. -- **`localized: false` is the default** that Studio pre-sets for references. The original holds the ids, and every translation resolves them in its own language (RFC-0041 §5). A translator never re-picks targets, and a venue's Turkish version appears on the Turkish event without any write. +- **`localized: false` is the default** that Studio pre-sets for references. The group's same-site original holds the ids, and every same-site translation resolves them in its own language (RFC-0041 §5). A translator never re-picks targets, and a venue's Turkish version appears on the Turkish event without any write. - **`localized: true`** is for real differences between languages, such as a regional partner. Each version holds its own group ids, and an empty value falls back to the original's (`fromOriginal`). - **Language editions** (RFC-0034; an original in another site of the project). Non-localized values are copied when the edition is made (RFC-0041 §5, through `forEdition`, `cross-site.ts:115`). Group ids are shared across sites, so a copied id stays the same id. - `forEdition` keeps an id when the edition site has a member of that group in a collection the edition's own field allows. Otherwise the id is dropped and reported ("2 references not carried: Main hall has no edition in site B yet"). + - A repeater item left with no value once its references are dropped is dropped too, and reported, as `remove_field` with `sub` does (§7), so the edition write never fails on "no empty item". A `requiredToPublish` field emptied this way is fine: the copy is a draft. + - **Inside an edition site,** the edition node's own translations do not inherit its non-localized values. This is RFC-0041's existing rule: `originalOf` looks for the group's node in the same site (`store/entry-fields.ts:54-62`; search does the same, `search-rows.ts:299`), and the group's node is in another site. Each such member holds its own references, and their rows get `shared = 0` (§2). Changing that rule for every field type is open question 12. - A reference never resolves across sites: it reads only members with this site's `site_id`. - This lands in phase 2 with the guard, because without it making an edition would be refused (`unknown_entry`) as soon as references exist. - **Repeaters are localized as a whole.** @@ -388,14 +460,14 @@ interface GraphSource { - per entry: 100 reference values (code, §1); - per render: **200 resolved targets**, shared by every `entryFields` call of one request (`MediaContext.budget` already carries per-request budgets). - Over budget, the remaining targets resolve to nothing with one `console.warn`, never an error. Their `cloudhal-ref-` tags are still added within the tag budget (§2). A listing of 20 events at depth 1 costs 20. + Over budget, the remaining targets resolve to nothing with one `console.warn`, never an error. Purges do not depend on what rendered: they go through the index (§2, Purges). A listing of 20 events at depth 1 costs 20. - **Batching.** - The snapshot source does no store reads: every lookup is in memory, and bundles (RFC-0010) need nothing new. - The live source (`cloudhal dev`, previews) makes one store read per level: `getGroupMembers(siteId, ids)`, a `json_each` IN list over `nodes_translation_locale`, plus the original by id. -- **`referencedBy` covers translations.** A row sits on the node holding the value, which is the original for `localized: false`. - - The live source reads `entry_refs` by `(site_id, target_id)`, expands each `source_group` to its members in the requested locale, and filters them by the view. - - The snapshot's reverse map is built from resolved values, so it includes inheriting members by construction. - - The test plan compares the two sources. +- **`referencedBy` equals the resolved values.** A row sits on the node holding the value. + - The live source reads `entry_refs` by `(site_id, target_id)`. It keeps the holder when the holder is in the requested locale, and expands a row only through §2's `shared` rule: `shared = 1` adds every inheriting member, `shared = 2` adds those without their own value of `root_key`, and `shared = 0` adds nobody. It then filters by the view. + - The snapshot's reverse map is built from `resolveFields` output, which applies the same rule. + - The test plan compares the two on the same snapshot of the working copy, edition chains and localized values held by translations included. - **Listings** (RFC-0042 §4): - **Filter by a reference: yes.** A `reference` field may be a list field, with ops `eq`, `ne`, `in`, `set` and `unset` on group ids: "events at this venue", "talks by this speaker". Its `entry_list_values` rows hold the group id in `text` (≤ 48 characters, below the 100-unit limit). A multiple reference writes one row per id with `has = 2`, like a multiple select (migration 0029). - **Sort by a reference, or filter by a referenced entry's field** ("events whose venue is in Istanbul"): **no.** Either needs denormalized target values that every target edit would invalidate. Agents and themes do it in two steps: `places` by `city eq istanbul`, then `venue in [...]` (at most 20 values). @@ -410,7 +482,10 @@ interface GraphSource { ``` - Exactly one of `property` and `targetField` is given (checked in code). - `targetField` is checked against the target collection's definitions with the preset table's type rules. So `location.address.streetAddress` can come from the venue's `street` field, and the motivation's "address retyped on every event" is solved, not just `location.name`. - - The value is read from the depth-1 member's resolved fields. + - **The data path.** `structuredDataFor` stays pure (`structured-data/values.ts:26-40`). Its `StructuredDataContext` gains `references?: ReadonlyMap`, one entry per reference field a preset source names. + - `withStructuredData`, the async renderer export that already loads the original and assets (RFC-0042 §2), fills the map: for each such field, `pickMember` through the view (gated), then `resolveFields(site, member, memberOriginal)`, so a translation member's non-localized `street` comes from its original. + - This is one member per field, independent of the card depth: cards stay depth 1. + - A missing member or value emits nothing for the path, and a missing *required* value drops the preset document, as RFC-0042 §2 does for any required value. - `ReferenceValueProperty` is a named `$defs` enum, as RFC-0042 §1 requires for Rust. ### 5. Studio @@ -480,9 +555,20 @@ All new UI follows `kumo-design`, `studio-ui` and DESIGN.md "Fields", imports on - A `null` sub-value inside a repeater item is stripped before checking: that sub-field is unset in that item. - A given reference or repeater value replaces the whole value. - Removals go in a separate **`clearFields: FieldKey[]`**, which works with either input. + - **Edge cases**, decided: + - A key absent from `fieldsStrict` also means unchanged. The input accepts any subset of keys, which is what makes splitting a schema work. + - An item whose sub-values are all `null` becomes `{}` once stripped. It is dropped from the list, and the result reports it (`droppedItems`), so the "no empty item" rule never fires on strict output. + - A key that is non-null in `fieldsStrict` and also listed in `clearFields` is refused in precheck (`invalid`, "set or clear, not both"). + - An unknown key in `clearFields` is `unknown_field`. + - `clearFields` on create is `invalid`, since there is nothing to clear. - `fieldsJsonSchema(defs, locale, defaultLocale, { strict: true })` emits that meaning: every property required, `type: [T, "null"]`, and `enum` with `null` for selects. `list_collections` returns it as `fieldsSchemaStrict`, with `strictCost: { properties, depth, enumValues, unions }` counted by `strictSchemaCost`. - **Over a provider's limit** (Anthropic: 16 unions per request), a caller builds a smaller schema from a subset of `fieldsSchemaStrict.properties`. Each property's schema stands alone, with no `$ref`, and splitting is safe precisely because `null` leaves other keys alone. - - Phase 5 tests the round trip: strict output that changes only `venue` leaves `startsAt` and `speakers` untouched; a create with nulls stores no value; nulls inside items are stripped. + - Phase 5 tests the round trip: + - strict output that changes only `venue` leaves `startsAt` and `speakers` untouched; + - a create with nulls stores no value; + - nulls inside items are stripped, and all-null items are dropped and reported; + - a subset of keys leaves the others alone; + - set-and-clear of one key is refused. - **Example.** An `events` collection with `startsAt` (datetime), `venue` (a reference to `places`) and `speakers` (a repeater: `person` → `people`, `role` select, `talk` text): ```json @@ -531,7 +617,7 @@ All new UI follows `kumo-design`, `studio-ui` and DESIGN.md "Fields", imports on ### 7. Field operations (`fields-operations/`) -The four operations, their phases, budgets and revert stay as RFC-0041 §6 built them (`fields-operations/types.ts:22-50`). Phase 2 already makes every rewrite carry its `entry_refs` rows and the target guard (§2.5). Phase 4 adds the new types: +The four operations, their phases, budgets and revert stay as RFC-0041 §6 built them (`fields-operations/types.ts:22-50`). Phase 2 already makes every rewrite carry its `entry_refs` rows and the target guard (§2.6). Phase 4 adds the new types: - **`sub`.** A repeater's sub-field is addressed with a new optional `sub` parameter (`rename_field {from: "speakers", sub: "name", to: "fullName"}`), and the marker records it (`FieldMigration.sub`, §1). - **`change_field_type` gains the inputs the new definitions require** (`FieldOperation` in `types.ts:36-46`, the MCP tool and `cloudhal fields retype`): @@ -563,10 +649,13 @@ New rows of the closed table (`conversions.ts:38`): - **Where the graph reads happen.** The table is pure and synchronous (`conversions.ts:1-3`, `ConvertContext` at `:26-31`). The two text ↔ reference rows need entries, so `ConvertContext` gains an optional, prefetched `entries?: { byText: ReadonlyMap; titles: ReadonlyMap }` (case-folded slug or title → group ids; group id → default-locale title). Who builds it: - each operation pass (dry run, phase B pass, revert pass), from one read of the target collections, before the pass; - the store's `EntryEngine.write`, which is async, before it calls `normalizeEntryFields` (`entry-fields.ts:69-78`), only while a marker with such a conversion is active; - - the snapshot source, from its in-memory entries, lazily and only while the snapshot carries such a marker. + - the snapshot source, from its in-memory entries, lazily and only while the snapshot carries such a marker; + - the live source, which is async like the store: it builds the maps from the working copy, the same graph phase B reads, before resolving an entry whose collection carries such a marker. - A caller without it (the live source) resolves an unconverted value to `undefined` with an issue, which renders nothing and is never wrongly typed. Readers, writers and phase B therefore convert identically, and RFC-0041 §6's "a person's edit and the operation converge" holds. -- **Phase B writes references only under the target guard** (§2.5). A missed guard skips the entry for the next pass, so a conversion can never write a deleted target. + Every converter therefore has the maps. Writers, phase B and the live source read the working copy, so they convert identically, and RFC-0041 §6's "a person's edit and the operation converge" holds. + + The snapshot source converts against the snapshot's own entries. That is the graph it renders, and it may legitimately differ from the working copy, as every other value does. The test plan compares the two sources only on a snapshot of the current working copy, where both read the same entries. +- **Phase B writes references only under the target guard** (§2.6). A missed guard skips the entry for the next pass, so a conversion can never write a deleted target. - **Revert** (`fields-operations/revert.ts`) restores earlier values under the same guard. A restored reference whose group has no member left in the site is dropped and reported in a new `FieldRevertResult.dropped: { entryId, field, targetId }[]`. - **Free definition edits:** - add or reorder sub-fields; @@ -578,28 +667,35 @@ New rows of the closed table (`conversions.ts:38`): | Threat | Mitigation | Test | |---|---|---| -| **A draft, archived, embargoed or hidden target leaks through a published referrer** (the snapshot holds drafts, and `Loaded` holds published entries before gates) | Resolution filters through the view's gates at its instant (§4); only gate-independent card parts are cached. A card carries only the member's public title, summary, URL and cover image, the image through the view's asset gates. Unresolved values render nothing: no id, no `data-` attribute, no empty link. JSON-LD's `reference` source uses the same cards. | RFC-0008's route sweep (extended by RFC-0042) gains a published referrer pointing at a draft, an archived, an embargoed and a `hide`-expired target, before and after their instants. No id, slug, title or asset of a hidden target may appear in HTML, JSON-LD, feeds, the sitemap, search, `llms.txt` or status pages. Cache tags may carry group ids (§2). | +| **A draft, archived, embargoed or hidden target leaks through a published referrer** (the snapshot holds drafts, and `Loaded` holds published entries before gates) | Resolution filters through the view's gates at its instant (§4); only gate-independent card parts are cached. A card carries only the member's public title, summary, URL and cover image, the image through the view's asset gates. Unresolved values render nothing: no id, no `data-` attribute, no empty link. JSON-LD's `reference` source uses the same cards. | RFC-0008's route sweep (extended by RFC-0042) gains a published referrer pointing at a draft, an archived, an embargoed and a `hide`-expired target, before and after their instants. No id, slug, title or asset of a hidden target may appear in HTML, JSON-LD, feeds, the sitemap, search, `llms.txt`, status pages or response headers. Purges go through the index, so no header carries target ids (§2). | | **IDOR in the picker and on write** | Pick and references routes run under the `/sites/:siteId` membership check and query by `site_id`. The target guard requires a member with `n.site_id = ?` (§2.1). An id of another site gets the same `unknown_entry` as an id that does not exist. Editions copy only ids the edition site can resolve (§3). | A shared-suite case: another site's entry or group id in `create_entry`, `update_entry`, the API and `push` is refused identically; the pick route with another site's id answers 404. | -| **An imported or pushed archive pins another tenant's entries** | The rebuild inserts only rows whose target has a member in the same site, and every guard and referrer query filters `site_id` (§2.3). | An archive referencing site Y's entry, imported into site X, must not block Y's delete or move, and must not list X's titles to Y. | -| **Lock-out by low-privilege actors** (`content:draft:own`, `content:write:own`) | Only published referrers block delete and move (§2.2). Contributors cannot publish, and drafts, review and archived entries never lock. | Policy-suite cases: a contributor's draft and an author's review entry referencing another author's entry do not block its delete; an author's published entry does. | +| **An imported or pushed archive pins another tenant's entries or assets** | Rows are site-scoped, and every guard filters both the row's `site_id` and the member's (§2). `asset_refs` gets the same filter as a phase-2 prerequisite (§2.5). | An archive referencing site Y's entry or asset, imported into site X, must not block Y's delete or move, and must not list X's titles to Y. | +| **Lock-out by low-privilege actors** (`content:draft:own`, `content:write:own`) | Only what a published entry shows blocks, by the `shared` rule (§2): a translation's own value is never inherited. Contributors cannot publish; drafts, review and archived entries never lock. | Policy-suite cases that must *not* block: a contributor's draft; an author's review entry; a draft translation's `localized: true` value while its original is published; an edition member's value. Cases that must block: an author's published entry; a published translation inheriting its original's `localized: false` value. | | **Fan-out DoS on render** | Caps of 50 per field, 50 items and 10 sub-fields (contract); 100 reference values per entry (code); depth ≤ 2 with a visited set; 200 targets per page; ≤ 2 live reads per page, 0 for snapshots; `referencedBy` ≤ 100. | A property test: random graphs with cycles and 10× the caps resolve within budget, never throw, and cost at most `depth` live reads. | -| **Write amplification and slow guards** | Each guard is one indexed probe on `(site_id, target_id)` or one `json_each` count; members are found through `nodes_translation_locale`. Delete and move cost one probe each. Rebuilds are chunked inside one batch. | RFC-0042's benchmark (#105), extended to 10,000 events × 3 references on Miniflare D1, reports the write, delete and move guard costs. | +| **Write amplification and slow guards** | Each guard is one indexed probe on `(site_id, target_id)` or one `json_each` count; members are found through `nodes_translation_locale`. Delete, move and group change cost one probe each, plus the site anchor when rows exist. The purge planner adds one probe per touched group. Rebuilds are chunked inside one batch. | RFC-0042's benchmark (#105), extended to 10,000 events × 3 references on Miniflare D1, reports the write, delete and move guard costs. | | **Picker search abuse** | `q` ≤ 100 characters, a substring match with no regex (rule 13), ≤ 20 results, and a per-member rate window. | A 100k-character `q` is refused before it is read. | | **Prompt injection through target titles** | Titles are data, quoted and truncated in approval prompts (`quote()`, 60 characters); `list_references` marks them as content. | Approval-prompt tests gain reference cases. | -| **An older site Worker crashes on the new types** (`COMPILE[def.type]`, `props.ts:46`, has no entry) | `GET /__cloudhal/features` adds `{ "fields": 2 }`. Writers refuse the new types with `409 fields_unsupported` until the site Worker reports it, as `gates: 2` guards `expiry` (`availability.ts:243`, `cloudflare/src/features.ts`). `resolveFields` skips unknown types from phase 1. | The site Worker features test; a renderer test with an unknown field type. | -| **Assets in repeaters deleted while used** | `fieldAssetRefs` walks items (§2.4), so the guard, rebuild and purge cover them. | An asset used only in a repeater item cannot be deleted; the rebuild keeps its row; a retired or foreign asset id in an item is refused. | +| **An older site Worker crashes on the new types** (`COMPILE[def.type]`, `props.ts:46`, has no entry) | `GET /__cloudhal/features` adds `{ "fields": 2 }`. Every write path refuses the new types with `409 fields_unsupported` until it is reported: interactive writes and `checkBundle` for import, sync, push, pull and restore (§1), as `gates: 2` guards `expiry` (`availability.ts:243`, `cloudflare/src/features.ts`). `resolveFields` skips unknown types from phase 1. | The site Worker features test; an import, a push and a restore of a bundle with a reference definition are refused before step 7; a renderer test with an unknown field type. | +| **Assets in repeaters deleted while used** | `fieldAssetRefs` walks items, `fieldAssetIssues` resolves sub-field types, and the purge planner's asset rule reads field and item assets (§2.5). | An asset used only in a repeater item cannot be deleted; the rebuild keeps its row; a retired, foreign or non-image asset in an image sub-field is refused at its item path; a changed item image purges its page. | +| **A group change strands references** (`translation.group` is writable) | Leaving a group is guarded like a delete, with the site anchor (§2.2, R6). | Relinking a referenced untranslated original as a translation, and dropping `translation`, are both refused while a published entry shows the reference; a draft referrer does not block. | **Policy.** No new scopes. - Field definitions stay site settings: `site:settings`; for agents, `site:write` plus a person's approval, person-classed as today (`field-tools.ts:264-267`). - Values are content: `content:write`, with the `:own` limits. - Reading targets in the picker and `list_references` needs `content:read`, which every role already holds (`roles.ts` `READ`). -- Referencing an entry never writes it, and only a published referrer can stop its delete or move. +- Referencing an entry never writes it. Only what a published entry shows can stop its delete, its move or its leaving the group (§2.2). + +### 9. Rollout (one PR each; the types stay refused on every write path until step 7) -### 9. Rollout (one PR each; the types stay refused until step 7) +0. **Prerequisite: `asset_refs` by site** (§2.5). + - `deleteAsset`'s guard filters by site; + - `refsRebuild` joins same-site assets and chunks its `INSERT`; + - the purge planner's asset rule reads field assets. + It is independent of this RFC's types, and fixes three existing gaps. 1. **Contract and pure core:** - the §1 diff, fixtures and `pnpm gen` through `contract-change`; - - `checkFieldDefinitions` with the new rules, **refusing both types** (`fields_unsupported`); + - the gate: `checkFieldDefinitions` **and** `checkBundle`'s `assertFieldTypes` refuse both types (`fields_unsupported`), so a phase-1 server takes no such definition from Studio, MCP, the API, import, sync, push, pull or restore; - `fieldPropSchema`, value checks, `resolveFields` (skipping unknown types); - `fieldsJsonSchema` (open and strict) with `strictSchemaCost`, `referenceIssues`, `pickMember`; - the dispatch-table entries of §1; @@ -607,59 +703,85 @@ New rows of the closed table (`conversions.ts:38`): No behaviour change. 2. **Store and integrity:** - - migration 0033; - - the target guard, the definitions anchor and the rows; - - the delete, move and publish rules with the policy (§2.2); - - site-checked, chunked `entryRefsRebuild` in import, sync and restore, and `referenceIssues` in `import --dry-run`; - - `fieldAssetRefs` walking items; - - rows and the target guard in field-operation rewrites, with operations on the new types refused; + - migration 0033, with `entry_refs` in `SITE_TABLES`; + - the target guard, the definitions anchor bound as planned, and the rows with `shared`/`root_key`; + - the delete, move, group-change and publish rules, with the site anchor and the policy (§2.2), and the races R1 to R11 as cases; + - chunked `entryRefsRebuild` in import, sync and restore, and `referenceIssues` in `import --dry-run`; + - the item walk in `fieldAssetRefs`, and `fieldAssetIssues` with sub-field types and item paths; + - rows and the target guard in field-operation rewrites, with operations naming the new types refused; - the kind-aware release `depsOf`; - - edition id mapping in `forEdition`; - - the `fields: 2` writer guard; - - API: entry routes, `pick` and `references`. + - edition id mapping in `forEdition`, emptied items dropped; + - the `fields: 2` capability in the gate; + - the API: entry routes, `pick` and `references`. The shared suites enable the types with a store option (`SiteGraphStoreOptions.fieldTypes`); production keeps refusing them. -3. **Renderer:** +3. **Renderer and purges:** - cards, depth and budget, gated per view; - - `groupMembers`/`referencedBy` on both sources; + - `groupMembers` and `referencedBy` on both sources, with the `shared` rule; - `EntryFields.astro` (links for references, a nested `
` per item); - - `cloudhal-ref-` tags and the purge planner rules; + - the conversion maps in the live and snapshot sources; + - the purge planner's referrer rule and `SnapshotReader.referrersOf` (no new tags); - features `fields: 2`; - the route sweep extension; - a frozen theme copy for `old-theme-compat.test.ts`. 4. **Field operations:** `sub`, the new `change_field_type` inputs, the conversion rows with the prefetched context, revert `dropped`, and the CLI flags. 5. **MCP:** - `list_collections` (`fieldsSchema`, `fieldsSchemaStrict`, `strictCost`); - - `fieldsStrict` and `clearFields`; + - `fieldsStrict` and `clearFields` with the edge cases of §6; - prechecks, `delete_entry`'s refusal and `list_references`; - the translation tools' repeater paths; - the operation tools' new inputs; - regenerated tool docs (`node scripts/mcp-tools-doc.ts`). 6. **Studio:** the five blocks, `FieldReference`, the builder, "Used by", the delete dialog and the publish warnings; `design:shots`; the `fields` namespace and `pnpm i18n:import`. -7. **Enable** (small): `checkFieldDefinitions` accepts the two types where the writer guard confirms `fields: 2`; `update_collection`'s input lists them; `docs/guides/fields.md`. It needs steps 2 to 6 released. -8. **Listings and structured data:** reference list fields (rows, `checkFieldQuery`, the property suite) and `ReferenceValueSource` with `targetField`. +7. **Enable** (small): + - the gate accepts the two types where the site Worker reports `fields: 2`, on every path; + - `update_collection`'s input lists them; + - the one-time per-site `entryRefsRebuild` job (§2.4); + - `docs/guides/fields.md`. + + It needs steps 0 and 2 to 6 released. +8. **Listings and structured data:** reference list fields (rows, `checkFieldQuery`, the property suite), and `ReferenceValueSource` with `targetField` (its own `contract-change`, §4). **Test plan.** Store behaviour goes into the shared suites (`packages/core-ts/src/testing-*.ts`), on SQLite, Miniflare D1 and Durable Object SQLite. - `testing-references.ts` (new): - writes, and canonicalizing a member's id to its group; - refusing a missing, cross-site, wrong-collection, own-group or archived new target; - - the definitions-anchor races: a reference field added, or an allowed collection removed, between plan and batch; - - delete: the last member versus a remaining member, and published versus draft referrers; - - move, of an original and of a translation; + - **every race R1 to R11 of §2.3**, as two interleaved batches each: the definitions anchor bound as planned (with a definitions change between `getSite` and the batch), and the move and delete site anchors; + - delete: the last member versus a remaining member in an allowed or a disallowed collection, and a published versus a draft referrer; + - moves of an original and of a translation; + - group changes: relinking a referenced original, and dropping `translation`; - publish re-checks; + - the `shared` rule: + - a draft translation's `localized: true` value does not block; + - a published translation inheriting a `localized: false` value does; + - edition members never inherit; - the same target in two items; - an edition site referencing a cross-site group; + - rows kept for a missing target, and still blocking once a member reappears; - rows after every write, delete, import, restore and operation phase; - - a foreign-tenant archive. + - a foreign-tenant archive; + - site deletion removing the rows. +- `testing-gates.ts`, beside its bundle-gate cases: a bundle with reference or repeater definitions is refused by import, push, pull and restore before step 7, and accepted with the capability. - `testing-fields.ts`: repeater values, `requiredToPublish` per item, the leaf and reference caps, assets in items. -- `testing-assets.ts`: an asset used only in a repeater is not deletable, and its row survives a rebuild. +- `testing-assets.ts`: + - an asset used only in a repeater is not deletable, and its row survives a rebuild; + - a non-image in an image sub-field is refused at its item path; + - another site's archive does not block an asset's delete. - `testing-field-operations.ts`: every new conversion row, `sub`, a target deleted during phase B, resume, revert with a deleted target. -- `testing-releases.ts` (today on SQLite and D1): a scheduled referrer of a non-live target is not held and renders without it; an edition's group id never fails the run. -- `testing-translations.ts`: edition mapping, and drops reported. +- `testing-releases.ts` (today on SQLite and D1): a scheduled referrer of a non-live target is not held and renders without it, and an edition's group id never fails the run. +- `testing-translations.ts`: edition mapping, drops and emptied items reported. - `testing-list-values*.ts`: reference list fields, the seeded SQL-versus-evaluator comparison included. -- Pure tests: the emitter's keywords and depth, root `object`, no recursion, `strictCost`; the strict round trip (phase 5); `referenceIssues`; `pickMember`; budget and cycles, property-based. -- Renderer: identical cards on the snapshot and live sources, gate instants, `referencedBy` through translations, cache tags and purges, including a target that becomes visible. -- Studio: block tests (keyboard, focus after add and remove, live region), `design:shots` at both widths, `design:lint`. +- **Pure tests:** + - the emitter's keywords and depth, root `object`, no recursion, `strictCost`; + - the strict round trip and its edge cases (phase 5); + - `referenceIssues` and `pickMember`; + - the definitions anchor's text equality for every fixture definition; + - budget and cycles, property-based. +- **Renderer:** + - identical cards and `referencedBy` on the snapshot and live sources, for a snapshot of the working copy; + - gate instants; + - purge plans for a target retitled, unpublished, newly published, newly translated and with a changed cover, with a theme that emits no new tags. +- **Studio:** block tests (keyboard, focus after add and remove, live region), `design:shots` at both widths, `design:lint`. ## Alternatives considered @@ -667,12 +789,15 @@ New rows of the closed table (`conversions.ts:38`): |---|---|---|---| | Store a per-site anchor member's id instead of the group id | No foreign ids in values | Editions need an id rewrite per site; the anchor changes when the original is deleted; every lookup maps member → anchor | The group id is project-stable and already indexed (§1) | | Anchor every entry write on the site hash | Simple | Any site edit (theme, menus) conflicts every concurrent entry write | Anchor on the collection's definitions only (§2.1) | -| Every reference locks, drafts included (Sanity's strong default) | Strictest | Contributors pin others' entries; abandoned drafts pin forever | Only published referrers lock (§2.2) | +| Every reference locks, drafts included (Sanity's strong default) | Strictest | Contributors pin others' entries; abandoned drafts pin forever | Only what a published entry shows locks, by the `shared` rule (§2.2) | | Weak references (no locking at all) | No refusals | Live pages silently lose venues on delete; "used by" still needs the index | Lock where the live site depends | | Cascade or "unlink and delete" | One click | Rewrites other people's entries as a side effect | Refuse and list; open question 1 | | Block publish or fail the build on unresolved references | Strict | Publish is a site snapshot; one draft venue blocks every fix | Warn (§2) | -| References as release dependencies (today's generic extractor) | A scheduled event takes its venue live with it | Locks every member of the target until the run; edition group ids fail runs; hard, where render is soft | Kind-aware `depsOf` drops them (§2.6) | -| Tag only rendered targets; forbid group ids in tags | Fewer tags | Newly visible targets never purge their referrers; per-member purges miss group-keyed referrers | `cloudhal-ref-` for every value read; tags are edge-only | +| References as release dependencies (today's generic extractor) | A scheduled event takes its venue live with it | Locks every member of the target until the run; edition group ids fail runs; hard, where render is soft | Kind-aware `depsOf` drops them (§2.7) | +| `cloudhal-ref-` cache tags on referrer responses (first revision) | Purges by group | Themes build tags themselves, so existing themes would never emit them; older site Workers answer 400 to new prefixes and every publish purges everything; ids in headers rest on unverified stripping; header budget | The purge planner adds referrers' existing node tags through `entry_refs` (§2, Purges) | +| Store something immutable (a new `refId` per entry) instead of the group id | Survives group changes | A new identity on every entry, its own index, migrations and editions mapping; translations still need group resolution | Group ids, with group changes guarded like deletes (§2.2) | +| Rewrite referrers when a target changes group | Group changes always succeed | Rewrites other people's published entries as a side effect of a translation link | Refuse while shown by a published entry (§2.2) | +| Keep rows only while the target has a member (first revision) | Smaller table | Nothing re-creates rows when a member reappears; cleanup on last-member delete is fiddly | Rows for every stored value; the site filter is the isolation (§2) | | `null` in strict output means "remove", as in `fields` | One input | Following the schema deletes every untouched field | `fieldsStrict` with `null` = unchanged, plus `clearFields` | | `anyOf` in `update_collection`'s input for strict modes | Looks strict-friendly | Optional properties and bounds still fail strict modes | Inputs stay non-strict; `$defs` fixed instead | | Reverse lookups by scanning `json_extract` over `nodes` | No table | Every delete and move scans the site | `entry_refs`, the `asset_refs` pattern | @@ -685,10 +810,14 @@ New rows of the closed table (`conversions.ts:38`): ## Compatibility & migration - **Contract:** two new variants, a new `$defs` and one optional marker property. Existing sites, snapshots, bundles and archives stay valid, and there is no `schemaVersion` bump. - - An older admin Worker refuses the new definitions, so servers upgrade first. + - An admin Worker before phase 1 refuses the new definitions by schema. One at phases 1 to 6 refuses them by the gate, on every write path, bundles included (§1). Servers upgrade first. - An older site Worker is covered by `fields: 2`. -- **Data:** none to migrate. The types are refused until step 7, so migration 0033 is DDL only, and the definitions anchor keeps `field_key_in_use` honest. -- **Themes:** additive. `entry.fields` keeps meaning what is stored: group ids and item objects. Old theme copies render the new types as nothing. `EntryFields.astro` ships in the renderer package. + - The purge endpoint and `isCloudhalCacheTag` are unchanged, so mixed admin and site Worker versions purge normally (§2). +- **Data:** none to migrate. The types are refused on every write path until step 7, so migration 0033 is DDL only. Step 7's one-time per-site rebuild is the safety net, and the definitions anchor keeps `field_key_in_use` honest. +- **Themes:** additive. `entry.fields` keeps meaning what is stored: group ids and item objects. + - Themes from v0.5.0 on render fields through the renderer's `entryFields` and `EntryFields.astro`, so they draw references and repeaters after a renderer upgrade. + - Older copies show nothing for them. + - Either way, no theme code is needed for correct purges, because referrers are purged through their own node tags (§2). - **Releases:** the kind-aware `depsOf` changes nothing for existing content, because no field value can be a group id before step 7. - **Rollback:** - A snapshot restores the site node and entries, and the index is rebuilt in its batch. @@ -715,11 +844,13 @@ New rows of the closed table (`conversions.ts:38`): 8. **`fieldsSchemaStrict` beside `fieldsSchema`**, with `strictCost`, and no automatic splitting into groups. *Recommended.* 9. **Publish warnings as a gate:** never; warnings only. *Recommended.* A site setting could later make them a person-approval point for agents' publishes, like RFC-0042 §3.5's expiry rule. 10. **Review referrers:** they do not lock, like drafts. *Recommended*, so that a contributor's review request never pins an entry. The alternative, locking from review, protects content about to go live, but it reopens the contributor lock-out. -11. **`asset_refs` follow-ups found here:** `refsRebuild` puts a whole site in one `INSERT`, which can exceed D1's 100 KB statement on large imports, and `deleteAsset`'s guard (`assets.ts:867`) and the rebuild have no `site_id` filter, the same cross-tenant pin as §2.3. *Recommended:* one small PR before phase 2, outside this RFC. +11. ~~`asset_refs` follow-ups~~ **Decided** in the second revision: §9 step 0 is a prerequisite of phase 2 (§2.5). +12. **Edition chains and non-localized values:** inside an edition site, the edition node's translations do not inherit its non-localized values of any type, as RFC-0041 behaves today (`originalOf`). References follow that rule (`shared = 0`). *Recommended:* keep it in stage 4, and fix it for every field type in its own change, defining the in-site root as the member with no same-site source. +13. **R8** (a translation published just after its inherited target was deleted): accept it with a warning. *Recommended*, because refusing would block a translator over a value they cannot edit. The alternative is to refuse the publish until the original's value is fixed. ## Review log (2026-11-10, PR #193) -An independent multi-agent review upheld 28 findings and split two. Each one, with the decision taken: +An independent multi-agent review upheld 28 findings and split two. Each one, with the decision taken. Section numbers in this first table refer to commit fe8ac929; the second round renumbered §2 (§2.4 rebuilds, §2.5 assets, §2.6 field operations, §2.7 releases). | # | Finding | Decision | |---|---|---| @@ -744,3 +875,26 @@ An independent multi-agent review upheld 28 findings and split two. Each one, wi | U25 | The Event address cannot come from a Place | `ReferenceValueSource.targetField` (§4) | | U26 | `change_field_type` lacks inputs | `collections`, `maxItems`, `subFields`, `itemLabel`, and CLI flags (§7) | | S2 | The release closure treats ids as dependencies | Kind-aware `depsOf` drops references; a `testing-releases` case (§2.6) | + +### Second round (2026-11-10, verification of fe8ac929) + +Three verifiers found 34 earlier findings resolved. The unresolved ones and the new issues, with the decisions: + +| Finding | Decision | +|---|---| +| Group ids change (`translation.group` is writable), so stored ids escape the guards (blocking, two verifiers) | Leaving a group is guarded like a delete, with the site anchor, and refused while a published entry shows the reference (§1, §2.2, R6, tests). Rewriting referrers and a new immutable id are both rejected (Alternatives). | +| Import, push, pull and restore bypass the gate (blocking, two verifiers) | `checkBundle` gains `assertFieldTypes`, covering every row path through `importSite`, with one capability shared with interactive writes (§1). Step 7 runs a one-time per-site rebuild (§2.4, §9). | +| The blocking expansion treats every row as inherited (blocking, three verifiers) | Rows carry `shared` and `root_key`, mirroring `resolveFields`. One SQL fragment serves the guards, the live `referencedBy` and the purge planner, so live equals snapshot. The policy is stated explicitly (§2, §4, §8). | +| The definitions anchor is read separately from the planned `Site` (blocking) | The anchor binds the definitions of the very `Site` object the write planned with, compared with `json(...) IS json(?)` (§2.1). | +| The move guard anchors the wrong collection, and the delete exception ignores allowed collections | Moves, deletes and group changes anchor on the hash of the planned site when rows exist. The exception counts only usable members in collections every blocking field allows (§2.2, R4, R5). | +| No emission path for `cloudhal-ref-` tags; old Workers refuse new prefixes; budget; privacy (blocking) | Tags dropped. The purge planner adds referrers' own node tags through `entry_refs`, covering newly visible targets and changed covers. No new prefix, no header ids, no theme code (§2, Purges). | +| Rows depend on membership and are not re-created | Rows are kept for every stored value; the site filter is the isolation (§2 invariant, R11). | +| `entry_refs` is missing from `SITE_TABLES` | Added in phase 2 (§2.9). | +| `fieldAssetIssues` and the purge planner do not follow | Both are specified: sub-field types, item paths, and `fieldAssetRefs` in the asset rule, which also fixes top-level image fields (§2.5). | +| The `asset_refs` cross-tenant fix was only recommended | Decided as rollout step 0, a prerequisite of phase 2 (§2.5, §9, OQ 11). | +| Edition chains have no original; mapping can empty an item | Edition members do not inherit (RFC-0041's rule, `shared = 0`; OQ 12). Emptied items are dropped and reported (§3). | +| The conversion-context claims contradict each other | The live source prefetches from the working copy. The snapshot source converts against its own entries. The comparison test runs on a snapshot of the working copy (§7). | +| `fieldsStrict` edge cases | Absent keys mean unchanged; all-null items are dropped and reported; set-and-clear is refused; an unknown key to clear is refused; clearing on create is refused (§6). | +| The Event address data path was undefined, and the phase-8 contract was unlisted | `StructuredDataContext.references`, filled by `withStructuredData` through `pickMember` and `resolveFields`, independent of card depth. The phase-8 contract change is listed in §1 (§4). | +| Move and delete races | Listed as R1 to R11 with their guards; R8 accepted and warned (§2.3, OQ 13). | + From 21abf4e1c207ef40e6fc7ea3d558f9c15dfdcc52 Mon Sep 17 00:00:00 2001 From: Madraka <8898640+Madraka@users.noreply.github.com> Date: Tue, 10 Nov 2026 15:54:03 +0300 Subject: [PATCH 4/8] docs(rfc): RFC-0045 third revision after the final verification Folds in the four blocking clusters and the non-blocking items of the verification of 9e457d12, extending the review log: - one membership rule everywhere, COALESCE(translation.group, id) = G, in an index-friendly form; relinking a node with dependants is refused, and readsOriginal requires an original to own its group - shared is a pure function of localized; the fragment decides who inherits at batch time - delete, move and group-change guards bind the planned pairs, so an unseen field's row forces a retry (R13) - purge closure to the render depth, "used by" pages, index mode for publishes and snapshot mode for rollbacks, release runs (re-planned at run time) and restores, bounded by maxReads/maxTags - set_field_localized flips shared in phase A; conversions into references refuse target deletes until phase C (R12) - an absent capability refuses in the bundle gate (CopySiteOptions, FIELDS_SUPPORTED); importSite stores canonical bodies - races R12-R15, the size-ratchet modules, the test cases and the open questions are added Co-Authored-By: Claude Opus 5.5 Signed-off-by: Madraka <8898640+Madraka@users.noreply.github.com> --- ...ctions-stage-4-references-and-repeaters.md | 214 +++++++++++++----- 1 file changed, 156 insertions(+), 58 deletions(-) diff --git a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md index 2fc47379..1da8af3a 100644 --- a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md +++ b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md @@ -26,7 +26,9 @@ Two new field types join RFC-0041's nine: - A **`reference`** field points to entries of one to five declared collections of the same site, one target or up to fifty. It stores the target's **translation group id**, the same id in every site of a project. It always resolves to that group's member in this site and in the page's language. - A **`repeater`** field is an ordered list of up to fifty items, each of up to ten flat sub-fields. A repeater cannot contain a repeater, so every agent-facing schema has a fixed depth and no recursion. -References are strong where it matters. A target cannot be deleted, moved, or taken out of its translation group while a published entry shows it, exactly as `resolveFields` would show it. References held by drafts, by entries in review, or by archived entries never lock anything. +References are strong where it matters. A target *group* cannot lose its last usable member in a site, whether by a delete, a move or a relink into another group, while a published entry shows a reference to it exactly as `resolveFields` would show it. One language version's card can still go when that version is deleted, with a warning. + +References held by drafts, by entries in review, or by archived entries never lock anything. A node's group is one value everywhere: `COALESCE(translation.group, id)`. A reverse index, `entry_refs`, holds one row per stored value, whatever the target's state. It is written in each entry's own batch, anchored on the entry's new content hash and on its collection's definitions *as planned*. Every guard filters it by site. @@ -165,7 +167,7 @@ Phase 8 adds one more variant, `ReferenceValueSource`, and the named enum `Refer | Type | Value | Checked on write | |---|---|---| - | `reference` | a group id `"entry_…"`, or with `multiple` a list of them, no repeats, at most `maxItems` | see §2.1: the group has a member in this site in one of `collections`; it is not the entry's own group; it meets `where`. Checked for changed keys, and for every own value when the entry becomes published (§2.2). A member's id is canonicalized to its group id (`translation.group ?? id`), as `canonicalImageRefs` canonicalizes asset references. | + | `reference` | a group id `"entry_…"`, or with `multiple` a list of them, no repeats, at most `maxItems` | see §2.1: the group has a member in this site in one of `collections`; it is not the entry's own group; it meets `where`. Checked for changed keys, and for every own value when the entry becomes published (§2.2). A member's id is canonicalized to its group id, `COALESCE(translation.group, id)` (§2, "Membership"), as `canonicalImageRefs` canonicalizes asset references. | | `repeater` | a list of at most `maxItems` objects. Each holds only sub-field keys, with RFC-0041 §2's values for its type. No empty item. | each item against the sub-fields. `requiredToPublish` sub-fields need a value in every item of an entry in review or published. Image and rich-text sub-values are checked and recorded as assets (§2.5). | - **Why a group id.** `Translation.group` is "the original node's id … May cross sites in one project" (`common.schema.json:121-124`). @@ -173,6 +175,8 @@ Phase 8 adds one more variant, `ReferenceValueSource`, and the named enum `Refer - The group id is therefore the one identity that is the same for every language version in every site. "The original in this site" is undefined in an edition site, where no member is the group's node. - The group id is used as a *key*: it is never read as a node of another site. A reference resolves, and is guarded, only through members whose `site_id` is this site (§2, §3). - **A node's group can change.** `checkTranslationLinks` (`store/translations.ts:107-145`) accepts a new `translation.source`, and with it a new group, and `PUT …/entries/:id` (`api/routes/entries.ts:138-175`) passes the whole entry. A stored id therefore stays valid only because §2.2 guards group changes exactly like deletes: leaving a referenced group is leaving it. + - Membership is the node's *current* group, never its id alone, so a relinked node really leaves its old group. + - A node whose id other same-site nodes use as their `translation.group` cannot relink or drop `translation` (`translation_group_in_use`). Its translations must move first, so a group never loses its original while members still inherit from it. - It is a bare id, not `{collection, id}` as in Payload's polymorphic values: it keeps a reference what an image field already is, and the store checks the collection on every write. - **Why flat items.** Repeater items are flat objects of scalars, which is exactly ADR-0030's `ArrayPropSchema` with `FlatObjectPropSchema` items (`blocks.ts:55-68`). - `fieldPropSchema` compiles a repeater into the subset, and `validateProps` checks it unchanged, eval-free (rule 6). @@ -192,9 +196,12 @@ Phase 8 adds one more variant, `ReferenceValueSource`, and the named enum `Refer - **Interactive writes.** `checkFieldDefinitions`, called from `store/site-fields.ts:87` and `fields-operations/state.ts:94`. - **Rows.** `checkBundle` (`sync-checks.ts:254-271`) gains `assertFieldTypes(siteId, bundle, support)` beside `assertGates`. That one function covers `importSite`, `pull`, `push`, `cloudhal import --replace` and `--revert`, and snapshot restores, all of which go through `importSite` (`sync-restore.ts:4`). Today `checkBundle` validates the schema only (`assertValidRows`, `:130-158`), which accepts the new variants as soon as phase 1 runs `pnpm gen`. - `support` is the same capability object for both: - - the shared suites' store option (`SiteGraphStoreOptions.fieldTypes`); - - after step 7, the site Worker's `fields: 2`, through the `GateSupport` pattern (`availability.ts:235-237`). + `support` is the same capability object for both, and **when it is absent, the types are refused**. This is unlike `assertGates`, which skips its check without a `GateSupport` (`sync-checks.ts:56-58`). The CLI's pull and import pass none today. The capability comes from: + - the shared suites: `SiteGraphStoreOptions.fieldTypes` for the store, and a new `CopySiteOptions.fieldTypes` for `importSite`, which takes a `StoragePair` and never sees store options; + - after step 7, on Cloudflare: the site Worker's `fields: 2`, through the `GateSupport` pattern (`availability.ts:235-237`); + - after step 7, on Node (`cloudhal dev`, `studio`, self-host): the in-process constant `FIELDS_SUPPORTED`, the twin of `GATES_ENFORCED` (`availability.ts:240`), passed by those callers. + + Without one of these, a store refuses. An older CLI's local store at phases 1 to 6, which has no `entry_refs`, therefore refuses a pull or a hand-made archive with the types. A server at phases 1 to 6 therefore refuses the types from Studio, MCP, the API and every bundle alike, so no value can exist without index rows. - **Adding a reference or repeater field is a free edit**, except over a key some entry already holds (`field_key_in_use`, one `json_type` count). The count races with entry writes in that collection; the definitions anchor of §2.1 closes the race. @@ -237,27 +244,34 @@ CREATE TABLE entry_refs ( source_collection TEXT NOT NULL, field TEXT NOT NULL, -- "venue", or "speakers.person" inside a repeater root_key TEXT NOT NULL, -- the top-level field key ("speakers") - shared INTEGER NOT NULL, -- who else shows the value (below): 0 nobody, 1 every inheriting member, 2 inheriting members without their own value + shared INTEGER NOT NULL, -- from the definition: 1 localized: false (every inheriting member shows it), 2 localized: true (only those without their own value) source_status TEXT NOT NULL, -- the holding node's own status PRIMARY KEY (source_id, field, target_id) ); CREATE INDEX entry_refs_target ON entry_refs (site_id, target_id, source_status); ``` -**Invariant (the index).** `entry_refs` holds exactly one row per `(node, field path, stored group id)`, whatever the target's state: present, missing, archived or foreign. Rows depend only on the holder's own body, so nothing re-creates or deletes them except a write of the holder, its deletion, or a rebuild. +**Invariant (the index).** `entry_refs` holds exactly one row per `(node, field path, stored group id)`, whatever the target's state: present, missing, archived or foreign. Rows depend on the holder's body and on its collection's definitions (`localized` decides `shared`). Nothing changes them except a write of the holder, its deletion, a rebuild, or `set_field_localized`'s phase A (§7). - **No cross-tenant pinning.** Every guard and lookup filters `r.site_id` (the holder's site) and `n.site_id` (the member's site). A row naming another tenant's id pins nothing there and never resolves. - **Repeats.** A target named in two repeater items (a host who also gives a talk) is one row: the writer deduplicates `(field, target)` pairs, and the `INSERT` is `INSERT OR IGNORE`. -- **`shared`** mirrors `resolveFields` (`fields/resolve.ts:37-65`). Only the group's same-site original, the node whose `id` is the group id, passes values on, and only to members of that group in the same site whose own `translation.source` is not in another site (`readsOriginal`). Hence: - - `shared = 1`: the holder is that original and the field is `localized: false`; - - `shared = 2`: the holder is that original and the field is `localized: true` (members without their own value fall back to it, `fromOriginal`); - - `shared = 0`: everything else. That covers a translation's own value, and any node of an edition site whose group lives in another site: `originalOf` returns nothing there (`store/entry-fields.ts:54-62`), so no edition member inherits (open question 12). +- **`shared`** is a pure function of the definition: 1 for `localized: false`, 2 for `localized: true`. It no longer encodes the holder's role, which the first revision computed at write time and which went stale when a node changed group. + - *Who* inherits is decided at batch time by the fragment below, exactly as `readsOriginal` decides it (`fields/resolve.ts:37-41`): members `m` of this site whose `translation.group` is the holder's id, who are not the holder, and whose own `translation.source` is not in another site. + - A translation's own value, and any value held in an edition site (where no node has the group's id, `originalOf`, `store/entry-fields.ts:54-62`), therefore inherits to nobody, whatever its `shared` (open question 12). +- **Membership.** A node `n` is a member of group G exactly when `COALESCE(json_extract(n.body, '$.translation.group'), n.id) = G`. Every probe uses this definition: the target guard, the fragment, `pickMember`, `groupMembers`/`getGroupMembers`, `referenceIssues`, the guards of §2.2 and the purge rules. In SQL it is written so that the primary key and `nodes_translation_locale` still serve it: + ```sql + ((n.id = :G AND json_extract(n.body, '$.translation.group') IS NULL) + OR json_extract(n.body, '$.translation.group') = :G) + ``` + The first revision's `n.id = G OR translation.group = G` let a relinked node stay a member of its old group by its id, which defeated the group-change guard (R6). + - The relink refusal of §1 keeps the inheritance side exact: a node with same-site dependants always has its own id as its group. + - `readsOriginal` and `originalOf` additionally require the original's own group to be its id. Legacy data relinked before this RFC then inherits nothing rather than reading from a node of another group. This is an RFC-0041 behaviour change for that edge case only, and the fragment agrees with it. **"Shown by a published entry"** is one SQL fragment. The guards, the live `referencedBy` and the purge planner all use it, so they mean the same as the snapshot's resolved values: ```sql -- rows of site S naming group G that a published entry shows SELECT r.* FROM entry_refs r WHERE r.site_id = ? AND r.target_id = ? - AND (r.source_status = 'published' OR (r.shared > 0 AND EXISTS ( + AND (r.source_status = 'published' OR (EXISTS ( SELECT 1 FROM nodes m WHERE m.site_id = r.site_id AND m.kind = 'contentEntry' AND json_extract(m.body, '$.translation.group') = r.source_id AND m.id <> r.source_id AND json_extract(m.body, '$.translation.source.siteId') IS NULL @@ -265,21 +279,25 @@ SELECT r.* FROM entry_refs r WHERE r.site_id = ? AND r.target_id = ? AND (r.shared = 1 OR json_type(m.body, '$.fields.' || r.root_key) IS NULL)))) ``` -The member probe uses `nodes_translation_locale` (migration 0025: `site_id, kind, json_extract(body, '$.translation.group')`; its partial `WHERE … IS NOT NULL` is implied by the equality). +The member probe uses `nodes_translation_locale` (migration 0025: `site_id, kind, json_extract(body, '$.translation.group')`; its partial `WHERE … IS NOT NULL` is implied by the equality). For legacy relinked holders, where `readsOriginal` now inherits nothing, the fragment can only over-block, which is safe. During a `rename_field`, `resolveFields` also reads `migration.fromKey` while the fragment checks `root_key`, which again can only over-block. **1. Writing a referrer** (`EntryEngine.write`, `store/entries.ts:30`). `planEntryFields` returns three more parts: -- **The target guard**, ANDed into the entry's CAS (`entries.ts:104`), for changed reference keys, and for every own reference value when the entry becomes published (point 2). Every named group must have a member in this site, in one of the field's allowed collections, that is not archived (for a newly chosen group). It is one count over a JSON list of `{t: group, c: [collections]}` pairs, like the asset guard (`entry-fields.ts:163`): +- **The target guard**, ANDed into the entry's CAS (`entries.ts:104`), for changed reference keys, and for every own reference value when the entry becomes published (point 2). Every named group must have a member in this site, by the membership rule, in one of the field's allowed collections; a newly chosen group's member must also not be archived. It is one count over a JSON list of `{t: group, c: [collections], f: fresh}` pairs, like the asset guard (`entry-fields.ts:163`): ```sql (SELECT count(*) FROM json_each(?) p WHERE EXISTS ( SELECT 1 FROM nodes n WHERE n.site_id = ? AND n.kind = 'contentEntry' - AND (n.id = json_extract(p.value, '$.t') OR json_extract(n.body, '$.translation.group') = json_extract(p.value, '$.t')) - AND json_extract(n.body, '$.collection') IN (SELECT value FROM json_each(json_extract(p.value, '$.c'))))) = ? + AND ((n.id = json_extract(p.value, '$.t') AND json_extract(n.body, '$.translation.group') IS NULL) + OR json_extract(n.body, '$.translation.group') = json_extract(p.value, '$.t')) + AND json_extract(n.body, '$.collection') IN (SELECT value FROM json_each(json_extract(p.value, '$.c'))) + AND (json_extract(p.value, '$.f') = 0 OR json_extract(n.body, '$.status') <> 'archived'))) = ? ``` - **The definitions anchor**, on every entry write that changes `fields` or `collection`. The writer binds its collection's definitions *as planned*: the canonical JSON of `site.collections[c].fields` from the very `Site` object it planned with (`entries.ts:39-41`), so there is no second read and no gap. ```sql json((SELECT json_extract(body, ?) FROM nodes WHERE id = ? AND kind = 'site')) IS json(?) ``` - The path is `$.collections."".fields`, and both sides are normalized by `json()`. `NULL IS NULL` holds for a collection without definitions. A unit test round-trips every fixture definition through the stored form (fractional bounds included), so text equality is exact. + The path is `$.collections."".fields`, and both sides are normalized by `json()`. `NULL IS NULL` holds for a collection without definitions. A unit test round-trips every fixture definition through the stored form (fractional bounds included). + + Equality needs stored bodies in canonical form. Store writes are canonical (`canonicalJson`, `node-sql.ts:62/91`), but `importSite` inserts a bundle's body text verbatim (`sync-plan.ts:300-301`), and SQLite's `json()` keeps key order. So from phase 2, `importSite` stores `canonicalJson(JSON.parse(body))` for every row. The hash is computed over canonical JSON already, so it does not change. A hand-made archive with unsorted keys therefore cannot leave a collection whose writes always conflict. The anchor conflicts only when that collection's definitions change, never on other site edits. The loser retries under the new definitions, where `field_key_in_use` then sees its value. This is RFC-0042's rule that list-value rows are valid only "under the fingerprint they were built with" (`list-values/fingerprint.ts`), applied to writes. - **The rows**, anchored on the entry's new hash: `DELETE FROM entry_refs WHERE source_id = ? AND `, then one `INSERT OR IGNORE … SELECT … FROM json_each(?) WHERE `, appended to `own` (`entries.ts:110`). Every write of the holder rewrites all its rows, so `source_status` and `shared` stay current. @@ -293,13 +311,26 @@ A target deleted or moved between check and batch fails the guard, and `diagnose - An author pins only through what published entries show, which the live site really depends on. - An editor can always unpublish to release a lock. - **Writes that make a group lose a member in this site.** That is a delete, a move to a collection no blocking field allows, or a change of `translation.group` (including dropping `translation`, `store/translations.ts:107-145`). Each such write of member X of group G carries: - - `NOT EXISTS () OR EXISTS ( X, not archived, in a collection every blocking row's field allows>)`. The allowed set is the intersection of those fields' `collections`, computed from the site as planned. + - The batch-time condition. P is the JSON list of `"."` pairs of the blocking rows the plan saw, and A is the intersection of those fields' `collections`, from the site as planned: + ```sql + NOT EXISTS () + OR ( EXISTS ( X, not archived, collection IN A>) + AND NOT EXISTS ( + AND r.source_collection || '.' || r.field NOT IN (SELECT value FROM json_each(:P))) ) + ``` + The second `NOT EXISTS` makes a blocking row from a field the plan did not see, committed in between, fail the batch, so the write retries and re-plans with that field. The first revision's form let it through: `events.venue` allows `[places, halls]` and `talks.room` allows `[places]`; a talk published meanwhile with `room` = G is stranded once the last member in `places` goes. - **The site anchor**, `EXISTS (SELECT 1 FROM nodes WHERE id = ? AND kind = 'site' AND hash = ?)`, with the hash of the `Site` object the write planned with. The allowed collections come from *referrer* collections' definitions, so anchoring only the target's own collection (the first draft) left a race (R4). These writes are rare, so whole-site anchoring costs nothing in practice. It is added only when the plan saw blocking rows. Without them, the write carries only `NOT EXISTS ()`, which is safe on its own: at worst it over-refuses. - A refusal answers `409 entry_referenced`, with up to 20 published referrers (id, title, collection, field) and the total. For a group change, it also names the group. - - Deleting a translation, or an original that still has another usable member, never blocks. References keep resolving to the remaining members (§3). -- **Moving a member** from C to D additionally refuses while a blocking row's field allows C but not D, and no other usable member remains (`409 reference_target_moved`). During a field operation, the pairs include the marker's `fromKey`. + - Deleting a translation, or an original that still has another usable member, never blocks. The guard protects the group's **existence**, not each language's card. + - The deleted member's locale then resolves to the fallback, or to nothing (§3). + - The delete dialog and `referenceIssues` name the locales that will lose the card ("Turkish event pages will show no venue"). +- **Moving a member** X from C to D: for each seen pair whose field does not allow D, the batch requires another usable member in a collection that field allows. Pairs whose field allows D are satisfied by X itself at D. The batch also carries the same `NOT EXISTS ( AND pair NOT IN :P)`, so an unseen field forces a retry. Refused: `409 reference_target_moved`. During a field operation, the pairs include the marker's `fromKey`. +- **Relinking a node with dependants** is refused (`409 translation_group_in_use`, §1). The write carries `NOT EXISTS (SELECT 1 FROM nodes WHERE site_id = ? AND kind = 'contentEntry' AND json_extract(body, '$.translation.group') = :X AND id <> :X)`. + - A translation created concurrently with X as its source carries the matching guard on its source: `EXISTS (SELECT 1 FROM nodes WHERE id = :source AND )`. + - That guard is new in `checkTranslationLinks`, which today checks the source's group with a read only (`store/translations.ts:131`). +- **While a field operation converts text into references** (`text → reference`, or a repeater gaining a reference sub-field from text), readers convert unconverted values on the fly (§7), so a published entry can show a group that has no row yet. Until phase C drops the marker, deletes, moves and group changes of entries in that field's target collections are refused (`409 field_operation_running`). The site anchor carries the marker, so an operation started concurrently fails the batch. - **Publishing a referrer.** A write that makes an entry published runs the target guard over *every* own reference value. A dangling id is refused (`unknown_entry`: remove or replace it), so a draft that lost its target cannot go live pointing at nothing. - Inherited values are not the translation's to fix. A translation that goes live while its original holds a dangling id gets a warning, and the reference renders as nothing (R8). - **Archiving and unpublishing** a target are allowed, and write nothing to `entry_refs`. The picker hides archived entries. @@ -309,23 +340,30 @@ A target deleted or moved between check and batch fails the guard, and `diagnose | # | Interleaving | Caught by | |---|---|---| -| R1 | referrer write ↔ deleting the target's last usable member | the referrer's target guard (members at batch time), or the delete's `NOT EXISTS ` (rows at batch time) | -| R2 | referrer write ↔ moving the target to a disallowed collection | the target guard (collection at batch time), or the move's blocking-row check | +| R1 | referrer write ↔ deleting the target's last usable member | the referrer's target guard (members at batch time, by the membership rule), or the delete's condition: blocking rows at batch time, with an unseen field forcing a retry | +| R2 | referrer write ↔ moving the target to a disallowed collection | the target guard (collection at batch time), or the move's per-pair condition plus the unseen-field check | | R3 | referrer write ↔ adding a reference field over its key, or removing an allowed collection | the definitions anchor, bound as planned (point 1). `field_key_in_use` counts committed values. The collection refusal counts values whose group has a member in the removed collection. | | R4 | moving a member ↔ narrowing a referrer field's `collections` | the move's site anchor. The narrowing's count sees the member once it has moved. | | R5 | delete ↔ any change to the definitions a blocking row's field depends on | the delete's site anchor | -| R6 | changing a member's group ↔ referrer write | the group change carries the delete guard and site anchor for its old group; the referrer's target guard checks membership at batch time | +| R6 | changing a member's group ↔ referrer write | the group change carries the delete guard and site anchor for its old group. The referrer's target guard checks membership at batch time, and a relinked node no longer matches its old group by id. A node with dependants cannot relink at all. | | R7 | a referrer becoming published ↔ deleting its target | the publish re-checks all own values with the target guard. The delete sees the referrer's new `source_status`, because its rows are rewritten in the same batch. | | R8 | publishing a translation that *inherits* the value ↔ deleting the target | when the publish is first, the delete's fragment reads member status at batch time. When the delete is first, nothing published showed the value at that moment. **Accepted:** the translation goes live with a warning and renders nothing; no guard can refuse a write the translator cannot fix. | | R9 | field-operation phase B ↔ deleting a target | the target guard ANDed into each rewritten entry's CAS (point 6) | | R10 | import, restore or sync ↔ anything | one atomic batch with the index rebuilt in it (point 4) | | R11 | a member reappearing (a new translation or edition member) | rows are kept whatever the membership (the invariant), so nothing needs re-creating | +| R12 | a reference shown only through an in-flight text → reference conversion ↔ deleting its target | `field_operation_running` refuses the delete until the marker is gone; the site anchor carries the marker | +| R13 | a blocking row from a field the plan did not see ↔ delete, move or group change | the unseen-field `NOT EXISTS` with the planned pairs `P` | +| R14 | a held node published at its live version (holds, `snapshots.ts:29-37`) whose working copy is a draft ↔ deleting its target | **Accepted**, like R8: holds are temporary, and the release's approval text and `referenceIssues` warn. Probing live versions of held nodes from the delete guard would need snapshot reads inside a write. | +| R15 | creating a translation of X ↔ relinking X | the source guard on the new translation, or the dependants guard on the relink | **4. Rebuilds** run in the same batch as the node set: `importSite`/`pull`/`push` (`sync-plan.ts:308`), `cloudhal import --replace` and `--revert`, and snapshot restores. - `entryRefsRebuild(siteId, rows)` builds the rows from the bodies, with the site's definitions from the site row, as `refsRebuild` does (`media/refs-rebuild.ts:12-37`), and `DELETE`s and `INSERT`s them under `site_id = ?`. - There is no membership join: the invariant keeps rows for every stored value, and the site filters keep them harmless. - The `INSERT` is chunked so each statement stays below 90 KB, inside the one atomic batch. -- **Step 7** also runs a one-time, budgeted, resumable `entryRefsRebuild` per site as an admin job, the pattern of `lists.backfill`. It is idempotent, and is a safety net: the gate (§1) means it should find nothing. +- **Statement budget.** The chunked `INSERT`s count against the import batch's 1,000-statement cap (`sync-plan.ts:311`), like every other statement there. +- **Step 7** also runs a one-time `entryRefsRebuild` per site as an admin job, the pattern of `lists.backfill`. Each site's rebuild is **one atomic batch**: the `DELETE` and its chunked `INSERT`s together, so there is never a window without rows. + - A site too large for one batch gets `INSERT OR IGNORE` chunks only, with no `DELETE`, resumable by cursor. That is safe, because it can only add missing rows. + - It is a safety net: the gate (§1) means it should find nothing. **5. Assets inside repeaters.** Each of these needs its own change, because none follows from the others: - **`fieldAssetRefs`** (`media/field-refs.ts:17-36`) reads only top-level string values (`if (typeof value !== "string") continue;`). It gains a walk over items. It records `fields..` in `asset_refs.field` (no index: the row is per asset and field), and returns each use's item path (`fields///`) beside it. @@ -375,29 +413,47 @@ R8 is the one accepted exception, and it is warned. Dangling ids can still exist - **Publish warns, never fails.** `referenceIssues(snapshot, now)`, pure and in `@cloudhal/core-ts/fields`, lists published entries whose references will not resolve. It appears in Studio's publish dialog, the MCP `publish` tool's approval text and `cloudhal snapshot --publish` output. A publish never fails on it: "a published snapshot must keep rendering" (RFC-0042 §4). - **Gates.** Every published entry's instants are in the snapshot's gate index already (`Loaded.gates`), so a referrer's cached page expires when its target's embargo opens or its `hide` expiry passes. -**Purges** (`cache-purge.ts`). There are no new tags. Referrer pages already carry their own node tag (`cloudhal-node-`), set by the theme for the entry a response renders and for the entries an archive lists (`cache-tags.ts:6-8`). The planner reaches them through the index. -- **The referrer rule.** Groups touched by the diff are collected from the before and after bodies (`translation.group ?? id`): - - every changed, added or removed entry; - - the entries the asset rule reaches; - - a changed group, which counts both its old and its new group. - - For those groups, the planner adds the node tags of: - - their referrers in the index that the fragment above shows as published, plus the inheriting members it names; - - the release's held referrers, read from the live snapshot, which the index may not match. - - Lookup is one `entry_refs` probe per group, run through a new `SnapshotReader.referrersOf(siteId, groups)`. The store implements it, and so does the release planner's virtual reader (`releases.ts:607`), on the same index. -- **What it covers.** Retitled, unpublished, archived and *newly visible* targets: a draft venue published, a new Turkish member, an embargo opening at a publish. Also changed covers, in every language and with any theme. -- **Costs and limits.** - - A group referenced by thousands of pages exceeds `maxTags` (300) and falls back to purging everything, as any large diff does today. - - `isCloudhalCacheTag` (`cache-tags.ts:48`) and the site Worker's purge endpoint (`cloudflare-entry.ts:312-325`) are unchanged, so an older site Worker accepts every purge. - - No ids of unresolved targets reach any header. -- The first draft's `cloudhal-ref-` tags are dropped. Existing themes build their own tag lists (`starters/blog/src/pages/[...path].astro:261-321`), so they would never have emitted them. The new prefixes would also have made older site Workers answer 400 and fall back to a full purge on every publish. And their privacy relied on unverified header stripping. +**Purges** (`cache-purge.ts`). There are no new tags. Pages already carry `cloudhal-node-` tags, set by the theme for the entry a response renders and for the entries an archive lists (`cache-tags.ts:6-8`). The planner reaches every page that can show a changed card. Without a purge, a page stays cached for `MAX_EDGE_AGE`, one year (`availability.ts:87`). Its edge age is capped at the next gate instant (`availability.ts:94-95`), which covers time-based visibility with no purge at all. + +1. **Seeds.** The groups touched by the diff, `COALESCE(translation.group, id)` of the before and after bodies: + - every changed, added or removed entry; + - the entries the asset rule reaches (with `fieldAssetRefs`, §2.5); + - both the old and the new group of a node that changed group. +2. **Closure to the render depth.** Pages show cards up to depth 2 (§4), so the planner walks referrers in two hops: + - hop 1: the referrers of the seed groups; + - hop 2: the referrers of the groups of the hop-1 referrers. + + Each page reached adds the node tags of its referrer and of every member of that referrer's group, so all of its language versions go. For purges, the walk uses **all rows, whatever `source_status`**: over-purging is cheap, and this keeps a status change in the working copy from hiding a cached page. +3. **"Used by" pages.** For every changed, added or removed entry, the planner also adds the node tags of the members of every group its before *or* after values name. A venue page that renders `referencedBy` ("events here") is purged when an event naming it appears, disappears or changes. That needs no theme code, because the target's page carries its own node tag. +4. **Which rows: the index or the snapshots.** + - **Index mode**, for an ordinary publish, where `to` was just snapshotted from the working copy. Referrers come from `entry_refs`, which then matches `to`. Any referrer that differs from `from` is in the diff, and is purged by the entry rule anyway. The release's held nodes are read from `to`'s bodies, since the index may not match them. + - **Snapshot mode**, for every other path: rollbacks (`store.ts:1127`, which moves only the pointer), release runs, and restores or imports that publish. Referrers come from the `from` and `to` snapshots' entry bodies, scanned with `fieldsReferences(defs, body)` as the asset rule already scans bodies (`cache-purge.ts:220-229`), under `maxReads`. + - **Release runs re-plan at run time.** The executor computes the referrer part of the plan after `publishRelease` commits, in snapshot mode, over base and result. Today it replays a plan pinned earlier (`releases.ts:154`, `releasePurgePlan`), which would miss a referrer published between pin and run. +5. **Every path that changes visibility, and how it purges:** + + | Path | Purge | + |---|---| + | publish (status changes, archive, delete, unpublish going live) | index mode | + | rollback | snapshot mode | + | release run | snapshot mode, re-planned at run time | + | import or restore that moves the published pointer | snapshot mode | + | embargo opening, `availability.until`, derived `hide`/`unlist` expiry | none: the gate-index TTL expires the page at that instant | + + Archive, delete and unpublish change only the working copy, and they go live through one of the paths above. +6. **Bounds and fallback.** + - Probes are batched: one `json_each` probe per hop over all its groups, so at most four index probes per plan (two hops, plus "used by"). + - Each probe and each body read counts against `maxReads` (2,000). Tags stop being collected at `maxTags + 1` (300 tags). + - Exceeding either purges everything, as any large diff does today (`cache-purge.ts`, "Falls back to purging everything"). + - A group referenced by thousands of pages therefore costs one full purge, never a stale card. +7. **Compatibility.** `isCloudhalCacheTag` (`cache-tags.ts:48`) and the site Worker's purge endpoint (`cloudflare-entry.ts:312-325`) are unchanged, so an older site Worker accepts every purge, and no id of an unresolved target reaches any header. + - `SnapshotReader.referrersOf(siteId, groups)` is new. The store implements it on the index, and the release executor's run-time reader implements it on snapshot bodies. + - The first revision's `cloudhal-ref-` tags are dropped. Existing themes build their own tag lists (`starters/blog/src/pages/[...path].astro:261-321`), new prefixes would make older site Workers answer 400, and ids in headers relied on unverified stripping. **Archives (RFC-0039).** An archive holds every node of the site, so references stay complete, and import rebuilds the index in its node batch (point 4). Dangling or foreign ids are reported, not refused (RFC-0041 open question 8): `cloudhal import --dry-run` adds `referenceIssues`. Until step 7, an archive *with* reference or repeater definitions is refused (`fields_unsupported`, §1). ### 3. The locale rule -A reference names a translation group (§1). In this site, the candidates are the group's members (`id = G` or `translation.group = G`) whose collection is in the field's `collections`. Resolution picks, among those the view shows: +A reference names a translation group (§1). In this site, the candidates are the group's members by the membership rule (§2: `COALESCE(translation.group, id) = G`) whose collection is in the field's `collections`. Resolution picks, among those the view shows: 1. the member whose `locale` is the page's locale; 2. otherwise, when `Site.localization.fallback` is `default-with-notice` (RFC-0033 §2), the member in the site's default locale, flagged `fromFallbackLocale` so themes set `lang` on it; @@ -410,7 +466,7 @@ The renderer, Studio's preview of a translation and `referenceIssues` ("Main hal - **Language editions** (RFC-0034; an original in another site of the project). Non-localized values are copied when the edition is made (RFC-0041 §5, through `forEdition`, `cross-site.ts:115`). Group ids are shared across sites, so a copied id stays the same id. - `forEdition` keeps an id when the edition site has a member of that group in a collection the edition's own field allows. Otherwise the id is dropped and reported ("2 references not carried: Main hall has no edition in site B yet"). - A repeater item left with no value once its references are dropped is dropped too, and reported, as `remove_field` with `sub` does (§7), so the edition write never fails on "no empty item". A `requiredToPublish` field emptied this way is fine: the copy is a draft. - - **Inside an edition site,** the edition node's own translations do not inherit its non-localized values. This is RFC-0041's existing rule: `originalOf` looks for the group's node in the same site (`store/entry-fields.ts:54-62`; search does the same, `search-rows.ts:299`), and the group's node is in another site. Each such member holds its own references, and their rows get `shared = 0` (§2). Changing that rule for every field type is open question 12. + - **Inside an edition site,** the edition node's own translations do not inherit its non-localized values. This is RFC-0041's existing rule: `originalOf` looks for the group's node in the same site (`store/entry-fields.ts:54-62`; search does the same, `search-rows.ts:299`), and the group's node is in another site. Each such member holds its own references, and the fragment finds no inheriting member for them, because no node there has the group's id (§2). Changing that rule for every field type is open question 12. - A reference never resolves across sites: it reads only members with this site's `site_id`. - This lands in phase 2 with the guard, because without it making an edition would be refused (`unknown_entry`) as soon as references exist. - **Repeaters are localized as a whole.** @@ -463,9 +519,9 @@ interface GraphSource { Over budget, the remaining targets resolve to nothing with one `console.warn`, never an error. Purges do not depend on what rendered: they go through the index (§2, Purges). A listing of 20 events at depth 1 costs 20. - **Batching.** - The snapshot source does no store reads: every lookup is in memory, and bundles (RFC-0010) need nothing new. - - The live source (`cloudhal dev`, previews) makes one store read per level: `getGroupMembers(siteId, ids)`, a `json_each` IN list over `nodes_translation_locale`, plus the original by id. + - The live source (`cloudhal dev`, previews) makes one store read per level: `getGroupMembers(siteId, ids)`, a `json_each` IN list with §2's membership form, served by `nodes_translation_locale` and the primary key. - **`referencedBy` equals the resolved values.** A row sits on the node holding the value. - - The live source reads `entry_refs` by `(site_id, target_id)`. It keeps the holder when the holder is in the requested locale, and expands a row only through §2's `shared` rule: `shared = 1` adds every inheriting member, `shared = 2` adds those without their own value of `root_key`, and `shared = 0` adds nobody. It then filters by the view. + - The live source reads `entry_refs` by `(site_id, target_id)`. It keeps the holder when the holder is in the requested locale, and expands a row only through §2's fragment. Inheriting members are those whose `translation.group` is the holder's id, with a same-site source. `shared = 1` adds them all, and `shared = 2` adds those without their own value of `root_key`. A translation's or an edition member's own row therefore adds nobody. It then filters by the view. - The snapshot's reverse map is built from `resolveFields` output, which applies the same rule. - The test plan compares the two on the same snapshot of the working copy, edition chains and localized values held by translations included. - **Listings** (RFC-0042 §4): @@ -505,7 +561,10 @@ All new UI follows `kumo-design`, `studio-ui` and DESIGN.md "Fields", imports on - Repeaters render inside the form with `RepeaterField`. Image sub-fields reuse `FieldImage`. - On a translation, a non-localized reference or repeater is read-only, with "From the original" and a link. A localized one shows the original's value as a placeholder. - **"Used by":** a `SectionCard` with every referrer (`GET …/entries/:id/references`), published ones first. - - **The delete dialog:** it disables Delete only while a *published* referrer exists and this is the group's last member ("Unpublish or edit these 2 published entries first"). It lists draft referrers as "Will show nothing for this field" and lets the delete go ahead. + - **The delete dialog:** it disables Delete only while a *published* referrer exists and this is the group's last usable member ("Unpublish or edit these 2 published entries first"). + - It lists draft referrers as "Will show nothing for this field" and lets the delete go ahead. + - When other members remain, it names the locales that will lose the card ("Turkish event pages will show no venue"). + - The same rules apply to unlinking a translation, which leaves its group. - **Fields builder** (`FieldDialog.tsx`): - **reference:** target collections (a multi-select of declared collections), "Several" (`multiple`), the maximum, and conditions when one collection is chosen; - **repeater:** a nested sub-field list (rows like `FieldRow`, with up/down and ⋯), the maximum and the item label; @@ -552,7 +611,7 @@ All new UI follows `kumo-design`, `studio-ui` and DESIGN.md "Fields", imports on `JSON_SCHEMA_KEYWORDS` (`json-schema.ts:29`) is unchanged. - **The strict profile and what `null` means.** OpenAI's strict mode needs every property required, with optional ones as a `null` union. Today's tools give `null` other meanings: `update_entry`'s merge patch deletes the key (`mergeFields`, `mcp-server/src/entry-fields.ts:45-53`), and `create_entry`'s checker refuses `null` as a value (`value-checks.ts:137-138`). Fed into `fields`, strict output would therefore delete untouched fields on update and be refused on create. So strict output gets its own input: - `create_entry` and `update_entry` gain **`fieldsStrict`**, which excludes `fields` (`invalid` when both are given). In it, `null` means **"leave as it is"**: on create that is no value, and on update the stored value stays. - - A `null` sub-value inside a repeater item is stripped before checking: that sub-field is unset in that item. + - A `null` sub-value inside a repeater item is stripped before checking: that sub-field is unset in that item. Inside an item, `null` means "unset", not "unchanged", because a given list replaces the whole list. The strict schema's item descriptions say so in words. - A given reference or repeater value replaces the whole value. - Removals go in a separate **`clearFields: FieldKey[]`**, which works with either input. - **Edge cases**, decided: @@ -630,9 +689,9 @@ The four operations, their phases, budgets and revert stay as RFC-0041 §6 built | Operation | Reference | Repeater | |---|---|---| -| `rename_field` | moves the key; rows' `field` follow each rewritten entry; list fields and sections follow as today (`fields-operations/references.ts`) | moves the key, or with `sub` the sub-key in every item; rows (`speakers.person`) and asset rows (`fields.speakers.photo`) follow | +| `rename_field` | moves the key; rows' `field` and `root_key` follow each rewritten entry (until then the fragment checks the old `root_key`, which can only over-block, §2); list fields and sections follow as today (`fields-operations/references.ts`) | moves the key, or with `sub` the sub-key in every item; rows (`speakers.person`) and asset rows (`fields.speakers.photo`) follow | | `change_field_type` | see below | see below; with `sub`, a sub-field converts by RFC-0041's table among sub-field types | -| `set_field_localized` | true→false keeps the original's ids and deletes translations' copies and their rows; false→true writes nothing | the same, for whole lists | +| `set_field_localized` | **phase A**, in the same batch as the site write and guarded on the site's new hash: `UPDATE entry_refs SET shared = <1 or 2> WHERE site_id = ? AND source_collection = ? AND root_key = ?`, so guards and purges follow `resolveFields` from that moment. **Phase B**, true→false: deletes translations' copies and their rows. false→true: no rewrite. | the same, for whole lists | | `remove_field` | deletes values and their rows; refused while list fields or presets use it (`list_field_in_use`) | deletes the list, or with `sub` the sub-value in every item (an item left empty is dropped) | New rows of the closed table (`conversions.ts:38`): @@ -656,6 +715,7 @@ New rows of the closed table (`conversions.ts:38`): The snapshot source converts against the snapshot's own entries. That is the graph it renders, and it may legitimately differ from the working copy, as every other value does. The test plan compares the two sources only on a snapshot of the current working copy, where both read the same entries. - **Phase B writes references only under the target guard** (§2.6). A missed guard skips the entry for the next pass, so a conversion can never write a deleted target. +- **Between phase A and phase C** of a conversion *into* references, readers already show converted groups that have no rows. Deletes, moves and group changes in the target collections are refused meanwhile (`field_operation_running`, §2.2, R12). Rows appear entry by entry as phase B rewrites. - **Revert** (`fields-operations/revert.ts`) restores earlier values under the same guard. A restored reference whose group has no member left in the site is dropped and reported in a new `FieldRevertResult.dropped: { entryId, field, targetId }[]`. - **Free definition edits:** - add or reorder sub-fields; @@ -670,14 +730,14 @@ New rows of the closed table (`conversions.ts:38`): | **A draft, archived, embargoed or hidden target leaks through a published referrer** (the snapshot holds drafts, and `Loaded` holds published entries before gates) | Resolution filters through the view's gates at its instant (§4); only gate-independent card parts are cached. A card carries only the member's public title, summary, URL and cover image, the image through the view's asset gates. Unresolved values render nothing: no id, no `data-` attribute, no empty link. JSON-LD's `reference` source uses the same cards. | RFC-0008's route sweep (extended by RFC-0042) gains a published referrer pointing at a draft, an archived, an embargoed and a `hide`-expired target, before and after their instants. No id, slug, title or asset of a hidden target may appear in HTML, JSON-LD, feeds, the sitemap, search, `llms.txt`, status pages or response headers. Purges go through the index, so no header carries target ids (§2). | | **IDOR in the picker and on write** | Pick and references routes run under the `/sites/:siteId` membership check and query by `site_id`. The target guard requires a member with `n.site_id = ?` (§2.1). An id of another site gets the same `unknown_entry` as an id that does not exist. Editions copy only ids the edition site can resolve (§3). | A shared-suite case: another site's entry or group id in `create_entry`, `update_entry`, the API and `push` is refused identically; the pick route with another site's id answers 404. | | **An imported or pushed archive pins another tenant's entries or assets** | Rows are site-scoped, and every guard filters both the row's `site_id` and the member's (§2). `asset_refs` gets the same filter as a phase-2 prerequisite (§2.5). | An archive referencing site Y's entry or asset, imported into site X, must not block Y's delete or move, and must not list X's titles to Y. | -| **Lock-out by low-privilege actors** (`content:draft:own`, `content:write:own`) | Only what a published entry shows blocks, by the `shared` rule (§2): a translation's own value is never inherited. Contributors cannot publish; drafts, review and archived entries never lock. | Policy-suite cases that must *not* block: a contributor's draft; an author's review entry; a draft translation's `localized: true` value while its original is published; an edition member's value. Cases that must block: an author's published entry; a published translation inheriting its original's `localized: false` value. | +| **Lock-out by low-privilege actors** (`content:draft:own`, `content:write:own`) | Only what a published entry shows blocks, by the `shared` rule (§2): a translation's own value is never inherited. Contributors cannot publish; drafts, review and archived entries never lock. | Policy-suite cases that must *not* block: a contributor's draft; an author's review entry; a draft translation's `localized: true` value while its original is published; a draft edition member whose edition translations are published (edition translations never inherit). Cases that must block: an author's published entry, an edition member's included; a published translation inheriting its original's `localized: false` value. | | **Fan-out DoS on render** | Caps of 50 per field, 50 items and 10 sub-fields (contract); 100 reference values per entry (code); depth ≤ 2 with a visited set; 200 targets per page; ≤ 2 live reads per page, 0 for snapshots; `referencedBy` ≤ 100. | A property test: random graphs with cycles and 10× the caps resolve within budget, never throw, and cost at most `depth` live reads. | | **Write amplification and slow guards** | Each guard is one indexed probe on `(site_id, target_id)` or one `json_each` count; members are found through `nodes_translation_locale`. Delete, move and group change cost one probe each, plus the site anchor when rows exist. The purge planner adds one probe per touched group. Rebuilds are chunked inside one batch. | RFC-0042's benchmark (#105), extended to 10,000 events × 3 references on Miniflare D1, reports the write, delete and move guard costs. | | **Picker search abuse** | `q` ≤ 100 characters, a substring match with no regex (rule 13), ≤ 20 results, and a per-member rate window. | A 100k-character `q` is refused before it is read. | | **Prompt injection through target titles** | Titles are data, quoted and truncated in approval prompts (`quote()`, 60 characters); `list_references` marks them as content. | Approval-prompt tests gain reference cases. | | **An older site Worker crashes on the new types** (`COMPILE[def.type]`, `props.ts:46`, has no entry) | `GET /__cloudhal/features` adds `{ "fields": 2 }`. Every write path refuses the new types with `409 fields_unsupported` until it is reported: interactive writes and `checkBundle` for import, sync, push, pull and restore (§1), as `gates: 2` guards `expiry` (`availability.ts:243`, `cloudflare/src/features.ts`). `resolveFields` skips unknown types from phase 1. | The site Worker features test; an import, a push and a restore of a bundle with a reference definition are refused before step 7; a renderer test with an unknown field type. | | **Assets in repeaters deleted while used** | `fieldAssetRefs` walks items, `fieldAssetIssues` resolves sub-field types, and the purge planner's asset rule reads field and item assets (§2.5). | An asset used only in a repeater item cannot be deleted; the rebuild keeps its row; a retired, foreign or non-image asset in an image sub-field is refused at its item path; a changed item image purges its page. | -| **A group change strands references** (`translation.group` is writable) | Leaving a group is guarded like a delete, with the site anchor (§2.2, R6). | Relinking a referenced untranslated original as a translation, and dropping `translation`, are both refused while a published entry shows the reference; a draft referrer does not block. | +| **A group change strands references** (`translation.group` is writable) | Membership is the current group, `COALESCE(translation.group, id)`, everywhere. Leaving a group is guarded like a delete, with the site anchor. A node with dependants cannot relink (§1, §2.2, R6, R15). | Relinking a referenced untranslated original as a translation, and dropping `translation`, are both refused while a published entry shows the reference; a draft referrer does not block. | **Policy.** No new scopes. - Field definitions stay site settings: `site:settings`; for agents, `site:write` plus a person's approval, person-classed as today (`field-tools.ts:264-267`). @@ -690,7 +750,8 @@ New rows of the closed table (`conversions.ts:38`): 0. **Prerequisite: `asset_refs` by site** (§2.5). - `deleteAsset`'s guard filters by site; - `refsRebuild` joins same-site assets and chunks its `INSERT`; - - the purge planner's asset rule reads field assets. + - the purge planner's asset rule reads field assets; + - the `unused` asset listing filter (`assets.ts:322`) filters by site, since a foreign archive can hide an asset from its own site's list today. It is independent of this RFC's types, and fixes three existing gaps. 1. **Contract and pure core:** @@ -707,10 +768,14 @@ New rows of the closed table (`conversions.ts:38`): - the target guard, the definitions anchor bound as planned, and the rows with `shared`/`root_key`; - the delete, move, group-change and publish rules, with the site anchor and the policy (§2.2), and the races R1 to R11 as cases; - chunked `entryRefsRebuild` in import, sync and restore, and `referenceIssues` in `import --dry-run`; - - the item walk in `fieldAssetRefs`, and `fieldAssetIssues` with sub-field types and item paths; + - the item walk in `fieldAssetRefs`, and `fieldAssetIssues` with sub-field types and item paths; the `skip` set built in `planEntryFields` (`entry-fields.ts:107`, `slice(0, 2)`) keeps item paths too; + - the membership rule everywhere, the relink refusal (`translation_group_in_use`) with the source guard on new translations, and the RFC-0041 `readsOriginal`/`originalOf` tightening; + - `field_operation_running` for references shown through an in-flight conversion; + - canonical bodies in `importSite`, and `CopySiteOptions.fieldTypes`; - rows and the target guard in field-operation rewrites, with operations naming the new types refused; - the kind-aware release `depsOf`; - edition id mapping in `forEdition`, emptied items dropped; + - new modules, to respect the size ratchet (rule 12): `core-ts/src/references/` holds the guards, the fragment, membership and the rebuild, and `core-ts/src/release-deps.ts` the kind-aware `depsOf`, so `releases.ts` (996 lines, in `scripts/size-baseline.json`) only imports them; - the `fields: 2` capability in the gate; - the API: entry routes, `pick` and `references`. @@ -720,7 +785,9 @@ New rows of the closed table (`conversions.ts:38`): - `groupMembers` and `referencedBy` on both sources, with the `shared` rule; - `EntryFields.astro` (links for references, a nested `
` per item); - the conversion maps in the live and snapshot sources; - - the purge planner's referrer rule and `SnapshotReader.referrersOf` (no new tags); + - the purge planner's referrer closure (two hops, "used by", index and snapshot modes) and `SnapshotReader.referrersOf` (no new tags), in `core-ts/src/references/purge.ts`; + - the release executor's run-time referrer plan; + - new renderer modules, `astro-renderer/src/references.ts` (members and reverse maps, `groupMembers`, `referencedBy`) and `reference-conversions.ts`, so `source.ts` (616 lines) does not approach 800; - features `fields: 2`; - the route sweep extension; - a frozen theme copy for `old-theme-compat.test.ts`. @@ -750,6 +817,11 @@ New rows of the closed table (`conversions.ts:38`): - delete: the last member versus a remaining member in an allowed or a disallowed collection, and a published versus a draft referrer; - moves of an original and of a translation; - group changes: relinking a referenced original, and dropping `translation`; + - **relink with a draft referrer, then publish the referrer (refused: the old id has no member), then delete**; + - relinking an original that has translations (refused, `translation_group_in_use`), and the concurrent translation-creation race (R15); + - the unseen-field interleaving (R13): `events.venue` allows `[places, halls]`, `talks.room` allows `[places]`, and a talk is published between plan and batch; + - `shared` after `set_field_localized`'s phase A, both directions; + - `field_operation_running` during a text → reference conversion (R12); - publish re-checks; - the `shared` rule: - a draft translation's `localized: true` value does not block; @@ -780,7 +852,8 @@ New rows of the closed table (`conversions.ts:38`): - **Renderer:** - identical cards and `referencedBy` on the snapshot and live sources, for a snapshot of the working copy; - gate instants; - - purge plans for a target retitled, unpublished, newly published, newly translated and with a changed cover, with a theme that emits no new tags. + - purge plans with a theme that emits no new tags: a target retitled, unpublished, newly published, newly translated and with a changed cover; a depth-2 page (A → B → C with C unpublished purges A); a "used by" page when a referrer appears or goes; a rollback whose referrer was since edited to draft; a release pinned before a referrer was published; the `maxTags` and `maxReads` fallbacks; + - the live-versus-snapshot `referencedBy` comparison runs with no field-operation marker active. - **Studio:** block tests (keyboard, focus after add and remove, live region), `design:shots` at both widths, `design:lint`. ## Alternatives considered @@ -811,7 +884,7 @@ New rows of the closed table (`conversions.ts:38`): - **Contract:** two new variants, a new `$defs` and one optional marker property. Existing sites, snapshots, bundles and archives stay valid, and there is no `schemaVersion` bump. - An admin Worker before phase 1 refuses the new definitions by schema. One at phases 1 to 6 refuses them by the gate, on every write path, bundles included (§1). Servers upgrade first. - - An older site Worker is covered by `fields: 2`. + - An older site Worker is covered by `fields: 2`. Node stores use `FIELDS_SUPPORTED`, and without a capability every store refuses (§1). - The purge endpoint and `isCloudhalCacheTag` are unchanged, so mixed admin and site Worker versions purge normally (§2). - **Data:** none to migrate. The types are refused on every write path until step 7, so migration 0033 is DDL only. Step 7's one-time per-site rebuild is the safety net, and the definitions anchor keeps `field_key_in_use` honest. - **Themes:** additive. `entry.fields` keeps meaning what is stored: group ids and item objects. @@ -845,8 +918,10 @@ New rows of the closed table (`conversions.ts:38`): 9. **Publish warnings as a gate:** never; warnings only. *Recommended.* A site setting could later make them a person-approval point for agents' publishes, like RFC-0042 §3.5's expiry rule. 10. **Review referrers:** they do not lock, like drafts. *Recommended*, so that a contributor's review request never pins an entry. The alternative, locking from review, protects content about to go live, but it reopens the contributor lock-out. 11. ~~`asset_refs` follow-ups~~ **Decided** in the second revision: §9 step 0 is a prerequisite of phase 2 (§2.5). -12. **Edition chains and non-localized values:** inside an edition site, the edition node's translations do not inherit its non-localized values of any type, as RFC-0041 behaves today (`originalOf`). References follow that rule (`shared = 0`). *Recommended:* keep it in stage 4, and fix it for every field type in its own change, defining the in-site root as the member with no same-site source. -13. **R8** (a translation published just after its inherited target was deleted): accept it with a warning. *Recommended*, because refusing would block a translator over a value they cannot edit. The alternative is to refuse the publish until the original's value is fixed. +12. **Edition chains and non-localized values:** inside an edition site, the edition node's translations do not inherit its non-localized values of any type, as RFC-0041 behaves today (`originalOf`). References follow that rule: the fragment finds no inheriting member there. *Recommended:* keep it in stage 4, and fix it for every field type in its own change, defining the in-site root as the member with no same-site source. +13. **R8 and R14** (a translation published just after its inherited target was deleted; a held node published at its live version): accept both with a warning. *Recommended*, because refusing would block a translator over a value they cannot edit. The alternative is to refuse the publish until the original's value is fixed. +14. **Tightening `readsOriginal`/`originalOf`** (an original must have its own id as its group): legacy data relinked before this RFC stops inheriting non-localized values. *Recommended*, because it is the only reading under which an original belongs to exactly one group. Phase 2 reports such nodes in `cloudhal import --dry-run` and on Studio's collection screen, beside `referenceIssues`. +15. **Depth-2 purges** cost a second index probe per publish. *Recommended* as specified. The alternative, dropping depth 2 until themes tag cards, was rejected because the index makes it theme-independent. ## Review log (2026-11-10, PR #193) @@ -898,3 +973,26 @@ Three verifiers found 34 earlier findings resolved. The unresolved ones and the | The Event address data path was undefined, and the phase-8 contract was unlisted | `StructuredDataContext.references`, filled by `withStructuredData` through `pickMember` and `resolveFields`, independent of card depth. The phase-8 contract change is listed in §1 (§4). | | Move and delete races | Listed as R1 to R11 with their guards; R8 accepted and warned (§2.3, OQ 13). | +### Third round (2026-11-10, final verification of 9e457d12) + +| Finding | Decision | +|---|---| +| Membership used `n.id = G OR group = G`, so a relinked node stayed in its old group, R6 failed, and a published reference could strand (blocking, three verifiers) | One definition everywhere, `COALESCE(translation.group, id) = G`, in the index-friendly form (§2, "Membership"). Relinking a node with dependants is refused (`translation_group_in_use`), with a source guard on new translations (R15). `readsOriginal`/`originalOf` require the original's own group to be its id (OQ 14). The relink-then-publish-then-delete case is in the test plan. | +| `shared` encoded the holder's role at write time | `shared` is a pure function of `localized`; who inherits is decided at batch time by the fragment, exactly as `readsOriginal` (§2). | +| Guards computed allowed collections from rows seen at plan time (blocking) | Planned pairs `P` are bound, and `NOT EXISTS (blocking rows with pair NOT IN :P)` forces a retry, for deletes, moves and group changes (§2.2, R13; the `events.venue`/`talks.room` case in the tests). | +| One-hop purges missed depth-2 cards, rollbacks, release runs and "used by" lists (blocking) | Closure to the render depth (two hops), "used by" pages via the groups changed entries name, index mode for ordinary publishes and snapshot mode for rollbacks, releases (re-planned at run time) and restores. Time gates rely on the gate-index TTL. Bounded by `maxReads`/`maxTags`, with purge-everything as the fallback (§2, Purges). | +| `set_field_localized` left `shared` stale; text→reference rendered references without rows (blocking) | Phase A flips `shared` in the site write's batch (§7). Conversions into references refuse deletes, moves and group changes of target entries until phase C (`field_operation_running`, R12). The `rename_field` `root_key` lag only over-blocks (§2). | +| The bundle gate accepted when support was absent | An absent capability refuses; `CopySiteOptions.fieldTypes` and the Node constant `FIELDS_SUPPORTED` (§1). | +| Imported bodies with unsorted keys break the anchor | `importSite` stores canonical bodies from phase 2 (§2.1). | +| The `skip` set cannot hold item paths | Changed in `planEntryFields` too (§9 step 2). | +| The `unused` asset filter is unscoped | Added to step 0 (§9). | +| `null` inside items | Means unset in that item, and the schema descriptions say so (§6). | +| Referrer probes are not counted against `maxReads` | One batched probe per hop, counted, stopping at `maxTags + 1` (§2, Purges). | +| A delete protects the group, not each locale's card | Stated, and the dialog and `referenceIssues` name the locales that lose the card (§2.2, §5). | +| Holds can publish a held referrer while its working copy is a draft | Listed as R14, accepted and warned (OQ 13). | +| The §8 policy-test wording for edition members | Reworded: edition translations never inherit, but an edition member's own published value blocks. | +| The target guard SQL lacked the archived check | A per-pair `fresh` flag (§2.1). | +| Step 7 on Node, and the atomicity of its rebuild | `FIELDS_SUPPORTED`; one atomic batch per site, or `INSERT OR IGNORE`-only chunks for very large sites; the statement cap is counted (§2.4). | +| Size ratchet | New modules named: `core-ts/src/references/`, `core-ts/src/release-deps.ts`, `astro-renderer/src/references.ts` and `reference-conversions.ts` (§9). | +| The live-versus-snapshot comparison during operations | Runs with no marker active (test plan). | + From fca37cbed91a0c3d6e62461b3722780c03316842 Mon Sep 17 00:00:00 2001 From: Madraka <8898640+Madraka@users.noreply.github.com> Date: Tue, 10 Nov 2026 16:04:39 +0300 Subject: [PATCH 5/8] docs(rfc): RFC-0045 fourth revision after the verification of 21abf4e1 - every delete, move and group change of an entry carries the site anchor, closing R12 for deletes planned before the conversion marker - purges use one method for every publish path: index rows corrected by a hash check against `to`, with a from/to body scan of the entries that differ (an older snapshot publish and writes between snapshot and publish are exact) - `shared` is read from the site row inside the single INSERT ... SELECT at batch time (R16) - the group check and a written-group guard run on any change to translation.group or source - the step-7 job canonicalizes rows; the field-refs.ts:81 skip filter uses item paths; "used by" purges use resolved values - wording, archived-move alignment, size-ratchet modules, phase-2 compatibility notes and the refusal list are completed; review log "Fourth round" Co-Authored-By: Claude Opus 5.5 Signed-off-by: Madraka <8898640+Madraka@users.noreply.github.com> --- ...ctions-stage-4-references-and-repeaters.md | 116 ++++++++++++------ 1 file changed, 78 insertions(+), 38 deletions(-) diff --git a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md index 1da8af3a..c9bade82 100644 --- a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md +++ b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md @@ -26,9 +26,9 @@ Two new field types join RFC-0041's nine: - A **`reference`** field points to entries of one to five declared collections of the same site, one target or up to fifty. It stores the target's **translation group id**, the same id in every site of a project. It always resolves to that group's member in this site and in the page's language. - A **`repeater`** field is an ordered list of up to fifty items, each of up to ten flat sub-fields. A repeater cannot contain a repeater, so every agent-facing schema has a fixed depth and no recursion. -References are strong where it matters. A target *group* cannot lose its last usable member in a site, whether by a delete, a move or a relink into another group, while a published entry shows a reference to it exactly as `resolveFields` would show it. One language version's card can still go when that version is deleted, with a warning. +References are strong where it matters. A target *group* cannot lose its last member in a site, whether by a delete, a move or a relink into another group, while a published entry shows a reference to it exactly as `resolveFields` would show it. The guard protects a member's *existence*, not its usability: archiving or unpublishing the last member is allowed and renders nothing. One language version's card can still go when that version is deleted, with a warning. -References held by drafts, by entries in review, or by archived entries never lock anything. A node's group is one value everywhere: `COALESCE(translation.group, id)`. +A reference locks only through an entry that is published and shows it. A draft's own references lock nothing until a published entry shows them; that happens only when a published translation inherits a draft original's value (§2.2). A node's group is one value everywhere: `COALESCE(translation.group, id)`. A reverse index, `entry_refs`, holds one row per stored value, whatever the target's state. It is written in each entry's own batch, anchored on the entry's new content hash and on its collection's definitions *as planned*. Every guard filters it by site. @@ -263,7 +263,7 @@ CREATE INDEX entry_refs_target ON entry_refs (site_id, target_id, source_status) OR json_extract(n.body, '$.translation.group') = :G) ``` The first revision's `n.id = G OR translation.group = G` let a relinked node stay a member of its old group by its id, which defeated the group-change guard (R6). - - The relink refusal of §1 keeps the inheritance side exact: a node with same-site dependants always has its own id as its group. + - The relink refusal of §1, with the group guard on every write that sets or changes `translation.group` (§2.2), keeps the inheritance side exact: a node with same-site dependants always has its own id as its group. - `readsOriginal` and `originalOf` additionally require the original's own group to be its id. Legacy data relinked before this RFC then inherits nothing rather than reading from a node of another group. This is an RFC-0041 behaviour change for that edge case only, and the fragment agrees with it. **"Shown by a published entry"** is one SQL fragment. The guards, the live `referencedBy` and the purge planner all use it, so they mean the same as the snapshot's resolved values: @@ -300,7 +300,19 @@ The member probe uses `nodes_translation_locale` (migration 0025: `site_id, kind Equality needs stored bodies in canonical form. Store writes are canonical (`canonicalJson`, `node-sql.ts:62/91`), but `importSite` inserts a bundle's body text verbatim (`sync-plan.ts:300-301`), and SQLite's `json()` keeps key order. So from phase 2, `importSite` stores `canonicalJson(JSON.parse(body))` for every row. The hash is computed over canonical JSON already, so it does not change. A hand-made archive with unsorted keys therefore cannot leave a collection whose writes always conflict. The anchor conflicts only when that collection's definitions change, never on other site edits. The loser retries under the new definitions, where `field_key_in_use` then sees its value. This is RFC-0042's rule that list-value rows are valid only "under the fingerprint they were built with" (`list-values/fingerprint.ts`), applied to writes. -- **The rows**, anchored on the entry's new hash: `DELETE FROM entry_refs WHERE source_id = ? AND `, then one `INSERT OR IGNORE … SELECT … FROM json_each(?) WHERE `, appended to `own` (`entries.ts:110`). Every write of the holder rewrites all its rows, so `source_status` and `shared` stay current. +- **The rows**, anchored on the entry's new hash: `DELETE FROM entry_refs WHERE source_id = ? AND `, then one `INSERT OR IGNORE … SELECT … FROM json_each(?) WHERE `, appended to `own` (`entries.ts:110`). Every write of the holder rewrites all its rows, so `source_status` stays current. + - **`shared` is never bound from the plan.** The same single `INSERT … SELECT` reads it from the site row *as it is when the batch runs*, so a status-only or title-only write that races `set_field_localized`'s phase A cannot write a stale value: + ```sql + INSERT OR IGNORE INTO entry_refs (site_id, target_id, source_id, source_collection, field, root_key, shared, source_status) + SELECT ?, json_extract(p.value, '$.t'), ?, ?, json_extract(p.value, '$.f'), json_extract(p.value, '$.k'), + (SELECT CASE json_extract(d.value, '$.localized') WHEN 1 THEN 2 ELSE 1 END + FROM nodes s, json_each(json_extract(s.body, ?)) d -- ? = '$.collections."".fields' + WHERE s.id = ? AND s.kind = 'site' AND json_extract(d.value, '$.key') = json_extract(p.value, '$.k')), + ? + FROM json_each(?) p WHERE + ``` + - Batches run one at a time, so the row reflects the site either before phase A, which phase A's `UPDATE` then corrects, or after it. + - A key without a definition at batch time (the definitions anchor makes this impossible for changed keys) yields `NULL`, which the `NOT NULL` column refuses, so the batch fails and retries. A target deleted or moved between check and batch fails the guard, and `diagnose()` names it (`unknown_entry`, `reference_target_moved`). One bound parameter carries the whole list (D1 allows 100 parameters), and 100 group ids stay far below 100 KB per statement. @@ -321,15 +333,15 @@ A target deleted or moved between check and batch fails the guard, and `diagnose The second `NOT EXISTS` makes a blocking row from a field the plan did not see, committed in between, fail the batch, so the write retries and re-plans with that field. The first revision's form let it through: `events.venue` allows `[places, halls]` and `talks.room` allows `[places]`; a talk published meanwhile with `room` = G is stranded once the last member in `places` goes. - **The site anchor**, `EXISTS (SELECT 1 FROM nodes WHERE id = ? AND kind = 'site' AND hash = ?)`, with the hash of the `Site` object the write planned with. The allowed collections come from *referrer* collections' definitions, so anchoring only the target's own collection (the first draft) left a race (R4). - These writes are rare, so whole-site anchoring costs nothing in practice. It is added only when the plan saw blocking rows. Without them, the write carries only `NOT EXISTS ()`, which is safe on its own: at worst it over-refuses. + These writes are rare, so **every** delete, move and group change of a `contentEntry` carries the site anchor, whether or not the plan saw blocking rows. The first revisions added it only when rows were seen. That left R12 open: during a text → reference conversion, the values that will become references have no rows yet, so a delete planned before the marker committed carried no anchor and went through. With the anchor always present, the batch fails once the site changes, the retry plans against the marker, and `field_operation_running` refuses it. - A refusal answers `409 entry_referenced`, with up to 20 published referrers (id, title, collection, field) and the total. For a group change, it also names the group. - Deleting a translation, or an original that still has another usable member, never blocks. The guard protects the group's **existence**, not each language's card. - The deleted member's locale then resolves to the fallback, or to nothing (§3). - The delete dialog and `referenceIssues` name the locales that will lose the card ("Turkish event pages will show no venue"). -- **Moving a member** X from C to D: for each seen pair whose field does not allow D, the batch requires another usable member in a collection that field allows. Pairs whose field allows D are satisfied by X itself at D. The batch also carries the same `NOT EXISTS ( AND pair NOT IN :P)`, so an unseen field forces a retry. Refused: `409 reference_target_moved`. During a field operation, the pairs include the marker's `fromKey`. +- **Moving a member** X from C to D: for each seen pair whose field does not allow D, the batch requires another usable member in a collection that field allows. Pairs whose field allows D are satisfied by X itself at D, unless X is archived. An archived X satisfies nothing, exactly as in the delete rule. The batch also carries the same `NOT EXISTS ( AND pair NOT IN :P)`, so an unseen field forces a retry. Refused: `409 reference_target_moved`. During a field operation, the pairs include the marker's `fromKey`. - **Relinking a node with dependants** is refused (`409 translation_group_in_use`, §1). The write carries `NOT EXISTS (SELECT 1 FROM nodes WHERE site_id = ? AND kind = 'contentEntry' AND json_extract(body, '$.translation.group') = :X AND id <> :X)`. - - A translation created concurrently with X as its source carries the matching guard on its source: `EXISTS (SELECT 1 FROM nodes WHERE id = :source AND )`. - - That guard is new in `checkTranslationLinks`, which today checks the source's group with a read only (`store/translations.ts:131`). + - **Every write that sets or changes `translation.group` or `translation.source`** carries a guard keyed on the *written* group X. This covers a new translation, a source change, and a group-only change, which today passes unchecked when the source stays the same (`store/translations.ts:107-113`). The guard is `NOT EXISTS (SELECT 1 FROM nodes WHERE id = :X AND site_id = ? AND json_extract(body, '$.translation.group') IS NOT NULL AND json_extract(body, '$.translation.group') <> :X)`: X's node, if it is in this site, still has its own id as its group. + - `checkTranslationLinks` runs its source-group check on the same condition, not only when `source` changes (`translations.ts:111-135`), and adds this guard to the write. Today it checks the source's group with a read only (`:131`). - **While a field operation converts text into references** (`text → reference`, or a repeater gaining a reference sub-field from text), readers convert unconverted values on the fly (§7), so a published entry can show a group that has no row yet. Until phase C drops the marker, deletes, moves and group changes of entries in that field's target collections are refused (`409 field_operation_running`). The site anchor carries the marker, so an operation started concurrently fails the batch. - **Publishing a referrer.** A write that makes an entry published runs the target guard over *every* own reference value. A dangling id is refused (`unknown_entry`: remove or replace it), so a draft that lost its target cannot go live pointing at nothing. - Inherited values are not the translation's to fix. A translation that goes live while its original holds a dangling id gets a warning, and the reference renders as nothing (R8). @@ -351,10 +363,11 @@ A target deleted or moved between check and batch fails the guard, and `diagnose | R9 | field-operation phase B ↔ deleting a target | the target guard ANDed into each rewritten entry's CAS (point 6) | | R10 | import, restore or sync ↔ anything | one atomic batch with the index rebuilt in it (point 4) | | R11 | a member reappearing (a new translation or edition member) | rows are kept whatever the membership (the invariant), so nothing needs re-creating | -| R12 | a reference shown only through an in-flight text → reference conversion ↔ deleting its target | `field_operation_running` refuses the delete until the marker is gone; the site anchor carries the marker | +| R12 | a reference shown only through an in-flight text → reference conversion ↔ deleting its target, including a delete planned *before* phase A commits | `field_operation_running` refuses the delete until the marker is gone. The site anchor, now on every delete, move and group change, fails a batch planned before the marker. | | R13 | a blocking row from a field the plan did not see ↔ delete, move or group change | the unseen-field `NOT EXISTS` with the planned pairs `P` | | R14 | a held node published at its live version (holds, `snapshots.ts:29-37`) whose working copy is a draft ↔ deleting its target | **Accepted**, like R8: holds are temporary, and the release's approval text and `referenceIssues` warn. Probing live versions of held nodes from the delete guard would need snapshot reads inside a write. | -| R15 | creating a translation of X ↔ relinking X | the source guard on the new translation, or the dependants guard on the relink | +| R15 | creating a translation of X, or setting any node's group to X ↔ relinking X | the written-group guard on the joining write, or the dependants guard on the relink | +| R16 | a status-only or title-only write of a holder ↔ `set_field_localized` phase A | `shared` is read from the site row inside the `INSERT … SELECT` at batch time, and phase A's `UPDATE` corrects rows written before it (§2.1) | **4. Rebuilds** run in the same batch as the node set: `importSite`/`pull`/`push` (`sync-plan.ts:308`), `cloudhal import --replace` and `--revert`, and snapshot restores. - `entryRefsRebuild(siteId, rows)` builds the rows from the bodies, with the site's definitions from the site row, as `refsRebuild` does (`media/refs-rebuild.ts:12-37`), and `DELETE`s and `INSERT`s them under `site_id = ?`. @@ -363,6 +376,7 @@ A target deleted or moved between check and batch fails the guard, and `diagnose - **Statement budget.** The chunked `INSERT`s count against the import batch's 1,000-statement cap (`sync-plan.ts:311`), like every other statement there. - **Step 7** also runs a one-time `entryRefsRebuild` per site as an admin job, the pattern of `lists.backfill`. Each site's rebuild is **one atomic batch**: the `DELETE` and its chunked `INSERT`s together, so there is never a window without rows. - A site too large for one batch gets `INSERT OR IGNORE` chunks only, with no `DELETE`, resumable by cursor. That is safe, because it can only add missing rows. + - The same job rewrites the site's node rows in canonical form (`canonicalJson(JSON.parse(body))`, hash unchanged). That covers rows imported verbatim before phase 2, and imports that planned `unchanged` (`sync-plan.ts:257, 295`), so the definitions anchor is live on every collection. - It is a safety net: the gate (§1) means it should find nothing. **5. Assets inside repeaters.** Each of these needs its own change, because none follows from the others: @@ -395,11 +409,11 @@ Each refusal names the operation to use (§7). **9. Site deletion.** `entry_refs` joins `SITE_TABLES` (`site-delete.ts:16`) in phase 2. The existing test fails on a `site_id` table missing from that list. **The invariant restated.** -- No write lets a group lose its last usable member in a site while a published entry there shows a reference to it. +- No write lets a group lose its last member in a site (a member not archived, in a collection the field allows) while a published entry there shows a reference to it. Archiving that member stays allowed. - No write stores a new own reference to a group without such a member. - The index always matches the stored values. -R8 is the one accepted exception, and it is warned. Dangling ids can still exist, and every reader treats them as unresolved without throwing: +R8 and R14 are the accepted exceptions, both warned (open question 13). Dangling ids can still exist, and every reader treats them as unresolved without throwing: - in drafts whose target went; - in hand-made or foreign archives; - in publish snapshots composed with holds, which leave out a held node that is not live (`store/snapshots.ts:29-37`); @@ -424,27 +438,26 @@ R8 is the one accepted exception, and it is warned. Dangling ids can still exist - hop 2: the referrers of the groups of the hop-1 referrers. Each page reached adds the node tags of its referrer and of every member of that referrer's group, so all of its language versions go. For purges, the walk uses **all rows, whatever `source_status`**: over-purging is cheap, and this keeps a status change in the working copy from hiding a cached page. -3. **"Used by" pages.** For every changed, added or removed entry, the planner also adds the node tags of the members of every group its before *or* after values name. A venue page that renders `referencedBy` ("events here") is purged when an event naming it appears, disappears or changes. That needs no theme code, because the target's page carries its own node tag. -4. **Which rows: the index or the snapshots.** - - **Index mode**, for an ordinary publish, where `to` was just snapshotted from the working copy. Referrers come from `entry_refs`, which then matches `to`. Any referrer that differs from `from` is in the diff, and is purged by the entry rule anyway. The release's held nodes are read from `to`'s bodies, since the index may not match them. - - **Snapshot mode**, for every other path: rollbacks (`store.ts:1127`, which moves only the pointer), release runs, and restores or imports that publish. Referrers come from the `from` and `to` snapshots' entry bodies, scanned with `fieldsReferences(defs, body)` as the asset rule already scans bodies (`cache-purge.ts:220-229`), under `maxReads`. - - **Release runs re-plan at run time.** The executor computes the referrer part of the plan after `publishRelease` commits, in snapshot mode, over base and result. Today it replays a plan pinned earlier (`releases.ts:154`, `releasePurgePlan`), which would miss a referrer published between pin and run. -5. **Every path that changes visibility, and how it purges:** - - | Path | Purge | - |---|---| - | publish (status changes, archive, delete, unpublish going live) | index mode | - | rollback | snapshot mode | - | release run | snapshot mode, re-planned at run time | - | import or restore that moves the published pointer | snapshot mode | - | embargo opening, `availability.until`, derived `hide`/`unlist` expiry | none: the gate-index TTL expires the page at that instant | - - Archive, delete and unpublish change only the working copy, and they go live through one of the paths above. +3. **"Used by" pages.** For every changed, added or removed entry, the planner also adds the node tags of the members of every group its before *or* after **resolved** values name. Resolution uses `resolveFields` with the same-site original read from the same snapshot, so a translation that only inherits its original's `venue` counts too. + - A venue page that renders `referencedBy` ("events here") is purged when an event naming it appears, disappears or changes, with no theme code, because the target's page carries its own node tag. + - `referencedBy` results are drawn as cards: title, URL, summary and cover. Drawing *their* fields and references on the target page is outside the purge guarantee, and `docs/guides/fields.md` says so. +4. **One mode for every path: the index, corrected by a hash check.** Which path published (`store.publish` of any snapshot, `POST …/snapshots/:id/publish`, an approved role request, a rollback, a release run, a restore) does not matter, and `PublishChange` stays `{from, to}`. Two steps: + 1. **Find the entries whose working copy differs from `to`.** One statement compares `to.nodes` with the working copy, `SELECT … FROM json_each(:toEntryRefs) t LEFT JOIN nodes n ON n.id = json_extract(t.value, '$.id') WHERE n.hash IS NOT json_extract(t.value, '$.hash')`, and lists the working entries absent from `to`. Held nodes are naturally among the differing ones. + 2. **Take the referrers from two sources.** + - For entries whose working copy equals `to`, the index answers exactly: their rows are their `to` values. A referrer that differs between `from` and `to` is in the diff, and the entry rule purges it anyway. + - For the differing entries, the planner reads their `from` and `to` bodies and extracts their reference values with `fieldsReferences(defs, body)`, as the asset rule scans bodies (`cache-purge.ts:220-229`). + - Working entries absent from `to` can only add index rows, so they over-purge, which is harmless. + + The approved-request case, an older snapshot published after its referrer was edited, and a write landing between `createSnapshot` and `publish` are thereby exact. + + **Release runs** re-plan the referrer part at run time, after `publishRelease` commits, with this same method. They do not replay the plan pinned earlier (`releases.ts:154`), which would miss a referrer published between pin and run. +5. **Every path that changes visibility:** publish (any snapshot), rollback, release run, and an import or restore that moves the published pointer all purge by point 4. Archive, delete and unpublish change only the working copy and go live through one of these. + - Time-based visibility (an embargo opening, `availability.until`, a derived `hide`/`unlist` expiry) needs no purge: the gate-index TTL expires the page at that instant (`availability.ts:94-95`). 6. **Bounds and fallback.** - - Probes are batched: one `json_each` probe per hop over all its groups, so at most four index probes per plan (two hops, plus "used by"). - - Each probe and each body read counts against `maxReads` (2,000). Tags stop being collected at `maxTags + 1` (300 tags). + - At most three batched index probes per plan, one `json_each` each: hop 1, hop 2 and "used by". There is one hash-comparison statement, and the member lookups for referrer groups run as one batched probe per hop. + - Every probe and body read counts against `maxReads` (2,000). Tags stop being collected at `maxTags + 1` (300 tags). - Exceeding either purges everything, as any large diff does today (`cache-purge.ts`, "Falls back to purging everything"). - - A group referenced by thousands of pages therefore costs one full purge, never a stale card. + - A rollback far back on a large site differs in many entries and will usually take that fallback. That is safe; a per-snapshot reverse map built once is a later optimization (follow-ups). 7. **Compatibility.** `isCloudhalCacheTag` (`cache-tags.ts:48`) and the site Worker's purge endpoint (`cloudflare-entry.ts:312-325`) are unchanged, so an older site Worker accepts every purge, and no id of an unresolved target reaches any header. - `SnapshotReader.referrersOf(siteId, groups)` is new. The store implements it on the index, and the release executor's run-time reader implements it on snapshot bodies. - The first revision's `cloudhal-ref-` tags are dropped. Existing themes build their own tag lists (`starters/blog/src/pages/[...path].astro:261-321`), new prefixes would make older site Workers answer 400, and ids in headers relied on unverified stripping. @@ -751,7 +764,8 @@ New rows of the closed table (`conversions.ts:38`): - `deleteAsset`'s guard filters by site; - `refsRebuild` joins same-site assets and chunks its `INSERT`; - the purge planner's asset rule reads field assets; - - the `unused` asset listing filter (`assets.ts:322`) filters by site, since a foreign archive can hide an asset from its own site's list today. + - the `unused` asset listing filter (`assets.ts:322`) filters by site, since a foreign archive can hide an asset from its own site's list today; + - size ratchet (rule 12): the asset rule moves out of `planCachePurge` (a baselined 233-line function) into `core-ts/src/cache-purge-assets.ts`. The edits to `assets.ts` (1,370 lines) and `store.ts` (1,212), both baselined, add no net lines. It is independent of this RFC's types, and fixes three existing gaps. 1. **Contract and pure core:** @@ -768,7 +782,7 @@ New rows of the closed table (`conversions.ts:38`): - the target guard, the definitions anchor bound as planned, and the rows with `shared`/`root_key`; - the delete, move, group-change and publish rules, with the site anchor and the policy (§2.2), and the races R1 to R11 as cases; - chunked `entryRefsRebuild` in import, sync and restore, and `referenceIssues` in `import --dry-run`; - - the item walk in `fieldAssetRefs`, and `fieldAssetIssues` with sub-field types and item paths; the `skip` set built in `planEntryFields` (`entry-fields.ts:107`, `slice(0, 2)`) keeps item paths too; + - the item walk in `fieldAssetRefs`, and `fieldAssetIssues` with sub-field types and item paths. The `skip` set built in `planEntryFields` (`entry-fields.ts:107`, `slice(0, 2)`) and the skip filter in `fieldAssetIssues` (`field-refs.ts:81`, which compares `r.field.replace('.', '/')`) both switch to the item path `fieldAssetRefs` returns; - the membership rule everywhere, the relink refusal (`translation_group_in_use`) with the source guard on new translations, and the RFC-0041 `readsOriginal`/`originalOf` tightening; - `field_operation_running` for references shown through an in-flight conversion; - canonical bodies in `importSite`, and `CopySiteOptions.fieldTypes`; @@ -785,8 +799,8 @@ New rows of the closed table (`conversions.ts:38`): - `groupMembers` and `referencedBy` on both sources, with the `shared` rule; - `EntryFields.astro` (links for references, a nested `
` per item); - the conversion maps in the live and snapshot sources; - - the purge planner's referrer closure (two hops, "used by", index and snapshot modes) and `SnapshotReader.referrersOf` (no new tags), in `core-ts/src/references/purge.ts`; - - the release executor's run-time referrer plan; + - the purge planner's referrer closure (two hops, "used by" by resolved values, the index corrected by the hash check) and `SnapshotReader.referrersOf` (no new tags), in `core-ts/src/references/purge.ts`; + - the release executor's run-time referrer plan, in `core-ts/src/references/release-purge.ts`, so `releases.ts` (996, baselined) only calls it; - new renderer modules, `astro-renderer/src/references.ts` (members and reverse maps, `groupMembers`, `referencedBy`) and `reference-conversions.ts`, so `source.ts` (616 lines) does not approach 800; - features `fields: 2`; - the route sweep extension; @@ -821,7 +835,9 @@ New rows of the closed table (`conversions.ts:38`): - relinking an original that has translations (refused, `translation_group_in_use`), and the concurrent translation-creation race (R15); - the unseen-field interleaving (R13): `events.venue` allows `[places, halls]`, `talks.room` allows `[places]`, and a talk is published between plan and batch; - `shared` after `set_field_localized`'s phase A, both directions; - - `field_operation_running` during a text → reference conversion (R12); + - `field_operation_running` during a text → reference conversion, including a delete planned before phase A commits and batched after it (R12); + - a status-only holder write racing `set_field_localized` phase A leaves the right `shared` (R16); + - a group-only `translation.group` change racing a relink (R15); - publish re-checks; - the `shared` rule: - a draft translation's `localized: true` value does not block; @@ -852,7 +868,7 @@ New rows of the closed table (`conversions.ts:38`): - **Renderer:** - identical cards and `referencedBy` on the snapshot and live sources, for a snapshot of the working copy; - gate instants; - - purge plans with a theme that emits no new tags: a target retitled, unpublished, newly published, newly translated and with a changed cover; a depth-2 page (A → B → C with C unpublished purges A); a "used by" page when a referrer appears or goes; a rollback whose referrer was since edited to draft; a release pinned before a referrer was published; the `maxTags` and `maxReads` fallbacks; + - purge plans with a theme that emits no new tags: a target retitled, unpublished, newly published, newly translated and with a changed cover; a depth-2 page (A → B → C with C unpublished purges A); a "used by" page when a referrer appears or goes; a rollback whose referrer was since edited to draft; a release pinned before a referrer was published; **an approved request publishing an older snapshot after the referrer was edited, then the target unpublished: the referrer's page is purged**; a referrer write landing between `createSnapshot` and `publish`; a translation that only inherits the reference appearing on a "used by" page; the `maxTags` and `maxReads` fallbacks; - the live-versus-snapshot `referencedBy` comparison runs with no field-operation marker active. - **Studio:** block tests (keyboard, focus after add and remove, live region), `design:shots` at both widths, `design:lint`. @@ -892,6 +908,12 @@ New rows of the closed table (`conversions.ts:38`): - Older copies show nothing for them. - Either way, no theme code is needed for correct purges, because referrers are purged through their own node tags (§2). - **Releases:** the kind-aware `depsOf` changes nothing for existing content, because no field value can be a group id before step 7. +- **Phase 2 changes behaviour for every site**, before the types exist, and its changeset says so: + - relinking or unlinking a node that other nodes use as their group is refused (`translation_group_in_use`); + - `readsOriginal`/`originalOf` require an original to have its own id as its group (OQ 14); + - group-only translation changes get the group check; + - imports store canonical bodies; + - deletes and moves of entries carry the site anchor. - **Rollback:** - A snapshot restores the site node and entries, and the index is rebuilt in its batch. - An operation reverts through its `before` snapshot, dropping deleted targets. @@ -903,7 +925,7 @@ New rows of the closed table (`conversions.ts:38`): §8 is the threat model and the policy. - No new scopes and no new approval points. - Every statement filters on `site_id`, every guard re-checks it, and nothing resolves across sites. -- New refusals, each with a hint in `field-hints.ts`: `entry_referenced`, `reference_target_moved`, `unknown_entry`, `too_many_references`, `field_key_in_use`, `fields_unsupported`, `field_operation_unsupported`, `translation_shape_mismatch`. +- New refusals, each with a hint in `field-hints.ts`: `entry_referenced`, `reference_target_moved`, `reference_target_unknown`, `unknown_entry`, `too_many_references`, `field_key_in_use`, `fields_unsupported`, `field_operation_unsupported`, `field_operation_running`, `translation_group_in_use`, `translation_shape_mismatch`, `list_field_type`. ## Open questions (each with a recommended answer) @@ -996,3 +1018,21 @@ Three verifiers found 34 earlier findings resolved. The unresolved ones and the | Size ratchet | New modules named: `core-ts/src/references/`, `core-ts/src/release-deps.ts`, `astro-renderer/src/references.ts` and `reference-conversions.ts` (§9). | | The live-versus-snapshot comparison during operations | Runs with no marker active (test plan). | +### Fourth round (2026-11-10, verification of 21abf4e1) + +| Finding | Decision | +|---|---| +| R12 relied on a site anchor that was added only when blocking rows were seen (blocking) | Every delete, move and group change of a `contentEntry` carries the site anchor, so a delete planned before the conversion marker fails once the marker commits, then is refused (§2.2, R12 and its test). | +| Index-mode purges assumed `to` equals the working copy, but any snapshot can be published (blocking, two verifiers) | Chosen: the hybrid. One method for every path: index rows for entries whose working copy equals `to`, plus a `from`/`to` body scan of the entries whose hash differs, found by one `json_each` comparison. `PublishChange` is unchanged. There are tests for an approved request publishing an older snapshot after the referrer was edited, and for a write between `createSnapshot` and `publish` (§2, Purges point 4). | +| `shared` was bound from plan-time definitions on writes the definitions anchor does not cover (blocking) | Chosen: derive it inside the single `INSERT … SELECT` from the site row at batch time. Phase A's `UPDATE` corrects rows written before it (§2.1, R16). | +| The group check ran only when `source` changed | It runs, with a guard keyed on the written group, whenever `translation.group` or `translation.source` is set or changed (§2.2, R15). | +| Legacy non-canonical rows | The step-7 job rewrites the site's rows canonically (§2.4). | +| The `field-refs.ts:81` skip filter path format | It switches to item paths, with the `planEntryFields` skip set (§9 step 2). | +| "Used by" should use resolved values; `referencedBy` cards | Resolved values with the same-site original; `referencedBy` results are cards, and drawing their fields is outside the guarantee (§2, Purges point 3). | +| Wording: "usable member", "drafts never lock", "the one accepted exception" | The guard protects a member's existence; a draft locks only through a published translation that inherits its value; R8 and R14 are both accepted (Summary, §2). | +| A move could let an archived X satisfy a pair | An archived X satisfies nothing, as in deletes (§2.2). | +| Size ratchet for baselined files | `cache-purge-assets.ts`, `references/release-purge.ts`; no net lines in `assets.ts`/`store.ts` (§9). | +| Phase-2 behaviour changes missing from Compatibility | Listed, for the changeset (Compatibility). | +| The refusal list was incomplete | Completed (Security & policy impact). | +| Probe-count wording; snapshot-scan cost | Three probes plus one hash comparison and batched member lookups; a far rollback usually falls back to purging everything; a per-snapshot reverse map is a later optimization (Purges point 6). | + From d8a436cc4c0a9444d3d7dc5c1e659a63daae5c0c Mon Sep 17 00:00:00 2001 From: Madraka <8898640+Madraka@users.noreply.github.com> Date: Tue, 10 Nov 2026 16:18:50 +0300 Subject: [PATCH 6/8] docs(rfc): RFC-0045 fifth revision, simplify after the fca37cbe check - the rows INSERT is a plain INSERT; shared is matched by key or migration.fromKey, and a NULL fails the batch (ConflictError, retry) instead of being skipped by OR IGNORE (R16, rename case) - purges read only the immutable from/to bodies, through snapshot bundles, on every publish path; the hash-check hybrid and index mode are dropped (a probe cannot see referrers edited away or deleted, and cheap whole-site change checks are unsound) - any field-operation marker on from or to purges everything - deletes, moves and group changes anchor on json($.collections), not the site hash, so publishes no longer cause 409s - non-blocking: the written-group guard is exempt for a node's own id; CAS canonicalization runs from phase 2; prefix skip match; the rebuild reads through fromKey; wording fixes - new "Implementation watch list" section; review log "Fifth round" Co-Authored-By: Claude Opus 5.5 Signed-off-by: Madraka <8898640+Madraka@users.noreply.github.com> --- ...ctions-stage-4-references-and-repeaters.md | 145 ++++++++++++------ 1 file changed, 100 insertions(+), 45 deletions(-) diff --git a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md index c9bade82..ea4bfaf8 100644 --- a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md +++ b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md @@ -1,6 +1,6 @@ # RFC-0045: Collections stage 4: reference fields and repeaters -- Status: Draft (revised twice on 2026-11-10, after an independent review of PR #193 and a verification pass; the review log is at the end) +- Status: Draft (revised five times on 2026-11-10, after an independent review of PR #193 and four verification passes; the review log and the implementation watch list are at the end) - Author: Halil (Madraka), drafted with Claude · Date: 2026-11-10 - Related: - [#114](https://github.com/cloudhal-dev/cloudhal/issues/114) (this stage); #71 (the collections plan). @@ -253,7 +253,7 @@ CREATE INDEX entry_refs_target ON entry_refs (site_id, target_id, source_status) **Invariant (the index).** `entry_refs` holds exactly one row per `(node, field path, stored group id)`, whatever the target's state: present, missing, archived or foreign. Rows depend on the holder's body and on its collection's definitions (`localized` decides `shared`). Nothing changes them except a write of the holder, its deletion, a rebuild, or `set_field_localized`'s phase A (§7). - **No cross-tenant pinning.** Every guard and lookup filters `r.site_id` (the holder's site) and `n.site_id` (the member's site). A row naming another tenant's id pins nothing there and never resolves. -- **Repeats.** A target named in two repeater items (a host who also gives a talk) is one row: the writer deduplicates `(field, target)` pairs, and the `INSERT` is `INSERT OR IGNORE`. +- **Repeats.** A target named in two repeater items (a host who also gives a talk) is one row: the writer deduplicates `(field, target)` pairs, and the writer's `INSERT` is a **plain** `INSERT`. The anchored `DELETE` runs first in the same batch, so no key conflict remains to ignore, and any constraint failure fails the batch. - **`shared`** is a pure function of the definition: 1 for `localized: false`, 2 for `localized: true`. It no longer encodes the holder's role, which the first revision computed at write time and which went stale when a node changed group. - *Who* inherits is decided at batch time by the fragment below, exactly as `readsOriginal` decides it (`fields/resolve.ts:37-41`): members `m` of this site whose `translation.group` is the holder's id, who are not the holder, and whose own `translation.source` is not in another site. - A translation's own value, and any value held in an edition site (where no node has the group's id, `originalOf`, `store/entry-fields.ts:54-62`), therefore inherits to nobody, whatever its `shared` (open question 12). @@ -300,25 +300,28 @@ The member probe uses `nodes_translation_locale` (migration 0025: `site_id, kind Equality needs stored bodies in canonical form. Store writes are canonical (`canonicalJson`, `node-sql.ts:62/91`), but `importSite` inserts a bundle's body text verbatim (`sync-plan.ts:300-301`), and SQLite's `json()` keeps key order. So from phase 2, `importSite` stores `canonicalJson(JSON.parse(body))` for every row. The hash is computed over canonical JSON already, so it does not change. A hand-made archive with unsorted keys therefore cannot leave a collection whose writes always conflict. The anchor conflicts only when that collection's definitions change, never on other site edits. The loser retries under the new definitions, where `field_key_in_use` then sees its value. This is RFC-0042's rule that list-value rows are valid only "under the fingerprint they were built with" (`list-values/fingerprint.ts`), applied to writes. -- **The rows**, anchored on the entry's new hash: `DELETE FROM entry_refs WHERE source_id = ? AND `, then one `INSERT OR IGNORE … SELECT … FROM json_each(?) WHERE `, appended to `own` (`entries.ts:110`). Every write of the holder rewrites all its rows, so `source_status` stays current. +- **The rows**, anchored on the entry's new hash: `DELETE FROM entry_refs WHERE source_id = ? AND `, then one plain `INSERT … SELECT … FROM json_each(?) WHERE `, appended to `own` (`entries.ts:110`). Every write of the holder rewrites all its rows, so `source_status` stays current. - **`shared` is never bound from the plan.** The same single `INSERT … SELECT` reads it from the site row *as it is when the batch runs*, so a status-only or title-only write that races `set_field_localized`'s phase A cannot write a stale value: ```sql - INSERT OR IGNORE INTO entry_refs (site_id, target_id, source_id, source_collection, field, root_key, shared, source_status) + INSERT INTO entry_refs (site_id, target_id, source_id, source_collection, field, root_key, shared, source_status) SELECT ?, json_extract(p.value, '$.t'), ?, ?, json_extract(p.value, '$.f'), json_extract(p.value, '$.k'), (SELECT CASE json_extract(d.value, '$.localized') WHEN 1 THEN 2 ELSE 1 END FROM nodes s, json_each(json_extract(s.body, ?)) d -- ? = '$.collections."".fields' - WHERE s.id = ? AND s.kind = 'site' AND json_extract(d.value, '$.key') = json_extract(p.value, '$.k')), + WHERE s.id = ? AND s.kind = 'site' + AND (json_extract(d.value, '$.key') = json_extract(p.value, '$.k') + OR json_extract(d.value, '$.migration.fromKey') = json_extract(p.value, '$.k'))), ? FROM json_each(?) p WHERE ``` - Batches run one at a time, so the row reflects the site either before phase A, which phase A's `UPDATE` then corrects, or after it. - - A key without a definition at batch time (the definitions anchor makes this impossible for changed keys) yields `NULL`, which the `NOT NULL` column refuses, so the batch fails and retries. + - **During a rename.** `rename_field`'s phase A renames the definition and keeps the old key in `migration.fromKey` (`fields-operations/plan.ts:131-139`). A holder write planned before phase A, which changes no fields and so carries no definitions anchor, still finds its definition through `fromKey`. `rename_field` does not change `localized`, so the value is right. + - **A key with no definition at all** at batch time yields `NULL`. With a plain `INSERT`, the `NOT NULL` column then fails the whole batch, which the store maps to a `ConflictError`, and the caller retries under the current definitions. (The fourth revision's `INSERT OR IGNORE` would have skipped the row silently: SQLite's conflict clause covers `NOT NULL` too. A verifier confirmed this in sqlite3.) A target deleted or moved between check and batch fails the guard, and `diagnose()` names it (`unknown_entry`, `reference_target_moved`). One bound parameter carries the whole list (D1 allows 100 parameters), and 100 group ids stay far below 100 KB per statement. **2. Who can lock what: delete, move, leaving a group, publishing.** - **The policy.** A reference blocks only while a *published* entry shows it, as `resolveFields` would show it: its own value, or one inherited from its same-site original (the fragment above). - - Drafts, entries in review and archived entries never lock anything. + - Drafts, entries in review and archived entries never lock anything *on their own*. Their values lock only through a published entry that shows them (a published translation inheriting a draft original's value). - A contributor (`content:draft:own`, `roles.ts`) cannot publish, so nothing a contributor writes locks on its own. A value it holds locks only once someone with publish rights publishes an entry that shows it, such as a translation inheriting it. That person's publish is the decision. - An author pins only through what published entries show, which the live site really depends on. - An editor can always unpublish to release a lock. @@ -331,9 +334,12 @@ A target deleted or moved between check and batch fails the guard, and `diagnose AND r.source_collection || '.' || r.field NOT IN (SELECT value FROM json_each(:P))) ) ``` The second `NOT EXISTS` makes a blocking row from a field the plan did not see, committed in between, fail the batch, so the write retries and re-plans with that field. The first revision's form let it through: `events.venue` allows `[places, halls]` and `talks.room` allows `[places]`; a talk published meanwhile with `room` = G is stranded once the last member in `places` goes. - - **The site anchor**, `EXISTS (SELECT 1 FROM nodes WHERE id = ? AND kind = 'site' AND hash = ?)`, with the hash of the `Site` object the write planned with. The allowed collections come from *referrer* collections' definitions, so anchoring only the target's own collection (the first draft) left a race (R4). + - **The collections anchor**, `json((SELECT json_extract(body, '$.collections') FROM nodes WHERE id = ? AND kind = 'site')) IS json(?)`, bound with the canonical JSON of `site.collections` of the `Site` object the write planned with. + - It holds every collection's definitions and every field-operation marker, so it covers the referrer fields whose allowed collections the guard used (R4) and the markers `field_operation_running` reads (R12). + - Unlike the site hash used in the fourth revision, it ignores `publishedSnapshotId`, theme, menus and other settings, so a publish no longer makes concurrent deletes and moves answer 409. + - Canonical bodies (§2.1, §2.4) keep the comparison live. - These writes are rare, so **every** delete, move and group change of a `contentEntry` carries the site anchor, whether or not the plan saw blocking rows. The first revisions added it only when rows were seen. That left R12 open: during a text → reference conversion, the values that will become references have no rows yet, so a delete planned before the marker committed carried no anchor and went through. With the anchor always present, the batch fails once the site changes, the retry plans against the marker, and `field_operation_running` refuses it. + These writes are rare, so **every** delete, move and group change of a `contentEntry` carries the collections anchor, whether or not the plan saw blocking rows. The first revisions added it only when rows were seen. That left R12 open: during a text → reference conversion, the values that will become references have no rows yet, so a delete planned before the marker committed carried no anchor and went through. With the anchor always present, the batch fails once the marker lands in `collections`, the retry plans against the marker, and `field_operation_running` refuses it. - A refusal answers `409 entry_referenced`, with up to 20 published referrers (id, title, collection, field) and the total. For a group change, it also names the group. - Deleting a translation, or an original that still has another usable member, never blocks. The guard protects the group's **existence**, not each language's card. - The deleted member's locale then resolves to the fallback, or to nothing (§3). @@ -341,8 +347,9 @@ A target deleted or moved between check and batch fails the guard, and `diagnose - **Moving a member** X from C to D: for each seen pair whose field does not allow D, the batch requires another usable member in a collection that field allows. Pairs whose field allows D are satisfied by X itself at D, unless X is archived. An archived X satisfies nothing, exactly as in the delete rule. The batch also carries the same `NOT EXISTS ( AND pair NOT IN :P)`, so an unseen field forces a retry. Refused: `409 reference_target_moved`. During a field operation, the pairs include the marker's `fromKey`. - **Relinking a node with dependants** is refused (`409 translation_group_in_use`, §1). The write carries `NOT EXISTS (SELECT 1 FROM nodes WHERE site_id = ? AND kind = 'contentEntry' AND json_extract(body, '$.translation.group') = :X AND id <> :X)`. - **Every write that sets or changes `translation.group` or `translation.source`** carries a guard keyed on the *written* group X. This covers a new translation, a source change, and a group-only change, which today passes unchecked when the source stays the same (`store/translations.ts:107-113`). The guard is `NOT EXISTS (SELECT 1 FROM nodes WHERE id = :X AND site_id = ? AND json_extract(body, '$.translation.group') IS NOT NULL AND json_extract(body, '$.translation.group') <> :X)`: X's node, if it is in this site, still has its own id as its group. + - It is **omitted when X is the written node's own id**: a translation becoming an original (`checkLocalizedNode` requires an original's group to be its id, `translations.ts:59-61`). There the guard would read the pre-update row and fail on every retry. - `checkTranslationLinks` runs its source-group check on the same condition, not only when `source` changes (`translations.ts:111-135`), and adds this guard to the write. Today it checks the source's group with a read only (`:131`). -- **While a field operation converts text into references** (`text → reference`, or a repeater gaining a reference sub-field from text), readers convert unconverted values on the fly (§7), so a published entry can show a group that has no row yet. Until phase C drops the marker, deletes, moves and group changes of entries in that field's target collections are refused (`409 field_operation_running`). The site anchor carries the marker, so an operation started concurrently fails the batch. +- **While a field operation converts text into references** (`text → reference`, or a repeater gaining a reference sub-field from text), readers convert unconverted values on the fly (§7), so a published entry can show a group that has no row yet. Until phase C drops the marker, deletes, moves and group changes of entries in that field's target collections are refused (`409 field_operation_running`). The collections anchor carries the marker, so an operation started concurrently fails the batch. - **Publishing a referrer.** A write that makes an entry published runs the target guard over *every* own reference value. A dangling id is refused (`unknown_entry`: remove or replace it), so a draft that lost its target cannot go live pointing at nothing. - Inherited values are not the translation's to fix. A translation that goes live while its original holds a dangling id gets a warning, and the reference renders as nothing (R8). - **Archiving and unpublishing** a target are allowed, and write nothing to `entry_refs`. The picker hides archived entries. @@ -355,28 +362,30 @@ A target deleted or moved between check and batch fails the guard, and `diagnose | R1 | referrer write ↔ deleting the target's last usable member | the referrer's target guard (members at batch time, by the membership rule), or the delete's condition: blocking rows at batch time, with an unseen field forcing a retry | | R2 | referrer write ↔ moving the target to a disallowed collection | the target guard (collection at batch time), or the move's per-pair condition plus the unseen-field check | | R3 | referrer write ↔ adding a reference field over its key, or removing an allowed collection | the definitions anchor, bound as planned (point 1). `field_key_in_use` counts committed values. The collection refusal counts values whose group has a member in the removed collection. | -| R4 | moving a member ↔ narrowing a referrer field's `collections` | the move's site anchor. The narrowing's count sees the member once it has moved. | -| R5 | delete ↔ any change to the definitions a blocking row's field depends on | the delete's site anchor | -| R6 | changing a member's group ↔ referrer write | the group change carries the delete guard and site anchor for its old group. The referrer's target guard checks membership at batch time, and a relinked node no longer matches its old group by id. A node with dependants cannot relink at all. | +| R4 | moving a member ↔ narrowing a referrer field's `collections` | the move's collections anchor. The narrowing's count sees the member once it has moved. | +| R5 | delete ↔ any change to the definitions a blocking row's field depends on | the delete's collections anchor | +| R6 | changing a member's group ↔ referrer write | the group change carries the delete guard and collections anchor for its old group. The referrer's target guard checks membership at batch time, and a relinked node no longer matches its old group by id. A node with dependants cannot relink at all. | | R7 | a referrer becoming published ↔ deleting its target | the publish re-checks all own values with the target guard. The delete sees the referrer's new `source_status`, because its rows are rewritten in the same batch. | | R8 | publishing a translation that *inherits* the value ↔ deleting the target | when the publish is first, the delete's fragment reads member status at batch time. When the delete is first, nothing published showed the value at that moment. **Accepted:** the translation goes live with a warning and renders nothing; no guard can refuse a write the translator cannot fix. | | R9 | field-operation phase B ↔ deleting a target | the target guard ANDed into each rewritten entry's CAS (point 6) | | R10 | import, restore or sync ↔ anything | one atomic batch with the index rebuilt in it (point 4) | | R11 | a member reappearing (a new translation or edition member) | rows are kept whatever the membership (the invariant), so nothing needs re-creating | -| R12 | a reference shown only through an in-flight text → reference conversion ↔ deleting its target, including a delete planned *before* phase A commits | `field_operation_running` refuses the delete until the marker is gone. The site anchor, now on every delete, move and group change, fails a batch planned before the marker. | +| R12 | a reference shown only through an in-flight text → reference conversion ↔ deleting its target, including a delete planned *before* phase A commits | `field_operation_running` refuses the delete until the marker is gone. The collections anchor, on every delete, move and group change, fails a batch planned before the marker. Purges of a snapshot carrying any marker purge everything (§2, Purges). | | R13 | a blocking row from a field the plan did not see ↔ delete, move or group change | the unseen-field `NOT EXISTS` with the planned pairs `P` | | R14 | a held node published at its live version (holds, `snapshots.ts:29-37`) whose working copy is a draft ↔ deleting its target | **Accepted**, like R8: holds are temporary, and the release's approval text and `referenceIssues` warn. Probing live versions of held nodes from the delete guard would need snapshot reads inside a write. | | R15 | creating a translation of X, or setting any node's group to X ↔ relinking X | the written-group guard on the joining write, or the dependants guard on the relink | -| R16 | a status-only or title-only write of a holder ↔ `set_field_localized` phase A | `shared` is read from the site row inside the `INSERT … SELECT` at batch time, and phase A's `UPDATE` corrects rows written before it (§2.1) | +| R16 | a status-only or title-only write of a holder ↔ `set_field_localized` or `rename_field` phase A | `shared` is read from the site row inside the plain `INSERT … SELECT` at batch time, matched by `key` or `migration.fromKey`. Phase A's `UPDATE` corrects rows written before it. A missing definition fails the batch (`ConflictError`, retry) instead of dropping rows (§2.1). | **4. Rebuilds** run in the same batch as the node set: `importSite`/`pull`/`push` (`sync-plan.ts:308`), `cloudhal import --replace` and `--revert`, and snapshot restores. - `entryRefsRebuild(siteId, rows)` builds the rows from the bodies, with the site's definitions from the site row, as `refsRebuild` does (`media/refs-rebuild.ts:12-37`), and `DELETE`s and `INSERT`s them under `site_id = ?`. + - It computes `shared` in JS from the same site row, so no `NULL` can arise. + - It reads body keys through any `migration.fromKey` the site row carries, as `resolveFields` does. A bundle exported mid-operation then keeps rows for the entries that were not yet rewritten. - There is no membership join: the invariant keeps rows for every stored value, and the site filters keep them harmless. - The `INSERT` is chunked so each statement stays below 90 KB, inside the one atomic batch. - **Statement budget.** The chunked `INSERT`s count against the import batch's 1,000-statement cap (`sync-plan.ts:311`), like every other statement there. - **Step 7** also runs a one-time `entryRefsRebuild` per site as an admin job, the pattern of `lists.backfill`. Each site's rebuild is **one atomic batch**: the `DELETE` and its chunked `INSERT`s together, so there is never a window without rows. - - A site too large for one batch gets `INSERT OR IGNORE` chunks only, with no `DELETE`, resumable by cursor. That is safe, because it can only add missing rows. - - The same job rewrites the site's node rows in canonical form (`canonicalJson(JSON.parse(body))`, hash unchanged). That covers rows imported verbatim before phase 2, and imports that planned `unchanged` (`sync-plan.ts:257, 295`), so the definitions anchor is live on every collection. + - A site too large for one batch gets `INSERT OR IGNORE` chunks only, with no `DELETE`, resumable by cursor. That is safe, because it can only add missing rows, and its `shared` is computed in JS, so it is never `NULL`. + - **Canonical rows, from phase 2, not step 7.** The anchors ship in phase 2, so phase 2 also ships an admin job that rewrites existing node rows canonically. It is per row and guarded by CAS (`UPDATE nodes SET body = ? WHERE id = ? AND hash = ?`, hash unchanged, rule 4), so it cannot race a write. That covers rows imported verbatim before phase 2, and imports that planned `unchanged` (`sync-plan.ts:257, 295`), so the anchors are live on every collection. Step 7 runs it once more. - It is a safety net: the gate (§1) means it should find nothing. **5. Assets inside repeaters.** Each of these needs its own change, because none follows from the others: @@ -437,29 +446,25 @@ R8 and R14 are the accepted exceptions, both warned (open question 13). Dangling - hop 1: the referrers of the seed groups; - hop 2: the referrers of the groups of the hop-1 referrers. - Each page reached adds the node tags of its referrer and of every member of that referrer's group, so all of its language versions go. For purges, the walk uses **all rows, whatever `source_status`**: over-purging is cheap, and this keeps a status change in the working copy from hiding a cached page. + Each page reached adds the node tags of its referrer and of every member of that referrer's group, so all of its language versions go. The walk takes referrers from **both** snapshots' bodies, whatever their status: over-purging is cheap, and a page cached from `from` is reached even when `to` no longer shows the reference. 3. **"Used by" pages.** For every changed, added or removed entry, the planner also adds the node tags of the members of every group its before *or* after **resolved** values name. Resolution uses `resolveFields` with the same-site original read from the same snapshot, so a translation that only inherits its original's `venue` counts too. - A venue page that renders `referencedBy` ("events here") is purged when an event naming it appears, disappears or changes, with no theme code, because the target's page carries its own node tag. - `referencedBy` results are drawn as cards: title, URL, summary and cover. Drawing *their* fields and references on the target page is outside the purge guarantee, and `docs/guides/fields.md` says so. -4. **One mode for every path: the index, corrected by a hash check.** Which path published (`store.publish` of any snapshot, `POST …/snapshots/:id/publish`, an approved role request, a rollback, a release run, a restore) does not matter, and `PublishChange` stays `{from, to}`. Two steps: - 1. **Find the entries whose working copy differs from `to`.** One statement compares `to.nodes` with the working copy, `SELECT … FROM json_each(:toEntryRefs) t LEFT JOIN nodes n ON n.id = json_extract(t.value, '$.id') WHERE n.hash IS NOT json_extract(t.value, '$.hash')`, and lists the working entries absent from `to`. Held nodes are naturally among the differing ones. - 2. **Take the referrers from two sources.** - - For entries whose working copy equals `to`, the index answers exactly: their rows are their `to` values. A referrer that differs between `from` and `to` is in the diff, and the entry rule purges it anyway. - - For the differing entries, the planner reads their `from` and `to` bodies and extracts their reference values with `fieldsReferences(defs, body)`, as the asset rule scans bodies (`cache-purge.ts:220-229`). - - Working entries absent from `to` can only add index rows, so they over-purge, which is harmless. - - The approved-request case, an older snapshot published after its referrer was edited, and a write landing between `createSnapshot` and `publish` are thereby exact. - - **Release runs** re-plan the referrer part at run time, after `publishRelease` commits, with this same method. They do not replay the plan pinned earlier (`releases.ts:154`), which would miss a referrer published between pin and run. -5. **Every path that changes visibility:** publish (any snapshot), rollback, release run, and an import or restore that moves the published pointer all purge by point 4. Archive, delete and unpublish change only the working copy and go live through one of these. +4. **One mode, exact by construction: the snapshots' own bodies.** The planner never consults the working copy for referrers. + - Referrers, both hops, inheriting members (`COALESCE(translation.group, id)`) and "used by" are all computed from the immutable `from` and `to` entry bodies: each body's resolved reference values (`fieldsReferences(defs, body)`, through that snapshot's own definitions, with its same-site original for inheritance). + - It reads both snapshots' **bundles** (RFC-0010: 256 bodies per shard, `bundles.ts`). Every snapshot made to be published gets one (`snapshots.ts:102-110`), and the live one has one, so a 10,000-entry site costs about 80 shard reads. A missing bundle falls back to node-by-node reads. + - This covers every publish path alike: the default path, an older snapshot, an approved request (`approvals.ts:343-356`), a rollback, a release run (re-planned at run time, not replayed from `releases.ts:154`), and a restore or import that moves the pointer. `PublishChange` stays `{from, to}`. + - **Any field-operation marker on `from`'s or `to`'s site node purges everything.** That covers text → reference conversions in flight and renames. Operations are rare, so this is cheap, and it is obviously safe. + - `entry_refs` stays the source for guards, Studio's "Used by", `list_references` and the live `referencedBy`, never for purges. + - **Why not the index on the default path?** The PM's decision for this round proposed index mode there, with a freshness check inside each hop's `SELECT`. It is not exact. A probe returns only rows that *still* name the target, so a referrer in `to` that was edited to stop naming it, or deleted, after the snapshot is never returned, and its hash cannot be compared. Detecting that needs a whole-site change check. The cheap ones are not sound: `count(*)` with `sum(rev)` and `max(rowid)` misses a delete of the newest row followed by an insert that reuses its rowid (`nodes` has no `AUTOINCREMENT`, migration 0001). Reading the bodies the snapshot already holds is both simpler and exact. +5. **Every path that changes visibility** purges by point 4: publish (any snapshot), rollback, release run, and an import or restore that moves the published pointer. Archive, delete and unpublish change only the working copy and go live through one of these. - Time-based visibility (an embargo opening, `availability.until`, a derived `hide`/`unlist` expiry) needs no purge: the gate-index TTL expires the page at that instant (`availability.ts:94-95`). 6. **Bounds and fallback.** - - At most three batched index probes per plan, one `json_each` each: hop 1, hop 2 and "used by". There is one hash-comparison statement, and the member lookups for referrer groups run as one batched probe per hop. - - Every probe and body read counts against `maxReads` (2,000). Tags stop being collected at `maxTags + 1` (300 tags). - - Exceeding either purges everything, as any large diff does today (`cache-purge.ts`, "Falls back to purging everything"). - - A rollback far back on a large site differs in many entries and will usually take that fallback. That is safe; a per-snapshot reverse map built once is a later optimization (follow-ups). + - Each bundle shard read, and each node read where no bundle exists, counts against `maxReads` (2,000). Tags stop being collected at `maxTags + 1` (300 tags). Exceeding either purges everything, as any large diff does today (`cache-purge.ts`, "Falls back to purging everything"). + - With bundles, a site of up to about 250,000 entries stays within `maxReads`. Without them, about 1,000 entries per snapshot do. + - The work is in memory, linear in the entries, as the snapshot source already does when it loads (`source.ts` `#load`). `SnapshotReader` gains `readSnapshotBundle` (`store.ts:949`). 7. **Compatibility.** `isCloudhalCacheTag` (`cache-tags.ts:48`) and the site Worker's purge endpoint (`cloudflare-entry.ts:312-325`) are unchanged, so an older site Worker accepts every purge, and no id of an unresolved target reaches any header. - - `SnapshotReader.referrersOf(siteId, groups)` is new. The store implements it on the index, and the release executor's run-time reader implements it on snapshot bodies. + - `SnapshotReader.readSnapshotBundle` is the one reader addition. The store has it already, and the release executor's run-time reader gets it too. - The first revision's `cloudhal-ref-` tags are dropped. Existing themes build their own tag lists (`starters/blog/src/pages/[...path].astro:261-321`), new prefixes would make older site Workers answer 400, and ids in headers relied on unverified stripping. **Archives (RFC-0039).** An archive holds every node of the site, so references stay complete, and import rebuilds the index in its node batch (point 4). Dangling or foreign ids are reported, not refused (RFC-0041 open question 8): `cloudhal import --dry-run` adds `referenceIssues`. Until step 7, an archive *with* reference or repeater definitions is refused (`fields_unsupported`, §1). @@ -574,7 +579,7 @@ All new UI follows `kumo-design`, `studio-ui` and DESIGN.md "Fields", imports on - Repeaters render inside the form with `RepeaterField`. Image sub-fields reuse `FieldImage`. - On a translation, a non-localized reference or repeater is read-only, with "From the original" and a link. A localized one shows the original's value as a placeholder. - **"Used by":** a `SectionCard` with every referrer (`GET …/entries/:id/references`), published ones first. - - **The delete dialog:** it disables Delete only while a *published* referrer exists and this is the group's last usable member ("Unpublish or edit these 2 published entries first"). + - **The delete dialog:** it disables Delete only while a *published* referrer exists and this is the group's last member that is not archived, in an allowed collection ("Unpublish or edit these 2 published entries first"). Archiving that member first does not release the lock; the dialog and `docs/guides/fields.md` say so. - It lists draft referrers as "Will show nothing for this field" and lets the delete go ahead. - When other members remain, it names the locales that will lose the card ("Turkish event pages will show no venue"). - The same rules apply to unlinking a translation, which leaves its group. @@ -745,12 +750,12 @@ New rows of the closed table (`conversions.ts:38`): | **An imported or pushed archive pins another tenant's entries or assets** | Rows are site-scoped, and every guard filters both the row's `site_id` and the member's (§2). `asset_refs` gets the same filter as a phase-2 prerequisite (§2.5). | An archive referencing site Y's entry or asset, imported into site X, must not block Y's delete or move, and must not list X's titles to Y. | | **Lock-out by low-privilege actors** (`content:draft:own`, `content:write:own`) | Only what a published entry shows blocks, by the `shared` rule (§2): a translation's own value is never inherited. Contributors cannot publish; drafts, review and archived entries never lock. | Policy-suite cases that must *not* block: a contributor's draft; an author's review entry; a draft translation's `localized: true` value while its original is published; a draft edition member whose edition translations are published (edition translations never inherit). Cases that must block: an author's published entry, an edition member's included; a published translation inheriting its original's `localized: false` value. | | **Fan-out DoS on render** | Caps of 50 per field, 50 items and 10 sub-fields (contract); 100 reference values per entry (code); depth ≤ 2 with a visited set; 200 targets per page; ≤ 2 live reads per page, 0 for snapshots; `referencedBy` ≤ 100. | A property test: random graphs with cycles and 10× the caps resolve within budget, never throw, and cost at most `depth` live reads. | -| **Write amplification and slow guards** | Each guard is one indexed probe on `(site_id, target_id)` or one `json_each` count; members are found through `nodes_translation_locale`. Delete, move and group change cost one probe each, plus the site anchor when rows exist. The purge planner adds one probe per touched group. Rebuilds are chunked inside one batch. | RFC-0042's benchmark (#105), extended to 10,000 events × 3 references on Miniflare D1, reports the write, delete and move guard costs. | +| **Write amplification and slow guards** | Each guard is one indexed probe on `(site_id, target_id)` or one `json_each` count; members are found through `nodes_translation_locale`. Delete, move and group change cost one probe each, plus the collections anchor, always. The purge planner reads the two snapshots' bundles. Rebuilds are chunked inside one batch. | RFC-0042's benchmark (#105), extended to 10,000 events × 3 references on Miniflare D1, reports the write, delete and move guard costs. | | **Picker search abuse** | `q` ≤ 100 characters, a substring match with no regex (rule 13), ≤ 20 results, and a per-member rate window. | A 100k-character `q` is refused before it is read. | | **Prompt injection through target titles** | Titles are data, quoted and truncated in approval prompts (`quote()`, 60 characters); `list_references` marks them as content. | Approval-prompt tests gain reference cases. | | **An older site Worker crashes on the new types** (`COMPILE[def.type]`, `props.ts:46`, has no entry) | `GET /__cloudhal/features` adds `{ "fields": 2 }`. Every write path refuses the new types with `409 fields_unsupported` until it is reported: interactive writes and `checkBundle` for import, sync, push, pull and restore (§1), as `gates: 2` guards `expiry` (`availability.ts:243`, `cloudflare/src/features.ts`). `resolveFields` skips unknown types from phase 1. | The site Worker features test; an import, a push and a restore of a bundle with a reference definition are refused before step 7; a renderer test with an unknown field type. | | **Assets in repeaters deleted while used** | `fieldAssetRefs` walks items, `fieldAssetIssues` resolves sub-field types, and the purge planner's asset rule reads field and item assets (§2.5). | An asset used only in a repeater item cannot be deleted; the rebuild keeps its row; a retired, foreign or non-image asset in an image sub-field is refused at its item path; a changed item image purges its page. | -| **A group change strands references** (`translation.group` is writable) | Membership is the current group, `COALESCE(translation.group, id)`, everywhere. Leaving a group is guarded like a delete, with the site anchor. A node with dependants cannot relink (§1, §2.2, R6, R15). | Relinking a referenced untranslated original as a translation, and dropping `translation`, are both refused while a published entry shows the reference; a draft referrer does not block. | +| **A group change strands references** (`translation.group` is writable) | Membership is the current group, `COALESCE(translation.group, id)`, everywhere. Leaving a group is guarded like a delete, with the collections anchor. A node with dependants cannot relink (§1, §2.2, R6, R15). | Relinking a referenced untranslated original as a translation, and dropping `translation`, are both refused while a published entry shows the reference; a draft referrer does not block. | **Policy.** No new scopes. - Field definitions stay site settings: `site:settings`; for agents, `site:write` plus a person's approval, person-classed as today (`field-tools.ts:264-267`). @@ -780,10 +785,10 @@ New rows of the closed table (`conversions.ts:38`): 2. **Store and integrity:** - migration 0033, with `entry_refs` in `SITE_TABLES`; - the target guard, the definitions anchor bound as planned, and the rows with `shared`/`root_key`; - - the delete, move, group-change and publish rules, with the site anchor and the policy (§2.2), and the races R1 to R11 as cases; + - the delete, move, group-change and publish rules, with the collections anchor and the policy (§2.2), and the races R1 to R16 as cases; - chunked `entryRefsRebuild` in import, sync and restore, and `referenceIssues` in `import --dry-run`; - - the item walk in `fieldAssetRefs`, and `fieldAssetIssues` with sub-field types and item paths. The `skip` set built in `planEntryFields` (`entry-fields.ts:107`, `slice(0, 2)`) and the skip filter in `fieldAssetIssues` (`field-refs.ts:81`, which compares `r.field.replace('.', '/')`) both switch to the item path `fieldAssetRefs` returns; - - the membership rule everywhere, the relink refusal (`translation_group_in_use`) with the source guard on new translations, and the RFC-0041 `readsOriginal`/`originalOf` tightening; + - the item walk in `fieldAssetRefs`, and `fieldAssetIssues` with sub-field types and item paths. The `skip` set built in `planEntryFields` (`entry-fields.ts:107`, `slice(0, 2)`) and the skip filter in `fieldAssetIssues` (`field-refs.ts:81`, which compares `r.field.replace('.', '/')`) both switch to item paths, with a **prefix** match: a ref is skipped when any skip path (`fields/speakers`, `fields/speakers/2`) is a prefix of its item path. That keeps today's top-level behaviour too; + - the membership rule everywhere, the relink refusal (`translation_group_in_use`) with the written-group guard on every group or source change (exempt when the group is the node's own id), and the RFC-0041 `readsOriginal`/`originalOf` tightening; - `field_operation_running` for references shown through an in-flight conversion; - canonical bodies in `importSite`, and `CopySiteOptions.fieldTypes`; - rows and the target guard in field-operation rewrites, with operations naming the new types refused; @@ -799,7 +804,7 @@ New rows of the closed table (`conversions.ts:38`): - `groupMembers` and `referencedBy` on both sources, with the `shared` rule; - `EntryFields.astro` (links for references, a nested `
` per item); - the conversion maps in the live and snapshot sources; - - the purge planner's referrer closure (two hops, "used by" by resolved values, the index corrected by the hash check) and `SnapshotReader.referrersOf` (no new tags), in `core-ts/src/references/purge.ts`; + - the purge planner's referrer closure (two hops, "used by" by resolved values, read from the `from`/`to` bundles, purge-everything under any marker) and `SnapshotReader.readSnapshotBundle` (no new tags), in `core-ts/src/references/purge.ts`; - the release executor's run-time referrer plan, in `core-ts/src/references/release-purge.ts`, so `releases.ts` (996, baselined) only calls it; - new renderer modules, `astro-renderer/src/references.ts` (members and reverse maps, `groupMembers`, `referencedBy`) and `reference-conversions.ts`, so `source.ts` (616 lines) does not approach 800; - features `fields: 2`; @@ -827,7 +832,7 @@ New rows of the closed table (`conversions.ts:38`): - `testing-references.ts` (new): - writes, and canonicalizing a member's id to its group; - refusing a missing, cross-site, wrong-collection, own-group or archived new target; - - **every race R1 to R11 of §2.3**, as two interleaved batches each: the definitions anchor bound as planned (with a definitions change between `getSite` and the batch), and the move and delete site anchors; + - **every race R1 to R16 of §2.3**, as two interleaved batches each: the definitions anchor bound as planned (with a definitions change between `getSite` and the batch), and the move and delete collections anchors; - delete: the last member versus a remaining member in an allowed or a disallowed collection, and a published versus a draft referrer; - moves of an original and of a translation; - group changes: relinking a referenced original, and dropping `translation`; @@ -836,6 +841,9 @@ New rows of the closed table (`conversions.ts:38`): - the unseen-field interleaving (R13): `events.venue` allows `[places, halls]`, `talks.room` allows `[places]`, and a talk is published between plan and batch; - `shared` after `set_field_localized`'s phase A, both directions; - `field_operation_running` during a text → reference conversion, including a delete planned before phase A commits and batched after it (R12); + - a status-only holder write planned before `rename_field` phase A and batched after it keeps its rows, through `migration.fromKey`; a key with no definition fails the batch instead of dropping rows (R16); + - a publish concurrent with a delete or move does not make it answer 409 (the collections anchor ignores `publishedSnapshotId`); + - a translation becoming its own original is not refused by the written-group guard; - a status-only holder write racing `set_field_localized` phase A leaves the right `shared` (R16); - a group-only `translation.group` change racing a relink (R15); - publish re-checks; @@ -868,7 +876,7 @@ New rows of the closed table (`conversions.ts:38`): - **Renderer:** - identical cards and `referencedBy` on the snapshot and live sources, for a snapshot of the working copy; - gate instants; - - purge plans with a theme that emits no new tags: a target retitled, unpublished, newly published, newly translated and with a changed cover; a depth-2 page (A → B → C with C unpublished purges A); a "used by" page when a referrer appears or goes; a rollback whose referrer was since edited to draft; a release pinned before a referrer was published; **an approved request publishing an older snapshot after the referrer was edited, then the target unpublished: the referrer's page is purged**; a referrer write landing between `createSnapshot` and `publish`; a translation that only inherits the reference appearing on a "used by" page; the `maxTags` and `maxReads` fallbacks; + - purge plans with a theme that emits no new tags: a target retitled, unpublished, newly published, newly translated and with a changed cover; a depth-2 page (A → B → C with C unpublished purges A); a "used by" page when a referrer appears or goes; a rollback whose referrer was since edited to draft; a release pinned before a referrer was published; **an approved request publishing an older snapshot after the referrer was edited, then the target unpublished: the referrer's page is purged**; a referrer write landing between `createSnapshot` and `publish`; a translation that only inherits the reference appearing on a "used by" page; **a referrer edited to stop naming the target, or deleted, between snapshot and purge, still purged from `to`'s bodies**; a translation in `to` that only inherits and was deleted since; a snapshot without a bundle (node reads, then the `maxReads` fallback); **any field-operation marker on `from` or `to` purges everything**; the `maxTags` and `maxReads` fallbacks; - the live-versus-snapshot `referencedBy` comparison runs with no field-operation marker active. - **Studio:** block tests (keyboard, focus after add and remove, live region), `design:shots` at both widths, `design:lint`. @@ -876,7 +884,7 @@ New rows of the closed table (`conversions.ts:38`): | Option | Pros | Cons | Why not | |---|---|---|---| -| Store a per-site anchor member's id instead of the group id | No foreign ids in values | Editions need an id rewrite per site; the anchor changes when the original is deleted; every lookup maps member → anchor | The group id is project-stable and already indexed (§1) | +| Store a per-collections anchor member's id instead of the group id | No foreign ids in values | Editions need an id rewrite per site; the anchor changes when the original is deleted; every lookup maps member → anchor | The group id is project-stable and already indexed (§1) | | Anchor every entry write on the site hash | Simple | Any site edit (theme, menus) conflicts every concurrent entry write | Anchor on the collection's definitions only (§2.1) | | Every reference locks, drafts included (Sanity's strong default) | Strictest | Contributors pin others' entries; abandoned drafts pin forever | Only what a published entry shows locks, by the `shared` rule (§2.2) | | Weak references (no locking at all) | No refusals | Live pages silently lose venues on delete; "used by" still needs the index | Lock where the live site depends | @@ -913,7 +921,7 @@ New rows of the closed table (`conversions.ts:38`): - `readsOriginal`/`originalOf` require an original to have its own id as its group (OQ 14); - group-only translation changes get the group check; - imports store canonical bodies; - - deletes and moves of entries carry the site anchor. + - deletes and moves of entries carry the collections anchor. - **Rollback:** - A snapshot restores the site node and entries, and the index is rebuilt in its batch. - An operation reverts through its `before` snapshot, dropping deleted targets. @@ -945,6 +953,35 @@ New rows of the closed table (`conversions.ts:38`): 14. **Tightening `readsOriginal`/`originalOf`** (an original must have its own id as its group): legacy data relinked before this RFC stops inheriting non-localized values. *Recommended*, because it is the only reading under which an original belongs to exactly one group. Phase 2 reports such nodes in `cloudhal import --dry-run` and on Studio's collection screen, beside `referenceIssues`. 15. **Depth-2 purges** cost a second index probe per publish. *Recommended* as specified. The alternative, dropping depth 2 until themes tag cards, was rejected because the index makes it theme-independent. +## Implementation watch list + +The riskiest SQL and interplay. Every phase PR's review checks the items its phase touches against this list. + +1. **Batch-time guard SQL** (phase 2). + - **The target guard.** Membership in its index-friendly form, the per-pair `fresh` flag, and `n.site_id = ?`. + - **The definitions anchor and the collections anchor.** Bound from the planned `Site` and never from a second read, compared with `json(...) IS json(?)` over canonical bodies. + - **The delete, move and group-change condition.** The blocking-rows fragment, the planned pairs `:P` with the unseen-field `NOT EXISTS`, and "another member, not archived, in `A`". + - **The rows `INSERT`.** It is a plain `INSERT`, never `OR IGNORE` on the writer path; `shared` is read in SQL by `key` or `migration.fromKey`; a `NULL` must fail the batch and map to `ConflictError`. + - **The written-group guard.** Present on every group or source change, absent when the group is the node's own id. + - **What breaks the index invariant silently.** Any `OR IGNORE`, any value bound from the plan where the batch should read it, and any probe without `site_id`. +2. **Purge reads** (phase 3). + - Purges read only the immutable `from`/`to` bodies (bundles first), never the working copy or `entry_refs`. + - Any marker on either snapshot purges everything. + - Release runs re-plan at run time. + - The `maxReads`/`maxTags` fallbacks are exercised by tests, including a snapshot without a bundle. +3. **Marker interplay** (phases 2 to 4). + - `rename_field`'s `root_key` and `fromKey` lag in the fragment (over-blocks only) and in the rows `INSERT` (`fromKey` match). + - Text → reference conversion: readers convert on the fly, `field_operation_running` refuses target deletes, moves and group changes, and any marker on `from` or `to` purges everything. + - `set_field_localized`'s phase-A `UPDATE` of `shared`. + - `entryRefsRebuild` reading through `fromKey`. +4. **Phase-2 behaviour changes on existing sites.** + - `translation_group_in_use`. + - The `readsOriginal`/`originalOf` tightening (OQ 14). + - The group check on group-only changes. + - Canonical bodies on import, and the CAS-guarded canonicalization job. + - The always-on collections anchor's liveness: watch 409 rates in the #105 benchmark and in agents' retry loops. + - Each of these is listed in the changeset. + ## Review log (2026-11-10, PR #193) An independent multi-agent review upheld 28 findings and split two. Each one, with the decision taken. Section numbers in this first table refer to commit fe8ac929; the second round renumbered §2 (§2.4 rebuilds, §2.5 assets, §2.6 field operations, §2.7 releases). @@ -1036,3 +1073,21 @@ Three verifiers found 34 earlier findings resolved. The unresolved ones and the | The refusal list was incomplete | Completed (Security & policy impact). | | Probe-count wording; snapshot-scan cost | Three probes plus one hash comparison and batched member lookups; a far rollback usually falls back to purging everything; a per-snapshot reverse map is a later optimization (Purges point 6). | +### Fifth round (2026-11-10, verification of fca37cbe; the PM chose to simplify) + +| Finding | Decision | +|---|---| +| R16's `INSERT OR IGNORE` drops a row with a `NULL` `shared` instead of failing, so a write racing `rename_field` phase A loses rows (blocking, two verifiers; one confirmed it in sqlite3) | The writer uses a plain `INSERT`. `shared` is matched by `key` or `migration.fromKey`. A `NULL` fails the batch as a `ConflictError`, and the caller retries. The false claim is corrected, and the rename interleaving is in the R16 test (§2.1, R16). | +| The hash-check hybrid's comparison and probes are separate reads; members came from the working copy (blocking, two verifiers) | Dropped, with index mode altogether. Purges read only the immutable `from`/`to` bodies, from their bundles, for every path. **Deviation from the PM's decision:** index mode on the default path, even with an in-statement freshness check, misses referrers edited away or deleted after the snapshot, because the probe never returns them. The cheap whole-site change checks are unsound (rowid reuse). Bundles make the snapshot read cost about 80 shard reads for 10,000 entries (§2, Purges point 4). | +| Purges ignored references shown through in-flight conversions (blocking) | Any field-operation marker on `from` or `to` purges everything (§2, Purges point 4). | +| The always-on site anchor covers `publishedSnapshotId`, so every publish causes 409s | The collections anchor, `json(json_extract(body, '$.collections'))`, holds the definitions and the markers. The threat-model line is corrected (§2.2, §8). | +| The written-group guard blocks a translation becoming its own original | Omitted when the group is the node's own id (§2.2). | +| The canonical rewrite needs CAS and must start with the anchors | A CAS `UPDATE … WHERE hash = ?` job ships in phase 2, and step 7 runs it again (§2.4). | +| Skip filter exact match misses list- and item-level issues | Prefix match (§9 step 2). | +| A large `json_each(:toEntryRefs)` parameter | Moot: purges no longer compare against the working copy. | +| A rebuild during a rename drops rows | `entryRefsRebuild` reads through `migration.fromKey` (§2.4). | +| Stale wording: Status line, §8 anchor line, "never lock", "last usable member", R1–R11, the source-guard name | All corrected. | +| An archived-only member cannot be deleted while referenced | Stated in the delete dialog and the guide (§5). | +| Liveness of the always-on anchor | Narrowed by the collections anchor; on the watch list. | +| Riskiest SQL | The new "Implementation watch list" section, which every phase PR's review checks. | + From cb28aa37c87ed1e0783f9cf4d98216661782dc76 Mon Sep 17 00:00:00 2001 From: Madraka <8898640+Madraka@users.noreply.github.com> Date: Tue, 10 Nov 2026 16:28:42 +0300 Subject: [PATCH 7/8] docs(rfc): RFC-0045 sixth revision after the d8a436cc check - entryRefsRebuild runs only when its import batch writes the node set (no rebuild on an unchanged import); the step-7 job is per node and anchored on each node's read hash, with no site-wide DELETE (R10, R17); step 0 closes the same asset_refs hole on main - purges use one hash-keyed body cache over both snapshots: one bundle (to's first), then only the missing hashes; per-shard streaming with byte and node budgets; short-circuit without reference definitions; RELEASE_EXECUTOR_COST covers the run-time re-plan - shared prefers the exact key; NULL -> ConflictError only for entry_refs.shared; dedup after canonicalization; SQL NULL for an absent collections; "used by" covers asset-rule entries - stale index-era wording removed; new "Owner decisions" section; review log "Sixth round" Co-Authored-By: Claude Opus 5.5 Signed-off-by: Madraka <8898640+Madraka@users.noreply.github.com> --- ...ctions-stage-4-references-and-repeaters.md | 127 ++++++++++++++---- 1 file changed, 100 insertions(+), 27 deletions(-) diff --git a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md index ea4bfaf8..43d9c636 100644 --- a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md +++ b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md @@ -1,6 +1,6 @@ # RFC-0045: Collections stage 4: reference fields and repeaters -- Status: Draft (revised five times on 2026-11-10, after an independent review of PR #193 and four verification passes; the review log and the implementation watch list are at the end) +- Status: Draft (revised six times on 2026-11-10, after an independent review of PR #193 and five verification passes; the fifth revision departed from the PM's index-mode request for purges, see "Owner decisions"; the review log and the implementation watch list are at the end) - Author: Halil (Madraka), drafted with Claude · Date: 2026-11-10 - Related: - [#114](https://github.com/cloudhal-dev/cloudhal/issues/114) (this stage); #71 (the collections plan). @@ -20,6 +20,22 @@ - [OpenAI](https://developers.openai.com/api/docs/guides/structured-outputs): types plus `anyOf`; no `allOf`/`not`/`if`; "Root objects must not be `anyOf`"; every property required, optional ones as a `null` union; up to 5,000 properties, 10 levels of nesting and 1,000 enum values; recursion allowed; `oneOf` not mentioned. - [Anthropic](https://platform.claude.com/docs/en/build-with-claude/structured-outputs): `anyOf`, and `$defs`/`$ref` without recursion; no `minimum`/`maximum`/`minLength`/`maxLength`; `minItems` only 0 or 1; per request at most 20 strict tools, 24 optional parameters, and 16 parameters with union types (`anyOf` or type arrays). +## Owner decisions + +The verification passes left these for the owner to ratify explicitly: + +1. **Purges read the two snapshots, not the index.** Every publish path computes referrer purges from the immutable `from` and `to` bodies: one bundle plus the missing hashes (§2, Purges point 4). It is exact by construction. An index probe cannot see a referrer edited away or deleted after the snapshot, which is why the fifth revision departed from the PM's index-mode request; the verifiers confirmed that departure. + - **Cost:** on a 5,000-entry site, about 25 shard reads and 21 MB per publish that touches references, adding roughly 0.5 to 1.5 s to the publish response. + - **Short-circuit:** sites without reference definitions skip it entirely. + - **Fallback:** without either bundle, or past the read, byte or node budgets, the publish purges the whole site cache. That is safe, but it spends a purge. +2. **Archiving a group's last member does not release a lock.** While a published entry shows a reference, the group's last member (not archived, in an allowed collection) cannot be deleted, moved away or relinked. Archiving it first is allowed but does not help. Unpublish or edit the referrers, as the delete dialog says (§2.2, §5). +3. **Phase 2 changes behaviour on every existing site**, before the types exist (Compatibility): + - `translation_group_in_use`; + - the `readsOriginal`/`originalOf` tightening (OQ 14); + - the group check on group-only changes; + - canonical import bodies; + - the always-on collections anchor. + ## Summary Two new field types join RFC-0041's nine: @@ -34,7 +50,7 @@ A reverse index, `entry_refs`, holds one row per stored value, whatever the targ Archiving or unpublishing a target is always allowed. The renderer resolves only what the published snapshot shows *at that instant*, gates included, so an unresolved reference renders as nothing, and publish warns but never fails. Assets used inside repeaters count in `asset_refs`, like any field's assets. -Resolution is breadth-first, at most two levels deep, under a budget per page. A publish purges referrers' own pages, found through `entry_refs`, whenever a target group changes, becomes visible, or changes its cover. This needs no new cache tags and no theme code. +Resolution is breadth-first, at most two levels deep, under a budget per page. A publish purges referrers' own pages, found from the published and the previous snapshot's bodies, whenever a target group changes, becomes visible, or changes its cover. This needs no new cache tags and no theme code. Studio gets an entry picker and nested repeater forms from new `@cloudhal/ui` blocks. The four field operations learn the new types. @@ -253,7 +269,7 @@ CREATE INDEX entry_refs_target ON entry_refs (site_id, target_id, source_status) **Invariant (the index).** `entry_refs` holds exactly one row per `(node, field path, stored group id)`, whatever the target's state: present, missing, archived or foreign. Rows depend on the holder's body and on its collection's definitions (`localized` decides `shared`). Nothing changes them except a write of the holder, its deletion, a rebuild, or `set_field_localized`'s phase A (§7). - **No cross-tenant pinning.** Every guard and lookup filters `r.site_id` (the holder's site) and `n.site_id` (the member's site). A row naming another tenant's id pins nothing there and never resolves. -- **Repeats.** A target named in two repeater items (a host who also gives a talk) is one row: the writer deduplicates `(field, target)` pairs, and the writer's `INSERT` is a **plain** `INSERT`. The anchored `DELETE` runs first in the same batch, so no key conflict remains to ignore, and any constraint failure fails the batch. +- **Repeats.** A target named in two repeater items (a host who also gives a talk) is one row: the writer deduplicates `(field, target)` pairs *after* canonicalizing member ids to group ids, so two members of one group in two items are one pair. The writer's `INSERT` is a **plain** `INSERT`. The anchored `DELETE` runs first in the same batch, so no key conflict remains to ignore, and any constraint failure fails the batch. - **`shared`** is a pure function of the definition: 1 for `localized: false`, 2 for `localized: true`. It no longer encodes the holder's role, which the first revision computed at write time and which went stale when a node changed group. - *Who* inherits is decided at batch time by the fragment below, exactly as `readsOriginal` decides it (`fields/resolve.ts:37-41`): members `m` of this site whose `translation.group` is the holder's id, who are not the holder, and whose own `translation.source` is not in another site. - A translation's own value, and any value held in an edition site (where no node has the group's id, `originalOf`, `store/entry-fields.ts:54-62`), therefore inherits to nobody, whatever its `shared` (open question 12). @@ -309,13 +325,16 @@ The member probe uses `nodes_translation_locale` (migration 0025: `site_id, kind FROM nodes s, json_each(json_extract(s.body, ?)) d -- ? = '$.collections."".fields' WHERE s.id = ? AND s.kind = 'site' AND (json_extract(d.value, '$.key') = json_extract(p.value, '$.k') - OR json_extract(d.value, '$.migration.fromKey') = json_extract(p.value, '$.k'))), + OR json_extract(d.value, '$.migration.fromKey') = json_extract(p.value, '$.k')) + ORDER BY json_extract(d.value, '$.key') = json_extract(p.value, '$.k') DESC LIMIT 1), ? FROM json_each(?) p WHERE ``` - Batches run one at a time, so the row reflects the site either before phase A, which phase A's `UPDATE` then corrects, or after it. - **During a rename.** `rename_field`'s phase A renames the definition and keeps the old key in `migration.fromKey` (`fields-operations/plan.ts:131-139`). A holder write planned before phase A, which changes no fields and so carries no definitions anchor, still finds its definition through `fromKey`. `rename_field` does not change `localized`, so the value is right. - - **A key with no definition at all** at batch time yields `NULL`. With a plain `INSERT`, the `NOT NULL` column then fails the whole batch, which the store maps to a `ConflictError`, and the caller retries under the current definitions. (The fourth revision's `INSERT OR IGNORE` would have skipped the row silently: SQLite's conflict clause covers `NOT NULL` too. A verifier confirmed this in sqlite3.) + - A new field may reuse a running rename's old key once phase B has emptied it. Then both definitions match, and the exact key wins (`ORDER BY … DESC LIMIT 1`). + - **A key with no definition at all** at batch time yields `NULL`. With a plain `INSERT`, the `NOT NULL` column then fails the whole batch, which the store maps to a `ConflictError`, and the caller retries under the current definitions. + - The mapping matches only `NOT NULL constraint failed: entry_refs.shared`, by message, as `errors.ts:51` already matches `UNIQUE constraint failed`. Any other constraint failure still surfaces as an error, never as an endless 409 loop. (The fourth revision's `INSERT OR IGNORE` would have skipped the row silently: SQLite's conflict clause covers `NOT NULL` too. A verifier confirmed this in sqlite3.) A target deleted or moved between check and batch fails the guard, and `diagnose()` names it (`unknown_entry`, `reference_target_moved`). One bound parameter carries the whole list (D1 allows 100 parameters), and 100 group ids stay far below 100 KB per statement. @@ -334,7 +353,7 @@ A target deleted or moved between check and batch fails the guard, and `diagnose AND r.source_collection || '.' || r.field NOT IN (SELECT value FROM json_each(:P))) ) ``` The second `NOT EXISTS` makes a blocking row from a field the plan did not see, committed in between, fail the batch, so the write retries and re-plans with that field. The first revision's form let it through: `events.venue` allows `[places, halls]` and `talks.room` allows `[places]`; a talk published meanwhile with `room` = G is stranded once the last member in `places` goes. - - **The collections anchor**, `json((SELECT json_extract(body, '$.collections') FROM nodes WHERE id = ? AND kind = 'site')) IS json(?)`, bound with the canonical JSON of `site.collections` of the `Site` object the write planned with. + - **The collections anchor**, `json((SELECT json_extract(body, '$.collections') FROM nodes WHERE id = ? AND kind = 'site')) IS json(?)`, bound with the canonical JSON of `site.collections` of the `Site` object the write planned with. When the planned site has no `collections`, the write binds SQL `NULL`, never the string `'null'` (`json('null') IS NULL` is false), so `NULL IS NULL` holds. - It holds every collection's definitions and every field-operation marker, so it covers the referrer fields whose allowed collections the guard used (R4) and the markers `field_operation_running` reads (R12). - Unlike the site hash used in the fourth revision, it ignores `publishedSnapshotId`, theme, menus and other settings, so a publish no longer makes concurrent deletes and moves answer 409. - Canonical bodies (§2.1, §2.4) keep the comparison live. @@ -368,7 +387,8 @@ A target deleted or moved between check and batch fails the guard, and `diagnose | R7 | a referrer becoming published ↔ deleting its target | the publish re-checks all own values with the target guard. The delete sees the referrer's new `source_status`, because its rows are rewritten in the same batch. | | R8 | publishing a translation that *inherits* the value ↔ deleting the target | when the publish is first, the delete's fragment reads member status at batch time. When the delete is first, nothing published showed the value at that moment. **Accepted:** the translation goes live with a warning and renders nothing; no guard can refuse a write the translator cannot fix. | | R9 | field-operation phase B ↔ deleting a target | the target guard ANDed into each rewritten entry's CAS (point 6) | -| R10 | import, restore or sync ↔ anything | one atomic batch with the index rebuilt in it (point 4) | +| R10 | import, restore or sync ↔ anything | one atomic batch with the index rebuilt in it, *only* when that batch writes the node set. An `unchanged` import rebuilds nothing, so it cannot overwrite a concurrent writer's rows (point 4). | +| R17 | the step-7 job ↔ a referrer write | the job's per-node statements are anchored on each node's read hash, and a node that moved is skipped, because its writer already wrote fresh rows (point 4) | | R11 | a member reappearing (a new translation or edition member) | rows are kept whatever the membership (the invariant), so nothing needs re-creating | | R12 | a reference shown only through an in-flight text → reference conversion ↔ deleting its target, including a delete planned *before* phase A commits | `field_operation_running` refuses the delete until the marker is gone. The collections anchor, on every delete, move and group change, fails a batch planned before the marker. Purges of a snapshot carrying any marker purge everything (§2, Purges). | | R13 | a blocking row from a field the plan did not see ↔ delete, move or group change | the unseen-field `NOT EXISTS` with the planned pairs `P` | @@ -377,14 +397,20 @@ A target deleted or moved between check and batch fails the guard, and `diagnose | R16 | a status-only or title-only write of a holder ↔ `set_field_localized` or `rename_field` phase A | `shared` is read from the site row inside the plain `INSERT … SELECT` at batch time, matched by `key` or `migration.fromKey`. Phase A's `UPDATE` corrects rows written before it. A missing definition fails the batch (`ConflictError`, retry) instead of dropping rows (§2.1). | **4. Rebuilds** run in the same batch as the node set: `importSite`/`pull`/`push` (`sync-plan.ts:308`), `cloudhal import --replace` and `--revert`, and snapshot restores. +- **Only when the batch writes the node set.** `nodeBatch` (`sync-plan.ts:295-308`) adds the node `DELETE`/`INSERT` only when `!changes.unchanged && !changes.conflict`. `entryRefsRebuild` is emitted under exactly that condition. + - On an import planned `unchanged` it is skipped. Its rows would come from bodies read at plan time, and they could overwrite a writer's fresher rows that committed between plan and batch. The writers' own rows are already current there. + - **`refsRebuild` for `asset_refs` has the same hole on main today**: it is pushed unconditionally at `sync-plan.ts:308`. Step 0 fixes it the same way (tracked in its own issue). - `entryRefsRebuild(siteId, rows)` builds the rows from the bodies, with the site's definitions from the site row, as `refsRebuild` does (`media/refs-rebuild.ts:12-37`), and `DELETE`s and `INSERT`s them under `site_id = ?`. - It computes `shared` in JS from the same site row, so no `NULL` can arise. - It reads body keys through any `migration.fromKey` the site row carries, as `resolveFields` does. A bundle exported mid-operation then keeps rows for the entries that were not yet rewritten. - There is no membership join: the invariant keeps rows for every stored value, and the site filters keep them harmless. - The `INSERT` is chunked so each statement stays below 90 KB, inside the one atomic batch. - **Statement budget.** The chunked `INSERT`s count against the import batch's 1,000-statement cap (`sync-plan.ts:311`), like every other statement there. -- **Step 7** also runs a one-time `entryRefsRebuild` per site as an admin job, the pattern of `lists.backfill`. Each site's rebuild is **one atomic batch**: the `DELETE` and its chunked `INSERT`s together, so there is never a window without rows. - - A site too large for one batch gets `INSERT OR IGNORE` chunks only, with no `DELETE`, resumable by cursor. That is safe, because it can only add missing rows, and its `shared` is computed in JS, so it is never `NULL`. +- **Step 7** also runs a one-time per-site rebuild as an admin job, the pattern of `lists.backfill`, whose chunks are guarded on the hash each row was computed from (`list-values/backfill.ts`). + - It never does a site-wide `DELETE`. For each entry node read with hash *h*, it emits one pair, `DELETE FROM entry_refs WHERE source_id = ? AND EXISTS (SELECT 1 FROM nodes WHERE id = ? AND hash = :h)` and the node's `INSERT … WHERE EXISTS (… hash = :h)`, in chunks, resumable by cursor. + - A node whose hash moved since the read matches nothing and is skipped: its writer has already rewritten its rows. + - Rows of nodes that no longer exist go with `DELETE FROM entry_refs WHERE site_id = ? AND source_id NOT IN (SELECT id FROM nodes WHERE site_id = ?)`. + - Its `shared` is computed in JS from the site row read with the chunk, and each chunk also carries the collections anchor, so a definitions change between read and batch fails the chunk, which is retried. - **Canonical rows, from phase 2, not step 7.** The anchors ship in phase 2, so phase 2 also ships an admin job that rewrites existing node rows canonically. It is per row and guarded by CAS (`UPDATE nodes SET body = ? WHERE id = ? AND hash = ?`, hash unchanged, rule 4), so it cannot race a write. That covers rows imported verbatim before phase 2, and imports that planned `unchanged` (`sync-plan.ts:257, 295`), so the anchors are live on every collection. Step 7 runs it once more. - It is a safety net: the gate (§1) means it should find nothing. @@ -447,22 +473,38 @@ R8 and R14 are the accepted exceptions, both warned (open question 13). Dangling - hop 2: the referrers of the groups of the hop-1 referrers. Each page reached adds the node tags of its referrer and of every member of that referrer's group, so all of its language versions go. The walk takes referrers from **both** snapshots' bodies, whatever their status: over-purging is cheap, and a page cached from `from` is reached even when `to` no longer shows the reference. -3. **"Used by" pages.** For every changed, added or removed entry, the planner also adds the node tags of the members of every group its before *or* after **resolved** values name. Resolution uses `resolveFields` with the same-site original read from the same snapshot, so a translation that only inherits its original's `venue` counts too. +3. **"Used by" pages.** For every changed, added or removed entry, and every entry the asset rule reaches (its cover changed), the planner also adds the node tags of the members of every group its before *or* after **resolved** values name. Resolution uses `resolveFields` with the same-site original read from the same snapshot, so a translation that only inherits its original's `venue` counts too. - A venue page that renders `referencedBy` ("events here") is purged when an event naming it appears, disappears or changes, with no theme code, because the target's page carries its own node tag. - `referencedBy` results are drawn as cards: title, URL, summary and cover. Drawing *their* fields and references on the target page is outside the purge guarantee, and `docs/guides/fields.md` says so. -4. **One mode, exact by construction: the snapshots' own bodies.** The planner never consults the working copy for referrers. +4. **One mode, exact by construction: the snapshots' own bodies.** The planner never consults the working copy or `entry_refs` for referrers. - Referrers, both hops, inheriting members (`COALESCE(translation.group, id)`) and "used by" are all computed from the immutable `from` and `to` entry bodies: each body's resolved reference values (`fieldsReferences(defs, body)`, through that snapshot's own definitions, with its same-site original for inheritance). - - It reads both snapshots' **bundles** (RFC-0010: 256 bodies per shard, `bundles.ts`). Every snapshot made to be published gets one (`snapshots.ts:102-110`), and the live one has one, so a 10,000-entry site costs about 80 shard reads. A missing bundle falls back to node-by-node reads. - - This covers every publish path alike: the default path, an older snapshot, an approved request (`approvals.ts:343-356`), a rollback, a release run (re-planned at run time, not replayed from `releases.ts:154`), and a restore or import that moves the pointer. `PublishChange` stays `{from, to}`. + - **Short-circuit.** When neither snapshot's site node defines a `reference` field, or a repeater with a `reference` sub-field, the referrer closure is skipped entirely. Sites without references keep today's purge cost, and phase 3 costs nothing anywhere until step 7 enables the types. + - **One hash-keyed body cache over both snapshots.** The planner already keeps one: `bodies`, keyed by content hash (`cache-purge.ts:122-131`). + - It loads **one** bundle: `to`'s when it exists, otherwise `from`'s (RFC-0010, 256 bodies per shard, `bundles.ts:65-114`). + - It then reads only the hashes still missing, one node each. Those are typically the `before` bodies of the diff, which today's planner reads anyway. Only those reads count against `maxReads`. + - Bundles are not guaranteed. A bundle is written only for reason `publish`, best effort (`snapshots.ts:104-107`). Release runs' `scheduled` snapshots get none (`release-apply.ts:224`). The `bundles/` prefix expires after 30 days (RFC-0010 Retention). A live snapshot regains one only when the site Worker reads through (`source.ts:390`). + - The cache covers all of these. A release run costs the live snapshot's bundle plus its items, an approval of a manual snapshot costs `from`'s bundle plus the diff, and only a far rollback with neither bundle falls back. + - **Streaming.** Shards are parsed one at a time. From each body the planner keeps only what the closure needs: id, group, collection, status, and the resolved reference values. The full body is dropped, except for the few the rest of the planner reads. + - This covers every publish path alike: the default path, an older snapshot, an approved request (`approvals.ts:343-356`), a rollback, a release run, and a restore or import that moves the pointer. `PublishChange` stays `{from, to}`. + - Release runs re-plan at run time: the executor calls `planCachePurge({ from: baseSnapshotId, to: snapshotId })` after `publishRelease` commits, instead of replaying the stored plan (`releases.ts:146-157`). + - `RELEASE_EXECUTOR_COST` (`releases.ts:124`, 16 today) rises by the re-plan's statements: two `getSnapshot` reads plus its node reads, budgeted by `maxReads`. Its comment says so. - **Any field-operation marker on `from`'s or `to`'s site node purges everything.** That covers text → reference conversions in flight and renames. Operations are rare, so this is cheap, and it is obviously safe. - `entry_refs` stays the source for guards, Studio's "Used by", `list_references` and the live `referencedBy`, never for purges. - - **Why not the index on the default path?** The PM's decision for this round proposed index mode there, with a freshness check inside each hop's `SELECT`. It is not exact. A probe returns only rows that *still* name the target, so a referrer in `to` that was edited to stop naming it, or deleted, after the snapshot is never returned, and its hash cannot be compared. Detecting that needs a whole-site change check. The cheap ones are not sound: `count(*)` with `sum(rev)` and `max(rowid)` misses a delete of the newest row followed by an insert that reuses its rowid (`nodes` has no `AUTOINCREMENT`, migration 0001). Reading the bodies the snapshot already holds is both simpler and exact. + - **Why not the index on the default path?** The PM's decision for the fifth round proposed index mode there, with a freshness check inside each hop's `SELECT`. It is not exact. A probe returns only rows that *still* name the target, so a referrer in `to` that was edited to stop naming it, or deleted, after the snapshot is never returned, and its hash cannot be compared. Detecting that needs a whole-site change check. The cheap ones are not sound: `count(*)` with `sum(rev)` and `max(rowid)` misses a delete of the newest row followed by an insert that reuses its rowid (`nodes` has no `AUTOINCREMENT`, migration 0001). Reading the bodies the snapshot already holds is both simpler and exact. The verification of d8a436cc confirmed this. 5. **Every path that changes visibility** purges by point 4: publish (any snapshot), rollback, release run, and an import or restore that moves the published pointer. Archive, delete and unpublish change only the working copy and go live through one of these. - Time-based visibility (an embargo opening, `availability.until`, a derived `hide`/`unlist` expiry) needs no purge: the gate-index TTL expires the page at that instant (`availability.ts:94-95`). 6. **Bounds and fallback.** - - Each bundle shard read, and each node read where no bundle exists, counts against `maxReads` (2,000). Tags stop being collected at `maxTags + 1` (300 tags). Exceeding either purges everything, as any large diff does today (`cache-purge.ts`, "Falls back to purging everything"). - - With bundles, a site of up to about 250,000 entries stays within `maxReads`. Without them, about 1,000 entries per snapshot do. - - The work is in memory, linear in the entries, as the snapshot source already does when it loads (`source.ts` `#load`). `SnapshotReader` gains `readSnapshotBundle` (`store.ts:949`). + - **Reads:** node reads beyond the one bundle count against `maxReads` (2,000), and so do bundle shard reads. + - **Size:** a byte budget on the bodies loaded (`maxBodyBytes`, default 32 MB) and a node budget (`maxNodes`, default 50,000) are checked *before* each shard is parsed. + - **Tags:** collection stops at `maxTags + 1` (300 tags). + - **The fallback:** exceeding any budget purges everything, as any large diff does today (`cache-purge.ts`, "Falls back to purging everything"). + - **Why the size budgets exist.** The purge runs inside the awaited publish request (`admin-worker.ts:453-468`, `store.ts:1107`). An isolate that runs out of memory would never reach the planner's `catch` and would skip the purge silently. The budgets turn that into an explicit purge-everything. + - **Cost on a realistic site** (5,000 entries, about 6,250 nodes, depth 2), as the verifier measured it: + - about 25 shard reads for one bundle, 6 at a time (`BLOB_CONCURRENCY`), against about 207 node reads one after another today; + - about 21 MB of JSON read; + - roughly 0.5 to 1.5 s more on the publish response, and 100 to 200 ms of CPU. + + Purge-everything does not become the common case. As a side benefit, the asset rule, which today reads every entry, page and term body one by one (`cache-purge.ts:222-231`), now reads from the cache. 7. **Compatibility.** `isCloudhalCacheTag` (`cache-tags.ts:48`) and the site Worker's purge endpoint (`cloudflare-entry.ts:312-325`) are unchanged, so an older site Worker accepts every purge, and no id of an unresolved target reaches any header. - `SnapshotReader.readSnapshotBundle` is the one reader addition. The store has it already, and the release executor's run-time reader gets it too. - The first revision's `cloudhal-ref-` tags are dropped. Existing themes build their own tag lists (`starters/blog/src/pages/[...path].astro:261-321`), new prefixes would make older site Workers answer 400, and ids in headers relied on unverified stripping. @@ -534,7 +576,7 @@ interface GraphSource { - per entry: 100 reference values (code, §1); - per render: **200 resolved targets**, shared by every `entryFields` call of one request (`MediaContext.budget` already carries per-request budgets). - Over budget, the remaining targets resolve to nothing with one `console.warn`, never an error. Purges do not depend on what rendered: they go through the index (§2, Purges). A listing of 20 events at depth 1 costs 20. + Over budget, the remaining targets resolve to nothing with one `console.warn`, never an error. Purges do not depend on what rendered: they read the snapshots' bodies (§2, Purges). A listing of 20 events at depth 1 costs 20. - **Batching.** - The snapshot source does no store reads: every lookup is in memory, and bundles (RFC-0010) need nothing new. - The live source (`cloudhal dev`, previews) makes one store read per level: `getGroupMembers(siteId, ids)`, a `json_each` IN list with §2's membership form, served by `nodes_translation_locale` and the primary key. @@ -745,10 +787,10 @@ New rows of the closed table (`conversions.ts:38`): | Threat | Mitigation | Test | |---|---|---| -| **A draft, archived, embargoed or hidden target leaks through a published referrer** (the snapshot holds drafts, and `Loaded` holds published entries before gates) | Resolution filters through the view's gates at its instant (§4); only gate-independent card parts are cached. A card carries only the member's public title, summary, URL and cover image, the image through the view's asset gates. Unresolved values render nothing: no id, no `data-` attribute, no empty link. JSON-LD's `reference` source uses the same cards. | RFC-0008's route sweep (extended by RFC-0042) gains a published referrer pointing at a draft, an archived, an embargoed and a `hide`-expired target, before and after their instants. No id, slug, title or asset of a hidden target may appear in HTML, JSON-LD, feeds, the sitemap, search, `llms.txt`, status pages or response headers. Purges go through the index, so no header carries target ids (§2). | +| **A draft, archived, embargoed or hidden target leaks through a published referrer** (the snapshot holds drafts, and `Loaded` holds published entries before gates) | Resolution filters through the view's gates at its instant (§4); only gate-independent card parts are cached. A card carries only the member's public title, summary, URL and cover image, the image through the view's asset gates. Unresolved values render nothing: no id, no `data-` attribute, no empty link. JSON-LD's `reference` source uses the same cards. | RFC-0008's route sweep (extended by RFC-0042) gains a published referrer pointing at a draft, an archived, an embargoed and a `hide`-expired target, before and after their instants. No id, slug, title or asset of a hidden target may appear in HTML, JSON-LD, feeds, the sitemap, search, `llms.txt`, status pages or response headers. Purges read snapshot bodies and add only existing node tags, so no header carries target ids (§2). | | **IDOR in the picker and on write** | Pick and references routes run under the `/sites/:siteId` membership check and query by `site_id`. The target guard requires a member with `n.site_id = ?` (§2.1). An id of another site gets the same `unknown_entry` as an id that does not exist. Editions copy only ids the edition site can resolve (§3). | A shared-suite case: another site's entry or group id in `create_entry`, `update_entry`, the API and `push` is refused identically; the pick route with another site's id answers 404. | | **An imported or pushed archive pins another tenant's entries or assets** | Rows are site-scoped, and every guard filters both the row's `site_id` and the member's (§2). `asset_refs` gets the same filter as a phase-2 prerequisite (§2.5). | An archive referencing site Y's entry or asset, imported into site X, must not block Y's delete or move, and must not list X's titles to Y. | -| **Lock-out by low-privilege actors** (`content:draft:own`, `content:write:own`) | Only what a published entry shows blocks, by the `shared` rule (§2): a translation's own value is never inherited. Contributors cannot publish; drafts, review and archived entries never lock. | Policy-suite cases that must *not* block: a contributor's draft; an author's review entry; a draft translation's `localized: true` value while its original is published; a draft edition member whose edition translations are published (edition translations never inherit). Cases that must block: an author's published entry, an edition member's included; a published translation inheriting its original's `localized: false` value. | +| **Lock-out by low-privilege actors** (`content:draft:own`, `content:write:own`) | Only what a published entry shows blocks, by the `shared` rule (§2): a translation's own value is never inherited. Contributors cannot publish; drafts, review and archived entries never lock on their own. | Policy-suite cases that must *not* block: a contributor's draft; an author's review entry; a draft translation's `localized: true` value while its original is published; a draft edition member whose edition translations are published (edition translations never inherit). Cases that must block: an author's published entry, an edition member's included; a published translation inheriting its original's `localized: false` value. | | **Fan-out DoS on render** | Caps of 50 per field, 50 items and 10 sub-fields (contract); 100 reference values per entry (code); depth ≤ 2 with a visited set; 200 targets per page; ≤ 2 live reads per page, 0 for snapshots; `referencedBy` ≤ 100. | A property test: random graphs with cycles and 10× the caps resolve within budget, never throw, and cost at most `depth` live reads. | | **Write amplification and slow guards** | Each guard is one indexed probe on `(site_id, target_id)` or one `json_each` count; members are found through `nodes_translation_locale`. Delete, move and group change cost one probe each, plus the collections anchor, always. The purge planner reads the two snapshots' bundles. Rebuilds are chunked inside one batch. | RFC-0042's benchmark (#105), extended to 10,000 events × 3 references on Miniflare D1, reports the write, delete and move guard costs. | | **Picker search abuse** | `q` ≤ 100 characters, a substring match with no regex (rule 13), ≤ 20 results, and a per-member rate window. | A 100k-character `q` is refused before it is read. | @@ -767,12 +809,12 @@ New rows of the closed table (`conversions.ts:38`): 0. **Prerequisite: `asset_refs` by site** (§2.5). - `deleteAsset`'s guard filters by site; - - `refsRebuild` joins same-site assets and chunks its `INSERT`; + - `refsRebuild` joins same-site assets, chunks its `INSERT`, and is emitted only when its import batch writes the node set, which closes the existing hole where an `unchanged` push overwrites fresh `asset_refs` rows (`sync-plan.ts:308`; separate issue); - the purge planner's asset rule reads field assets; - the `unused` asset listing filter (`assets.ts:322`) filters by site, since a foreign archive can hide an asset from its own site's list today; - size ratchet (rule 12): the asset rule moves out of `planCachePurge` (a baselined 233-line function) into `core-ts/src/cache-purge-assets.ts`. The edits to `assets.ts` (1,370 lines) and `store.ts` (1,212), both baselined, add no net lines. - It is independent of this RFC's types, and fixes three existing gaps. + It is independent of this RFC's types, and fixes four existing gaps: the unscoped guard, the unscoped rebuild, field assets missing from purges, and the `unchanged`-import overwrite. 1. **Contract and pure core:** - the §1 diff, fixtures and `pnpm gen` through `contract-change`; - the gate: `checkFieldDefinitions` **and** `checkBundle`'s `assertFieldTypes` refuse both types (`fields_unsupported`), so a phase-1 server takes no such definition from Studio, MCP, the API, import, sync, push, pull or restore; @@ -844,6 +886,12 @@ New rows of the closed table (`conversions.ts:38`): - a status-only holder write planned before `rename_field` phase A and batched after it keeps its rows, through `migration.fromKey`; a key with no definition fails the batch instead of dropping rows (R16); - a publish concurrent with a delete or move does not make it answer 409 (the collections anchor ignores `publishedSnapshotId`); - a translation becoming its own original is not refused by the written-group guard; + - a referrer write landing between an `unchanged` push's plan and its batch keeps its rows: no rebuild runs (R10); + - the step-7 job racing a referrer write: the moved node is skipped, and its fresh rows stay (R17); + - a delete or move on a site with no `collections` (the anchor binds SQL `NULL`); + - a new field reusing a running rename's old key: the exact key wins in `shared`; + - two members of one group in two repeater items: one row, no constraint error; + - an unrelated constraint failure is not turned into a `ConflictError`; - a status-only holder write racing `set_field_localized` phase A leaves the right `shared` (R16); - a group-only `translation.group` change racing a relink (R15); - publish re-checks; @@ -876,7 +924,7 @@ New rows of the closed table (`conversions.ts:38`): - **Renderer:** - identical cards and `referencedBy` on the snapshot and live sources, for a snapshot of the working copy; - gate instants; - - purge plans with a theme that emits no new tags: a target retitled, unpublished, newly published, newly translated and with a changed cover; a depth-2 page (A → B → C with C unpublished purges A); a "used by" page when a referrer appears or goes; a rollback whose referrer was since edited to draft; a release pinned before a referrer was published; **an approved request publishing an older snapshot after the referrer was edited, then the target unpublished: the referrer's page is purged**; a referrer write landing between `createSnapshot` and `publish`; a translation that only inherits the reference appearing on a "used by" page; **a referrer edited to stop naming the target, or deleted, between snapshot and purge, still purged from `to`'s bodies**; a translation in `to` that only inherits and was deleted since; a snapshot without a bundle (node reads, then the `maxReads` fallback); **any field-operation marker on `from` or `to` purges everything**; the `maxTags` and `maxReads` fallbacks; + - purge plans with a theme that emits no new tags: a target retitled, unpublished, newly published, newly translated and with a changed cover; a depth-2 page (A → B → C with C unpublished purges A); a "used by" page when a referrer appears or goes; a rollback whose referrer was since edited to draft; a release pinned before a referrer was published; **an approved request publishing an older snapshot after the referrer was edited, then the target unpublished: the referrer's page is purged**; a referrer write landing between `createSnapshot` and `publish`; a translation that only inherits the reference appearing on a "used by" page; **a referrer edited to stop naming the target, or deleted, between snapshot and purge, still purged from `to`'s bodies**; a translation in `to` that only inherits and was deleted since; a release run whose `scheduled` snapshot has no bundle (one bundle plus its items, no purge-everything); an approval of a manual snapshot without a bundle; a site without reference definitions (no bundle read at all); the byte and node budgets falling back to purge-everything; **any field-operation marker on `from` or `to` purges everything**; the `maxTags` and `maxReads` fallbacks; - the live-versus-snapshot `referencedBy` comparison runs with no field-operation marker active. - **Studio:** block tests (keyboard, focus after add and remove, live region), `design:shots` at both widths, `design:lint`. @@ -884,14 +932,14 @@ New rows of the closed table (`conversions.ts:38`): | Option | Pros | Cons | Why not | |---|---|---|---| -| Store a per-collections anchor member's id instead of the group id | No foreign ids in values | Editions need an id rewrite per site; the anchor changes when the original is deleted; every lookup maps member → anchor | The group id is project-stable and already indexed (§1) | +| Store a per-site anchor member's id instead of the group id | No foreign ids in values | Editions need an id rewrite per site; the anchor changes when the original is deleted; every lookup maps member → anchor | The group id is project-stable and already indexed (§1) | | Anchor every entry write on the site hash | Simple | Any site edit (theme, menus) conflicts every concurrent entry write | Anchor on the collection's definitions only (§2.1) | | Every reference locks, drafts included (Sanity's strong default) | Strictest | Contributors pin others' entries; abandoned drafts pin forever | Only what a published entry shows locks, by the `shared` rule (§2.2) | | Weak references (no locking at all) | No refusals | Live pages silently lose venues on delete; "used by" still needs the index | Lock where the live site depends | | Cascade or "unlink and delete" | One click | Rewrites other people's entries as a side effect | Refuse and list; open question 1 | | Block publish or fail the build on unresolved references | Strict | Publish is a site snapshot; one draft venue blocks every fix | Warn (§2) | | References as release dependencies (today's generic extractor) | A scheduled event takes its venue live with it | Locks every member of the target until the run; edition group ids fail runs; hard, where render is soft | Kind-aware `depsOf` drops them (§2.7) | -| `cloudhal-ref-` cache tags on referrer responses (first revision) | Purges by group | Themes build tags themselves, so existing themes would never emit them; older site Workers answer 400 to new prefixes and every publish purges everything; ids in headers rest on unverified stripping; header budget | The purge planner adds referrers' existing node tags through `entry_refs` (§2, Purges) | +| `cloudhal-ref-` cache tags on referrer responses (first revision) | Purges by group | Themes build tags themselves, so existing themes would never emit them; older site Workers answer 400 to new prefixes and every publish purges everything; ids in headers rest on unverified stripping; header budget | The purge planner adds referrers' existing node tags, found from the snapshots' bodies (§2, Purges) | | Store something immutable (a new `refId` per entry) instead of the group id | Survives group changes | A new identity on every entry, its own index, migrations and editions mapping; translations still need group resolution | Group ids, with group changes guarded like deletes (§2.2) | | Rewrite referrers when a target changes group | Group changes always succeed | Rewrites other people's published entries as a side effect of a translation link | Refuse while shown by a published entry (§2.2) | | Keep rows only while the target has a member (first revision) | Smaller table | Nothing re-creates rows when a member reappears; cleanup on last-member delete is fiddly | Rows for every stored value; the site filter is the isolation (§2) | @@ -951,7 +999,7 @@ New rows of the closed table (`conversions.ts:38`): 12. **Edition chains and non-localized values:** inside an edition site, the edition node's translations do not inherit its non-localized values of any type, as RFC-0041 behaves today (`originalOf`). References follow that rule: the fragment finds no inheriting member there. *Recommended:* keep it in stage 4, and fix it for every field type in its own change, defining the in-site root as the member with no same-site source. 13. **R8 and R14** (a translation published just after its inherited target was deleted; a held node published at its live version): accept both with a warning. *Recommended*, because refusing would block a translator over a value they cannot edit. The alternative is to refuse the publish until the original's value is fixed. 14. **Tightening `readsOriginal`/`originalOf`** (an original must have its own id as its group): legacy data relinked before this RFC stops inheriting non-localized values. *Recommended*, because it is the only reading under which an original belongs to exactly one group. Phase 2 reports such nodes in `cloudhal import --dry-run` and on Studio's collection screen, beside `referenceIssues`. -15. **Depth-2 purges** cost a second index probe per publish. *Recommended* as specified. The alternative, dropping depth 2 until themes tag cards, was rejected because the index makes it theme-independent. +15. **Depth-2 purges** cost a second hop over bodies already in the cache, so nothing extra is read. *Recommended* as specified. The alternative, dropping depth 2 until themes tag cards, was rejected: reading the snapshots' bodies makes it theme-independent. ## Implementation watch list @@ -963,12 +1011,20 @@ The riskiest SQL and interplay. Every phase PR's review checks the items its pha - **The delete, move and group-change condition.** The blocking-rows fragment, the planned pairs `:P` with the unseen-field `NOT EXISTS`, and "another member, not archived, in `A`". - **The rows `INSERT`.** It is a plain `INSERT`, never `OR IGNORE` on the writer path; `shared` is read in SQL by `key` or `migration.fromKey`; a `NULL` must fail the batch and map to `ConflictError`. - **The written-group guard.** Present on every group or source change, absent when the group is the node's own id. - - **What breaks the index invariant silently.** Any `OR IGNORE`, any value bound from the plan where the batch should read it, and any probe without `site_id`. + - **What breaks the index invariant silently.** Any `OR IGNORE` on the writer path, any value bound from the plan where the batch should read it, and any probe without `site_id`. + - **Deduplication.** The writer deduplicates `(field, target)` *after* canonicalizing to group ids. + - **The `ConflictError` mapping** matches only the `entry_refs.shared` `NOT NULL` message. + - **The collections anchor** binds SQL `NULL` for an absent `collections`. + - **Rebuilds** never run beside an unchanged node set. The step-7 job is per node and anchored on each node's read hash; it never does a site-wide `DELETE`. 2. **Purge reads** (phase 3). - Purges read only the immutable `from`/`to` bodies (bundles first), never the working copy or `entry_refs`. - Any marker on either snapshot purges everything. - Release runs re-plan at run time. - The `maxReads`/`maxTags` fallbacks are exercised by tests, including a snapshot without a bundle. + - One hash-keyed cache, one bundle, then the missing hashes only. + - Shards are streamed, and the byte and node budgets are checked before each parse, so an out-of-memory isolate can never skip a purge silently. + - The short-circuit for sites without reference definitions. + - `RELEASE_EXECUTOR_COST` is updated for the re-plan. 3. **Marker interplay** (phases 2 to 4). - `rename_field`'s `root_key` and `fromKey` lag in the fragment (over-blocks only) and in the rows `INSERT` (`fromKey` match). - Text → reference conversion: readers convert on the fly, `field_operation_running` refuses target deletes, moves and group changes, and any marker on `from` or `to` purges everything. @@ -1091,3 +1147,20 @@ Three verifiers found 34 earlier findings resolved. The unresolved ones and the | Liveness of the always-on anchor | Narrowed by the collections anchor; on the watch list. | | Riskiest SQL | The new "Implementation watch list" section, which every phase PR's review checks. | +### Sixth round (2026-11-10, final verification of d8a436cc) + +| Finding | Decision | +|---|---| +| Rebuilds from rows read before their batch can overwrite a writer's fresh rows: the `unchanged` import path and the step-7 job (blocking) | `entryRefsRebuild` is emitted only when its batch writes the node set (`sync-plan.ts:295-308`, R10). The step-7 job is per node, anchored on each node's read hash like `lists.backfill`, with no site-wide `DELETE` (R17). The same `asset_refs` hole on main is fixed in step 0 (separate issue). | +| The fifth revision's departure from index mode | Confirmed by the verifiers, and put to the owner in "Owner decisions". | +| Purge cost: bundles are often missing (release runs, manual snapshots, expired prefixes) | One hash-keyed body cache over both snapshots: one bundle (`to`'s first), then only the missing hashes, counted. The re-plan is reflected in `RELEASE_EXECUTOR_COST` (§2, Purges points 4 and 6). | +| Memory, and an out-of-memory isolate silently skipping the purge | Per-shard streaming, with byte (`maxBodyBytes`) and node (`maxNodes`) budgets checked before each parse, falling back to purge-everything. | +| Cost for sites without references | Short-circuit: no bundle reads when neither snapshot defines a reference field; phase 3 costs nothing before step 7. | +| `shared` subquery ambiguity | The exact key match wins (`ORDER BY … = k DESC LIMIT 1`). | +| NULL → `ConflictError` too broad | Only the `entry_refs.shared` `NOT NULL` failure, by message (`errors.ts:51` precedent). | +| Deduplication | After canonicalization to group ids. | +| An absent `collections` | The anchor binds SQL `NULL`. | +| Rule 3 and the asset rule | "Used by" also covers entries the asset rule reaches. | +| Stale index-era wording (Summary, §4, §8, Alternatives, OQ 15), "per-collections", the bundle claim | All corrected. | +| An owner summary | The new "Owner decisions" section near the top. | + From ed588df4fc4e48df440e04e1c41a59148c9dc1d8 Mon Sep 17 00:00:00 2001 From: Madraka <8898640+Madraka@users.noreply.github.com> Date: Tue, 10 Nov 2026 17:01:44 +0300 Subject: [PATCH 8/8] docs(rfc): accept RFC-0045 (collections stage 4) The owner accepted the RFC on 2026-11-10 with the three owner decisions ratified: snapshot-only purges with their cost and fallbacks, archiving the last member does not release a lock, and the phase-2 behaviour changes on existing sites. Sets the status, turns "Owner decisions" into "Decisions (2026-11-10)", and links #114 and #196 (fixed by rollout step 0). The docs site lists RFCs from the folder itself (apps/docs/src/lib/pages.ts), so there is no index file to update. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Madraka <8898640+Madraka@users.noreply.github.com> --- ...ctions-stage-4-references-and-repeaters.md | 26 ++++++++++++------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md index 43d9c636..789780da 100644 --- a/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md +++ b/docs/rfcs/0045-collections-stage-4-references-and-repeaters.md @@ -1,9 +1,10 @@ # RFC-0045: Collections stage 4: reference fields and repeaters -- Status: Draft (revised six times on 2026-11-10, after an independent review of PR #193 and five verification passes; the fifth revision departed from the PM's index-mode request for purges, see "Owner decisions"; the review log and the implementation watch list are at the end) +- Status: Accepted (2026-11-10, by Halil), with the three Owner decisions ratified; see "Decisions (2026-11-10)". The ADR lands with rollout step 2. + Drafted in PR #193 and revised six times after an independent review and five verification passes; the review log and the implementation watch list are at the end. - Author: Halil (Madraka), drafted with Claude · Date: 2026-11-10 - Related: - - [#114](https://github.com/cloudhal-dev/cloudhal/issues/114) (this stage); #71 (the collections plan). + - [#114](https://github.com/cloudhal-dev/cloudhal/issues/114) (this stage); #71 (the collections plan); [#196](https://github.com/cloudhal-dev/cloudhal/issues/196) (the `unchanged`-import `asset_refs` overwrite, fixed by rollout step 0). - RFC-0040 / ADR-0042 (the registry), RFC-0041 / ADR-0043 (typed fields, field operations), RFC-0042 / ADR-0044 (§2 presets, §4 listings and `entry_list_values`, §6 "out: stage 4"). - RFC-0004 §4.4 (`asset_refs`, the model for the reverse index), RFC-0008 §6 and §7 (releases and their dependency closure, holds, gates, the gate index), RFC-0010 (snapshot bundles), RFC-0015 / ADR-0030 (the props subset), RFC-0033 / ADR-0035 (one node per locale, translation groups, `localization.fallback`), RFC-0034 (language editions), RFC-0039 / ADR-0041 (archives). - Prior art (checked 2026-11-10; "unverified" where a primary page could not be read): @@ -20,16 +21,16 @@ - [OpenAI](https://developers.openai.com/api/docs/guides/structured-outputs): types plus `anyOf`; no `allOf`/`not`/`if`; "Root objects must not be `anyOf`"; every property required, optional ones as a `null` union; up to 5,000 properties, 10 levels of nesting and 1,000 enum values; recursion allowed; `oneOf` not mentioned. - [Anthropic](https://platform.claude.com/docs/en/build-with-claude/structured-outputs): `anyOf`, and `$defs`/`$ref` without recursion; no `minimum`/`maximum`/`minLength`/`maxLength`; `minItems` only 0 or 1; per request at most 20 strict tools, 24 optional parameters, and 16 parameters with union types (`anyOf` or type arrays). -## Owner decisions +## Decisions (2026-11-10) -The verification passes left these for the owner to ratify explicitly: +Ratified by the owner (Halil) on 2026-11-10, as the verification passes left them for him to decide: -1. **Purges read the two snapshots, not the index.** Every publish path computes referrer purges from the immutable `from` and `to` bodies: one bundle plus the missing hashes (§2, Purges point 4). It is exact by construction. An index probe cannot see a referrer edited away or deleted after the snapshot, which is why the fifth revision departed from the PM's index-mode request; the verifiers confirmed that departure. +1. **Snapshot-only purges.** Every publish path computes referrer purges from the immutable `from` and `to` bodies: one bundle plus the missing hashes (§2, Purges point 4). It is exact by construction. An index probe cannot see a referrer edited away or deleted after the snapshot, which is why the fifth revision departed from the project manager's index-mode request; the verifiers confirmed that departure, and the owner ratified it. - **Cost:** on a 5,000-entry site, about 25 shard reads and 21 MB per publish that touches references, adding roughly 0.5 to 1.5 s to the publish response. - **Short-circuit:** sites without reference definitions skip it entirely. - **Fallback:** without either bundle, or past the read, byte or node budgets, the publish purges the whole site cache. That is safe, but it spends a purge. -2. **Archiving a group's last member does not release a lock.** While a published entry shows a reference, the group's last member (not archived, in an allowed collection) cannot be deleted, moved away or relinked. Archiving it first is allowed but does not help. Unpublish or edit the referrers, as the delete dialog says (§2.2, §5). -3. **Phase 2 changes behaviour on every existing site**, before the types exist (Compatibility): +2. **Archiving the last member does not release a lock.** While a published entry shows a reference, the group's last member (not archived, in an allowed collection) cannot be deleted, moved away or relinked. Archiving it first is allowed but does not help. Unpublish or edit the referrers, as the delete dialog says (§2.2, §5). +3. **Phase-2 behaviour changes on existing sites** are accepted, before the types exist (Compatibility). They are listed in the phase-2 changeset: - `translation_group_in_use`; - the `readsOriginal`/`originalOf` tightening (OQ 14); - the group check on group-only changes; @@ -399,7 +400,7 @@ A target deleted or moved between check and batch fails the guard, and `diagnose **4. Rebuilds** run in the same batch as the node set: `importSite`/`pull`/`push` (`sync-plan.ts:308`), `cloudhal import --replace` and `--revert`, and snapshot restores. - **Only when the batch writes the node set.** `nodeBatch` (`sync-plan.ts:295-308`) adds the node `DELETE`/`INSERT` only when `!changes.unchanged && !changes.conflict`. `entryRefsRebuild` is emitted under exactly that condition. - On an import planned `unchanged` it is skipped. Its rows would come from bodies read at plan time, and they could overwrite a writer's fresher rows that committed between plan and batch. The writers' own rows are already current there. - - **`refsRebuild` for `asset_refs` has the same hole on main today**: it is pushed unconditionally at `sync-plan.ts:308`. Step 0 fixes it the same way (tracked in its own issue). + - **`refsRebuild` for `asset_refs` has the same hole on main today**: it is pushed unconditionally at `sync-plan.ts:308`. Step 0 fixes it the same way ([#196](https://github.com/cloudhal-dev/cloudhal/issues/196)). - `entryRefsRebuild(siteId, rows)` builds the rows from the bodies, with the site's definitions from the site row, as `refsRebuild` does (`media/refs-rebuild.ts:12-37`), and `DELETE`s and `INSERT`s them under `site_id = ?`. - It computes `shared` in JS from the same site row, so no `NULL` can arise. - It reads body keys through any `migration.fromKey` the site row carries, as `resolveFields` does. A bundle exported mid-operation then keeps rows for the entries that were not yet rewritten. @@ -807,9 +808,9 @@ New rows of the closed table (`conversions.ts:38`): ### 9. Rollout (one PR each; the types stay refused on every write path until step 7) -0. **Prerequisite: `asset_refs` by site** (§2.5). +0. **Prerequisite: `asset_refs` by site** (§2.5; fixes [#196](https://github.com/cloudhal-dev/cloudhal/issues/196)). - `deleteAsset`'s guard filters by site; - - `refsRebuild` joins same-site assets, chunks its `INSERT`, and is emitted only when its import batch writes the node set, which closes the existing hole where an `unchanged` push overwrites fresh `asset_refs` rows (`sync-plan.ts:308`; separate issue); + - `refsRebuild` joins same-site assets, chunks its `INSERT`, and is emitted only when its import batch writes the node set, which closes the existing hole where an `unchanged` push overwrites fresh `asset_refs` rows (`sync-plan.ts:308`; [#196](https://github.com/cloudhal-dev/cloudhal/issues/196)); - the purge planner's asset rule reads field assets; - the `unused` asset listing filter (`assets.ts:322`) filters by site, since a foreign archive can hide an asset from its own site's list today; - size ratchet (rule 12): the asset rule moves out of `planCachePurge` (a baselined 233-line function) into `core-ts/src/cache-purge-assets.ts`. The edits to `assets.ts` (1,370 lines) and `store.ts` (1,212), both baselined, add no net lines. @@ -985,6 +986,8 @@ New rows of the closed table (`conversions.ts:38`): ## Open questions (each with a recommended answer) +The owner's acceptance ratified the three decisions above. The answers below are the recommendations each phase PR follows; a phase PR that needs a different answer brings it back to the owner. + 1. **Deleting a referenced entry:** refuse while a published referrer exists and this is the group's last member in the site; no cascade. *Recommended.* Add an explicit `unlink_entry` operation (dry run, approval, snapshot, revert) if people ask for it. 2. **Per-sub-field localization in a shared repeater:** not in stage 4. *Recommended.* Revisit with stable item keys (`_key`, which cannot collide with a `FieldKey`). 3. **`datetime`, multiple selects and references inside repeaters** (an agenda's session times): not in stage 4, because ADR-0030's array items are flat scalars. *Recommended:* the first follow-up, allowing a flat object one level below an item for `datetime` only (an ADR-0030 amendment), at a strict depth of 5 of 10. @@ -1164,3 +1167,6 @@ Three verifiers found 34 earlier findings resolved. The unresolved ones and the | Stale index-era wording (Summary, §4, §8, Alternatives, OQ 15), "per-collections", the bundle claim | All corrected. | | An owner summary | The new "Owner decisions" section near the top. | +### Acceptance (2026-11-10) + +The owner accepted the RFC with the three decisions ratified ("Kabul ediyorum"). "Owner decisions" became "Decisions (2026-11-10)", and [#196](https://github.com/cloudhal-dev/cloudhal/issues/196) is linked from step 0.