diff --git a/docs/concepts.md b/docs/concepts.md index 04e584b..f14b5e0 100644 --- a/docs/concepts.md +++ b/docs/concepts.md @@ -51,7 +51,7 @@ Each `.md` file with a `type` frontmatter field is a **typed page** — it repre - **Heading with `[type:: x]`** or **anchor-implied type** (e.g. `## My Goal ^goal1`) → becomes a child node. - **Relationship headings** (e.g. `### Assumptions`) → when matched to a relationship definition in the schema, signals that following content (list items or table rows) should be typed as that relationship's child type, without requiring explicit inline annotations. -- **Untyped headings** → update the depth stack for parent resolution but do not become nodes. +- **Untyped headings** → update the depth stack for parent resolution but do not become nodes. They stay in the content of the node whose section contains them; see [Node content](#node-content). - **Typed bullet items** (`- [type:: solution] Title`) → become child nodes at any nesting depth. - **Table rows** under a relationship heading → become child nodes of the relationship type. - **YAML blocks** and **unbracketed `key:: value` paragraph fields** → merged into the current node's `schemaData`. @@ -87,7 +87,7 @@ Key properties: #### Preamble -**Preamble** is content in a `space on a page` document that appears before the first heading. It is parsed but discarded — not associated with any node. +**Preamble** is content in a `space on a page` document that appears before the first heading. It is parsed but discarded — not associated with any node, and not part of any node's `content`. --- @@ -99,6 +99,8 @@ Node types are defined by the schema in use and may vary across schemas. Example Each node retains its physical `source` location (`file`, and `line` / `column` where available) so that validation diagnostics, edge errors, and downstream tooling pinpoint the exact location in the vault. +A node's `content` is the markdown source of its body; see [Node content](#node-content). + > `space_on_a_page` and `dashboard` are not `space node` types — they are `tooling types`. ### Embedded node @@ -107,6 +109,20 @@ An **embedded node** is a `space node` defined *within* a containing document ra A `typed page` may contain embedded nodes in its body. Those nodes become full members of the parsed node set, with `parent references` wired to their containing page or enclosing heading. +### Node content + +A node's `content` is the markdown of its body, copied unchanged from the file: blank lines, list looseness, line endings and code fences all stay as written. The content of a list item node is dedented by the item's content indent, so it reads as markdown on its own. Leading blank lines and trailing whitespace are trimmed. + +A node owns the source from its heading or list item up to the next heading that ends its section, with these parts removed: + +- Its own heading line or list marker text. +- The sections of nested embedded nodes. +- Field syntax that supplies data: frontmatter and other YAML blocks, and lines that hold only `key:: value` or `[key:: value]` fields. A field written inside a sentence stays in the prose, and its value is still read as a field. + +An untyped heading belongs to the content of the node whose section contains it. An untyped heading at the same level as an embedded node's heading, or at a higher level (fewer `#`), ends that node's section, so the heading and what follows it go to the enclosing node, or to the page. An untyped heading at a deeper level stays inside the section. Inline fields below an untyped heading are assigned the same way. + +In a `space_on_a_page` file, content before the first heading (the preamble) is not assigned to any node. + ### Type alias A **type alias** is an alternative name accepted in the `type` field for a given `space node` type. Aliases allow teams to use their own vocabulary while still receiving schema validation. For example, a schema might accept `outcome` as an alias for `goal`. @@ -117,7 +133,7 @@ A `space node`'s resolved type (`resolvedType`) is its canonical type after alia ## Typed page -A **typed page** is a markdown file whose frontmatter declares a `space node` type (e.g. `type: goal`). The file itself represents one node, and its body may additionally contain `embedded nodes`. Body content appearing before any embedded headings (or the entire body when no embedded headings are present) is preserved as the page node's `content`. +A **typed page** is a markdown file whose frontmatter declares a `space node` type (e.g. `type: goal`). The file itself represents one node, and its body may additionally contain `embedded nodes`. The page node's `content` is the body markdown outside any embedded node (the entire body when there are none). Typed pages are distinct from `space on a page` files: a typed page *is* a `space node`; a `space_on_a_page` file is merely a container. diff --git a/docs/schemas.md b/docs/schemas.md index 3fce5d6..5035dc4 100644 --- a/docs/schemas.md +++ b/docs/schemas.md @@ -71,7 +71,7 @@ Main types: ### `okf.json` -[Open Knowledge Format](https://github.com/GoogleCloudPlatform/open-knowledge-format) (OKF v0.2) bundles, validated as plain markdown spaces. Any concept `type` is accepted and unknown fields are tolerated; provenance (`sources`), trust (`generated`, `verified`), lifecycle (`status`, `stale_after`) and the Attested Computation contract are checked when present. Reserved `index.md` and `log.md` are reported as excluded during parsing. +[Open Knowledge Format](https://github.com/GoogleCloudPlatform/open-knowledge-format) (OKF v0.2) bundles, validated as plain markdown spaces. Any concept `type` is accepted and unknown fields are tolerated; provenance (`sources`), trust (`generated`, `verified`), lifecycle (`status`, `stale_after`) and the Attested Computation contract are checked when present. An Attested Computation needs its computation: a `computation` field, or a `# Computation` section in the body (found by matching that line in the body text, so a `# Computation` line inside a code fence also counts). Reserved `index.md` and `log.md` are reported as excluded during parsing. ## Custom format annotations @@ -284,6 +284,8 @@ With this config, an application page may include `### capability` with wikilink - [[Zephyr]] ← populates application.tools (fieldOn: parent) ``` +A node's body becomes its `content`; see [Node content](concepts.md#node-content). + ## Composition and merge semantics `$ref` and `$defs` are for **validation only**: a `$ref` to another schema (or to one of its `$defs`) reuses its JSON Schema definitions but brings none of its `$metadata`. diff --git a/schemas/_okf.json b/schemas/_okf.json index cd5410e..3cd54d8 100644 --- a/schemas/_okf.json +++ b/schemas/_okf.json @@ -29,13 +29,11 @@ "check": "$not($exists(current.sources)) or $count(current.sources[$exists(usage_count) and $not($exists(usage_window)) and $not($exists($$.current.usage_window))]) = 0" }, { - // Body content is plain text (headings are not kept), so a `# Computation` section can't be told apart - // from other body text: any body counts. "id": "okf-computation-present", "category": "validation", "type": "Attested Computation", - "description": "An Attested Computation needs its computation: a `computation` path or a body `# Computation` section", - "check": "$exists(current.computation) or $exists(current.content)" + "description": "An Attested Computation needs its computation: a `computation` path or a body `# Computation` section. Known limitation: the section is found by matching a `# Computation` line in the body text, so such a line inside a code fence also counts", + "check": "$exists(current.computation) or $contains(current.content, /^# Computation[ \\t]*$/m)" } ] }, diff --git a/schemas/_sctx_base.json b/schemas/_sctx_base.json index c4d4e69..18847b0 100644 --- a/schemas/_sctx_base.json +++ b/schemas/_sctx_base.json @@ -9,7 +9,7 @@ }, "content": { "type": "string", - "description": "Body content" + "description": "Markdown source of the node's body, excluding its own heading or list marker and any nested embedded nodes" }, "summary": { "type": "string", diff --git a/src/plugins/markdown/parse-embedded.ts b/src/plugins/markdown/parse-embedded.ts index 6a91bd9..4bfe378 100644 --- a/src/plugins/markdown/parse-embedded.ts +++ b/src/plugins/markdown/parse-embedded.ts @@ -26,6 +26,8 @@ type StackEntry = { nodeType: string; /** Preferred wikilink key used when this heading acts as a parent. */ refTarget: string; + /** Node whose section this heading opens; absent for untyped placeholders and the page's own entry. */ + node?: BaseNode; }; /** @@ -197,14 +199,108 @@ export function defaultNodeType(stack: StackEntry[], hierarchy: readonly string[ return hierarchy[idx + 1]!; } -function appendContent(node: BaseNode, text: string): void { - if (!text) return; - const existing = node.schemaData.content as string | undefined; - const updated = existing ? `${existing}\n${text}` : text; - node.schemaData.content = updated; - node.content = updated; +type Position = NonNullable; + +/** A half-open `[start, end)` range of offsets into the markdown body. */ +type SourceRange = { start: number; end: number }; + +/** + * Tracks which parts of the body source each node owns. A node's content is its ranges sliced + * from the body and joined, so it stays contiguous, verbatim source minus whatever other nodes + * (or consumed field syntax) took out of it. + */ +type SourceClaims = { + ranges: Map; + /** Nodes whose next claim starts at its block, because a consumed block already kept the blank lines before it. */ + gapKept: Set; + /** List item nodes mapped to the column where their content starts; their content is dedented by it. */ + contentIndent: Map; +}; + +/** Start and end offsets of an mdast node. Throws when the parser gave no position. */ +function offsetsOf(node: { type: string; position?: Position }): SourceRange { + const start = node.position?.start.offset; + const end = node.position?.end.offset; + if (start === undefined || end === undefined) { + throw new Error(`Markdown ${node.type} node has no source position; cannot slice its content`); + } + return { start, end }; +} + +function addRange(claims: SourceClaims, node: BaseNode, start: number, end: number): void { + if (end <= start) return; + const ranges = claims.ranges.get(node) ?? []; + const last = ranges[ranges.length - 1]; + if (last && last.end === start) last.end = end; + else ranges.push({ start, end }); + claims.ranges.set(node, ranges); +} + +/** + * Give a node a block of source, plus the blank lines between `from` and the block's `start`. + * Those lines are skipped when a consumed block just before already kept its own. + */ +function claim(claims: SourceClaims, node: BaseNode, from: number, start: number, end: number): void { + addRange(claims, node, claims.gapKept.delete(node) ? start : from, end); +} + +/** + * Take a block out of a node's content (it was consumed as a node, field or other syntax). + * The blank lines before it stay, so the content on either side does not run together. + */ +function consume(claims: SourceClaims, node: BaseNode, from: number, start: number): void { + addRange(claims, node, claims.gapKept.delete(node) ? start : from, start); + claims.gapKept.add(node); +} + +/** Sets each node's content from its claimed source. Leading blank lines and trailing whitespace are dropped. */ +function applyClaims(claims: SourceClaims, body: string): void { + for (const [node, ranges] of claims.ranges) { + let content = ranges + .map((r) => body.slice(r.start, r.end)) + .join('') + .replace(/^(?:[ \t]*\r?\n)+/, '') + .trimEnd(); + const indent = claims.contentIndent.get(node); + if (indent) { + const strip = new RegExp(`^[ \\t]{1,${indent}}`); + content = content + .split(/(?<=\n)/) + .map((line) => line.replace(strip, '')) + .join(''); + } + if (!content) continue; + node.schemaData.content = content; + node.content = content; + } +} + +/** Source lines (with terminators) of a range, as absolute offsets. */ +function sourceLines(body: string, range: SourceRange): (SourceRange & { text: string })[] { + const lines: (SourceRange & { text: string })[] = []; + const re = /[^\n]*\n|[^\n]+$/g; + const text = body.slice(range.start, range.end); + for (const m of text.matchAll(re)) { + lines.push({ + start: range.start + m.index, + end: range.start + m.index + m[0].length, + text: m[0].replace(/\r?\n$/, ''), + }); + } + return lines; } +/** Whether a line holds only inline fields: bracketed `[key:: value]` pieces, or one `key:: value`. */ +function isFieldOnlyLine(line: string): boolean { + const { cleanText, fields } = extractBracketedFields(line); + const hasBracketed = Object.keys(fields).length > 0; + if (cleanText === '') return hasBracketed; + return /^[a-zA-Z][a-zA-Z0-9_-]*:: *.*$/.test(cleanText); +} + +/** Cursor over the body while embedded list items are walked: everything before `pos` is already assigned. */ +type ListWalk = { claims: SourceClaims; pos: number }; + function processListItem( item: ListItem, parentRef: string | undefined, @@ -213,17 +309,20 @@ function processListItem( makeLabel: (title: string) => string, buildLinkTargets: (title: string) => string[], typeAliases: Record, - fieldMap?: Record, - pendingType?: string, - parentFieldAppend?: { node: BaseNode; field: string }, - activeNodeFieldAppend?: { node: BaseNode; field: string }, - file?: string, - lineOffset?: number, + fieldMap: Record | undefined, + pendingType: string | undefined, + parentFieldAppend: { node: BaseNode; field: string } | undefined, + activeNodeFieldAppend: { node: BaseNode; field: string } | undefined, + file: string | undefined, + lineOffset: number | undefined, + walk: ListWalk, ): void { + const itemEnd = offsetsOf(item).end; const firstPara = item.children.find((c) => c.type === 'paragraph') as Paragraph | undefined; if (!firstPara) { - appendContent(contentTarget, `- ${mdastToString(item)}`); + claim(walk.claims, contentTarget, walk.pos, offsetsOf(item).start, itemEnd); + walk.pos = itemEnd; return; } @@ -234,11 +333,15 @@ function processListItem( if (wikiTarget) { const linkRef = `[[${wikiTarget}]]`; if (parentFieldAppend) { + consume(walk.claims, contentTarget, walk.pos, offsetsOf(item).start); appendParentField(parentFieldAppend.node, parentFieldAppend.field, linkRef); + walk.pos = itemEnd; return; } if (activeNodeFieldAppend) { + consume(walk.claims, contentTarget, walk.pos, offsetsOf(item).start); appendParentField(activeNodeFieldAppend.node, activeNodeFieldAppend.field, linkRef); + walk.pos = itemEnd; return; } // No field append context — fall through to node creation or content append @@ -281,13 +384,18 @@ function processListItem( ...(source ? { source } : {}), }; nodes.push(newNode); + const contentColumn = firstPara.position?.start.column; + if (contentColumn !== undefined) walk.claims.contentIndent.set(newNode, contentColumn - 1); if (parentFieldAppend) { appendParentField(parentFieldAppend.node, parentFieldAppend.field, `[[${linkTargets[0] ?? title}]]`); } const nestedParentRef = `[[${linkTargets[0] ?? title}]]`; + consume(walk.claims, contentTarget, walk.pos, offsetsOf(item).start); + walk.pos = offsetsOf(firstPara).end; for (const child of item.children) { + if (child === firstPara) continue; if (child.type === 'list') { for (const subItem of (child as List).children) { processListItem( @@ -304,13 +412,19 @@ function processListItem( undefined, file, lineOffset, + walk, ); } + } else { + const { start, end } = offsetsOf(child); + claim(walk.claims, newNode, walk.pos, start, end); + walk.pos = end; } } } else { - appendContent(contentTarget, `- ${rawText}`); + claim(walk.claims, contentTarget, walk.pos, offsetsOf(item).start, itemEnd); } + walk.pos = itemEnd; } export interface ExtractEmbeddedOptions { @@ -569,7 +683,30 @@ export function extractEmbeddedNodes(body: string, options: ExtractEmbeddedOptio let grouping: GroupingState | null = null; let activeNode: BaseNode = rootNode; + const claims: SourceClaims = { ranges: new Map(), gapKept: new Set(), contentIndent: new Map() }; + /** End of the previous top-level block; a block's span starts here so the blank lines before it stay with it. */ + let prevEnd = 0; + for (const child of tree.children) { + const { start: blockStart, end: blockEnd } = offsetsOf(child); + const spanStart = prevEnd; + prevEnd = blockEnd; + /** Give a node this block as source content, minus any `removed` ranges inside it. */ + const claimBlock = (node: BaseNode, removed: SourceRange[] = []): void => { + let from = blockStart; + let first = true; + for (const gap of removed) { + if (first) claim(claims, node, spanStart, blockStart, gap.start); + else addRange(claims, node, from, gap.start); + first = false; + from = gap.end; + } + if (first) claim(claims, node, spanStart, blockStart, blockEnd); + else addRange(claims, node, from, blockEnd); + }; + /** Take this whole block out of a node's content, keeping the blank lines before it. */ + const consumeBlock = (node: BaseNode): void => consume(claims, node, spanStart, blockStart); + if (parseState === 'done') { if (child.type === 'heading') { const rawTitle = mdastToString(child); @@ -588,7 +725,10 @@ export function extractEmbeddedNodes(body: string, options: ExtractEmbeddedOptio if (child.type === 'heading') { const heading = child as Heading; const depth = heading.depth; - if (depth > 5) continue; + if (depth > 5) { + if (parseState === 'active') claimBlock(activeNode); + continue; + } parseState = 'active'; @@ -610,6 +750,10 @@ export function extractEmbeddedNodes(body: string, options: ExtractEmbeddedOptio stack.push({ depth, title, nodeType: '', refTarget: title }); // Discard any pending grouping (untyped heading has no implied type) grouping = null; + // An untyped heading is part of the body of the nearest enclosing node. Popping above removed every + // node at the same or a higher level, so those sections end here; deeper nodes' sections continue. + activeNode = stack.findLast((entry) => entry.node)?.node ?? rootNode; + claimBlock(activeNode); continue; } @@ -655,7 +799,7 @@ export function extractEmbeddedNodes(body: string, options: ExtractEmbeddedOptio // Push to stack BEFORE resolving semantic parent — stack[-2] is the correct parent. const refTarget = linkTargets[0] ?? title; - stack.push({ depth, title, nodeType: type, refTarget }); + stack.push({ depth, title, nodeType: type, refTarget, node: headingNode }); // If this match came from a relationship/hierarchy and no explicit type was given, // create a grouping — delay adding the node until we see the following content. @@ -699,6 +843,7 @@ export function extractEmbeddedNodes(body: string, options: ExtractEmbeddedOptio ? { node: semanticParent.node, field: definition.field } : undefined; + const walk: ListWalk = { claims, pos: spanStart }; for (const item of list.children) { processListItem( item, @@ -714,10 +859,12 @@ export function extractEmbeddedNodes(body: string, options: ExtractEmbeddedOptio activeNodeFieldAppendArg, options.file, options.lineOffset, + walk, ); } grouping = null; } else { + const walk: ListWalk = { claims, pos: spanStart }; for (const item of list.children) { processListItem( item, @@ -733,6 +880,7 @@ export function extractEmbeddedNodes(body: string, options: ExtractEmbeddedOptio undefined, options.file, options.lineOffset, + walk, ); } } @@ -853,7 +1001,7 @@ export function extractEmbeddedNodes(body: string, options: ExtractEmbeddedOptio } grouping = null; } else { - appendContent(activeNode, mdastToString(child)); + claimBlock(activeNode); } } } else { @@ -867,7 +1015,7 @@ export function extractEmbeddedNodes(body: string, options: ExtractEmbeddedOptio if (child.type === 'paragraph') { const rawText = mdastToString(child); const { cleanText: afterBracketed, fields: bracketedFields } = extractBracketedFields(rawText); - const { remainingText, fields: unbracketedFields } = extractUnbracketedFields(afterBracketed); + const { fields: unbracketedFields } = extractUnbracketedFields(afterBracketed); const allFields = applyFieldMap({ ...unbracketedFields, ...bracketedFields }, fieldMap); if ('type' in allFields) { @@ -878,7 +1026,21 @@ export function extractEmbeddedNodes(body: string, options: ExtractEmbeddedOptio } Object.assign(activeNode.schemaData, allFields); - if (remainingText) appendContent(activeNode, remainingText); + // Lines holding only fields are consumed as fields; any other line, including prose that mentions a field, is content. + const lines = sourceLines(body, { start: blockStart, end: blockEnd }); + const fieldLines = lines.filter((l) => isFieldOnlyLine(l.text)); + if (fieldLines.length === lines.length) { + consumeBlock(activeNode); + } else { + // Each field line goes with its terminator; a last line without one takes the preceding line break instead. + claimBlock( + activeNode, + fieldLines.map((l) => { + if (/\n$/.test(body.slice(l.start, l.end))) return { start: l.start, end: l.end }; + return { start: l.start - (body[l.start - 2] === '\r' ? 2 : 1), end: l.end }; + }), + ); + } activeNode.contentLinks ??= []; activeNode.contentLinks.push(...extractLinksFromAstNode(child)); } else if (child.type === 'code' && (child as Code).lang?.trim() === 'yaml') { @@ -886,19 +1048,21 @@ export function extractEmbeddedNodes(body: string, options: ExtractEmbeddedOptio const parsed = parseYaml(code.value); if (parsed && !Array.isArray(parsed) && typeof parsed === 'object') { Object.assign(activeNode.schemaData, applyFieldMap(parsed as Record, fieldMap)); + consumeBlock(activeNode); } else if (Array.isArray(parsed)) { throw new Error(`YAML block must be an object at "${activeNode.label}".`); } else { - appendContent(activeNode, code.value); + claimBlock(activeNode); } } else { - appendContent(activeNode, mdastToString(child)); + claimBlock(activeNode); activeNode.contentLinks ??= []; activeNode.contentLinks.push(...extractLinksFromAstNode(child)); } } } + applyClaims(claims, body); const preambleContent = rootNode.schemaData.content as string | undefined; const { type: _rootType, content: _rootContent, ...preambleFields } = rootNode.schemaData; return { diff --git a/src/types.ts b/src/types.ts index 03fc923..65b6243 100644 --- a/src/types.ts +++ b/src/types.ts @@ -134,7 +134,7 @@ export type BaseNode = { linkTargets: string[]; /** Raw type string from content, as written by the user. */ type: string; - /** Body content of the node, if any. First-class accessor mirroring schemaData.content. */ + /** Markdown source of the node's body, if any. First-class accessor mirroring schemaData.content. */ content?: string; /** * Links extracted from content body and non-edge string fields at parse time. diff --git a/tests/fixtures/okf/known-limitations/computation-heading-in-fence.md b/tests/fixtures/okf/known-limitations/computation-heading-in-fence.md new file mode 100644 index 0000000..5b3025f --- /dev/null +++ b/tests/fixtures/okf/known-limitations/computation-heading-in-fence.md @@ -0,0 +1,12 @@ +--- +type: Attested Computation +description: d +runtime: python +--- + +The computation lives elsewhere, but this example mentions the heading in a code fence: + +```python +# Computation +x = 1 +``` diff --git a/tests/fixtures/okf/rule-violations/body-without-computation.md b/tests/fixtures/okf/rule-violations/body-without-computation.md new file mode 100644 index 0000000..df196f5 --- /dev/null +++ b/tests/fixtures/okf/rule-violations/body-without-computation.md @@ -0,0 +1,9 @@ +--- +type: Attested Computation +description: d +runtime: python +--- + +Some notes, but no computation section. + +## Computation notes diff --git a/tests/plugins/markdown/parse-embedded-content.test.ts b/tests/plugins/markdown/parse-embedded-content.test.ts new file mode 100644 index 0000000..b946362 --- /dev/null +++ b/tests/plugins/markdown/parse-embedded-content.test.ts @@ -0,0 +1,260 @@ +import { describe, expect, it } from 'bun:test'; +import { extractEmbeddedNodes } from '../../../src/plugins/markdown/parse-embedded'; +import type { HierarchyLevel } from '../../../src/types'; +import { makeLevel, withHierarchy } from '../../test-helpers'; + +const HIERARCHY: HierarchyLevel[] = [makeLevel('goal'), makeLevel('task', { templateFormat: 'list' })]; +const metadata = withHierarchy(HIERARCHY); + +describe('extractEmbeddedNodes - content is markdown source', () => { + it('keeps inline formatting, blocks and fences of a page preamble verbatim', () => { + const body = `Some **bold** text with a [link](https://example.com). + +- item one +- item two + +\`\`\`ts +const x = 1; +\`\`\` +`; + const { preambleContent } = extractEmbeddedNodes(body, { pageTitle: 'Page', pageType: 'goal', metadata }); + expect(preambleContent).toBe( + 'Some **bold** text with a [link](https://example.com).\n\n- item one\n- item two\n\n```ts\nconst x = 1;\n```', + ); + }); + + it('includes untyped headings in the enclosing node content', () => { + const body = `Intro. + +# Computation + + SELECT 1 + +## Detail + +More. +`; + const { preambleContent } = extractEmbeddedNodes(body, { pageTitle: 'Page', pageType: 'goal', metadata }); + expect(preambleContent).toBe('Intro.\n\n# Computation\n\n SELECT 1\n\n## Detail\n\nMore.'); + }); + + it('gives an embedded node the source under its heading, excluding nested embedded nodes', () => { + const body = `# Parent [type:: goal] + +Parent *text*. + +## Child [type:: goal] + +Child text with \`code\`. + +- [type:: task] Do it +- plain item +`; + const { nodes } = extractEmbeddedNodes(body, { pageTitle: 'Page', pageType: 'goal', metadata }); + expect(nodes.find((n) => n.title === 'Parent')?.content).toBe('Parent *text*.'); + expect(nodes.find((n) => n.title === 'Child')?.content).toBe('Child text with `code`.\n\n- plain item'); + expect(nodes.find((n) => n.title === 'Do it')?.content).toBeUndefined(); + }); + + it('returns an untyped heading to the enclosing node once the embedded section ends', () => { + const body = `# Child [type:: goal] + +Child text. + +# Notes + +Page level notes. +`; + const { nodes, preambleContent } = extractEmbeddedNodes(body, { pageTitle: 'Page', pageType: 'goal', metadata }); + expect(nodes.find((n) => n.title === 'Child')?.content).toBe('Child text.'); + expect(preambleContent).toBe('# Notes\n\nPage level notes.'); + }); + + it('gives a list item node the markdown under it, excluding nested embedded items', () => { + const body = `# Goal [type:: goal] + +### task + +- Write **docs** + + Extra detail for the task. + + - Sub task +`; + const { nodes } = extractEmbeddedNodes(body, { pageTitle: 'Page', pageType: 'goal', metadata }); + const task = nodes.find((n) => n.title === 'Write **docs**'.replace(/\*/g, '')); + expect(task?.content).toBe('Extra detail for the task.'); + expect(nodes.find((n) => n.title === 'Sub task')).toBeDefined(); + }); +}); + +describe('extractEmbeddedNodes - list item content is dedented', () => { + const task = (items: string) => `# Goal [type:: goal]\n\n### task\n\n${items}`; + const run = (body: string) => extractEmbeddedNodes(body, { pageTitle: 'Page', pageType: 'goal', metadata }); + const contentOf = (body: string) => run(body).nodes.find((n) => n.title === 'Write docs')?.content; + + it('dedents a continuation paragraph by the item content indent', () => { + expect(contentOf(task('- Write docs\n\n First para.\n\n Second para.\n'))).toBe( + 'First para.\n\n Second para.'.replace(' ', ''), + ); + }); + + it('dedents by the content column of an ordered item', () => { + expect(contentOf(task('10. Write docs\n\n Detail.\n'))).toBe('Detail.'); + }); + + it('keeps relative indent for a nested plain sub-list', () => { + expect( + contentOf('# Goal [type:: goal]\n\n- Write docs [type:: task]\n\n Detail.\n\n - plain one\n - plain two\n'), + ).toBe('Detail.\n\n- plain one\n - plain two'); + }); + + it('dedents a fenced code block and keeps extra indent inside it', () => { + expect(contentOf(task('- Write docs\n\n ```js\n if (x) {\n y();\n }\n ```\n'))).toBe( + '```js\nif (x) {\n y();\n}\n```', + ); + }); + + it('preserves CRLF line endings', () => { + expect(contentOf(task('- Write docs\r\n\r\n Line one.\r\n\r\n indented\r\n'))).toBe( + 'Line one.\r\n\r\n indented', + ); + }); + + it('leaves heading content unchanged', () => { + const body = '# Goal [type:: goal]\n\n## Child [type:: task]\n\n Indented text.\n'; + expect(run(body).nodes.find((n) => n.title === 'Child')?.content).toBe(' Indented text.'); + }); +}); + +const extract = (body: string) => extractEmbeddedNodes(body, { pageTitle: 'Page', pageType: 'goal', metadata }); +const contentOf = (body: string, title: string) => extract(body).nodes.find((n) => n.title === title)?.content; + +describe('extractEmbeddedNodes - content is contiguous source', () => { + it('keeps a loose list loose', () => { + expect(extract('- a\n\n- b\n\n- c\n').preambleContent).toBe('- a\n\n- b\n\n- c'); + }); + + it('keeps two separate lists separate, with their original blank lines', () => { + expect(extract('- a\n\n\n- b\n\ntext\n\n* c').preambleContent).toBe('- a\n\n\n- b\n\ntext\n\n* c'); + }); + + it('keeps CRLF line endings', () => { + const body = 'Para one\r\nline two\r\n\r\n# Computation\r\n\r\n SELECT 1\r\n'; + expect(extract(body).preambleContent).toBe('Para one\r\nline two\r\n\r\n# Computation\r\n\r\n SELECT 1'); + }); + + it('keeps code fences, including headings inside them, verbatim', () => { + const body = '```python\n# Computation\nx = 1\n\n\ny = 2\n```\n\nAfter.\n'; + expect(extract(body).preambleContent).toBe('```python\n# Computation\nx = 1\n\n\ny = 2\n```\n\nAfter.'); + }); + + it('keeps trailing whitespace inside the content and trims it from the end', () => { + expect(extract('line with trailing \nnext\t\n\n \n').preambleContent).toBe('line with trailing \nnext'); + }); + + it('gives a node with an empty body no content', () => { + const { nodes } = extract('# Child [type:: goal]\n\n# Other [type:: goal]\n\n \n'); + expect(nodes.map((n) => n.content)).toEqual([undefined, undefined]); + expect(extract('').preambleContent).toBeUndefined(); + }); + + it('keeps blank lines either side of a nested embedded node', () => { + const body = 'Before.\n\n- [type:: task] Do it\n\nAfter.\n'; + expect(extract(body).preambleContent).toBe('Before.\n\nAfter.'); + }); + + it('joins the pieces around a nested embedded list item from the original text', () => { + const body = '- plain one\n- [type:: task] Do it\n- plain two\n'; + expect(extract(body).preambleContent).toBe('- plain one\n- plain two'); + }); + + it('keeps a plain list item with a nested typed sub-item whole, as the sub-item is not embedded', () => { + const body = '# Child [type:: goal]\n\n- plain\n - sub [type:: task] T\n'; + const { nodes } = extract(body); + expect(nodes.map((n) => n.title)).toEqual(['Child']); + expect(nodes[0]?.content).toBe('- plain\n - sub [type:: task] T'); + }); +}); + +describe('extractEmbeddedNodes - inline field lines', () => { + it('strips field-only lines and paragraphs from content', () => { + const body = `# Child [type:: goal] + +owner:: alice + +[priority:: high] + +Prose line. +effort:: 3 +More prose. +`; + const child = extract(body).nodes[0]; + expect(child?.content).toBe('Prose line.\nMore prose.'); + expect(child?.schemaData).toMatchObject({ owner: 'alice', priority: 'high', effort: '3' }); + }); + + it('strips a last field line that has no terminator, with CRLF', () => { + const child = extract('# Child [type:: goal]\r\n\r\nProse.\r\nowner:: bob').nodes[0]; + expect(child?.content).toBe('Prose.'); + expect(child?.schemaData.owner).toBe('bob'); + }); + + it('keeps a field written mid-sentence as part of the prose', () => { + const child = extract('# Child [type:: goal]\n\nSet the [priority:: high] flag before launch.\n').nodes[0]; + expect(child?.content).toBe('Set the [priority:: high] flag before launch.'); + expect(child?.schemaData.priority).toBe('high'); + }); + + it('keeps blank lines either side of a stripped field paragraph and a YAML block', () => { + const body = '# Child [type:: goal]\n\nA.\n\nowner:: x\n\n```yaml\nsize: 3\n```\n\nB.\n'; + expect(extract(body).nodes[0]?.content).toBe('A.\n\nB.'); + }); +}); + +describe('extractEmbeddedNodes - untyped headings and section ownership', () => { + const body = (untyped: string) => `# Parent [type:: goal] + +Parent text. + +## Child [type:: goal] + +Child text. + +${untyped} + +Under untyped. +owner:: bob +`; + + it('ends the section of an embedded node at an untyped heading of the same level', () => { + const { nodes } = extract(body('## Notes')); + expect(nodes.find((n) => n.title === 'Child')?.content).toBe('Child text.'); + expect(nodes.find((n) => n.title === 'Child')?.schemaData.owner).toBeUndefined(); + const parent = nodes.find((n) => n.title === 'Parent'); + expect(parent?.content).toBe('Parent text.\n\n## Notes\n\nUnder untyped.'); + expect(parent?.schemaData.owner).toBe('bob'); + }); + + it('ends the section of an embedded node at an untyped heading of a higher level', () => { + const { nodes, preambleContent } = extract(body('# Notes')); + expect(nodes.find((n) => n.title === 'Child')?.content).toBe('Child text.'); + expect(nodes.find((n) => n.title === 'Parent')?.content).toBe('Parent text.'); + expect(preambleContent).toBe('# Notes\n\nUnder untyped.'); + expect(extract(body('# Notes')).nodes.find((n) => n.schemaData.owner)).toBeUndefined(); + }); + + it('keeps an untyped deeper heading inside the embedded node section', () => { + const { nodes } = extract(body('### Notes')); + const child = nodes.find((n) => n.title === 'Child'); + expect(child?.content).toBe('Child text.\n\n### Notes\n\nUnder untyped.'); + expect(child?.schemaData.owner).toBe('bob'); + expect(nodes.find((n) => n.title === 'Parent')?.content).toBe('Parent text.'); + }); + + it('keeps a heading deeper than the supported depth in the section content', () => { + expect(contentOf('# Child [type:: goal]\n\nText.\n\n###### Deep\n\nMore.\n', 'Child')).toBe( + 'Text.\n\n###### Deep\n\nMore.', + ); + }); +}); diff --git a/tests/validate/okf.test.ts b/tests/validate/okf.test.ts index 301ecc4..ed98a4b 100644 --- a/tests/validate/okf.test.ts +++ b/tests/validate/okf.test.ts @@ -6,6 +6,7 @@ import { makeSpaceContext } from '../helpers/context'; const OKF_SCHEMA_PATH = join(bundledSchemasDir, 'okf.json'); const BUNDLE_DIR = join(import.meta.dir, '../fixtures/okf/bundle'); +const KNOWN_LIMITATIONS_DIR = join(import.meta.dir, '../fixtures/okf/known-limitations'); const RULE_VIOLATIONS_DIR = join(import.meta.dir, '../fixtures/okf/rule-violations'); const validateNode = createValidator(OKF_SCHEMA_PATH); @@ -90,9 +91,17 @@ describe('OKF bundle validation', () => { const result = await validateSpace(makeSpaceContext(RULE_VIOLATIONS_DIR, OKF_SCHEMA_PATH)); expect(result.nodeErrors).toEqual([]); expect(result.ruleViolations.map((v) => `${v.file}:${v.ruleId}`).sort()).toEqual([ + 'body-without-computation.md:okf-computation-present', 'no-computation.md:okf-computation-present', 'stale-before-generated.md:okf-stale-after-generated', 'usage-without-window.md:okf-usage-count-window', ]); }); + + it('passes okf-computation-present on a `# Computation` line inside a code fence (known limitation)', async () => { + // The rule matches the heading text with a regex, so it cannot tell a fenced comment from a real heading. + const result = await validateSpace(makeSpaceContext(KNOWN_LIMITATIONS_DIR, OKF_SCHEMA_PATH)); + expect(result.nodeErrors).toEqual([]); + expect(result.ruleViolations).toEqual([]); + }); });