Skip to content

Deepen Markdown document module; deprecate offset-based parser helpers - #98

Merged
scalvert merged 2 commits into
mainfrom
refactor/markdown-document
Sep 30, 2026
Merged

scalvert merged 2 commits into
mainfrom
refactor/markdown-document

Conversation

@scalvert

@scalvert scalvert commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Summary

Introduces a MarkdownDocument module that owns every read and edit of a Markdown/MDX file's code blocks. sync, check, extract, discovery, and eject now go through it instead of each re-deriving fence lines from raw offsets.

Recent fixes show why: #82 (eject metadata) and #86 (line endings) each had to patch three separate splicers — replaceCodeBlock, extract's annotation insert, and eject's directive removal. Those now live in one place.

const doc = MarkdownDocument.parse(content, { mdx: isMdxPath(path) });
for (const block of doc.codeBlocks) {
  if (block.directive) doc.setBody(block, resolved);
}
doc.toString();
  • src/markdown-document.ts — parse → codeBlocks (language, content, directive?, line, column, lineEnding) → setBody / setDirective → toString(). Edits are queued and applied bottom-up; offsets never leave the module. Strings only — callers do file I/O.
  • src/snippet-directive.ts — parseSnippetDirective (moved; fix: validate snippet directives and ranges #85's quote-aware scanner), formatSnippetDirective (canonical #L10, #L10-, #L10-L20; quotes paths containing whitespace), and findSnippetDirectiveToken, which the document uses so fence edits touch exactly the token the parser read.
  • src/line-endings.ts — getLineEnding / normalizeLineEndings moved out of parser.ts (still re-exported there) to avoid an import cycle.
  • CONTEXT.md — domain glossary: Markdown document, Code block, Snippet directive, Snippet.

Behaviour changes

  • eject removes quoted directives (snippet="a b.ts") whole; previously the regex left b.ts" behind.
  • extract quotes generated references when the markdown filename contains spaces, so they parse back.
  • setBody only keeps a closing fence that matches the opening fence (same character, at least as long); an unclosed block no longer treats its last content line as the closing fence.
  • check's out-of-sync message now reads snippet=a.ts#L10 (open-ended: #L10-) instead of snippet://a.ts#L10-L10. Pinned by a new integration test.

Deprecations (no breaking change)

parseMarkdownFile, parseMarkdownForExtraction, replaceCodeBlock, and the CodeBlock / MarkdownFile types are @deprecated thin wrappers over the new module, to be removed in 2.0. New public exports: MarkdownDocument, isMdxPath, formatSnippetDirective, DocumentCodeBlock, ParseOptions.

Tests

  • New test/markdown-document.test.ts: string-in/string-out cases — md and MDX, list/JSX indentation, ~~~ and ```` fences, meta preservation, LF/CRLF bodies, quoted directives, empty and unclosed blocks, multi-edit, add→remove round-trip.
  • parser.test.ts: parse/replace suites collapsed to one smoke test per deprecated wrapper; fix: preserve markdown line endings #86's CRLF case moved to the document tests.
  • snippet-directive.test.ts: format + parse round-trip cases.
  • cli.test.ts: eject end-to-end (piped confirmation) and init --extract → eject byte-for-byte round trip.
  • integration.test.ts: pins the out-of-sync message.
  • Existing integration, CLI, eject, extract, and MDX suites pass unchanged.

npm run lint, npm run typecheck, npm run build, npm test (254 passed) all green locally.

  • I have run npm run lint and fixed any issues.
  • I have run npm run build to verify the project builds.
  • I have run npm test to ensure all tests pass.

Notes

  • MarkdownFile stays a separate @deprecated interface rather than an alias of MarkdownDocument: the wrappers still return the old shape, and an alias would be breaking.
  • DocumentCodeBlock.lineEnding is exposed so sync/extract can normalise snippet content to the block (fix: preserve markdown line endings #86).
  • createMarkdownProcessor (never exported from index.ts) is gone from parser.ts.

Follow-ups (not in this PR)

  • Directive edits scan the raw fence line while parsing reads remark's decoded meta; they can disagree only for backslash escapes / entities inside the directive.
  • Body re-indentation assumes column - 1 spaces, so fences inside blockquotes or tab-indented lists are not re-prefixed correctly (pre-existing on main).

Introduce MarkdownDocument: parse a Markdown/MDX string once, expose its
code blocks (language, content, directive, line, column, lineEnding),
queue setBody/setDirective edits, and serialise with toString(). Fence
handling (marker, indentation, meta, line endings) now lives in one
place instead of three offset splicers in sync, extract, and eject —
each of which recent fixes (#82, #86) had to patch separately.

Move the snippet directive grammar into snippet-directive.ts:
parseSnippetDirective (moved, quote-aware scanner from #85),
formatSnippetDirective (canonical output, quotes paths with whitespace),
and findSnippetDirectiveToken, which the document uses so fence edits
touch exactly the token the parser read. Line-ending helpers move to
line-endings.ts and remain re-exported from parser.ts.

Behaviour fixes that fall out of the shared grammar:
- eject removes quoted directives (snippet="a b.ts") whole instead of
  leaving a dangling fragment
- extract quotes generated references whose directory contains spaces,
  so they parse back correctly
- setBody no longer treats the last content line of an unclosed fence
  as its closing fence
- check's out-of-sync message uses the directive form
  (snippet=a.ts#L10) instead of snippet://a.ts#L10-L10

parseMarkdownFile, parseMarkdownForExtraction, replaceCodeBlock, and the
CodeBlock/MarkdownFile types are deprecated wrappers over the new module
and will be removed in 2.0.

Adds CONTEXT.md with the project's domain glossary.
@scalvert scalvert added enhancement New feature or request internal labels Sep 30, 2026
- Only keep a closing fence in setBody when it matches the opening
  fence's character and is at least as long, so an unclosed ~~~ block
  ending in ``` no longer loses content.
- setDirective is a no-op when the directive is unchanged (including
  after an intermediate edit), so fence lines are rewritten only when
  their directive actually changes.
- formatSnippetDirective falls back to single quotes when the reference
  contains a double quote, and rejects references containing both, so
  formatted directives always parse back.
- Move readMarkdownDocument into parser.ts; the deprecated wrappers and
  eject reuse it instead of a duplicate helper and a sync.js import.
- Rename the resolved directive local in sync.ts to match CONTEXT.md.
- Pin check's out-of-sync message in an integration test.
- Update CONTRIBUTING.md project structure for the new modules.
@scalvert

Copy link
Copy Markdown
Owner Author

Second-pass review

Two independent passes: Standards (AGENTS.md / CONTRIBUTING.md + smell baseline) and Spec (the agreed design for this refactor).

Fixed in 398df33

  • Closing-fence detection ignored the opening fence's character/length; an unclosed ~~~ block ending in ``` would keep that line as its "closing" fence and then never re-sync. Now requires same character and ≥ length.
  • setDirective with an unchanged directive still rewrote the fence into canonical form (spec: rewrite "only when their directive changes"). Now a no-op, including after an intermediate edit.
  • formatSnippetDirective could emit an unparseable directive for paths containing ". Falls back to single quotes; throws if both quote kinds appear.
  • Duplicated read helper / odd import: readMarkdownDocument moved into parser.ts; wrappers and eject reuse it.
  • Untested user-visible output: check's out-of-sync message is now pinned in integration.test.ts.
  • Docs: CONTRIBUTING.md project structure lists the new modules.
  • Naming: resolved-directive local in sync.ts renamed to match CONTEXT.md.

Accepted as-is

  • MarkdownFile kept as a deprecated interface, not an alias (an alias would break the wrappers' return type).
  • lineEnding exposed on DocumentCodeBlock — needed after fix: preserve markdown line endings #86 for snippet normalisation.
  • Module-level WeakMap of offsets exists only for the deprecated wrappers; goes away in 2.0.

Deferred (pre-existing or edge-case)

  • Raw fence line vs remark-decoded meta can disagree for backslash escapes / HTML entities inside a directive.
  • Re-indentation assumes column - 1 spaces; blockquote and tab-indented fences aren't re-prefixed (same on main).

CI green on Node 22 and 24; 254 tests pass locally.

@scalvert
scalvert merged commit 39c600b into main Sep 30, 2026
2 checks passed
@scalvert
scalvert deleted the refactor/markdown-document branch September 30, 2026 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request internal

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant