Repository navigation
Conversation
S_out decoded every code point, re-encoded it and appended it to the buffer one at a time, even when the renderer writes it back unchanged (LITERAL escaping, or the plaintext renderer whose outc is the identity). Runs of such characters are now validated and copied with a single put. Spaces when wrapping, newlines in LITERAL, invalid UTF-8 and extensions with a commonmark_escape_func still take the per-character path, and a run never starts once a break opportunity is pending, so wrapping is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cmark_utf8proc_encode_char writes these two code points as a single byte, so copying their original three bytes changed the output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
|
A correction to the description: the output was not byte-identical for U+FFFE and U+FFFF. One more thing I should have said upfront: the 2x is at width 0, which is markly's default for |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi, thanks for maintaining cmark-gfm. We use it through the Ruby gem markly, and
to_plaintextshowed up in our profiles.S_outinrender.cdecodes every code point, hands it to the renderer'soutc, which re-encodes it, and appends it to the buffer one at a time. The plaintext renderer'soutcis the identity, and every renderer writesLITERALtext back unchanged too, so most of that work rebuilds the bytes it started from. This PR letsS_outvalidate a run of such characters and copy it with a singlecmark_strbuf_put, updatingcolumn,begin_lineandbegin_contentthe way the per-character loop would. Spaces when wrapping, newlines inLITERAL, invalid UTF-8 and extensions with acommonmark_escape_funcstill take the existing path, and a run never starts once a break opportunity is pending, so wrapping is unchanged.plaintext.copts in by passing aNULLoutctocmark_render;render.his internal and not installed, so the public API and ABI stay the same.Parsing then rendering to plain text with
CMARK_OPT_DEFAULTgoes from 36.9 µs to 18.7 µs per text (1.97x) on 916 real texts from our app (French product descriptions and posts, median 927 bytes), and from 384k to 163k instructions per text (−58%). On 1,084 Markdown files taken from 265 Ruby gems (READMEs, changelogs, docs) it goes from 1.85M to 1.10M instructions per text (−41%). Instructions retired come from/usr/bin/time -l, median of 5; times are the median of 5 interleaved rounds; Apple M5, macOS, Apple clang 21.0.0,CMAKE_BUILD_TYPE=Release,masterat 27d942c (0.29.0.gfm.13).The output is byte-identical to
master: 66,832 inputs (the 916 real texts, 60,000 random soups of Markdown tokens, 5,000 random byte strings, whitespace-mangled copies of the real texts) through 5 option and extension sets, rendered to plaintext, commonmark, latex and man at widths 0, 1, 5, 8, 30 and 80 plus HTML, along with API-built text, code and code block nodes holding invalid UTF-8: 8,354,225 outputs with the same SHA-256 on both sides. A mutant that miscounts the column of long runs changes that hash, so the harness does reach the new path.ctestpasses. The newapi_testcase covers wrapping inside a list item, multibyte characters and a code block's prefix; it passes onmastertoo, since the change is meant to be invisible.The tradeoff is that
S_outnow keeps its bookkeeping in two places, so a later change to the per-character loop has to be mirrored in the run path. I kept that path to about 30 lines right next to the loop to make it hard to miss.The same commit is proposed to markly's fork in socketry#7 and socketry/markly#35. Would you be open to taking it here as well?
🤖 Generated with Claude Code, but prompted, read, and verified by @czj, a caring human :)