Skip to content

perf(resolution): strip comments by slices, not an array of characters - #3147

Merged
colbymchenry merged 1 commit into
mainfrom
claude/spd-strip-slices
Oct 31, 2026
Merged

colbymchenry merged 1 commit into
mainfrom
claude/spd-strip-slices

Conversation

@colbymchenry

Copy link
Copy Markdown
Owner

Each stripper split a file into an array of one-character strings, blanked
comments in it and joined it back: 8 bytes a character and two passes, for every
file, on every resolver thread, for each of the several memos that read stripped
text. BlankedText keeps the ranges and copies what's between them by the slice.

Stripping 3,772 files of ten repos: 1,142 ms to 359 ms. django subset: CPU 21.6 s
to 20.7 s, peak RSS 1,412 to 1,302 MB.

Output matches the old strippers byte for byte on 48,283 inputs: those files in
their own language, a sample of them in every other language, and 40,000 random
cuts and splices that leave strings and comments open.

Validation

Adds offset tests for every shape of comment to strip-comments.test.ts; strip-cstyle-differential.test.ts already pins the old output for C-style.

  • Graphs: scripts/dump-graph.mjs dumps byte-identical to main on 14 repos: tokio, gin, flask, jsoup, excalidraw, serilog, ripgrep, taxonomy, and subsets of django, rocksdb, elasticsearch (two), rails and laravel. That's with all of this stack's commits in, on one build.
  • Full suite on the stack tip 38c97bd0, which holds all nine commits of this work on a0174ce (kernel built from that tree, CODEGRAPH_KERNEL_EXPECT=1, 4 workers, Vitest 4): 848 files passed, 6 skipped; 10,515 tests passed, 2 expected fails, 87 skipped, 0 failed.
  • Numbers are interleaved on a machine other agents were using, so read the ratios. REG's quiet-machine run gives the final ones.

Part of the work to bring main back to 1.6.2's speed. Independent of the other PRs of it: it compiles alone on main.

🤖 Generated with Claude Code

Each stripper split a file into an array of one-character strings, blanked
comments in it and joined it back: 8 bytes a character and two passes, for every
file, on every resolver thread, for each of the several memos that read stripped
text. BlankedText keeps the ranges and copies what's between them by the slice.

Stripping 3,772 files of ten repos: 1,142 ms to 359 ms. django subset: CPU 21.6 s
to 20.7 s, peak RSS 1,412 to 1,302 MB.

Output matches the old strippers byte for byte on 48,283 inputs: those files in
their own language, a sample of them in every other language, and 40,000 random
cuts and splices that leave strings and comments open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@colbymchenry
colbymchenry merged commit f76b468 into main Oct 31, 2026
@colbymchenry
colbymchenry deleted the claude/spd-strip-slices branch October 31, 2026 19:19
colbymchenry added a commit that referenced this pull request Nov 16, 2026
…aracters (#3312)

blankStringContents split every text it blanked into an array of one-character
strings and joined it back, the cost #3147 took out of the comment strippers:
8 bytes a character and two passes, for every file a reader blanks, on every
resolver thread. It now copies the text between blanked runs by the slice, as
BlankedText does, and the JS scope reader blanks regex-literal contents the
same way instead of splitting the blanked file again.

The output is byte-identical: on the 11,717 JS/TS/Vue/Svelte files of eight
repos (69 MB), stripped and raw, the old and new versions agree on every text
and every regex-literal offset; a differential test keeps the old version as
its oracle. Blanking that corpus: 4,437 ms to 1,001 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant