Repository navigation
perf(resolution): strip comments by slices, not an array of characters - #3147
Merged
Merged
Conversation
Each stripper split a file into an array of one-character strings, blanked comments in it and joined it back: 8 bytes a character and two passes, for every file, on every resolver thread, for each of the several memos that read stripped text. BlankedText keeps the ranges and copies what's between them by the slice. Stripping 3,772 files of ten repos: 1,142 ms to 359 ms. django subset: CPU 21.6 s to 20.7 s, peak RSS 1,412 to 1,302 MB. Output matches the old strippers byte for byte on 48,283 inputs: those files in their own language, a sample of them in every other language, and 40,000 random cuts and splices that leave strings and comments open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
colbymchenry
added a commit
that referenced
this pull request
Nov 16, 2026
…aracters (#3312) blankStringContents split every text it blanked into an array of one-character strings and joined it back, the cost #3147 took out of the comment strippers: 8 bytes a character and two passes, for every file a reader blanks, on every resolver thread. It now copies the text between blanked runs by the slice, as BlankedText does, and the JS scope reader blanks regex-literal contents the same way instead of splitting the blanked file again. The output is byte-identical: on the 11,717 JS/TS/Vue/Svelte files of eight repos (69 MB), stripped and raw, the old and new versions agree on every text and every regex-literal offset; a differential test keeps the old version as its oracle. Blanking that corpus: 4,437 ms to 1,001 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each stripper split a file into an array of one-character strings, blanked
comments in it and joined it back: 8 bytes a character and two passes, for every
file, on every resolver thread, for each of the several memos that read stripped
text. BlankedText keeps the ranges and copies what's between them by the slice.
Stripping 3,772 files of ten repos: 1,142 ms to 359 ms. django subset: CPU 21.6 s
to 20.7 s, peak RSS 1,412 to 1,302 MB.
Output matches the old strippers byte for byte on 48,283 inputs: those files in
their own language, a sample of them in every other language, and 40,000 random
cuts and splices that leave strings and comments open.
Validation
Adds offset tests for every shape of comment to
strip-comments.test.ts;strip-cstyle-differential.test.tsalready pins the old output for C-style.scripts/dump-graph.mjsdumps byte-identical to main on 14 repos: tokio, gin, flask, jsoup, excalidraw, serilog, ripgrep, taxonomy, and subsets of django, rocksdb, elasticsearch (two), rails and laravel. That's with all of this stack's commits in, on one build.CODEGRAPH_KERNEL_EXPECT=1, 4 workers, Vitest 4): 848 files passed, 6 skipped; 10,515 tests passed, 2 expected fails, 87 skipped, 0 failed.Part of the work to bring main back to 1.6.2's speed. Independent of the other PRs of it: it compiles alone on main.
🤖 Generated with Claude Code