Skip to content

Parse bun.lock once per scan (#1606) - #1700

Merged
Mikola Lysenko (mikolalysenko) merged 4 commits into
mainfrom
agent/fix-bun-store-liveness-walk-cost
Oct 23, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 4 commits into
mainfrom
agent/fix-bun-store-liveness-walk-cost

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

LLM Description written by Claude Code:claude-opus-5-5

Fixes #1606

Summary

A Bun scan parsed the same bun.lock three times: once in the lock inventory, once in VEX discovery and once in the hosted redirect. The vendored preflight also parsed it once per patched package. On the bench's 3000-package lock, each parse costs about 29M instructions, more than half of what the .bun store walk (#599) added. The readers now share one parse of an unchanged lock text. bun-isolated/rescan is 8–14% faster than main and back to the issue's base cc938aa6.

Root cause

Callgrind of bun-isolated/rescan on the bench fixture:

build instructions
cc938aa6 (issue base) 524.1M
main 3114a28 570.0M
this PR 500.5M

The +46M from cc938aa6 to main is the .bun store liveness walk (#599, live_bun_store_entries_sync / read_bun_store_node_modules_sync), as #1693 suspected. That walk is already lean: most of its cost is the per-entry node_modules listings the orphan check needs, and cutting them would weaken #599. The largest avoidable cost, on both builds, is older: BunTextLock::parse running 3× per scan, about 16% of all instructions. It also explains the non-isolated bun/hosted regression in the issue.

Fix

  • formats::bun::BunTextLock::parse_shared keeps the last lock it parsed (Arc<BunTextLock>). A text whose split('\n') lines equal that lock's lines gets the same parse back. Any other text is parsed again, so a changed lock never reads stale. A refused lock isn't kept. No copy of the text is kept either; the match is checked against the lines the parse already holds.
  • Readers using it: lock_inventory::bun::bun_text_entries (inventory and VEX discovery), parse_bun_hosted_lock (hosted redirect and hosted restore), and the vendored preflight in vendor/bun_lock.rs.
  • The hosted redirect never edits the shared parse. It splices through a Cow of the lines, which copies them only on the first edit. The hosted restore always edits, so it copies up front.

Benchmarks

socket-patch-bench compare, 20 interleaved pairs on this sandbox (4 vCPU). Base is main 3114a28:

scenario base wall head wall Δ wall [95% CI] Δ CPU verdict
bun/hosted 116.7 ms 107.5 ms -7.5% [-13.7, -1.4] -7.6% ≈
bun/rescan 93.6 ms 86.3 ms -8.5% [-12.2, -2.7] -8.0% ≈
bun-isolated/hosted 144.6 ms 132.9 ms -7.3% [-12.1, -0.8] -3.9% ≈
bun-isolated/rescan 126.1 ms 110.0 ms -13.6% [-18.2, -5.0] -7.7% ✅ faster
npm/rescan, pnpm/rescan, bundler/* (controls) -3.2% … +0.3% ≈

Against the issue's base cc938aa6, bun-isolated/rescan went from +21.3% to +5.3% [+0.6, +9.0] in the paired compare (bench run medians: 116.3 ms base, 117.0 ms this PR). bun/hosted and bun/rescan are at -0.1% and -0.4%.

Peak RSS, from socket-patch-bench run medians of 15 runs: this PR adds about 0.7 MiB over main on the Bun rescans (+2%) and nothing on npm, because the last lock's parse stays alive. The compare table's RSS column moved ±7–10% even on npm/pnpm, which never read bun.lock, so this run's RSS deltas are noise. main is already about 3.3 MiB above cc938aa6 on npm/rescan as well as on Bun, so that growth isn't Bun-specific (see #1607's RSS slice).

Tests

case test before after
same text → one parse formats::bun::tests::same_text_is_parsed_once n/a (new API) pass
changed text → parsed again, never stale changed_text_is_parsed_again n/a pass
prefix, extra newline, CRLF → a different text a_prefix_or_extension_is_another_text n/a pass
a refusal isn't kept, last good parse survives refusals_are_not_kept n/a pass
shared parse = plain parse shared_parse_reads_like_the_parse n/a pass
#1606 regression socket-patch-bench tables above +21.3% vs cc938aa6 +5.3% (within the 10% gate)

Scan output is unchanged: the JSON of bun-isolated/rescan is byte-identical between main and this PR, and every bench scenario passes the harness's validation.

Local runs:

  • cargo fmt --all -- --check and cargo clippy --workspace --all-features -- -D warnings: clean.
  • cargo test -p socket-patch-core --all-features: 6705 lib tests pass, plus every integration test binary. Four lib tests fail the same way on main in this sandbox: they are permission-denied tests, and the sandbox runs as root so chmod can't make writes fail (copy_tree, vlt_heal, pypi_poetry, pypi_requirements).
  • cargo test -p socket-patch-cli --all-features --lib: 933 pass.
  • -- --include-ignored on e2e_redirect_bun_build, e2e_vendor_bun_build, mode_migration_bun, in_process_vendor_bun_takeover, vendor_eject_bun_lockb, e2e_vex_redirect and e2e_vex_vendor, with real Bun 1.4.4: all pass.
  • e2e_bun_lockb: 22 pass, 2 fail identically on main here. binary_shared_bundled_record_hosted_pin_is_managed needs patches-api.socket.dev. workspace_text_migration_heals_on_rerun asserts Bun's own bun.lockb → bun.lock migration output, which local Bun 1.4.4 writes differently from CI's pinned 1.4.2.

CI on cdbe831: every check passed or was skipped by the PR tier, including ci-ok, coverage, clippy and the Bun 1.4.2 e2e leg. Cursor Bugbot found no issues.

Review notes (/code-review high)

Fixed in cdbe831: the vendored preflight parsing once per package, the redirect always cloning the lines, and the extra BunEntries wrapper.

Not fixed:

  • The memo is process-wide and never cleared. It holds at most one lock's parse (about 0.7 MiB at 3000 packages) for the life of the process. A per-scan cache on DiskSnapshot would free it at the end of a scan, but the redirect and the vendored preflight don't receive the snapshot, so that means threading it through those call paths.
  • One slot, so alternating locks miss. A hosted restore over several member bun.lock files parses each one, as before. The added line comparison stops at the first differing line, and member locks differ in the workspaces block near the top.
  • Refusals aren't kept. A lock bun's grammar refuses is rare and is still parsed by each reader, as before. Keeping it would need the text to recognize it again, which is the memory this PR avoids.

The npm, PyPI and gem wrappers only dispatch the binary, so they need no change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YB4wa2mtpEejKGP7bZ3NZF

Assisted-by: Claude Code:claude-opus-5-5
A Bun scan read the same bun.lock three times (the lock inventory, VEX
discovery and the hosted redirect), and on a 3000-package lock each
parse cost more than the .bun store walk #599 added. Those readers now
share one parse of an unchanged text, so Bun rescans and hosted scans
spend less CPU. A changed lock is always parsed again.

Refs #1606

Assisted-by: Claude Code:claude-opus-5-5
@mikolalysenko Mikola Lysenko (mikolalysenko) changed the title Fix bun-isolated rescan cost of the .bun store walk (#1606) Parse bun.lock once per scan (#1606) Oct 23, 2026
The shared bun.lock parse kept a copy of the whole text to recognize
it, which added to a scan's peak memory. It now checks a text against
the lines it already holds, and does not keep a refused lock.

Refs #1606

Assisted-by: Claude Code:claude-opus-5-5
Address review: the readers now get the shared parse itself instead of
a wrapper, the hosted redirect copies the lock's lines only when it
splices one, and the vendored preflight, which read bun.lock once per
patched package, uses the shared parse too.

Refs #1606

Assisted-by: Claude Code:claude-opus-5-5
@mikolalysenko
Mikola Lysenko (mikolalysenko) marked this pull request as ready for review October 23, 2026 21:53
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

BugBot review


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit cdbe831. Configure here.

Merged via the queue into main with commit 0b30e37 Oct 23, 2026
86 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the agent/fix-bun-store-liveness-walk-cost branch October 23, 2026 22:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Perf regression: bun-isolated/rescan wall +21% (cc938aa6..17793173; #1279 candidate)

3 participants