[Mikola Lysenko (mikolalysenko)]

LLM Description written by Claude Code:claude-opus-5-5

Fixes #1570

Summary

Hosted vlt scans did lock work once per patched package that only needs doing once per scan. This PR makes vlt/hosted 47% and vlt/rescan 50% faster on the scan bench, with the same rewrite results.

scenario	pkgs	main wall	this PR	Δ wall [95% CI]	Δ CPU	Δ peak RSS	requests
vlt/hosted	1500	482.4 ms	254.9 ms	-46.6% [-49.0, -46.2]	-45.5%	-2.5%	94 (unchanged)
vlt/rescan	1500	447.2 ms	222.6 ms	-50.1% [-50.7, -48.3]	-46.2%	-0.9%	95 (unchanged)

(socket-patch-bench compare --runs 25 -f '^vlt/', interleaved pairs on one 4-vCPU runner, base = origin/main @ aa69125.) The bench harness checks every run did the work the project calls for before keeping its timing, so the same lock edits are made on both sides.

Root cause

For each npm override, rewrite_dep (hosted rewrite) and preflight_scope (artifact preflight) walked every node of vlt-lock.json:

1. DepID re-splitting. partition_instances, the outside-registry check and has_vendored_node each called split_dep_id (segment decode, semver and registry-identity checks) on every node id, i.e. up to 3 full walks per override in the rewrite plus 2 in the preflight.
2. Full JSON re-parses. every_instance_pinned re-parsed the whole lock text into a serde_json::Value once per override in the preflight (on the unchanged lock) and once per override in the rewrite's residual gate (on the spliced candidate). With the index in place, callgrind showed this gate alone at 35% of all instructions (27.8% parse + 7.8% dropping the tree).

Fix

• NodeIndex (built once in parse_hosted_lock): every node id is split once; registry nodes are indexed by (name, version) and git/file/remote nodes by slot [1]. partition_instances, outside_instances (new, extracted), has_vendored_node and the lock-level discarding_lock_codes read from it. Lock key order is kept (positions ascend), and ids that don't split are left out, exactly as before (they matched nothing).
• Preflight checks existing pins with already_pinned against the lock it already parsed (sniff_lock uses the same serde_json::from_str) instead of parsing the text again.
• Residual gate: parse_nodes_of parses the whole candidate with a serde visitor that keeps only the nodes being checked and skips (IgnoredAny) everything else. Broken JSON anywhere is still refused, and the last nodes / last duplicate key still win, as with Value. Skipping does not range-check numbers or pair \u surrogates in skipped values; the candidate is a lock sniff_lock already parsed whole, with only node lines re-rendered, so this can't change a verdict (documented on the function).

Tests

• the_node_index_matches_a_scan_of_every_node: on a mixed lock (both DepID eras, a foreign named registry, peer extras, git/remote/file/vendored/workspace nodes, an undecodable id), the index gives the same defaults, foreign, outside and vendored answers in the same order as the old per-node scan, for 4 identities.
• the_gate_parse_matches_a_full_value_parse: 23 documents (duplicate nodes, duplicate keys, escaped keys, non-object nodes of every JSON type, broken JSON elsewhere in the document, trailing garbage, trailing comma, invalid escape, truncated, non-object top level, empty) give the same result as the old Value parse.
• /code-review high: no correctness findings. Taken: the gate is now private to the rewrite (its skipping parse is only sound on sniffed text) and the index no longer copies slot [1]. Not taken, with reason: avoiding the per-override line clone/join in the gate (the join is needed for the whole-document check that catches duplicate nodes sections), reusing the index's split for the ledger extra and allocation-free (name, version) lookups (one split / two small allocations per instance, not per node), and sharing one HostedLock between preflight and rewrite (a wider API change across hosted::vlt::plan).
• The existing hosted vlt tests (including residual_gate_refuses_when_the_parsed_node_is_not_the_spliced_line) pass unchanged.

For a perf fix, the red→green evidence is the bench table above (main vs. this branch, same machine, interleaved).

Local runs

• cargo fmt --all -- --check: ok
• cargo clippy --workspace --all-features -- -D warnings: ok
• cargo test --workspace --all-features: 234 suites ok. 10 tests in 4 targets fail only because this sandbox runs as root: they chmod a dir to 0o555 and expect a write or remove to fail (vlt_heal unremovable hidden lock, copy_tree symlinked root, poetry/requirements wire-write failure, in_process_redirect write failures, repair unremovable lock). The other one, e2e_bun_lockb::binary_shared_bundled_record_hosted_pin_is_managed, fails the same way with this PR's patch/redirect/ reverted to main, so it is environmental too. None of these run vlt hosted code that this PR changes.
• Real-vlt e2e (e2e_redirect_vlt_build, vlt 1.2.0): not runnable here. Its harness fetches packages from npmjs with reqwest's webpki roots, which don't trust this sandbox's TLS proxy. CI's vlt leg covers it: e2e (ubuntu-latest, e2e_redirect_vlt_build, 1.2.0) passed on c4aa9f1.

Checklist

• Slow scan: vlt hosted 2.6x median ms/pkg (vlt::partition_instances / split_dep_id per patch) #1570: vlt hosted per-patch DepID splitting and lock re-parsing removed (NodeIndex, already_pinned, parse_nodes_of); bench -47% / -50%.

No wrapper (npm/, pypi/, gem/) changes are needed: this is internal to the core hosted rewriter.

https://claude.ai/code/session_01FKVp3SxTz9Nyp97dCtuR2r

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c4aa9f1. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Labeled Ready for review (burn-down agent).

• Head: c4aa9f1
• CI: 54/54 settled green on this head (35 success / 19 skipped, including the real-vlt e2e leg). No main-wide failures affect it.
• Bugbot: reviewed c4aa9f1, no findings; no open review threads.
• Mergeable with main; no CHANGELOG.md change.
• For the reviewer: perf-only change to the hosted vlt rewrite/preflight (Slow scan: vlt hosted 2.6x median ms/pkg (vlt::partition_instances / split_dep_id per patch) #1570). Look at parse_nodes_of in vlt.rs, the serde visitor that skips non-checked nodes in the residual gate. Its soundness rests on the candidate having been fully parsed by sniff_lock first (documented on the function).
• Slack: not announced. The Slack send tool isn't available in this run, so the next run will retry.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Final review brief (head c4aa9f1)

What it does: Hosted vlt scans re-split every vlt-lock.json node id and re-parsed the whole lock once per patched package (#1570). This builds a NodeIndex once per scan (registry nodes by (name, version), git/file/remote nodes by slot [1]), checks existing pins against the already-parsed lock, and makes the per-override residual gate a skipping serde parse that only builds the nodes it checks. Bench: vlt/hosted -47%, vlt/rescan -50%, same lock edits and request counts.

Risk: low. Perf-only refactor of internal functions in vlt.rs with no API or behavior change. I traced each old per-node scan against its indexed replacement: installs_outside_registry returns false unless slot [1] equals the name and the kind is git/file/remote, so keying non_registry by slot [1] drops nothing. registry_identity is None for non-registry kinds, and the vendored check is unchanged. The preflight's already_pinned reads the same serde_json::Value that sniff_lock built from the same text.

Look here:

• vlt.rs:77 NodeIndex and vlt.rs:166-196 partition/outside_instances
• vlt.rs:383 parse_nodes_of: the skipping visitor. Its soundness rests on the candidate being a sniffed lock with only node lines re-rendered (documented)
• vlt.rs:631 residual gate call and vlt_preflight.rs:213 already_pinned
• Equivalence tests: vlt.rs:882 (index vs. full scan, 4 identities on a mixed lock) and vlt.rs:972 (23 documents vs. a Value parse, including duplicate keys and broken JSON)

Verified: I read the full diff. cargo test -p socket-patch-core --lib vlt gave 204 passed, 1 failed: vlt_heal::an_unremovable_hidden_lock_keeps_every_store_entry, which fails only because this sandbox runs as root (chmod 0o555 is not enforced). cargo clippy -p socket-patch-core --all-targets with the local 1.93.1 toolchain reports nothing in the two touched files. CI: 54/54 settled green on c4aa9f1 (35 success / 19 path-skipped, including the real-vlt e2e leg); ci-ok and clippy are green. Bugbot reviewed c4aa9f1 with no findings. There are no open threads, CHANGELOG.md is untouched, and the PR merges cleanly into main aa69125.

Changes I made: none.

Open questions: none.

Auto-merge (squash) is armed, so approving sends this PR straight to the merge queue.

---

Generated by Claude Code