Compare purls through one PurlKey type

[Mikola Lysenko (mikolalysenko)]

Problem

Architecture audit §3.B (purl equality). At least six relations answered "do these two purls name the same package release", and they disagreed:

• purl_eq folded percent-encoding and composer only;
• purl_identity_key stripped qualifiers and did composer only;
• two different functions were both named canonical_base_purl (one in vex::discover and one in rollout);
• policy::canon and package_spec_matches each had their own folding;
• the ledger had five ownership matchers with different canonicalization (covers_purl, purl_keys_cover, lookup_entry, apply's Go --check set, the hosted_pin_of / hosted_vendored_overlap keys);
• there were several local normalize_purl(strip_purl_qualifiers(..)) closures used as keys.

The canonical_purl doc comment also claimed an identity it doesn't provide.

User-visible effects:

• B20: scan --prune / --sync dropped a live manifest entry, and GC'd its blobs, when the API spelled the purl pkg:nuget/Newtonsoft.Json@13.0.3 and the NuGet global-cache crawl reported pkg:nuget/newtonsoft.json@13.0.3. Update detection and redirect-candidate matching missed the same entry. docker_e2e_nuget.rs worked around this by serving a lowercase purl from the fixture. A PEP 503 spelling (typing_extensions vs typing-extensions) was pruned the same way.
• B73: remove / rollback pkg:pypi/typing_extensions@4.12.2 found nothing when the recorded key was typing-extensions. The same happened for a NuGet case variant.

Related: #553 (the takeover ledger lookup now uses the same key, but there's no e2e for it yet), #748 / C20 (this is the identity half; the builders are untouched), #484 (Go + vs %2B is folded by the key).

Change

• New utils::purl_key
  • PurlKey is the release identity. It strips qualifiers and subpath, percent-decodes, lowercases the type, PEP 503-folds PyPI names, case-folds NuGet names and versions, case-folds Composer names, and keys Composer versions by release identity (3.0.2 = v3.0.2 = 3.0.2.0).
  • PurlKey::qualified keeps release variants apart.
  • canonical_base_purl moved here. It is the display spelling the key is built from, and VEX product purls still use it.
• Call sites. Every purl identity comparison in core and the CLI now builds PurlKeys (the exceptions are listed under Deferred):
  • ledger ownership;
  • prune GC;
  • update detection;
  • redirect candidates;
  • lockfile-only checks (one predicate; the duplicate notInstalled check now calls it);
  • takeover ledger drop;
  • hosted pins;
  • VEX source matching;
  • rollout budget;
  • policy reports;
  • get presence;
  • Go-patches orphan prune;
  • vlt heal targets;
  • Gradle pin rows;
  • the hosted memory engine's batch-search owner map, lock_inventory::lookup, the vendored blob harvest, apply's mismatch-blob record filter, rollback's superseded_by_hosted, and the rollout RecordedIndex (review round, see below).
• Vendored key sets are now HashSet<PurlKey>, so the compiler routes every construction through the key.
• package_spec_matches compares the folded key case-insensitively. User filter specs stay as lenient as before, and a versioned Composer spec still matches by exact release identity.
• canonical_purl keeps only its display role. Its doc no longer claims identity.
• Docker e2e: docker_e2e_nuget.rs now serves the API's real mixed-case spelling for the patch, and scan --sync must keep that entry.

Duplicate copies deleted

Purl equality relations went from 11 to 1 (PurlKey, with canonical_base_purl as its display spelling). Deleted:

• utils::purl::purl_eq
• composer_version::{purl_identity_key, composer_purl_identity, composer_purls_equivalent, composer_bases_equivalent}
• rollout::canonical_base_purl
• policy::canon
• vex::discover::{canonical_base_purl, same_package}
• rollout::stage::qualified_key (a copy of PurlKey::qualified)
• lock_inventory::lookup's own PEP 503 / exact-version matcher
• vex_sources::same_package
• policy's canonical_pypi_purl

The ledger matchers (covers_purl, purl_keys_cover, lookup_entry_kv) are now one-line PurlKey comparisons. The ad-hoc key closures in scan/mod.rs, ledgers.rs, list.rs, hosted.rs, get.rs and apply.rs are gone.

Behavior notes

• Policy filtered[] / retained[] purls and rollout base_purls are the PurlKey spelling, so PyPI and NuGet names there now appear folded. Composer already showed its identity form.
• VEX output is unchanged except for one case: a Composer dev- branch name keeps its case in canonical_base_purl. Before this PR it was lowercased.

Review round (rebased onto main @ 431b818)

• Rebased onto 14 new main commits (Fix VEX attesting beside an unpatched same-lock copy (#935, #938, #939) #940, Fix gem takeover un-hosting a grouped gem (#775) #776, Fix hosted yarn classic pins missing berry warning (#907) #917, Fix gem crawler missing Bundler .bundle root (#967) #968, Fix pnpm lock/workspace readers missing a BOM (#903, #904, #905) #909, Fix rollback/remove of an agent record superseded by a hosted pin (#933) #934, Honor HTTP-date Retry-After on vendor-service retries through api::retry (#677) #889, then Fix composer e2e flake on packagist connect timeouts #1037, Fix yarn berry project gates drifting between modes (#628, #629) #657, Fix Hatch environments being invisible to stale-install checks and VEX (#335) #700, Fix .bundle/config values keeping a trailing # comment (#951) #953, Scan every vendored-write file for vendored references (#832, #958) #1015, Fix remove/rollback missing PyPI name spellings (#1024) #1025, Fix stale docs and comments, and the npm wrapper's schema and exit code #1022).
• Fix remove/rollback missing PyPI name spellings (#1024) #1025 (fixes remove and rollback don't PEP 503-normalise PyPI purl identifiers, so remove pkg:pypi/typing_extensions@4.7.1 exits 1 "No patch found" while get accepts the same identifier #1024, the PyPI half of B73) had brought back utils::purl::purl_eq with a PEP 503 rule. This branch deletes it again: patch_matches already folds PEP 503 through PurlKey. Fix remove/rollback missing PyPI name spellings (#1024) #1025's unit test now checks equality with PurlKey::qualified, and its remove/pypi_name_spellings.rs CLI tests pass unchanged. The get.rs conflict was resolved so Fix VEX attesting beside an unpatched same-lock copy (#935, #938, #939) #940's hosted-claim set is a HashSet<PurlKey> (not canonical_purl strings). The new gem-takeover pin lookup in vendor.rs uses PurlKey::same, and the new rollback::superseded_by_hosted uses PurlKey instead of the deleted helpers.
• scan --prune --dry-run now shares the wet vendor GC's manifest-key drop (vendor::unused_vendored_manifest_keys). Before this, the preview used a qualifier-strip relation, so NuGet case, PEP 503 and composer padding variants were pruned for real but missing from the preview.
• hosted::memory::discover::batch_search matches response packages to their asking roots by PurlKey. Before this, a respelled answer (such as Newtonsoft.Json) was credited to every root in the chunk.
• lock_inventory::lookup matches by PurlKey, so a composer API @3.0.2.0 finds the lock's 3.0.2.
• LockfileSupplement.purls is a HashSet<PurlKey> built once. The lockfile-only check is now one hash lookup, not an O(n) re-key on every miss.
• hosted_vendored_overlap deduplicates by PurlKey, so one release spelled two ways gives one takeover warning.
• Composer: PurlKey keeps the sentinel-free composer_version_key so its string stays printable in policy and rollout reports. A new test over the vector file checks that a rejected spelling can never key the same as an accepted one: every accepted key is itself accepted, bare and behind a v. The module docs now say so.

Core API break (for the release notes; CHANGELOG untouched)

The following socket-patch-core public items were removed or moved:

• utils::purl::purl_eq
• vex::canonical_base_purl (re-export) and rollout::canonical_base_purl; use utils::purl_key::canonical_base_purl
• composer_version::{composer_purl_identity, purl_identity_key, composer_purls_equivalent}
• rollout::stage::qualified_key

These signatures changed from HashSet<String> to HashSet<PurlKey>:

• vendor::vendored_purl_keys
• VendorState::purl_keys
• vendor::purl_keys_cover

lock_inventory::lookup now also matches composer padding and NuGet/composer name case. Nothing else in the workspace references the removed items.

Testing

Failing first: on main plus only the new tests, all three regression tests failed. On this branch they pass.

• cargo test -p socket-patch-core --lib -- folds_pep503_and_nuget_case nuget_case_spellings_are_one_vendored_package: 2 failed.
• cargo test -p socket-patch-cli --lib -- detect_prunable_keeps_case_and_pep503: failed. It pruned both Newtonsoft.Json and typing_extensions.

New tests:

• purl_key property-style table tests (hand-written loops, not proptest):
  • every spelling variant of a release shares one key, and no other release does;
  • the Composer key equals release equivalence over the whole composer-version-vectors.json;
  • qualified keys keep variants apart;
  • NuGet and PEP 503 cases.
• B20 prune test (gc.rs).
• B73 identifier test (purl.rs).
• NuGet scan --mode vendored over a hosted pin skips the takeover (purl case mismatch), keeps the hosted feed wired, writes no vendor ledger and still reports success #553-shape ledger test (state.rs).

Commands run on this branch (macOS):

• cargo test -p socket-patch-core -p socket-patch-cli --lib
  • CLI: 862 passed.
  • Core: 5580 passed. 1 failed, digest::production_digests_go_through_the_helpers, which was already red on main and is fixed by Fix main CI red on stale digest pending-list entries #1016 (now merged; I rebased onto it after the run).
• cargo test -p socket-patch-core --tests --no-fail-fast: all 36 integration suites pass. The lib failure is the same pre-existing one.
• cargo test -p socket-patch-cli with these integration tests: scan, remove, rollback, apply, get, vendor, cli, e2e_scan, e2e_nuget, e2e_composer, e2e_composer_version_identity, e2e_socket_yml_policy, scan_rollout_e2e, hosted_memory_{rollout,engine,parity}, covgap_commands_{scan_mod,scan_hosted,rollback,get,vendor,vex}, global_scope_project_state, in_process_{remove_repair_lifecycle,rollback_vendored,vendor}, e2e_vex, e2e_vex_vendor, e2e_vex_redirect, e2e_embedded_vex, coverage_fix_scan_hosted_dryrun_vendored. All passed.
• cargo clippy -p socket-patch-core -p socket-patch-cli --all-targets -- -D warnings: no findings in any touched file.
  • The local clippy 1.93 reports pre-existing lints in untouched files, such as jvm_jar.rs, prebuilt_common/mod.rs and a macOS-only unused variable in python_crawler.rs.
  • I rustfmt'd only the changed hunks. Files that were already unformatted on main keep their old formatting.
• After the rebase: focused unit tests for purl, gc, state, policy and rollout pass.
• After the review-round rebase and fixes (macOS):
  • cargo build -p socket-patch-core -p socket-patch-cli --all-targets
  • cargo clippy -p socket-patch-core -p socket-patch-cli --all-targets: 0 findings in touched files. The pre-existing lints in untouched files listed above remain.
  • cargo test -p socket-patch-core --lib: 5641 passed (the digest failure is gone now that Fix main CI red on stale digest pending-list entries #1016 is on main).
  • cargo test -p socket-patch-cli --lib --test scan --test vendor --test rollback --test apply --test get --test remove --test gradle_agent_cli --test in_process_rollback_vendored --test in_process_vendor: all passed (lib 872, scan 118, vendor 91, rollback 38, apply 106, get 81, remove 94 (incl. Fix remove/rollback missing PyPI name spellings (#1024) #1025's PyPI spelling tests), gradle_agent_cli 34, in_process_rollback_vendored 16, in_process_vendor 121).
  • New regression tests: batch_search_credits_a_respelled_response_only_to_its_asker, lookup_matches_by_purl_identity, overlap_reports_one_entry_per_release_across_spellings, covers_every_spelling_of_the_release (unused_vendored_manifest_keys), and sentinel_free_key_keeps_rejected_spellings_apart. The first three fail against the pre-fix code: a spelling-keyed owner falls back to every chunk root, exact-version lookup misses 3.0.2.0, and spelling dedup gives 4 entries.
• Not run locally: docker_e2e_nuget (Linux/Docker). It is the end-to-end proof for B20 (mixed-case manifest key against the lowercase global-cache crawl, through scan --sync, GC, apply and agent VEX). It must pass in CI before this lands.

Deferred

• Name-level PyPI comparisons inside lock parsers (canonicalize_pypi_name(a) == canonicalize_pypi_name(b)) are left alone. They compare names, not purls.
• Same-source qualifier grouping in vendor.rs / apply.rs (variant_groups, vendored_bases over manifest keys) is left as-is, since both sides already share one spelling.
• Moving Ecosystem and canonicalize_pypi_name to core (E39 / Move canonicalize_pypi_name and the PEP 508 name scanner out of crawlers and vendor into one PyPI name module #883) is a separate refactor.
• The four target grammars (scan --package, get fuzzy, remove/rollback, UUID shortcut) are untouched.
• Left as literal spelling comparisons, deliberately:
  • get.rs's failure dedup on normalize_purl is display-only.
  • ecosystem_dispatch::merge_qualified matches a crawler's echo of the exact base purl it was asked for, so both sides are the same string.
  • The Gradle/Maven scan keys use canonical_base_purl. Maven coordinates are case-sensitive, so the canonical spelling is the identity, and the lock-file GAVs they are checked against are built as strings.
• The vendored blob harvest matches PurlKey::new(base_purl), not lookup_entry. It keeps the old qualifier-insensitive match for manifest keys, whereas lookup_entry is qualifier-sensitive.
• No maintainer decision was needed.

🤖 Generated with Claude Code

---

Note

Medium Risk
Wide refactor of identity comparisons across apply, scan, GC, rollback, and policy paths; behavior changes are intentional but any missed call site could still mis-match or over-prune patches.

Overview
Introduces PurlKey as the single release-identity for package URLs (encoding, qualifiers, NuGet/PyPI/Composer spelling variants) and routes prune GC, lockfile-only detection, vendor ownership, rollout, policy, VEX, and hosted batch search through it instead of a dozen overlapping helpers (purl_eq, policy::canon, duplicate canonical_base_purl, composer purl identity helpers, etc.).

User-visible fixes: scan --prune/--sync no longer drops live manifest entries when the API uses mixed-case NuGet or alternate PyPI spellings vs the crawl; remove/rollback identifiers now match those recorded keys. Vendored key sets become HashSet<PurlKey>; dry-run prune preview shares wet vendor GC manifest-key logic via unused_vendored_manifest_keys; hosted memory batch search credits respelled API purls only to the asking root.

Several removed/moved socket-patch-core public APIs (purl_eq, old canonical_base_purl locations, composer identity exports); vendored helpers now take HashSet<PurlKey>.

^{Reviewed by Cursor Bugbot for commit daa5ef8. Configure here.}

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Lockfile-only skips installed spelling variants
  • Changed lockfile_supplement to use PurlKey normalization for exclusion checks, matching the membership check behavior and preventing spelling variants from being incorrectly classified as not installed.

Or push these changes by commenting:

@cursor push aae44b9bf5

Preview (aae44b9bf5)

diff --git a/crates/socket-patch-cli/src/commands/scan/discovery.rs b/crates/socket-patch-cli/src/commands/scan/discovery.rs
--- a/crates/socket-patch-cli/src/commands/scan/discovery.rs
+++ b/crates/socket-patch-cli/src/commands/scan/discovery.rs
@@ -75,7 +75,7 @@
     if entries.is_empty() {
         return out;
     }
-    let crawled_purls: HashSet<&str> = crawled.iter().map(|p| p.purl.as_str()).collect();
+    let crawled_purls: HashSet<PurlKey> = crawled.iter().map(|p| PurlKey::new(&p.purl)).collect();
     let in_scope = |purl: &str| {
         only.is_none_or(|list| {
             socket_patch_core::crawlers::Ecosystem::from_purl(purl)
@@ -83,7 +83,7 @@
         })
     };
     for entry in entries {
-        if crawled_purls.contains(entry.purl.as_str()) || !in_scope(&entry.purl) {
+        if crawled_purls.contains(&PurlKey::new(&entry.purl)) || !in_scope(&entry.purl) {
             continue;
         }
         let Some(pkg) = crawled_from_purl(&entry.purl, &common.cwd) else {

_{You can send follow-ups to the cloud agent here.}

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit daa5ef8. Configure here.}

[Mikola Lysenko (mikolalysenko)]

[agent] Ready for review at daa5ef8bfba15b90700b40b8680235601206098f.

• CI: required ci-ok green. 527 success / 6 skipped / 0 failing; 30 non-required macOS/Windows legs still queued or running. yarn-berry 4.1.0 (ubuntu) failed on attempt 1 with a corepack yarn@2.4.3 unavailable fetch race (a sibling yarn@2.4.3 test passed 0.6s later) and passed on re-run.
• Bugbot reviewed daa5ef8: no unresolved findings. No open review threads.
• Mergeable, no conflicts.
• Reviewer focus: PurlKey equality and normalization, which now backs every purl comparison.

---

Generated by Claude Code