Skip to content

Move the Socket origin classifier into one module - #2002

Open
Mikola Lysenko (mikolalysenko) wants to merge 14 commits into
mainfrom
lock-model/1-origin
Open

Mikola Lysenko (mikolalysenko) wants to merge 14 commits into
mainfrom
lock-model/1-origin

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Stack: PR 1 of the lock-model/* stack, based on #2001 (design: docs/design/lock-model-unification.md).

What

This PR moves the Socket-origin classifier into one new module, crates/socket-patch-core/src/origin.rs. The code comes from two places:

  • From vex/discover/mod.rs:
    • hosted_uuid with its cached plain-URL fast path, now on a small OriginPolicy that holds the run's origin allowlist;
    • vendor_ref, vendor_ref_decorated, vendor_uuid_dir, strip_leaf_suffix, vendored_leaf_purl;
    • locate, Located, LocateOpts, Wired;
    • socket_patch_name_uuid;
    • the names_vendor_dir / mentions_vendor_dir / root_anchored_spelling helpers.
  • From patch/redirect/mod.rs: hosted_patch_uuid, hosted_patch_url_uuids and SOCKET_PATCH_SERVER_HOST.

The tests move with the code: hosted_patch_uuid_tests verbatim, and the six discovery tests that cover the moved functions.

Why

This is the first step to one classifier for every reader. Today, discovery checks a Socket host allowlist, while the lock inventory accepts any host whose URL leaf matches. Later PRs point the inventory and the writers at this module. It sits outside formats/ on purpose: the formats/mod.rs architecture test keeps format models free of host policy.

No behavior change

  • The code moves verbatim. I diffed every removed line against every added line. The only differences are:
    • paths (crate::patch::redirect::X → X);
    • the OriginPolicy struct replacing DiscoverCtx's patch_server_origins / hosted_hosts fields;
    • two one-line delegations on DiscoverCtx (hosted_uuid, locate);
    • module docs;
    • UUID_A → UUID in the moved tests.
  • Callers. vex::discover re-exports the moved items its extractors use. patch::redirect's re-export is gone (review finding 7): its four callers import from crate::origin.
  • The review's socket_identities change (finding 3) reuses the policy's host matcher instead of building one per call; the answers are the same (the recognition, memo and golden tests pass unchanged).

Size budget

Group Base (#2001) This PR Why
lock-readers (now includes origin.rs) 30,462 30,544 49 lines moved in from patch/redirect, plus OriginPolicy and re-export glue
lock-writers 88,362 88,315 the 49 lines moved out (+2 from rustfmt wrapping a crate::origin:: path)

Counted with #2001's fixed counter. The routine merge 4242ebc resolved the loc-budget.json conflict with the base's path list, which dropped origin.rs from lock-readers (it showed 30,259). The next merge (88df7f9) puts it back. #2009 and #2029 had inherited the same drop, and their restack merges fix it too.

Review findings

Each finding was checked against the code first. Commit fbe644d ("Address the review of the origin move") applies them, after a merge of #2001's review fixes.

  1. Fixed (confirmed). The extractor guard (extractors_read_content_only_through_the_recognizing_ctx_helpers) banned only hosted_patch_uuid( / hosted_patch_url_uuids(, so an extractor could build its own OriginPolicy and bypass DiscoverCtx::hosted_uuid. It now also bans OriginPolicy and crate::origin::. The needle is crate::origin::, not bare origin::, because extractors legitimately import crate::vendor::npm_origin::. vex::discover no longer re-exports OriginPolicy; mod.rs imports it from crate::origin directly.

  2. Fixed, except one needle. The lock_inventory entry-model guard now also bans OriginPolicy, crate::origin:: and hosted_uuid(. The formats/ guard bans OriginPolicy, hosted_uuid( and SOCKET_PATCH_SERVER_HOST, but not crate::origin::. That guard's rule is "no I/O and no host policy", and origin also holds the pure, root-anchored path grammar (vendor_ref), which applies no host policy. The pnpm planner core in Plan pnpm rewires with one core for both modes (lock-model 2b) #2029 (formats::pnpm::plan::match_key) uses it on purpose: it is the rule N5 asks every reader to use.

  3. Fixed (confirmed). socket_identities rebuilt the host list and its Aho-Corasick matcher on every call. It now takes &OriginPolicy, and the policy memoizes:

    • one lowercased host set, shared with hosted_uuid's fast path (which used the un-lowercased host_str(); reqwest already lowercases http(s) hosts, so no answer changes);
    • one case-insensitive matcher (OriginPolicy::host_finder).

    DiscoverCtx passes its own policy, so a discovery run builds the matcher once. hosted_uuids_in_text, the upstream restore's probe, keeps its &[String] signature and builds a policy per call, as before. The vex::discover golden test passes.

  4. Fixed. origin.rs's module doc now states the one-classifier rule as the goal and names what still classifies on its own:

    • the inventory's PyPI view (lock_inventory/pypi.rs hosted_pypi_reference, with its own url-tail grammar);
    • discovery's PyPI diagnostics (pypi_other.rs, which compares hosts directly);
    • the upstream restore's known-origin list;
    • the inventory's any-host matching.
  5. Fixed. socket_patch_name_uuid is pub(crate) again, as it was in vex::discover.

  6. Fixed. Removed the duplicate inner use is_canonical_uuid in hosted_patch_url_uuids.

  7. Fixed, with one exception. The patch::redirect re-export of hosted_patch_uuid / hosted_patch_url_uuids / SOCKET_PATCH_SERVER_HOST is gone.

    • Its callers (redirect::vlt_heal, redirect::upstream, lock_inventory::pypi, vex::discover) use crate::origin:: directly.
    • So do the docs: discover rule 4, socket_identities, redirect::generation and vendor::path.
    • The exception: vex::discover keeps re-exporting the vendored-path and locate helpers, and now SOCKET_PATCH_SERVER_HOST for pypi_other.rs:576. Finding 1's guard bans crate::origin:: in extractor files, so the extractors import every identity helper through super::. The public vex::discover::vendor_ref family stays too, because the CLI's vex_sources.rs imports it.
  8. Fixed. The new test vex::discover::tests::ctx_hosted_uuid_honours_the_runs_configured_origins goes through DiscoverCtx::with_origins(..).hosted_uuid(..). It checks a configured-origin URL, a registry URL, and the same URL with no origins configured.

  9. Fixed (confirmed). The public module's doc linked pub(crate) items (OriginPolicy::hosted_uuid, OriginPolicy::locate, and now socket_patch_name_uuid). They are plain code spans now. cargo doc -p socket-patch-core with rustdoc::private_intra_doc_links reports nothing for origin.rs.

Testing

  • cargo test -p socket-patch-core --lib -- origin:: vex:: formats:: vendor::lock_inventory:: vendor::pnpm patch::redirect:: hosted::: 2,518 passed, 0 failed. This includes:
    • vex::discover::testing::golden::tests::committed_fixture_corpus_matches_golden;
    • the extractor, formats and lock_inventory architecture guards;
    • the moved tests under origin::.
  • cargo clippy --locked --workspace --all-features -- -D warnings: clean. cargo fmt --all -- --check: clean.
  • python3 -B -m unittest discover -s scripts/tests -p 'test_loc_budget.py': 31 tests, OK. scripts/loc-budget.py --check: both groups at their budget.

Note

Medium Risk
Centralizes attestation-critical hosted URL and vendored path recognition; mistakes could misclassify lockfile wiring, though the PR positions this as a verbatim move with existing tests and architecture guards.

Overview
Introduces crates/socket-patch-core/src/origin.rs as the single place for classifying lockfile/config strings as Socket hosted URLs, vendored .socket/vendor/… paths, or socket-patch-<uuid> names. Logic moves here from patch::redirect (hosted_patch_uuid, hosted_patch_url_uuids, SOCKET_PATCH_SERVER_HOST) and vex::discover (vendor ref helpers, locate/Located, identity sweep support).

OriginPolicy holds the run’s patch-server allowlist and memoizes host sets and an Aho–Corasick matcher. DiscoverCtx delegates hosted_uuid and locate to it; socket_identities takes &OriginPolicy instead of rebuilding matchers per call. Callers (redirect::upstream, vlt_heal, lock inventory PyPI view, discover extractors) import from crate::origin; discover re-exports what extractors need and tightens architecture guards so extractors cannot bypass the ctx.

Tests move with the code; patch::redirect re-exports are removed. scripts/loc-budget.json adds origin.rs to the lock-readers group. Documented intent: no behavior change—refactor toward one classifier for all readers (follow-up PRs in the lock-model stack).

Reviewed by Cursor Bugbot for commit a322027. Configure here.


Generated by Claude Code

Lockfile discovery (vex/discover), the lock inventory
(vendor/lock_inventory) and the hosted and vendored writers each read the
same project files their own way: two Socket-origin classifiers, three
Python precedence rules, second walks of package-lock, yarn, vlt and pnpm
locks, a second parse of uv/pylock/script locks, five Cargo.toml
dependency-table walks, three readers of Maven's checksums.sha256. The
inventory also has no NuGet, JVM, Hatch or Deno models.

docs/design/lock-model-unification.md describes the target, one
per-project model whose entries carry an Origin (Registry, Hosted,
Vendored, Other, Rejected), file selection by install tree (precedence
within a tree, union across trees), and the stack of PRs that gets there.

scripts/loc-budget.py counts implementation lines (no comments, no
cfg(test) items, no test-only modules) for two groups in
scripts/loc-budget.json: lock-readers (30,898 lines today) and
lock-writers (88,363). The lint-ecosystems job runs it with --warn, which
annotates a group over budget but never fails, so bug fixes in these files
are not blocked; each PR in the stack lowers the budget as it deletes
code.

No product code changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
formats/ models are guarded to stay free of I/O and of the host policy
(formats/mod.rs architecture_tests bans hosted_patch_uuid there), so the
classifier is a top-level `origin` module and a project layer applies it.
Spell out the three layers and where each part of the classifier moves
from.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decisions: inconsistencies between discovery, the inventory and the
two modes are bugs to remove, not behavior to preserve; hosted and
vendored share one core and differ only in where a rewired entry points;
when in doubt the hosted mode's behavior wins, and every such call is
listed in the doc so it can be overruled. Behavior changes are allowed
per PR, listed, pinned by tests and shown as golden diffs.

First decision recorded: pnpm 11's env-lockfile document counts as its
own install tree, as the hosted planner already treats it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/design/lock-model-divergences.md lists every question on which
discovery, the inventory, hosted and vendored mode answer differently
today, across all ecosystems, with the resolution under the principles
and flags for false attestations and target-specific differences.

Owner decisions recorded: the modes are ground truth over the readers;
vendored mode's Maven profile and reactor support and its scala-cli wiring
are back-ported to hosted as the planners merge (the JVM PR splits into
readers, Maven planner, and Gradle/sbt/scala-cli). One call is pending:
whether hosted keeps rewriting a shrinkwrap.yaml that pnpm never reads.

Corrects the install-tree section: npm-shrinkwrap.json does not shadow
package-lock.json; every consumer but the inventory already reads both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decisions: a dependency the user patched themselves is never
patched or attested by default, in every mode and package manager, unless
the user explicitly overrides; and no reader or writer reads project files
outside the git repository root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Discovery's classifier (the memoized hosted_uuid fast path, vendor_ref
and its variants, locate, Wired, the socket-patch-<uuid> name grammar)
and the redirect's hosted URL rule (hosted_patch_uuid,
hosted_patch_url_uuids, SOCKET_PATCH_SERVER_HOST) now live in
crates/socket-patch-core/src/origin.rs, with their tests.

The code moves verbatim. DiscoverCtx holds an OriginPolicy (the
per-run origin allowlist and its memoized host set) and delegates
hosted_uuid and locate to it. vex::discover and patch::redirect
re-export the moved items, so no caller changes.

origin.rs sits outside formats/ on purpose: format models are guarded to
apply no host policy (formats/mod.rs architecture_tests).

Budget: lock-readers 30,898 -> 30,976 (49 lines moved in from
patch/redirect, ~30 lines of OriginPolicy and re-export glue);
lock-writers 88,363 -> 88,314.

No behavior change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #2001 found the counter miscounting in several shapes, each
now pinned by a test in scripts/tests/test_loc_budget.py:

- a cfg(test) struct field, enum variant, parameter or match arm drove
  the scan below the item's depth and swallowed the code after it; the
  item now ends at a top-level `,` or before the bracket closing its
  container
- `#[cfg(test)] use a::{..};` left its trailing `;` line counted
- cfg predicates are parsed: `all(unix, test)` and nested `all` count,
  `not(test)` doesn't, and a file opening with `#![cfg(test)]` is
  test-only
- `#[path]` is found among any attributes around the item and resolves
  against the declaring file's directory, as rustc does
- `mod x;` inside a test-only inline module (vex::discover's
  `testing { mod golden; }`) marks that file test-only
- `git ls-files -z` keeps paths with spaces whole, and only the group
  files plus their module declarers are read
- `--files` with an unknown group is a usage error, not a KeyError

golden.rs (435 lines) and two `};` lines leave the counts, so the
budget drops to 30462 / 88362.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of #2001: the Decisions table said every read is bounded to the
repository root, while G1, J13 and P4 still said to read an ancestor
go.work, parent NuGet configs and every include. The owner narrowed
the rule on 2026-11-05, and the table now records it with the rest of
that day's calls:

- the bound applies to workspace runs only; --global / -g /
  --global-prefix runs are unbounded, and without a git checkout upward
  lookups behave as before
- a marker file above the repository root (below home) makes hosted and
  vendored refuse with a message naming it
- out-of-repository -r includes stay bounded in workspace runs
- N11: shrinkwrap.yaml beside pnpm-lock.yaml is shadowed, unless the
  project pins pnpm 1 / 2, where it governs and vendored refuses
- discovery withholds pnpm root and member-lock attestations whenever a
  pnpm-lock.<branch>.yaml exists
- user patches: the override is --allow-patch-conflicts, and vex never
  attests under a user patch

G1, J9, J13, P4, P9, X12, N1, N2, N4, N5, N11, N12 and N14 name the open
PRs that fix them, and the fixes-on-main table gains a PR column. The
problem table now counts four checksums.sha256 readers, as J9 does. The
stack table splits PR 2 into 2a / 2b with their numbers, and the size
budget section says why CI only warns and gives the corrected counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude (claude) and others added 5 commits November 6, 2026 07:16
Resolve the scripts/loc-budget.json conflict: the base fixed the
counter's cfg(test) and module bugs, so both budgets are recounted
with the fixed counter and lowered to the exact current totals
(lock-readers 30259, lock-writers 88313), keeping this PR's
'both groups at their budget' intent.

Co-Authored-By: Claude <noreply@anthropic.com>
…g code

loc-budget.py only recognized `pub name: Type` as a field, so a
#[cfg(test)] on a `crate name: Type` field or a `const N: usize`
generic parameter was read as an item. _item_end then didn't stop at the
comma and excluded the rest of the struct or parameter list from the
implementation count.

Treat `crate` followed by a name like `pub`, and read `const` as a
parameter when the attribute follows `<` or `,`, where no item can
start. A `crate::path` statement and a plain const item keep their
current handling. Reported by Bugbot on #2001.

Co-Authored-By: Claude <noreply@anthropic.com>
Follow-up PRs grep for the names this doc cites. OriginPolicy::hosted_uuid
exists nowhere in the tree; the function is DiscoverCtx::hosted_uuid (its
memoized fast path is the hosted_hosts cell). The npm flavor router lives
at vendor/lock_inventory/npm_family.rs, since doc paths are relative to src/.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011f9aeNYq2GHyrsNnPBepcC
Brings in the counter's crate-field and const-generic fix and the doc
symbol fixes. Also restores crates/socket-patch-core/src/origin.rs to
the lock-readers group: the previous merge (4242ebc) resolved the
loc-budget.json conflict with the base's path list and dropped it, so
the 281 lines this PR moves into origin.rs went uncounted (30259). With
origin.rs counted, lock-readers is 30540 and lock-writers 88313.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The extractor guard also bans OriginPolicy and crate::origin:: (not
  bare origin::, which would catch crate::vendor::npm_origin::), and
  vex::discover no longer re-exports OriginPolicy; mod.rs imports it
  from crate::origin.
- The lock_inventory model guard bans OriginPolicy, crate::origin:: and
  hosted_uuid(; the formats guard bans OriginPolicy, hosted_uuid( and
  SOCKET_PATCH_SERVER_HOST. formats may still use origin's pure path
  grammar (vendor_ref), which applies no host policy.
- socket_identities takes &OriginPolicy. The policy memoizes one
  lowercased host set, shared by hosted_uuid's fast path and a
  case-insensitive matcher (host_finder), so a discovery run builds the
  matcher once instead of on every swept file. hosted_uuids_in_text
  keeps its signature. Answers are unchanged: reqwest already
  lowercases http(s) hosts.
- origin.rs's module doc states one classifier as the goal and names the
  readers that still classify on their own; it no longer links
  pub(crate) items (rustdoc private_intra_doc_links).
- socket_patch_name_uuid is pub(crate) again; hosted_patch_url_uuids
  loses its duplicate is_canonical_uuid import.
- patch::redirect stops re-exporting hosted_patch_uuid,
  hosted_patch_url_uuids and SOCKET_PATCH_SERVER_HOST; vlt_heal,
  upstream, lock_inventory::pypi and discover use crate::origin
  directly. Extractors keep importing origin helpers through
  vex::discover's re-export (now with SOCKET_PATCH_SERVER_HOST for
  pypi_other), as the guard requires.
- New test: DiscoverCtx::with_origins(..).hosted_uuid(..) honours a
  configured origin and only for that run.

Budget raised by hand: lock-readers 30540 -> 30544 (the shared host
matcher and the re-export note), lock-writers 88313 -> 88315 (rustfmt
wraps the upstream origin path).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mikola Lysenko (mikolalysenko) added a commit that referenced this pull request Nov 6, 2026
Brings in #2002's review fixes (OriginPolicy imported from crate::origin,
the redirect re-export dropped, the shared host matcher, the stricter
architecture guards) and #2001's later counter fixes.

Restores crates/socket-patch-core/src/origin.rs to the lock-readers
group, which the previous merge (1d27a20) inherited the loss of from
1-origin's 4242ebc. Budget at this branch's counts: lock-readers
30767, lock-writers 88315.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mikola Lysenko (mikolalysenko) added a commit that referenced this pull request Nov 6, 2026
Brings in #2002's review fixes and #2001's later counter fixes through
2a. Restores crates/socket-patch-core/src/origin.rs to the lock-readers
group (lost in the earlier 1-origin merge). Budget at this branch's
counts: lock-readers 30766, lock-writers 88114, both below 2a's 30767
and 88315.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from lock-model/0-design to main November 7, 2026 01:56
# Conflicts:
#	scripts/loc-budget.json
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit a322027. Configure here.

@mikolalysenko
Mikola Lysenko (mikolalysenko) marked this pull request as ready for review November 7, 2026 04:37
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Burn-down agent: Bugbot reviewed a322027d and found nothing, there are no unresolved review threads, and the draft CI subset is green (7 pass, 31 skipped). The description is complete, so I've marked this ready for review to run the full CI. The Ready for review label goes on once that's green. #2009 and #2029 are stacked on this PR.


Generated by Claude Code

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 7, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Burn-down agent: labeled Ready for review at head a322027.


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final review brief (head a322027)

What it does: Moves the Socket-origin classifier (hosted-URL uuid recognition, the vendored-path grammar, locate, and SOCKET_PATCH_SERVER_HOST) out of vex::discover and patch::redirect into one new module, origin.rs. A small OriginPolicy takes over DiscoverCtx's origin fields and memoizes the host set and matcher. This is step 1 of the lock-model stack, and behavior should not change.

Risk: low. I diffed every removed line against every added line. The only differences are import paths, the OriginPolicy glue, the two DiscoverCtx delegations, UUID_A → UUID in the moved tests, and docs. The one semantic touch is the shared host set, which now lowercases host_str(). That gives the same answers: url already lowercases http(s) hosts, and the identity sweep's matcher was already case-insensitive.

Look here

  • origin.rs:181-245: OriginPolicy: hosts(), host_finder(), hosted_uuid fast path
  • origin.rs:45-90: SOCKET_PATCH_SERVER_HOST, hosted_patch_uuid, hosted_patch_url_uuids (moved from patch::redirect)
  • vex/discover/mod.rs:1954: socket_identities now takes &OriginPolicy (one matcher per discovery run instead of one per call)
  • vex/discover/mod.rs:205-215: the re-exports extractors and the CLI still use
  • vex/discover/mod.rs:4471 and formats/mod.rs / lock_inventory/mod.rs IMPURE lists: the architecture guards now ban OriginPolicy / hosted_uuid( (and crate::origin:: outside formats/)

Verified

  • Moved-line diff as above. Base Plan one lock model for every lockfile reader #2001 is merged, and the diff against main is these 13 files only. CHANGELOG.md is untouched, and there is no debug code.
  • cargo clippy --locked -p socket-patch-core --all-features -- -D warnings (CI's flags): clean.
  • cargo test -p socket-patch-core --lib -- origin:: vex::discover:: patch::redirect:: formats:: vendor::lock_inventory::: 1805 passed, 1 failed. The failure is vlt_heal::an_unremovable_hidden_lock_keeps_every_store_entry, a permission test that can't hold when run as root in my container. This PR doesn't touch it, and it passes in CI.
  • CI is green on the head (ci-ok and clippy included). Bugbot reviewed a322027 with no findings. No review threads are unresolved.

Changes I made: none.

Open questions (non-blocking)

  • scripts/loc-budget.py --check reports both lock groups over budget here, but main is over too (lock-readers 31,346 / lock-writers 88,837 on main against a 30,462 / 88,362 budget). So the size table in the description is stale against today's main. It isn't caused by this PR, and CI treats it as a warning.

Auto-merge is armed, so approving sends this straight to the merge queue.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants