Fix PyPI vendored to hosted takeover being refused (#328)

[Mikola Lysenko (mikolalysenko)]

LLM Description written by Claude Code:claude-opus-5-5

Fixes #328

Root cause

When scan --mode hosted runs over a project socket-patch has already vendored, it first reverts each purl's vendored wiring ("takeover") and then redirects it. The gate that decides which purls get that treatment (takeover_capable in crates/socket-patch-cli/src/commands/scan/hosted.rs) admitted only pkg:cargo/, pkg:npm/ and pkg:golang/. A vendored PyPI purl therefore went straight to the hosted rewriters. Those rewriters treat any non-registry source as user-authored, including the vendored one socket-patch wrote itself, and refused it:

• Poetry: redirect_poetry_lock_unsupported
• Pipenv: redirect_pipenv_refused
• Hatch: redirect_hatch_unsupported
• uv: redirect_uv_project_unsupported
• requirements.txt: redirect_requirements_entry_not_found

The scan still exited 0 with redirected: 0, leaving the project vendored.

Fix

• Add pkg:pypi/ to the takeover gate. The per-purl vendor --revert machinery (revert_pypi_opts, which covers every PyPI flavor) restores the recorded registry entry and removes the ledger entry and wheel. Then the normal hosted rewrite pins it.
• Guard 1: if a revert leaves vendored wiring in place, refuse the takeover with redirect_vendored_revert_failed and keep the ledger entry and artifact. "In place" means a drift-skipped record, or a reverted file that still references the artifact; the shared revert_keeps_wiring predicate checks both. Previously the takeover dropped the ledger entry anyway, which breaks the RevertOutcome contract. --dry-run predicts the same refusal from the same signals. This applies to every takeover-capable ecosystem.
• Guard 2: if a taken-over package's wiring was reverted but the package was then not pinned to hosted, it is now unpatched in both modes. Causes include a refused lock, unavailable hosted wheel METADATA, or a vendored ledger update that failed after the revert. The run now reports redirect_takeover_unpatched with status: "partial_failure" and exit 1, and it is printed under --silent too. Human output gives no "Migrated …" line and no "keep the hosted patches" next steps for it.
• CLI_CONTRACT.md documents all of the above.

No wrapper changes are needed: the npm/, pypi/ and gem/ wrappers only dispatch to the binary.

Follow-up (not in this PR, suggested in review): fetch and validate the hosted wheel metadata before reverting, so a PyPI takeover refuses up front with nothing written, as the yarn berry gates do.

Tests (new suite crates/socket-patch-cli/tests/mode_migration_pypi.rs, hermetic, wiremock)

Test	Without fix	With fix
requirements_vendored_to_hosted	❌ redirect_requirements_entry_not_found, redirected 0	✅
requirements_sole_pin_vendored_to_hosted (six==1.16.0 alone, from the 2026-10-01 comment)	❌ same	✅
poetry_vendored_to_hosted	❌ redirect_poetry_lock_unsupported	✅
pipenv_vendored_to_hosted	❌ redirect_pipenv_refused	✅
uv_vendored_to_hosted	❌ redirect_uv_project_unsupported	✅
hatch_vendored_to_hosted	❌ redirect_hatch_unsupported	✅
uv_takeover_without_wheel_metadata_fails_loudly	❌ with only the gate change: exit 0, package unpatched	✅ exit 1, redirect_takeover_unpatched
drifted_vendored_line_refuses_takeover	❌ with only the gate change: ledger dropped	✅ refused, ledger and artifact kept
dry_run_predicts_drifted_takeover_refusal (Bugbot)	❌ on a0863a2	✅
stranded_takeover_human_output_is_not_a_migration (Bugbot)	❌ on a0863a2	✅
stranded_takeover_is_reported_under_silent (Bugbot)	❌ on a0863a2	✅
ledger_update_failure_after_revert_is_stranded (review P1; Unix, skips where permissions aren't enforced)	❌ on 0be19d0: exit 0, success	✅ exit 1, partial_failure

The existing covgap_commands_scan_hosted::ledger_save_failure_after_successful_revert_fails_closed now expects the stranded contract too: exit 1, partial_failure and redirect_takeover_unpatched.

Each lane test vendors the project, runs scan --mode hosted, and asserts all of: redirected: 1, redirect_takeover_reverted_vendored, no .socket/vendor/ reference in any wiring file, the hosted URL wired, and the vendored artifact reclaimed.

Local checks on aada684:

• cargo clippy --workspace --all-features -- -D warnings: clean.
• Run as an unprivileged user (so the read-only-permission tests really run): mode_migration_pypi 12/12, mode_migration_npm 15/15, mode_migration_cargo 5/5, covgap_commands_scan_hosted 50/50, covgap_commands_vendor 44/44, covgap_commands_rollback 62/62, covgap_commands_get 75/75, in_process_redirect 104/104.
• cargo test -p socket-patch-cli --all-features --lib, in_process_vendor, in_process_vendor_bun_takeover, coverage_fix_scan_hosted_dryrun_vendored, scan: all pass.
• The full cargo test --workspace didn't fit in this session's disk allowance, so CI runs the rest.
• cargo fmt --all -- --check: fails on main already (about 460 diffs in files this PR doesn't touch; CI doesn't run it). The hunks this PR adds are rustfmt-clean.

CI on aada684: all 467 checks pass (461 success, 6 skipped).

Per-issue checklist

• Poetry hosted ⇄ vendored mode switch is refused, and blames a "user-authored" source that socket-patch wrote itself #328 vendored → hosted on Poetry / Pipenv / Hatch / uv / requirements.txt → the five *_vendored_to_hosted tests plus requirements_sole_pin_vendored_to_hosted. The hosted → vendored direction was already fixed on main (see the issue comments).

🤖 Generated with Claude Code

https://claude.ai/code/session_018kguGYyuuizF1dwKyH6oZh

---

Note

Medium Risk
Changes hosted scan exit semantics (exit 1 / partial_failure when revert leaves the project on unpatched registry releases) and cross-mode takeover logic; behavior is contract-tested but affects CI and migration workflows.

Overview
Fixes vendored → hosted migration for PyPI (#328) by including pkg:pypi/ in the hosted scan’s vendored takeover gate, so scan --mode hosted reverts socket-patch’s own vendored wiring (via the same per-purl vendor --revert path as cargo/npm/golang) before lock rewriters run—those rewriters previously treated vendored sources as user-authored and skipped redirect with exit 0.

Tightens takeover safety for all takeover-capable ecosystems: if revert would leave vendored wiring in place (drift-skipped records or residual artifact references), the takeover is refused with redirect_vendored_revert_failed and the ledger/artifact stay put; --dry-run uses the same signals instead of previewing a takeover.

“Stranded” takeovers—revert succeeded but the package never got a hosted pin (failed redirect, missing wheel metadata, or ledger save after revert)—now emit redirect_takeover_unpatched, set JSON status: "partial_failure", and exit 1 (including under --silent). Human output skips misleading “Migrated …” and commit/reinstall next steps for those purls.

CLI_CONTRACT.md documents PyPI takeover, revert refusal, and stranded behavior. New hermetic tests in mode_migration_pypi.rs cover requirements/Poetry/Pipenv/uv/Hatch happy paths plus drift, dry-run, stranded UX, and ledger-save failure; an existing covgap hosted test expects the new stranded contract.

^{Reviewed by Cursor Bugbot for commit 9d46d12. Configure here.}

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[agent] CI: native (ubuntu-latest, 1.0.0) failed on 0be19d0. The only failing case was bun 1.0.0 crlf hosted, on the vexCheckoutPatchedBytes check (37/38 passed). That check runs a fresh bun install --frozen-lockfile on a checkout of the committed lock (scripts/backtest-bun.py:1095-1098).

I don't think this PR caused it:

• That case is a plain hosted scan with no vendored state, so it never reaches the takeover code this PR changes.
• The same job passed on a0863a2.
• 0be19d0 only changes code that runs during a vendored → hosted takeover (the dry-run drift prediction, and the human and --silent output for an unpatched takeover).

I'll re-run the failed job once when the workflow finishes, which isn't possible while other jobs are still running (403 "already running"). If it fails a second time I'll treat it as real and dig into the job artifact.

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

[agent] Ready for review at 0be19d02d0bae3ee54fb3baa1b46a3d274a2e17e.

• CI: 478/478 non-skipped checks green (484 total, 6 skipped). The earlier native (ubuntu-latest, 1.0.0) failure (bun 1.0.0 crlf hosted, fresh bun install) passed on re-run.
• Mergeability: up to date with main (0 behind). Waiting only on required approval.
• Bugbot: reviewed 0be19d0 with no new issues. The 3 findings on a0863a2 (dry-run prediction, stranded-takeover human output, --silent reporting) are fixed and covered by tests. All threads are resolved.
• For reviewers: this widens the hosted takeover gate to pkg:pypi/ and adds two guards for every takeover-capable ecosystem: redirect_vendored_revert_failed and redirect_takeover_unpatched. The second one exits 1. See CLI_CONTRACT.md.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Reviewed 0be19d02d0bae3ee54fb3baa1b46a3d274a2e17e. Recommendation: changes needed.

P1 — A ledger-save failure still reports success after removing the PyPI patch (hosted.rs:1533). stranded_takeovers only sees takeover_migrated, but vendored_takeover records that list after save_state succeeds. If the revert succeeds and ledger persistence fails, the purl is skipped before it reaches this guard.

Reproduced by vendoring requirements.txt with six==1.16.0, making .socket/vendor read-only, then running the hosted scan: it returns exit 0 / status: "success", restores the unpatched registry requirement, deletes the vendored wheel, and reports redirected: 0 / rewrittenFiles: []. The only warning is redirect_vendored_revert_failed, so --silent also misses the new stranded warning. Track loss of the vendored wiring independently of successful ledger persistence and return the same partial failure/nonzero exit as other stranded takeovers.

Validation: 11 PyPI migration tests plus 15 npm and 5 Cargo migration tests passed. An additional review-only CLI regression test for this read-only-ledger case failed with the output above. Full workspace and real-package-manager matrices were not rerun.

[Tanmay Singla (Tanmay182003)]

The vendored→hosted takeover for PyPI reverts the vendored patch before fetching the hosted wheel metadata (vendored_takeover at hosted.rs:802, fetch at :818+). If that fetch fails (network, a uv project whose metadata can't be fetched), the package is left patched in neither mode. The PR catches this and fails loudly (partial_failure, exit 1), and re-running recovers it. But CLI_CONTRACT promises the opposite for yarn berry ("refuses before reverting … never left unpatched in both modes"), and #470 adds exactly that check for berry. Can we file a follow-up to do the same for PyPI: fetch and validate the hosted metadata first, and only revert the vendored copy once that succeeds?

[Mikola Lysenko (mikolalysenko)]

[agent] Replying to both review comments.

P1 (ledger-save failure reported as success): fixed in 1a1f2e9. The cause is as described. The save_state failure branch in vendored_takeover pushed the purl only to refused, so stranded_takeovers never saw it, even though its vendored wiring and wheel were already gone. Takeover now carries an unrecorded list for exactly this case, and those purls join stranded. The run now:

• reports redirect_takeover_unpatched (alongside the existing redirect_vendored_revert_failed that names the ledger cause)
• returns status: "partial_failure" and exit 1
• prints the warning under --silent too
• skips the "Migrated …" line and the hosted next steps

This applies to every takeover-capable ecosystem, not only PyPI. CLI_CONTRACT.md lists the ledger-update failure as one of the stranded causes.

Regression test: ledger_update_failure_after_revert_is_stranded. It vendors six==1.16.0 in requirements.txt, makes .socket/vendor read-only, and runs the hosted scan. Run as an unprivileged user, it fails on 0be19d0 (exit 0, no redirect_takeover_unpatched) and passes on 1a1f2e9. The test probes whether directory permissions are actually enforced and skips if they aren't (root), so it can't silently pass for the wrong reason. All 12 PyPI migration tests pass, as do the npm and Cargo migration suites, the scan/vendor takeover suites and the CLI unit tests. Clippy is clean.

Tanmay Singla (@Tanmay182003), pre-validating hosted metadata before the revert: agreed. Fetching and validating the hosted wheel METADATA before reverting the vendored copy would let a PyPI takeover refuse up front with nothing written, the same way the berry gates do. It also matches the contract's "never left unpatched in both modes" wording. That reorders the takeover against the metadata fetch for every Python flavor, which is bigger than this PR, so I'd keep it as a follow-up. This PR guarantees the fallback: it fails loudly, and a re-run recovers. I'll leave filing that follow-up issue to the maintainers.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Follow-up reviewed aada6847c51b74beb7d9397e644e2489e8e3be4d. The earlier P1 is fixed; ready to merge from this review's perspective.

A takeover whose vendored wiring was removed but whose ledger update failed is now included among stranded packages. It reports redirect_takeover_unpatched, returns partial_failure / exit 1, and remains visible under --silent. The later test-only commit correctly updates the existing ledger-failure coverage to that contract. No new blocker found in these changes.

Validation: 12 PyPI migration tests and the original independent ledger-write-failure reproduction pass with the fix. The 7 ledger-related hosted coverage tests also pass at this head. These are focused checks; the full workspace suite was not rerun locally.

[Mikola Lysenko (mikolalysenko)]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 9d46d12. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Second pass reviewed 9d46d12786c60b79aa320f2d819243620c820b01: ready from code review; no additional code change needed. I checked the new main merge and its conflict resolution. The only manual resolution is in CLI_CONTRACT.md; it preserves the PyPI takeover contract and main's corrected Bundler configuration precedence. Production code merged automatically.

Validation at this exact head: all 12 mode_migration_pypi tests passed, covering the package-manager takeovers, drift and dry-run refusals, unavailable metadata, human/silent output, and the ledger-save-failure regression. The permission-based ledger test ran without skipping. The earlier P1 remains fixed: a stranded takeover reports redirect_takeover_unpatched and exits 1.

CI follow-up: this same head now has 478 successful checks and 6 skipped, with no unresolved review threads and a clean merge against main 203e092b. The Windows vlt retry succeeded. Ready to merge from code review; no code changes needed.

[Mikola Lysenko (mikolalysenko)]

[agent] CI on 9d46d12: install-proof (windows-latest, 1.0.0-rc.22) was cancelled. All other checks are green: 461 passed, 6 skipped.

The job hung in vlt_pinned_matrix_vendored_durability: its last output is "has been running for over 60 seconds" at 16:45:22, and it was cancelled at 17:28:40. Every other vlt case in the job had passed by then.

I don't think this PR caused it:

• That test exercises vlt in vendored mode on Windows and never reaches the hosted takeover code this PR changes.
• The same job passed in under 2 minutes on aada684, this PR's head before main was merged in.
• The only difference between the two heads is the 16:31 merge of main, which brought in vlt/Bun changes (e.g. Fix Bun/vlt bundled copies left unpatched (#469, #471) #472).

No fix exists yet. I've re-run the failed job once. If it hangs again, I'll treat it as real and dig in.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

[burn-down agent] Re-verified Ready for review at 9d46d12786c60b79aa320f2d819243620c820b01 (main merge after the earlier label at 0be19d0).

• CI: 478/484 check runs green on this head (6 skipped by path/matrix filters), 0 failing or pending. The previously cancelled install-proof (windows-latest, 1.0.0-rc.22) job is now green.
• Mergeable: yes, no conflicts. The branch is 12 commits behind main; GitHub reports it merges cleanly.
• Bugbot: the Cursor Bugbot check passed on 9d46d12; 0 unresolved review threads.
• Reviewer focus: unchanged. The hosted takeover gate now includes pkg:pypi/, and the new redirect_takeover_unpatched guard exits 1.

Slack announcement not sent (no Slack send tool available this run).

---

Generated by Claude Code