diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index d9895a2ce..4ee20fe76 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -165,6 +165,8 @@ For a **9.0 root lock**, the CLI ensures `pnpm-workspace.yaml` carries `trustLoc The rewriter reads a fixed set of candidate files from the project root: the npm-family locks (`package-lock.json`, `npm-shrinkwrap.json`, `pnpm-lock.yaml`, `shrinkwrap.yaml`, `yarn.lock`, plus `.yarnrc.yml` for the berry cache-config gate, `bun.lock` / `bun.lockb`, and `vlt-lock.json` with `vlt.json` and `node_modules/.vlt-lock.json` read only), `requirements.txt` / `uv.lock` / `Pipfile.lock` (pipfile-spec 6; see the Pipenv section below) / `poetry.lock` (every Poetry lock generation from 1.0 on — the 0.12 `[metadata.hashes]` layout is refused because that installer ignores URL sources; a Poetry < 1.4 writer additionally gets `redirect_poetry_stale_install_risk`, see `docs/testing/poetry-compatibility.md`) / `pdm.lock` (PDM lock formats `2` and `4.3`–`4.5.1`; the identity-losing `3.1` / `4.0`–`4.2` formats and unknown future formats are refused with `redirect_pdm_refused`, and a lock-format-`2` writer additionally gets `redirect_pdm_legacy_sync_required`, see `docs/testing/pdm-compatibility.md`; when `uv.lock` or `poetry.lock` sits beside it they drive and `pdm.lock` is left alone), `Cargo.toml` / `Cargo.lock` / `.cargo/config.toml` (plus the legacy extensionless `.cargo/config` — cargo reads that spelling in preference when both exist, so the managed `[registries.…]` block is written into whichever one is present; **cargo also reads every workspace-member manifest** — the `[workspace] members` globs minus `exclude` — and every in-root path-dependency manifest, recursively, reached without crossing a symbolic link and never under `.socket/`, and pins the crate in each one that declares it, so those `/Cargo.toml` files can appear in `rewrittenFiles`. A crate is redirected only when every declaration pins and every other `Cargo.lock` package depending on it is a planned member: one a registry or git crate — or a path package outside the root or behind a link — also depends on is refused `redirect_cargo_transitive_dependents` (a pin reaches only the declarations it sits on), a crate no manifest declares keeps `redirect_cargo_toml_dep_not_found` with a transitive-only detail naming `--mode vendored`, a crate every declaration of which requires another version (no requirement accepts the patched version) is refused `redirect_cargo_toml_dep_unrewritable`, and so is a requirement that also matches another locked version of the crate — each a transactional skip, never recorded or attested. With NO `Cargo.lock` there is no resolved graph to ask, so the dependents question is answered from the manifests instead: a crate declared beside any other dependency — anything but a path dependency on a manifest this run also pins, or a `workspace = true` inheritor of a table it scans — or beside a workspace member this run did not read (a `members` glob, or a member outside the project or behind a symbolic link, which member discovery drops) is refused `redirect_cargo_lockless_dependents`, whose detail names the remedies (commit a lockfile, or `--mode vendored`); a project whose only dependency is the patched crate has nothing that could pull it in and still redirects. All-CRLF manifests, locks and configs are rewritten with CRLF kept (mixed endings keep refusing where the grammar does not match), and `remove` / rollback match the recorded fragments across a later CRLF↔LF checkout conversion), `composer.lock`, `nuget.config` / `packages.lock.json`, `Gemfile` / `Gemfile.lock`, `pom.xml` (+ `.mvn/maven.config` / `.mvn/checksums/checksums.sha256` for maven Trusted Checksums merge, and the Gradle build scripts read only to trigger the manual-snippet warning). **npm-family flavor coverage**: package-lock / npm-shrinkwrap, pnpm (root OR any nested `*/pnpm-lock.yaml`), yarn classic, **yarn berry** (the pin yarn writes for a root `resolutions` entry: the root `package.json` — edited only beside a berry `yarn.lock` — gains one `"@npm:": ""` selector per locked range (`redirect_yarn_berry_resolution` edits), and only that `yarn.lock` entry is re-keyed `"@"` with the same `resolution:` + `yarnBerry10c0` checksum (`redirect_yarn_berry_entry`), moved to yarn's key order; never an `npm:` locator, whose fetcher sends npm registry auth to the patch host, nor a tarball locator under an `npm:` key, which hardened mode rejects (YN0078). An older release's `npm:::__archiveUrl=` pin is still recognized and is re-pinned on the next run; rollback rebuilds the key from the selectors and drops them. Refused, nothing written: a user-authored `resolutions` entry for the package `redirect_yarn_berry_resolutions_conflict`, no root manifest `redirect_yarn_berry_manifest_missing`, a builtin `patch:` entry wrapping the same descriptor `redirect_yarn_berry_shared_descriptor`, an artifact URL yarn cannot fetch as a tarball `redirect_yarn_berry_artifact_url_unsupported`; cacheKey `10c0` and `.yarnrc.yml compressionLevel 0` gated by `redirect_yarn_berry_cache_unsupported`), and **bun** (text `bun.lock` lockfileVersion 0, 1 or 2 — 0 is the `--save-text-lockfile` opt-in lock of Bun 1.1.39–1.1.45, 1 the 1.2–1.3 default, 2 the 1.4+ default; all three emit one `packages` grammar, so the registry 4-tuple → URL 3-tuple rewrite is version-independent and the lock's own version line is kept. Any other or missing version, or a `packages` section outside bun's single-line grammar, is refused `redirect_bun_lock_unsupported` — the detail is the shared version gate's text (a newer version: update socket-patch, re-locking would reproduce it; no integer: re-lock with Bun ≥ 1.2), identical to the vendored refusal. A version-0 lock holding `workspace:` packages is refused `redirect_bun_workspace_unsupported` (its 2-tuple workspace grammar cannot keep the hosted tuple through a frozen install); the remedy is to delete `bun.lock` and re-run `bun install` with Bun ≥ 1.2, which writes lockfileVersion 1 (accepted). A plain in-place `bun install` bumps the version only when a workspace depends on another workspace (e.g. root → member — the shape the matrix measured); otherwise Bun 1.2.0 keeps version 0 and Bun 1.2.23+ fail to resolve, so the in-place bump is not the documented remedy. Bun lock version, grammar and workspace compatibility are checked before a vendored takeover, including during dry-run: these refusals preserve the existing lock, artifact and vendor ledger. Version-1 and version-2 workspace locks are rewritten, nested versions included. A granted dep with no rewritable entry warns `redirect_bun_entry_not_found`, a grant without a sha512 `redirect_bun_missing_sha512`; a CRLF lock keeps `\r\n` on the rewritten line, and a hosted URL left by an earlier grant of the same `name@version` is re-pinned in place. **Digest-less re-saves (Bun 1.1.39–1.3.9)**: every text-lock Bun below 1.3.10 re-saves a URL tuple WITHOUT its `sha512` whenever the lock is re-saved for another reason (`bun add`, `bun install` after a package.json or workspace change), leaving the 2-tuple `["name@", {meta}]` — the spec Bun installs from is intact. The CLI treats that spelling as its own wiring: a repeat hosted run counts the dep as redirected (no `redirect_bun_entry_not_found`) and HEALS the line back to the 3-tuple with the current `sha512`, recording the heal as a further `redirect_bun_lock_package` edit whose `original` is the 2-tuple (a stale URL is re-pinned from either spelling); `rollback`, scoped `rollback ` / `remove ` and the vendored takeover accept the digest-less spelling of a recorded `new` line (same key, spec and meta, only the trailing `"sha512-…"` missing) and restore the recorded original over it, so the chain always unwinds to the pristine registry line. Anything else — another uuid/token, another version, a re-laid meta object — is still drift. **Native `bun.lockb`**: when no text `bun.lock` exists, binary format versions 1, 2 and 3 are read and rewritten directly. Socket Patch does not invoke Bun or convert the project to a text lockfile. Exact matching package records are rewritten to hosted tarballs with the granted integrity, preserving dependency resolution IDs, workspace/dependency topology and unrelated package metadata; binary pointers and the package metadata hash are updated. Per-package `redirect_bun_lockb_package` snapshots support scoped rollback, repeat runs, superseding grants and hosted ↔ vendored takeover. A regular binary lock is discoverable even with no Bun runtime or `node_modules`; a dry run previews the same binary edits without writing them. A malformed, unreadable, unsupported or unverified binary structure is `redirect_bun_lockb_invalid` (exit 0, `redirected: 0`), and it refuses the npm rewrite before any takeover or sibling npm-family lock mutation. A symlinked binary write target is `redirect_symlinked_file_unsupported` (exit 1, including dry-run). `bun.lock` wins when both spellings exist. Binary-only projects do not receive `redirect_npm_no_lockfile`. Measured boundaries and the real-Bun matrix: `docs/testing/bun-compatibility.md`), and **vlt** (`vlt-lock.json` without `lockfileVersion`, `0` or `1`; see the vlt hosted-mode contract below). **Rush monorepos**: when `rush.json` is present the rewriter also reads `common/config/rush/pnpm-lock.yaml` and each `common/config/subspaces//pnpm-lock.yaml` (sorted for determinism) under their repo-relative keys and repoints them in place; editing them emits `redirect_rush_repo_state_stale` when `common/config/rush/repo-state.json` exists (the `pnpmShrinkwrapHash` desync is refreshed by `rush update`, which the redirect survives). **maven** is fail-closed via version suffixing: a `mavenSuffixedVersion` + `mavenPomSha256` override pins the Socket-only `-socket.` by rewriting the literal `` (`redirect_maven_dep_version`) or adding a `` entry (`redirect_maven_dep_management_added`), plus optional Trusted Checksums (`redirect_maven_trusted_checksums`, conflicts as `redirect_maven_trusted_checksums_conflict`; when `.mvn/wrapper/maven-wrapper.properties` pins a Maven older than 3.9.4, which ignores those files, the additive warning `redirect_maven_trusted_checksums_unenforced`); a `${property}` version is refused (`redirect_maven_dep_unpinned`), a non-matching literal skipped (`redirect_maven_dep_version_mismatch`), and an override without a suffixed version falls back to same-GAV repository injection (`redirect_maven_same_gav_fallback`, NOT fail-closed). +**Non-UTF-8 candidate files (#721)**: the rewriters edit UTF-8 text only. A candidate file that exists but is not UTF-8 (for example a UTF-16 `requirements.txt`, which is what Windows PowerShell 5.1's `pip freeze >` writes and which pip installs from) is never read as absent. When a candidate of its ecosystem could rewrite it, the run is refused with `candidate_file_unreadable` (exit 1, `--dry-run` included), the message names the file, and nothing is written; the remedy is to re-save the file as UTF-8. A vendored→hosted takeover checks this before it reverts anything, so a refused run leaves the vendored wiring, ledger entry and artifact byte-identical. Vendored mode likewise refuses a non-UTF-8 `requirements.txt` or `-r` include by name (`pypi_no_requirements`) instead of wiring around it. Lock-only discovery reads `requirements.txt` and its in-root `-r` includes the way pip decodes them (a UTF-16 or UTF-32 byte-order mark selects that encoding), so such a project's pins are still found instead of reporting "No packages found". + **Gem stale-install guard (additive warning — the canonical narrative; other mentions point here)**: the gem hosted rewrite is pure Gemfile/lock text, so a gem ALREADY materialized under the project's bundle paths keeps its upstream bytes — the next `bundle install` prints `Using ` and never refetches, on **every** bundler major (live-verified 2026-08-19 on 1.17.3 / 2.7.2 / 4.0.18: bundler 4's CHECKSUMS verify at download time only, and nothing is downloaded; `bundle install --force`/`--redownload` re-install from the stale cached `.gem` instead of re-fetching — bundler 1 silently, bundler 4 with an exit-37 checksum refusal that still leaves the upstream bytes installed; the **verified** remedy is removing the installed dir + cache `.gem` + `specifications` entry, then `bundle install`). After the rewrite, a hosted run therefore probes the installed-gem discovery paths (the same ruby-crawler discovery `apply` uses, honoring `--global`/`--global-prefix` like scan's own discovery) for each confirmed gem redirect and judges the materialization against the patch record's `afterHash` file map. Judgment rules: records are found **by uuid** among this run's fetched records (v5.0: hosted mode persists no records, so a purl whose `/patches/view` fetch failed this run is not judged; the warning re-fires on every re-scan whose fetch succeeds, until the stale materialization is gone); a materialization with every file at `afterHash` is already patched and never warns (an agent→hosted migration stays quiet by construction), and when several confirmed variant purls resolve to one installed dir, ANY of them judging it patched keeps it quiet; staleness needs **positive evidence** — at least one record file whose bytes were actually read and hash to neither state's expectation — so missing or unreadable files never produce a warning. Warnings emit `redirect_gem_stale_install` (JSON `redirect.warnings[]` + a code-tagged stderr line) in three flavors: a PROJECT-LOCAL dir gets the verified delete-list remedy (installed dir, cache `.gem`, `specifications` entry — plus the project's committed `/.gem` when present and not proven to be the patched artifact, since bundler installs from its cache dir in preference to fetching); a SHARED gem-env home gets a caveat that the home is shared machine-wide and prefers migrating the project to a local bundle path over deleting shared files; and a committed cache-dir archive whose sha256 differs from the patched artifact's warns standalone even with no installed dir at all (a fresh checkout with a committed stale cache re-materializes the upstream bytes forever). A stale-flagged purl is additionally **excluded from the same run's `--vex` `assume_applied` set** — the envelope must never attest a CVE its own warning says is live; the purl falls back to normal installed-tree verification (a patched install still attests, a stale one is omitted). The cache dir is bundler's `cache_path` setting (`Bundler.app_cache`), resolved in `Bundler::Settings` priority: `BUNDLE_CACHE_PATH:` in the bundler app config (`$BUNDLE_APP_CONFIG/config`, else `.bundle/config`) first, then the `BUNDLE_CACHE_PATH` environment variable, then `BUNDLE_CACHE_PATH:` in the global config (`bundle config set --global`: `$BUNDLE_CONFIG`, else `$BUNDLE_USER_CONFIG`, else `$BUNDLE_USER_HOME/config`, else `~/.bundle/config`), else `vendor/cache`; a relative value is read against the project root. The same global tier, below the app config and the environment, applies to `BUNDLE_GEMFILE:` and, for agent-mode install-root discovery, to `BUNDLE_PATH:`. A present local or environment `path`, `path.system`, or `disable_shared_gems` setting (including an empty string or false flag) shadows the global path tier, matching the tested Bundler 2.6/4 behavior; Bundler 1.x's legacy global-path shortcut is not modeled. An empty higher-tier `gemfile` setting also shadows the global value but leaves an existing nonempty `BUNDLE_GEMFILE` environment value in effect, or uses default manifest discovery when there is none. With `BUNDLE_IGNORE_CONFIG` set (any value) bundler reads no config file, so the app and global configs are skipped here too and only the environment and the default count — the same holds for the `BUNDLE_GEMFILE:` app-config setting. The probe is read-only (nothing is deleted) and skipped on `--dry-run` — deliberately explicit, since nothing was rewritten. Exit code and `status` are unchanged (warning-only, the hosted-refusal posture); a same-run `--vex` may still fail on "nothing to attest" per the embedded-VEX contract. **Pipenv hosted redirect (`Pipfile.lock`, pipfile-spec 6)**: every category other than `_meta` (`default`, `develop`, and Pipenv 2022+ named categories) that pins the package at the patched version is rewritten to the hosted reference — `{"file" | "path": "#sha256=", "hashes": ["sha256:"]}` with `markers`/`extras`/`index` kept exactly as Pipenv wrote them (present or absent: whether Pipenv records `index` depends on its release, the Pipfile spelling and the locking environment, so only the entry itself knows) and `version` dropped; `_meta` (the Pipfile content hash) and the Pipfile itself are never touched, so `pipenv install --deploy`/`sync`/`verify` keep passing. The reference KEY depends on the installing Pipenv: releases 7–11 only install `path` references, 2018 and later `file` ones (0–6 write pipfile-spec < 6 and are refused). The release is probed once per command with `pipenv --version`, resolved on ABSOLUTE `PATH` entries only (a relative entry would run a `pipenv` planted in the scanned repository; `.bat`/`.cmd` shims are found through `PATHEXT` on Windows), only when a pypi patch actually targets an entry of the lock, and `SOCKET_PIPENV_MAJOR=` pins the answer without spawning anything. An unknown installer selects `file` and warns `redirect_pipenv_installer_unknown` only when the lock was rewritten. **Refusal scope**: a pin/source CONFLICT (another version pinned, a foreign `file`/`path` source, a VCS/editable dependency) refuses the whole dependency atomically across categories as `redirect_pipenv_refused` AND vetoes the sibling Python rewriters (requirements.txt / uv.lock / pyproject) for that patch — the project's Pipenv install could not pick the patch up, so a half-redirected checkout is refused; anything else (no entry for the package, an old pipfile-spec, an unparseable lock, a digest-less patch) is `redirect_pipenv_skipped` and leaves the siblings alone (a stale Pipfile.lock in a uv/Poetry/requirements project must not block them). The veto applies to a LIVE lock only: a `Pipfile.lock` with no `Pipfile` beside it is abandoned, so its conflict refuses that file but never the siblings. Hash enforcement at install time is split by era — the `#sha256=` URL fragment is what Pipenv 2023+ verifies, the `hashes` list what 2018–2022 verify, Pipenv 11 either — so both are load-bearing. **Pipenv stale-install guard**: Pipenv never reinstalls a release that is already present (`pipenv install`, `install --deploy` and `sync` all exit 0 and keep the installed bytes — measured on 11.10.4, 2018.11.26 and 2026.8.0, hosted and vendored), so after the rewrite the run probes the Python crawler's site-packages (VIRTUAL_ENV, `./.venv`, `./venv`, Pipenv's out-of-tree `WORKON_HOME` venv; `--global`/`--global-prefix` honoured) for each confirmed Pipfile.lock redirect with the same rules as the gem guard (records by uuid from this run's fetch, PATCHED = `verify_patch_record` Ok, STALE needs positive evidence, read-only, skipped on `--dry-run`, stale purls excluded from the same-run `--vex` `assume_applied` set) and the Python stale-install guard (`redirect_pypi_stale_install`, see above) names the site-packages dir and the Pipenv-specific verified remedy: `pipenv run pip uninstall -y && pipenv sync` (or `pipenv --rm && pipenv sync`), with the `sync` arguments following the lock, since plain `pipenv sync` installs only `default`: the targeted form re-syncs the categories that pin the package (`--dev` for `develop`, `--categories ""` for a named category) and the `--rm` form re-syncs every non-empty category — NOT `pipenv uninstall`, which rewrites the Pipfile and re-locks the patch away. The vendored backend emits the twin `pypi_pipenv_stale_install` (`skipped` warning event). **Rollback** (v5.0, upstream restore): each hosted entry gets its registry shape back — `"version": "=="`, the entry's own `index` carried back unchanged (refused unless it — and the Pipfile's explicit `index`, if any — names a PyPI source in `_meta.sources`), and every release file's sha256 from PyPI's JSON API (`SOCKET_PYPI_JSON_API`), sorted by filename as Pipenv records them; an entry that pins another version beside the hosted reference is refused with the `git checkout` remedy (see "Hosted unwind coverage"). A Pipfile names no project, so a same-run `--vex` on a Pipenv project needs `--vex-product` (or a git remote) to detect a product purl. **Discovery**: `Pipfile.lock` is part of the lockfile inventory (every category's `==` pins, with the lock's digest set as `Sha256AnyOf` integrity so a lock-only checkout can be vendored by fetching the pure wheel through PyPI's JSON API — only when `_meta.sources` name the public index; a private-index lock stays discovery-only and never reaches pypi.org), and Socket's own hosted / vendored references stay discoverable as the package they replace, so a re-scan of an already-redirected or already-vendored lock-only checkout re-confirms it (`--vex` attests, vendored reports `already_vendored`) instead of finding nothing. diff --git a/crates/socket-patch-cli/src/commands/scan/hosted.rs b/crates/socket-patch-cli/src/commands/scan/hosted.rs index 94a4bc247..ce3000dd4 100644 --- a/crates/socket-patch-cli/src/commands/scan/hosted.rs +++ b/crates/socket-patch-cli/src/commands/scan/hosted.rs @@ -1781,6 +1781,26 @@ async fn vendored_takeover( .filter(|_| entry.is_some_and(vlt_entry)) }) }; + // NON-UTF-8 PRE-CHECK (#721) — the GUARD's undecodable-file rule + // (`engine::undecodable_guard`), checked BEFORE any revert dispatches + // (and under --dry-run too): a takeover that reverted first and was + // then refused by the guard would leave the reverted purls unpatched + // in both modes. + if takeover.iter().any(|(_, entry)| entry.is_some()) { + let view = socket_patch_core::vendor::lock_inventory::ProjectView::Disk(&common.cwd); + let read = socket_patch_core::hosted::engine::read_candidate_files( + &view, + &std::collections::BTreeSet::new(), + candidates, + ) + .await; + if let Some(refusal) = socket_patch_core::hosted::engine::undecodable_guard( + &read.undecodable_reads, + candidates, + ) { + return Err(refusal); + } + } // SYMLINK PRE-CHECK for the takeover reverts — the same rule as the // SYMLINK GUARD below, applied to each ledger entry's recorded wiring // (the revert backends also stage and rename over the file). Checked diff --git a/crates/socket-patch-cli/tests/in_process_get_hosted_ecosystems.rs b/crates/socket-patch-cli/tests/in_process_get_hosted_ecosystems.rs index db2aab79f..efeaa64c5 100644 --- a/crates/socket-patch-cli/tests/in_process_get_hosted_ecosystems.rs +++ b/crates/socket-patch-cli/tests/in_process_get_hosted_ecosystems.rs @@ -315,6 +315,61 @@ async fn pypi_requirements_hosted_rewrites_pep440_equivalent_pin() { } } +/// #721: Windows PowerShell 5.1 writes `pip freeze > requirements.txt` as +/// UTF-16 with a BOM, and pip installs from it. The hosted grant must not +/// treat that file as absent and exit 0 with the project unpatched: it is +/// refused by name (`candidate_file_unreadable`, exit 1), nothing written. +#[tokio::test] +#[serial] +async fn pypi_requirements_hosted_refuses_a_utf16_file() { + const UUID: &str = "a1a1a1a1-a1a1-4a1a-8a1a-a1a1a1a1a1a3"; + const PURL: &str = "pkg:pypi/requests@2.31.0"; + const SHA256: &str = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"; + let url = format!( + "http://patch.test/patch/pypi/requests/2.31.0/{TOKEN}/{UUID}/requests-2.31.0-py3-none-any.whl" + ); + + let text = "flask==2.0.1\r\nrequests==2.31.0\r\n"; + let le: Vec = [0xFF, 0xFE] + .into_iter() + .chain(text.encode_utf16().flat_map(u16::to_le_bytes)) + .collect(); + let be: Vec = [0xFE, 0xFF] + .into_iter() + .chain(text.encode_utf16().flat_map(u16::to_be_bytes)) + .collect(); + for (what, bytes) in [("utf-16le", le), ("utf-16be", be)] { + let server = MockServer::start().await; + mock_view(&server, UUID, PURL).await; + mock_reference( + &server, + UUID, + PURL, + &url, + serde_json::json!({ "sha256": SHA256 }), + serde_json::Value::Null, + ) + .await; + + let tmp = tempfile::tempdir().unwrap(); + std::fs::write(tmp.path().join("requirements.txt"), &bytes).unwrap(); + + let code = + socket_patch_cli::commands::get::run(get_hosted_args(UUID, tmp.path(), server.uri())) + .await; + assert_eq!( + code, 1, + "{what}: a requirements.txt hosted mode cannot read must refuse, not exit 0 unpatched" + ); + assert_eq!( + std::fs::read(tmp.path().join("requirements.txt")).unwrap(), + bytes, + "{what}: the refused file must stay byte-identical" + ); + assert_no_manifest_no_blobs(tmp.path()); + } +} + // --------------------------------------------------------------------------- // maven — pom.xml fail-closed suffixed-version pin (rewrite_maven_pom) // --------------------------------------------------------------------------- diff --git a/crates/socket-patch-cli/tests/mode_migration_pypi.rs b/crates/socket-patch-cli/tests/mode_migration_pypi.rs index c2d92de1e..27ba3c3ba 100644 --- a/crates/socket-patch-cli/tests/mode_migration_pypi.rs +++ b/crates/socket-patch-cli/tests/mode_migration_pypi.rs @@ -534,6 +534,64 @@ async fn uv_takeover_without_wheel_metadata_fails_loudly() { /// the revert (the artifact and ledger entry are kept). The takeover must /// then refuse — keeping the ledger — rather than drop the entry and leave /// the project half vendored with no record of it. +/// #721: a non-UTF-8 candidate file (here a UTF-16 `pip freeze` export +/// beside a vendored Poetry project) refuses the hosted run BEFORE the +/// takeover reverts anything, wet and `--dry-run` alike: refusing only at +/// the rewrite would leave the reverted poetry.lock unpatched in both modes. +#[tokio::test] +async fn undecodable_candidate_refuses_before_the_takeover_reverts() { + let (_tmp, root) = project(); + std::fs::write( + root.join("pyproject.toml"), + "[tool.poetry]\nname = \"demo\"\nversion = \"0.1.0\"\ndescription = \"\"\nauthors = [\"x \"]\npackage-mode = false\n\n[tool.poetry.dependencies]\npython = \">=3.9\"\nsix = \"1.16.0\"\n", + ) + .unwrap(); + std::fs::write( + root.join("poetry.lock"), + POETRY_LOCK + .replace("WHEEL_SHA", WHEEL_SHA) + .replace("SDIST_SHA", SDIST_SHA), + ) + .unwrap(); + vendor_project(&root, &["poetry.lock", "pyproject.toml"]); + let mut utf16 = vec![0xFF, 0xFE]; + for unit in "six==1.16.0\r\n".encode_utf16() { + utf16.extend(unit.to_le_bytes()); + } + std::fs::write(root.join("requirements.txt"), &utf16).unwrap(); + let lock = std::fs::read_to_string(root.join("poetry.lock")).unwrap(); + let state = root.join(".socket/vendor/state.json"); + + let server = MockServer::start().await; + mount_hosted_api(&server, true).await; + let uri = server.uri(); + for dry_run in [true, false] { + let mut args = hosted_scan_args(&uri); + if dry_run { + args.push("--dry-run"); + } + let (code, env) = run_cli(&root, &args, &[]); + assert_eq!(code, 1, "dry_run={dry_run}: {env:#}"); + let text = env.to_string(); + assert!( + text.contains("candidate_file_unreadable") && text.contains("requirements.txt"), + "dry_run={dry_run}: {env:#}" + ); + assert!( + !text.contains("redirect_takeover_reverted_vendored"), + "dry_run={dry_run}: nothing is reverted: {env:#}" + ); + assert_eq!( + std::fs::read_to_string(root.join("poetry.lock")).unwrap(), + lock, + "dry_run={dry_run}: the vendored lock is untouched" + ); + assert!(std::fs::read_to_string(&state).unwrap().contains(UUID)); + assert!(root.join(format!(".socket/vendor/pypi/{UUID}")).exists()); + assert_eq!(std::fs::read(root.join("requirements.txt")).unwrap(), utf16); + } +} + #[tokio::test] async fn drifted_vendored_line_refuses_takeover() { let (_tmp, root) = project(); diff --git a/crates/socket-patch-cli/tests/scan_requirements_lock_only.rs b/crates/socket-patch-cli/tests/scan_requirements_lock_only.rs index 9ab939607..3ed496c3c 100644 --- a/crates/socket-patch-cli/tests/scan_requirements_lock_only.rs +++ b/crates/socket-patch-cli/tests/scan_requirements_lock_only.rs @@ -5,7 +5,9 @@ //! release: //! //! * #523: whitespace around `==` and the legacy `name (==X)` form; -//! * #412: pins reached through in-root `-r` includes. +//! * #412: pins reached through in-root `-r` includes; +//! * #721: a UTF-16 file with a BOM (Windows PowerShell 5.1's +//! `pip freeze >` output), which pip decodes. //! //! Driven through the built binary against a mock patch API; the //! assertion is what discovery sends to the batch endpoint and the @@ -87,6 +89,11 @@ async fn batch_purls(mock: &MockServer) -> Vec { } async fn assert_lock_only_discovers(files: &[(&str, &str)], expected: &[&str]) { + let files: Vec<(&str, &[u8])> = files.iter().map(|(r, c)| (*r, c.as_bytes())).collect(); + assert_lock_only_discovers_bytes(&files, expected).await; +} + +async fn assert_lock_only_discovers_bytes(files: &[(&str, &[u8])], expected: &[&str]) { for mode in [&[][..], &["--vendor"][..]] { let mock = MockServer::start().await; mount_empty_batch(&mock).await; @@ -148,3 +155,29 @@ async fn lock_only_scan_discovers_included_pins() { ) .await; } + +/// #721: pip decodes a requirements file by its BOM, so a UTF-16 file +/// (what Windows PowerShell 5.1's `pip freeze >` writes) is discovered, +/// in either byte order, instead of reading as "No packages found". +#[tokio::test] +async fn lock_only_scan_discovers_utf16_pins() { + let text = "sp-fixture-idna==3.7\r\nsp-fixture-six==1.16.0\r\n"; + let le: Vec = [0xFF, 0xFE] + .into_iter() + .chain(text.encode_utf16().flat_map(u16::to_le_bytes)) + .collect(); + let be: Vec = [0xFE, 0xFF] + .into_iter() + .chain(text.encode_utf16().flat_map(u16::to_be_bytes)) + .collect(); + for bytes in [le, be] { + assert_lock_only_discovers_bytes( + &[("requirements.txt", &bytes)], + &[ + "pkg:pypi/sp-fixture-idna@3.7", + "pkg:pypi/sp-fixture-six@1.16.0", + ], + ) + .await; + } +} diff --git a/crates/socket-patch-core/src/hosted/engine.rs b/crates/socket-patch-core/src/hosted/engine.rs index c197f36a9..b1cab9030 100644 --- a/crates/socket-patch-core/src/hosted/engine.rs +++ b/crates/socket-patch-core/src/hosted/engine.rs @@ -69,7 +69,9 @@ pub const SYMLINK_REFUSAL: &str = "redirect_symlinked_file_unsupported"; /// Refusal code for a candidate file that exists but whose content the /// in-memory host did not provide (oversize, an LFS pointer, -/// presence-only); disk would read and rewrite it. +/// presence-only), and, on disk and in memory alike, for one that is not +/// UTF-8 text (#721): no rewriter can edit it, and reading it as absent +/// would leave its pins unpatched behind an exit-0 run. pub const UNREADABLE_REFUSAL: &str = "candidate_file_unreadable"; /// Rush's repo-state file, whose `pnpmShrinkwrapHash` a lock edit @@ -155,6 +157,18 @@ fn unreadable_refusal(rel: &str) -> Refusal { } } +fn undecodable_refusal(rel: &str) -> Refusal { + Refusal { + code: UNREADABLE_REFUSAL.to_string(), + message: format!( + "{rel} is not UTF-8 text (for example UTF-16, which Windows PowerShell 5.1 \ + writes for `pip freeze > requirements.txt`), so it cannot be rewritten \ + alongside the other lockfiles; re-save it as UTF-8 and re-run; nothing was \ + written" + ), + } +} + /// Reference grants → candidates. A selection without a usable grant is /// recorded in `skipped` (`not_found`, the reference status, `bad_purl`, /// `no_url`). @@ -303,6 +317,11 @@ pub struct CandidateFiles { /// project whose candidates could rewrite (or whose rewrite depends on) /// one is refused, since the rewriters would treat it as absent. pub unreadable_reads: Vec, + /// Candidate files that exist but are not UTF-8 text (a UTF-16 + /// requirements.txt pip reads, #721), on disk and in memory alike. They + /// are left out of `files`; a project whose candidates could rewrite + /// one is refused rather than read as if the file were absent. + pub undecodable_reads: Vec, /// Set when bundler is configured (`BUNDLE_GEMFILE`) to load a manifest /// the gem rewriter cannot edit: every gem manifest and lock was left /// out of `files`, and the rewrite reports this instead of a redirect. @@ -322,7 +341,14 @@ impl CandidateFiles { // (non-blocking open + fstat regular-file check), so a FIFO // under a candidate name is skipped like a missing file instead // of wedging the run in open(2). - ProjectView::Disk(_) | ProjectView::Snapshot(_) => view.read_text(rel).await.ok(), + ProjectView::Disk(_) | ProjectView::Snapshot(_) => match view.read_text(rel).await { + Ok(text) => Some(text), + Err(e) if e.kind() == std::io::ErrorKind::InvalidData => { + self.undecodable_reads.push(rel.to_string()); + None + } + Err(_) => None, + }, ProjectView::Memory(project) => { if project.is_symlink(rel) { self.symlinked_reads.push(rel.to_string()); @@ -332,13 +358,17 @@ impl CandidateFiles { self.unreadable_reads.push(rel.to_string()); return false; } - // Disk reads any UTF-8 regular file; a non-UTF-8 one is - // absent to it as well. + // Disk reads any UTF-8 regular file and records a non-UTF-8 + // one as undecodable; so does memory. match project.get(rel) { Some(MemoryEntry::Text(text)) => Some(text.to_string()), - Some(MemoryEntry::Binary(bytes)) => { - std::str::from_utf8(bytes).ok().map(str::to_string) - } + Some(MemoryEntry::Binary(bytes)) => match std::str::from_utf8(bytes) { + Ok(text) => Some(text.to_string()), + Err(_) => { + self.undecodable_reads.push(rel.to_string()); + None + } + }, _ => None, } } @@ -547,6 +577,8 @@ pub async fn read_candidate_files( out.symlinked_reads.dedup(); out.unreadable_reads.sort(); out.unreadable_reads.dedup(); + out.undecodable_reads.sort(); + out.undecodable_reads.dedup(); out } @@ -595,6 +627,7 @@ async fn keep_bundler_loaded_gem_files(view: &ProjectView<'_>, out: &mut Candida out.files.retain(|rel, _| !dropped(rel)); out.symlinked_reads.retain(|rel| !dropped(rel)); out.unreadable_reads.retain(|rel| !dropped(rel)); + out.undecodable_reads.retain(|rel| !dropped(rel)); out.gem_manifest_unsupported = loaded.unsupported_detail().map(|detail| RewriteWarning { code: "redirect_gem_bundle_gemfile_unsupported".into(), detail, @@ -763,6 +796,7 @@ pub struct Rewritten { pub files: BTreeMap, pub symlinked_reads: Vec, pub unreadable_reads: Vec, + pub undecodable_reads: Vec, /// The rewriters' override slice (the candidates' deps). pub overrides: Vec, pub rewrite: RewriteResult, @@ -912,6 +946,7 @@ pub async fn rewrite( rush_lock_keys, symlinked_reads, unreadable_reads, + undecodable_reads, gem_manifest_unsupported, } = read; // The rewriters' override slice — materialized ONCE, after the last @@ -1066,6 +1101,7 @@ pub async fn rewrite( files, symlinked_reads, unreadable_reads, + undecodable_reads, overrides, rewrite, rewritten, @@ -1595,6 +1631,19 @@ fn file_ecosystem(rel: &str) -> Option<&'static str> { .then_some("pypi") } +/// The [`guard`]'s non-UTF-8 rule on its own (#721): the first of +/// `undecodable` (a [`CandidateFiles::undecodable_reads`]) whose ecosystem +/// has a candidate refuses the run. The vendored→hosted takeover runs it +/// before reverting anything, so a refusal never strands a reverted purl. +pub fn undecodable_guard(undecodable: &[String], candidates: &[Candidate]) -> Option { + undecodable + .iter() + .find(|rel| { + file_ecosystem(rel).is_some_and(|eco| candidates.iter().any(|c| c.dep.ecosystem == eco)) + }) + .map(|rel| undecodable_refusal(rel)) +} + /// SYMLINK GUARD — fail-closed, whole rewrite, before the ledger and before /// any write (hosted rewrites are transactional). The writer stages next to /// the path and renames over it, which REPLACES a symbolic link with a @@ -1602,6 +1651,9 @@ fn file_ecosystem(rel: &str) -> Option<&'static str> { /// bytes but never the link. Applies to every ecosystem's files and to dry /// runs, so a dry run predicts the refusal. /// +/// On disk and in memory: a candidate file that is not UTF-8 text, when a +/// candidate of its ecosystem could rewrite it (#721). +/// /// In memory, additionally: a candidate file read through a link (its bytes /// are unknown) or present without content, when a candidate of its /// ecosystem could rewrite it. @@ -1622,13 +1674,16 @@ pub fn guard( if let Some(linked) = written().find(|k| view.is_symlink(k)) { return Some(symlink_refusal(linked)); } - let ProjectView::Memory(project) = view else { - return None; - }; + if let Some(refusal) = undecodable_guard(&done.undecodable_reads, candidates) { + return Some(refusal); + } let candidate_ecosystems: BTreeSet<&str> = candidates .iter() .map(|c| c.dep.ecosystem.as_str()) .collect(); + let ProjectView::Memory(project) = view else { + return None; + }; if let Some(linked) = done .symlinked_reads .iter() @@ -1786,6 +1841,95 @@ mod tests { assert!(read.unreadable_reads.is_empty()); } + /// #721: a candidate file that is not UTF-8 (a UTF-16 requirements.txt, + /// which pip reads) is refused by name, on disk and in memory alike, + /// when a candidate of its ecosystem could rewrite it, instead of being + /// treated as absent (exit 0, nothing pinned, no diagnostic). + #[tokio::test] + async fn an_undecodable_candidate_file_refuses_its_ecosystem() { + let purl = "pkg:pypi/six@1.16.0"; + let uuid = "u-721"; + let mut refs = HashMap::new(); + refs.insert( + uuid.to_string(), + reference(serde_json::json!({ + "status": "granted", + "url": format!("https://patch.example/patch/pypi/six/1.16.0/tok/{uuid}/six-1.16.0-py2.py3-none-any.whl"), + "purl": purl, + "artifacts": [{"kind": "tarball", "url": null, "integrity": {"sha256": "ab"}}], + "registryOverride": null + })), + ); + let selected = vec![(purl.to_string(), uuid.to_string())]; + let mut skipped = Vec::new(); + let candidates = build_candidates(&selected, &refs, &mut skipped); + assert_eq!(candidates.len(), 1, "{skipped:?}"); + let utf16: Vec = [0xFF, 0xFE] + .into_iter() + .chain( + "idna==3.7\r\nsix==1.16.0\r\n" + .encode_utf16() + .flat_map(u16::to_le_bytes), + ) + .collect(); + let outer = OuterAllowRemote::default; + let options = || RewriteOptions { + dry_run: false, + targets_pipenv_lock: false, + pipenv_major: None, + pipenv_unknown_detail: String::new(), + trust_lockfile_config: true, + npm_allow_remote_config: true, + npm_outer: &outer, + blocking: false, + }; + + let tmp = tempfile::tempdir().unwrap(); + std::fs::write(tmp.path().join("requirements.txt"), &utf16).unwrap(); + let mut memory = MemoryProject::new(); + memory.insert( + "requirements.txt", + MemoryEntry::Binary(utf16.clone().into()), + ); + for view in [ProjectView::Disk(tmp.path()), ProjectView::Memory(&memory)] { + let read = read_candidate_files(&view, &BTreeSet::new(), &candidates).await; + assert_eq!(read.undecodable_reads, vec!["requirements.txt"]); + let done = rewrite( + &view, + read, + &candidates, + BTreeMap::new(), + &BTreeSet::new(), + &[], + options(), + ) + .await; + let refusal = guard(&view, &done, &candidates).expect("refused"); + assert_eq!(refusal.code, UNREADABLE_REFUSAL); + assert!( + refusal.message.contains("requirements.txt") && refusal.message.contains("UTF-8"), + "{}", + refusal.message + ); + + // Another ecosystem's run is not blocked by it. + let (cargo_selected, cargo_refs) = cargo_reference("u-2"); + let cargo = build_candidates(&cargo_selected, &cargo_refs, &mut Vec::new()); + let read = read_candidate_files(&view, &BTreeSet::new(), &cargo).await; + let done = rewrite( + &view, + read, + &cargo, + BTreeMap::new(), + &BTreeSet::new(), + &[], + options(), + ) + .await; + assert!(guard(&view, &done, &cargo).is_none()); + } + } + /// A hosted URL left in a berry project's `package.json` `resolutions` /// while `yarn.lock` still resolves the registry entry confirms nothing: /// only the lock pin installs (#404). diff --git a/crates/socket-patch-core/src/hosted/memory/mod.rs b/crates/socket-patch-core/src/hosted/memory/mod.rs index 85a0d3c48..ec5322775 100644 --- a/crates/socket-patch-core/src/hosted/memory/mod.rs +++ b/crates/socket-patch-core/src/hosted/memory/mod.rs @@ -1336,6 +1336,7 @@ mod tests { files: BTreeMap::new(), symlinked_reads: Vec::new(), unreadable_reads: Vec::new(), + undecodable_reads: Vec::new(), overrides: Vec::new(), rewrite, rewritten: files.iter().map(|(rel, _)| (*rel).to_string()).collect(), diff --git a/crates/socket-patch-core/src/utils/requirements.rs b/crates/socket-patch-core/src/utils/requirements.rs index b13f634d3..d22edd761 100644 --- a/crates/socket-patch-core/src/utils/requirements.rs +++ b/crates/socket-patch-core/src/utils/requirements.rs @@ -14,6 +14,42 @@ //! `--hash=sha256:ab#cd` are data. Exactly one leading BOM is encoding, not //! data (pip decodes with utf-8-sig; uv strips it too). +/// Decode a requirements file the way pip's `auto_decode` does: a UTF-16 +/// or UTF-32 byte-order mark selects that encoding and is dropped; anything +/// else is UTF-8, its one leading BOM kept for [`logical_lines`] to drop. +/// Windows PowerShell 5.1 writes `pip freeze > requirements.txt` as UTF-16 +/// LE with a BOM, and pip installs from it (#721). pip tries the UTF-16 +/// marks first, so a UTF-32 LE mark (`FF FE 00 00`) reads as UTF-16 LE, as +/// it does for pip. `None` when the bytes are not valid in that encoding. +/// (pip's last resort, the locale's encoding for a mark-less non-UTF-8 +/// file, is machine-dependent and not modelled.) +pub(crate) fn decode(bytes: &[u8]) -> Option { + fn utf16(body: &[u8], unit: fn([u8; 2]) -> u16) -> Option { + if !body.len().is_multiple_of(2) { + return None; + } + char::decode_utf16(body.chunks_exact(2).map(|c| unit([c[0], c[1]]))) + .collect::>() + .ok() + } + if let Some(body) = bytes.strip_prefix(&[0xFF, 0xFE]) { + return utf16(body, u16::from_le_bytes); + } + if let Some(body) = bytes.strip_prefix(&[0xFE, 0xFF]) { + return utf16(body, u16::from_be_bytes); + } + if let Some(body) = bytes.strip_prefix(&[0x00, 0x00, 0xFE, 0xFF]) { + if !body.len().is_multiple_of(4) { + return None; + } + return body + .chunks_exact(4) + .map(|c| char::from_u32(u32::from_be_bytes([c[0], c[1], c[2], c[3]]))) + .collect(); + } + String::from_utf8(bytes.to_vec()).ok() +} + /// One logical requirements line. pub(crate) struct LogicalLine { /// 0-based index of the first physical line. @@ -236,6 +272,34 @@ pub(crate) fn url_sha256_fragment(location: &str) -> Option { mod tests { use super::*; + /// #721: pip's `auto_decode` BOM table, in pip's order. + #[test] + fn decode_follows_pips_byte_order_marks() { + let text = "six==1.16.0\r\n"; + let le: Vec = text.encode_utf16().flat_map(u16::to_le_bytes).collect(); + let be: Vec = text.encode_utf16().flat_map(u16::to_be_bytes).collect(); + let be32: Vec = text + .chars() + .flat_map(|c| (c as u32).to_be_bytes()) + .collect(); + let with = |bom: &[u8], body: &[u8]| [bom, body].concat(); + assert_eq!(decode(text.as_bytes()).as_deref(), Some(text)); + // The UTF-8 mark is left for `logical_lines`. + let bom8 = with(&[0xEF, 0xBB, 0xBF], text.as_bytes()); + assert_eq!(decode(&bom8).as_deref(), Some("\u{feff}six==1.16.0\r\n")); + assert_eq!(decode(&with(&[0xFF, 0xFE], &le)).as_deref(), Some(text)); + assert_eq!(decode(&with(&[0xFE, 0xFF], &be)).as_deref(), Some(text)); + assert_eq!( + decode(&with(&[0x00, 0x00, 0xFE, 0xFF], &be32)).as_deref(), + Some(text) + ); + // Not valid in the encoding the mark selects (or mark-less and not + // UTF-8): unreadable, never guessed. + assert_eq!(decode(&with(&[0xFF, 0xFE], &le[1..])), None); + assert_eq!(decode(&with(&[0xFF, 0xFE], &[0x00, 0xD8])), None); + assert_eq!(decode(&[b's', 0xC3, 0x28]), None); + } + #[test] fn requires_hashes_reads_pip_hash_checking_mode() { for hashed in [ diff --git a/crates/socket-patch-core/src/vendor/lock_inventory/pypi.rs b/crates/socket-patch-core/src/vendor/lock_inventory/pypi.rs index b1d154c9a..d21b6a87b 100644 --- a/crates/socket-patch-core/src/vendor/lock_inventory/pypi.rs +++ b/crates/socket-patch-core/src/vendor/lock_inventory/pypi.rs @@ -724,11 +724,17 @@ async fn inventory_requirements_txt(view: &ProjectView<'_>) -> Option) -> Option> { use crate::vendor::pypi_requirements::{is_in_root_rel, requirements_includes}; const ROOT: &str = "requirements.txt"; - let root = view.read_text(ROOT).await.ok()?; + let read = |rel: String| async move { + let bytes = view.read_bytes(&rel).await.ok()?; + crate::utils::requirements::decode(&bytes) + }; + let root = read(ROOT.to_string()).await?; let mut visited = std::collections::HashSet::from([ROOT.to_string()]); let mut stack: Vec = requirements_includes(ROOT, &root); stack.reverse(); @@ -737,7 +743,7 @@ async fn requirements_tree(view: &ProjectView<'_>) -> Option> { if !is_in_root_rel(&rel) || !visited.insert(rel.clone()) { continue; } - let Ok(text) = view.read_text(&rel).await else { + let Some(text) = read(rel.clone()).await else { continue; }; let mut includes = requirements_includes(&rel, &text); diff --git a/crates/socket-patch-core/src/vendor/lock_inventory/tests.rs b/crates/socket-patch-core/src/vendor/lock_inventory/tests.rs index 135103a78..8746ae1f1 100644 --- a/crates/socket-patch-core/src/vendor/lock_inventory/tests.rs +++ b/crates/socket-patch-core/src/vendor/lock_inventory/tests.rs @@ -3248,6 +3248,56 @@ async fn requirements_in_root_includes_are_inventoried() { assert_eq!(sorted_pairs(&in_memory), sorted_pairs(&entries)); } +/// #721: pip decodes a requirements file by its BOM, so a UTF-16 root +/// file and a UTF-16 include (what Windows PowerShell 5.1's `pip freeze >` +/// writes) are inventoried like their UTF-8 text, on disk and in memory. +#[tokio::test] +async fn requirements_utf16_files_are_inventoried() { + fn utf16(text: &str, le: bool) -> Vec { + let mut out = if le { + vec![0xFF, 0xFE] + } else { + vec![0xFE, 0xFF] + }; + for unit in text.encode_utf16() { + out.extend(if le { + unit.to_le_bytes() + } else { + unit.to_be_bytes() + }); + } + out + } + for le in [true, false] { + let root_bytes = utf16("-r requirements/base.txt\r\nidna==3.7\r\n", le); + let base_bytes = utf16("six==1.16.0\r\n", !le); + let tmp = tempfile::tempdir().unwrap(); + std::fs::create_dir_all(tmp.path().join("requirements")).unwrap(); + std::fs::write(tmp.path().join("requirements.txt"), &root_bytes).unwrap(); + std::fs::write(tmp.path().join("requirements/base.txt"), &base_bytes).unwrap(); + let entries = inventory_pypi_locks(tmp.path()).await.unwrap(); + assert_eq!( + sorted_pairs(&entries), + vec![ + ("idna".to_string(), "3.7".to_string()), + ("six".to_string(), "1.16.0".to_string()), + ], + "le={le}: {entries:?}" + ); + + let mut project = MemoryProject::new(); + project.insert("requirements.txt", MemoryEntry::Binary(root_bytes.into())); + project.insert( + "requirements/base.txt", + MemoryEntry::Binary(base_bytes.into()), + ); + let in_memory = super::pypi::inventory_pypi_locks_in(&ProjectView::Memory(&project)) + .await + .unwrap(); + assert_eq!(sorted_pairs(&in_memory), sorted_pairs(&entries)); + } +} + /// pip applies an index option from ANY file of the tree globally, so an /// `--index-url` inside an include keeps the root file's hashed pins /// unverifiable too (the `public_index` rule spans the whole tree). diff --git a/crates/socket-patch-core/src/vendor/pypi_requirements.rs b/crates/socket-patch-core/src/vendor/pypi_requirements.rs index be8602984..5b2c6f562 100644 --- a/crates/socket-patch-core/src/vendor/pypi_requirements.rs +++ b/crates/socket-patch-core/src/vendor/pypi_requirements.rs @@ -833,6 +833,16 @@ async fn collect_requirements_files(root: &Path) -> Result, (&'stat }); Ok(true) } + // pip decodes a UTF-16 file by its BOM (#721), so a pin inside one + // is installed; wiring around it would leave that pin unpatched. + Err(e) if e.kind() == std::io::ErrorKind::InvalidData => Err(( + "pypi_no_requirements", + format!( + "{} is not UTF-8 text (for example UTF-16, which Windows PowerShell 5.1 \ + writes for `pip freeze > requirements.txt`); re-save it as UTF-8 and re-run", + path.display() + ), + )), Err(_) if out.is_empty() => Err(( "pypi_no_requirements", format!("cannot read {}", path.display()), @@ -1178,6 +1188,43 @@ mod tests { tmp } + /// #721: pip installs from a UTF-16 requirements file (what Windows + /// PowerShell 5.1's `pip freeze >` writes), so vendoring must refuse it + /// by name, as the root file or as an include, never wire around it. + #[tokio::test] + async fn a_utf16_requirements_file_is_refused_by_name() { + let utf16 = |text: &str| -> Vec { + let mut out = vec![0xFF, 0xFE]; + for unit in text.encode_utf16() { + out.extend(unit.to_le_bytes()); + } + out + }; + let tmp = tempfile::tempdir().unwrap(); + std::fs::write( + tmp.path().join("requirements.txt"), + utf16("six==1.16.0\r\n"), + ) + .unwrap(); + let err = wire_requirements(tmp.path(), "six", "1.16.0", REL_WHEEL, SHA) + .await + .unwrap_err(); + assert_eq!(err.0, "pypi_no_requirements"); + assert!( + err.1.contains("requirements.txt is not UTF-8 text"), + "{}", + err.1 + ); + + let tmp = write_root("-r inc.txt\nidna==3.7\n").await; + std::fs::write(tmp.path().join("inc.txt"), utf16("six==1.16.0\r\n")).unwrap(); + let err = wire_requirements(tmp.path(), "six", "1.16.0", REL_WHEEL, SHA) + .await + .unwrap_err(); + assert!(err.1.contains("inc.txt is not UTF-8 text"), "{}", err.1); + assert_eq!(read_root(tmp.path()).await, "-r inc.txt\nidna==3.7\n"); + } + async fn read_root(root: &Path) -> String { tokio::fs::read_to_string(root.join("requirements.txt")) .await