diff --git a/crates/socket-patch-cli/CLI_CONTRACT.md b/crates/socket-patch-cli/CLI_CONTRACT.md index e2bc9b3d9..710c5ab1e 100644 --- a/crates/socket-patch-cli/CLI_CONTRACT.md +++ b/crates/socket-patch-cli/CLI_CONTRACT.md @@ -1614,6 +1614,7 @@ command's integration tests check its output. | `vendor_artifact_missing` | reason | The recorded artifact is missing; repair requires an online exact redownload. | | `vendor_artifact_corrupt` | reason | The artifact does not match the recorded hash or inventory; repair requires an online exact redownload. | | `vendor_artifact_reused` | `skipped` (verbose note) | vendor / scan `--mode vendored` (pypi): the wiring was dropped by a relock but the committed wheel the ledger vouches for verified, so it was re-wired as-is — no service download, no rebuild; the lock pins the first run's sha again. | +| `vendor_requirements_constraints_link` | `skipped` (advisory) | vendor / scan `--mode vendored` (pypi / requirements.txt), dry run included: a requirements file the run wired (or would wire) a vendored wheel path into is read as `-c` / `--constraint` by a sibling requirements file (a `*.txt` / `*.in` in the project root or a root `requirements*/` directory, or a requirements-named `*.txt` / any `*.in` in another top-level directory, such as `docs/requirements.txt`; hidden and dependency directories are skipped). pip 20.3.x and 21.0.x refuse a link in a constraints file ("Links are not allowed as constraints"), so installing with that file fails on those pips. One advisory per reading file; the package is still vendored (#740). | | `vendor_redownload_failed` | `failed` | vendor: the same-UUID artifact could not be downloaded and verified against its original ledger. Existing files and fingerprints are preserved. | | `vendor_artifact_redownload_failed` | `failed` | repair: download unavailable, integrity mismatch, or downloaded bytes/inventory differ from the ledger. Existing files are preserved. | | `vendor_artifact_unrepairable` | `failed` | repair: the ledger identity or patch record cannot be trusted or recovered. | @@ -1665,6 +1666,7 @@ command's integration tests check its output. | `redirect_takeover_not_pinned` | `skipped` (hosted event) | scan/get `--mode hosted`: the skip reason of a retracted takeover when no rewriter warning names the cause. | | `redirect_hatch_lock_regenerated` | `warnings[]` (warning) | scan/get `--mode hosted` (pypi / Hatch 1.17+): the project's Hatch environments are locked (`lock-envs = true`, an environment's `locked = true` or `lock-filename`), so its `pylock*.toml` is derived from pyproject and Hatch regenerates it on the next sync. The package is wired in the Hatch declarations as well as in the lock (rewriting only the lock was discarded by Hatch, #479); run `hatch dep lock` to refresh the lock (`hatch dep lock --check` reports it stale until then). Vendored routes such a project to the Hatch lane and its pip-installer refusal. | | `redirect_requirements_direct_reference` | `warnings[]` (warning) | scan/get `--mode hosted` (pypi / requirements.txt): the root `requirements.txt` installs the patched release from a PEP 508 direct reference the user wrote (`name @ ` whose archive names that release: a `files.pythonhosted.org` file, a private mirror, an internal fork build, a `file://` path), not from socket-patch's own hosted artifact. It is the user's own source choice, so the line is left byte-for-byte and the package is not redirected (#542). Before v5.0 the line was swapped for the hosted build, and rollback then wrote a plain `name==version` index pin. | +| `redirect_requirements_constraints_link` | `warnings[]` (warning) | scan/get `--mode hosted` (pypi / requirements.txt): the root `requirements.txt` was rewritten, and a sibling requirements file (a `*.txt` / `*.in` in the project root or a root `requirements*/` directory, or a requirements-named `*.txt` / any `*.in` in another top-level directory, such as `docs/requirements.txt`; hidden and dependency directories are skipped) reads it as a constraints file (`-c` / `--constraint`). The rewritten pin is a URL, and pip 20.3.x and 21.0.x refuse a link in a constraints file ("Links are not allowed as constraints"), so installing with that file fails on those pips; pip 20.2 and 21.1+ accept it. One warning per reading file; the redirect still happens (#740). A constraints file named only on the pip command line is not detected. | | `redirect_requirements_takeover_unreachable` | `warnings[]` (warning) | scan/get `--mode hosted` (pypi / requirements.txt): a vendored → hosted takeover of a package that vendored mode wired through a pin in a `-r` include, or through a `(transitive)` line it appended to the root `requirements.txt`. Hosted mode only rewrites an existing pin in the root `requirements.txt`, so the staged takeover is retracted (wet and `--dry-run`): the vendored wiring, ledger entry and wheel stay byte-identical, the purl is skipped with this code as `skipped` (hosted event), and nothing is redirected for it. Exit 0. The detail names the remedy and its reach: run `socket-patch vendor --revert` (it reverts EVERY vendored package in the project, not just this one), move the pin from the include into the root `requirements.txt` and delete it from the include (or, for a `(transitive)` line, add an exact `==` pin to the root file), then re-run `scan --mode hosted`. | | `redirect_uv_takeover_version_unreachable` | `warnings[]` (warning) | scan/get `--mode hosted` (pypi / uv): a vendored → hosted takeover of a package whose recorded pre-vendor `uv.lock` entry is at another version than the patch (vendored uv pins the lock entry down to the patch's version, for example when the lock resolved a newer release). Reverting would bring the lock's own version back, and hosted mode only pins the version the lock resolves, so the staged takeover is retracted (wet and `--dry-run`): the vendored wiring, ledger entry and wheel stay byte-identical, the purl is skipped with this code as `skipped` (hosted event), and nothing is redirected for it. Exit 0. The detail names the remedy and its reach: run `socket-patch vendor --revert` (it reverts EVERY vendored package in the project, not just this one), make the project resolve the patch's version (for example an exact `==` requirement) and re-lock, then re-run `scan --mode hosted`. | | `redirect_vlt_missing_sha512` / `redirect_vlt_entry_not_found` / `redirect_vlt_entry_vendored` / `redirect_vlt_unsupported_lock_key` | `warnings[]` (warning) | scan/get `--mode hosted` (vlt): the grant has no sha512 / the lock has no default-registry node for `name@version` (not emitted when `redirect_vlt_custom_registry_skipped` already covers the dep's only nodes) / the only match is a vendored `file` node under `.socket/vendor/npm//` / a default-registry instance is outside vlt's node-line grammar or still unpatched after the splice. Per dep; none of the dep's instances is written. `redirect_vlt_missing_sha512` and `redirect_vlt_unsupported_lock_key` refuse the dep: it is never confirmed, whichever lock drives (a sibling lock may still carry its rewritten URL). `redirect_vlt_entry_not_found` and `redirect_vlt_entry_vendored` only say `vlt-lock.json` does not wire it: while vlt drives it is not confirmed; otherwise a sibling lock's rules may confirm it. Exit 0. | diff --git a/crates/socket-patch-cli/tests/mode_migration_pypi.rs b/crates/socket-patch-cli/tests/mode_migration_pypi.rs index 5494b8126..4e8d9cf2b 100644 --- a/crates/socket-patch-cli/tests/mode_migration_pypi.rs +++ b/crates/socket-patch-cli/tests/mode_migration_pypi.rs @@ -433,6 +433,162 @@ async fn requirements_sole_pin_vendored_to_hosted() { assert_vendored_to_hosted(&root, &["requirements.txt"]).await; } +/// #740: a sibling requirements file that reads the root requirements.txt +/// as `-c` constraints breaks on pip 20.3.x / 21.0.x once the root pin is a +/// link ("Links are not allowed as constraints"). Both writers still +/// rewrite the pin, but now name every such reader in a warning: hosted +/// `redirect_requirements_constraints_link`, vendored (dry run and wet) +/// `vendor_requirements_constraints_link`. A line that names the file as +/// `-r` (or `-r` after `-c`, which pip follows instead) is not a reader, +/// and a project with no reader gets no warning. +#[tokio::test] +async fn requirements_read_as_constraints_warn_in_both_modes() { + fn stage(root: &Path, readers: bool) { + std::fs::write(root.join("requirements.txt"), "six==1.16.0\n").unwrap(); + std::fs::write(root.join("include.txt"), "-r requirements.txt\n").unwrap(); + std::fs::write( + root.join("both.txt"), + "-c requirements.txt -r requirements.txt\n", + ) + .unwrap(); + if readers { + std::fs::write( + root.join("requirements-dev.txt"), + "-c requirements.txt\nidna==3.7\n", + ) + .unwrap(); + std::fs::create_dir_all(root.join("requirements")).unwrap(); + std::fs::write( + root.join("requirements/test.txt"), + "--constraint=../requirements.txt # pins\n", + ) + .unwrap(); + } + } + fn reported(env: &Value, code: &str) -> Vec { + let mut readers: Vec = Vec::new(); + let mut walk = vec![env]; + while let Some(v) = walk.pop() { + match v { + Value::Object(map) => { + if ["code", "errorCode"] + .iter() + .any(|key| map.get(*key).and_then(Value::as_str) == Some(code)) + { + let text = v.to_string(); + for reader in [ + "requirements-dev.txt", + "requirements/test.txt", + "both.txt", + "include.txt", + ] { + if text.contains(&format!("{reader} reads requirements.txt")) { + readers.push(reader.to_string()); + } + } + assert!( + text.contains("Links are not allowed as constraints"), + "{text}" + ); + } + walk.extend(map.values()); + } + Value::Array(items) => walk.extend(items), + _ => {} + } + } + readers.sort(); + readers + } + let expected = vec![ + "requirements-dev.txt".to_string(), + "requirements/test.txt".to_string(), + ]; + for readers in [true, false] { + let want = if readers { + expected.clone() + } else { + Vec::new() + }; + + let (_tmp, root) = project(); + stage(&root, readers); + let server = MockServer::start().await; + mount_hosted_api(&server, true).await; + let (code, env) = hosted_scan(&root, &server); + assert_eq!(code, 0, "{env:#}"); + assert_eq!( + hosted_pinned(&env), + 1, + "the pin is still redirected: {env:#}" + ); + assert_eq!( + reported(&env, "redirect_requirements_constraints_link"), + want, + "{env:#}" + ); + // A re-run rewrites nothing; the URL pin still breaks the readers. + let (code, env) = hosted_scan(&root, &server); + assert_eq!(code, 0, "{env:#}"); + assert_eq!( + reported(&env, "redirect_requirements_constraints_link"), + want, + "re-run: {env:#}" + ); + + let (_tmp, root) = project(); + stage(&root, readers); + stage_manifest(&root); + let (code, env) = run_cli(&root, &["vendor", "--dry-run"], &[]); + assert_eq!(code, 0, "{env:#}"); + assert_eq!( + reported(&env, "vendor_requirements_constraints_link"), + want, + "dry run: {env:#}" + ); + assert!( + !readers || env.to_string().contains("would pin six==1.16.0"), + "{env:#}" + ); + let (code, env) = run_cli(&root, &["vendor"], &[]); + assert_eq!(code, 0, "{env:#}"); + assert!(std::fs::read_to_string(root.join("requirements.txt")) + .unwrap() + .contains(&format!(".socket/vendor/pypi/{UUID}/"))); + assert_eq!( + reported(&env, "vendor_requirements_constraints_link"), + want, + "{env:#}" + ); + assert!( + !readers || env.to_string().contains("now pins six==1.16.0"), + "{env:#}" + ); + let (code, env) = run_cli(&root, &["vendor"], &[]); + assert_eq!(code, 0, "{env:#}"); + assert_eq!( + reported(&env, "vendor_requirements_constraints_link"), + want, + "in-sync re-run: {env:#}" + ); + // An in-sync run that rebuilds a missing wheel keeps the wiring too. + std::fs::remove_dir_all(root.join(format!(".socket/vendor/pypi/{UUID}"))).unwrap(); + let (code, env) = run_cli(&root, &["vendor"], &[]); + assert_eq!(code, 0, "{env:#}"); + assert!( + env["events"] + .as_array() + .is_some_and(|events| events.iter().any(|e| e["action"] == "rebuilt")), + "{env:#}" + ); + assert_eq!( + reported(&env, "vendor_requirements_constraints_link"), + want, + "in-sync rebuild: {env:#}" + ); + } +} + /// #410: a requirements.txt in which every requirement is the hosted pin /// (a lone `six==1.16.0`, or one beside `-e .`) can be unwound again. /// Hosted `rollback`, `remove` and the hosted → vendored takeover restore diff --git a/crates/socket-patch-core/src/hosted/engine.rs b/crates/socket-patch-core/src/hosted/engine.rs index 85f235c54..a8395aacd 100644 --- a/crates/socket-patch-core/src/hosted/engine.rs +++ b/crates/socket-patch-core/src/hosted/engine.rs @@ -28,6 +28,7 @@ use crate::constants::npm_family::{ RUSH_COMMON_LOCK_REL, RUSH_SUBSPACES_DIR, VLT_HIDDEN_LOCK_REL, VLT_LOCK, }; use crate::formats::cargo::source_replacement::CratesIoSource; +use crate::formats::governing_locks::PYPI_REQUIREMENTS; use crate::patch::redirect::npmrc::{ effective_replace_registry_host, plan_npmrc_allow_remote_with, replace_registry_host_rewrites, NpmrcPlan, OuterAllowRemote, NPMRC_ALLOW_REMOTE_EDIT_KIND, NPMRC_REL, @@ -1765,6 +1766,8 @@ pub async fn rewrite( ) .await; if vetoed.is_empty() { + let constraints = constraints_link_warnings(view, &done, &kept).await; + done.rewrite.warnings.extend(constraints); done.unattributed = unattributed; done.rewrite.warnings.extend(lockless); done.rewrite.warnings.extend(copy_left); @@ -1777,6 +1780,48 @@ pub async fn rewrite( } } +/// A `redirect_requirements_constraints_link` warning for each sibling +/// requirements file that reads the root requirements.txt as `-c` +/// constraints while a hosted pin there is a URL (#740), which pip 20.3 +/// and 21.0 refuse in a constraints file. Said on every run that confirms +/// such a pin, not only the one that writes it: the hazard lasts as long +/// as the pin does. +async fn constraints_link_warnings( + view: &ProjectView<'_>, + done: &Rewritten, + kept: &[Candidate], +) -> Vec { + let confirmed = &done.rewrite.confirmed_requirements_uuids; + let mut pinned: Vec<&str> = kept + .iter() + .filter(|c| confirmed.contains(&c.dep.patch_uuid)) + .map(|c| c.dep.name.as_str()) + .collect(); + pinned.sort_unstable(); + pinned.dedup(); + if pinned.is_empty() { + return Vec::new(); + } + crate::vendor::pypi_requirements::constraint_readers_in(*view, &[PYPI_REQUIREMENTS.to_string()]) + .await + .into_iter() + .map(|found| { + warning( + "redirect_requirements_constraints_link", + format!( + "{} reads {} as a constraints file (`-c`), and the hosted pin of {} in {} is a \ + URL; {}", + found.reader, + found.target, + pinned.join(", "), + found.target, + crate::vendor::pypi_requirements::CONSTRAINT_LINK_REMEDY + ), + ) + }) + .collect() +} + /// Lockfile discovery over the project as `done` would leave it, read as /// the management commands read it ([`HostedInventory`]): the skips for /// the confirmed candidates whose pin would be contested wiring, a diff --git a/crates/socket-patch-core/src/vendor/pypi.rs b/crates/socket-patch-core/src/vendor/pypi.rs index e7d72eacc..eaf49a13f 100644 --- a/crates/socket-patch-core/src/vendor/pypi.rs +++ b/crates/socket-patch-core/src/vendor/pypi.rs @@ -1385,6 +1385,14 @@ async fn pypi_prelude<'p>( .await); } if artifact_present || dry_run { + // The wiring stays a link in place: keep naming its `-c` + // readers on every run (#740). + if let Ok(prior) = reuse::prior_entry(project_root, "pypi", record, None).await { + warnings.extend( + constraint_link_warnings(project_root, &prior.wiring, &canon_name, version) + .await, + ); + } return Err(done( already_patched_result(base, Path::new(""), &record.files), None, @@ -1601,6 +1609,7 @@ async fn wire_in_sync_sibling( return done(result, None, warnings); }; if dry_run { + let mut targets = requirements_record_files(&entry.wiring); if export_pending { warnings.push(VendorWarning::new( "pypi_requirements_sibling_wired", @@ -1610,7 +1619,21 @@ async fn wire_in_sync_sibling( crate::formats::governing_locks::PYPI_REQUIREMENTS ), )); + targets.extend( + super::pypi_requirements::planned_files( + project_root, + canon_name, + version, + &entry.artifact.path, + ) + .await, + ); + targets.sort(); + targets.dedup(); } + warnings.extend( + constraint_reader_warnings(project_root, &targets, canon_name, version, true).await, + ); if !pylocks.is_empty() { warnings.push(pylock_dry_run_notice(&pylocks, canon_name, version)); } @@ -1690,6 +1713,10 @@ async fn wire_in_sync_sibling( } } } + // Every requirements file the entry wires, this run's or earlier: the + // link pins stay in place, so their `-c` readers are named each run. + warnings + .extend(constraint_link_warnings(project_root, &entry.wiring, canon_name, version).await); done(result, Some(entry), warnings) } @@ -1905,6 +1932,12 @@ pub async fn vendor_pypi_with_pipenv_version<'a>( return done(result, None, warnings); } } + let targets = + planned_requirements_targets(project_root, &plan, &canon_name, version, &rel_wheel) + .await; + warnings.extend( + constraint_reader_warnings(project_root, &targets, &canon_name, version, true).await, + ); return done(result, None, warnings); } let Some(artifact) = artifact else { @@ -2012,6 +2045,11 @@ pub async fn vendor_pypi_with_pipenv_version<'a>( ) .await; } + if let Ok(prior) = reuse::prior_entry(project_root, "pypi", record, None).await { + warnings.extend( + constraint_link_warnings(project_root, &prior.wiring, &canon_name, version).await, + ); + } return done(result, None, warnings); } @@ -2297,9 +2335,99 @@ pub async fn vendor_pypi_with_pipenv_version<'a>( MetaSlot::Pipenv(m) => entry.pipenv = Some(m), MetaSlot::None => {} } + warnings + .extend(constraint_link_warnings(project_root, &entry.wiring, &canon_name, version).await); done(result, Some(entry), warnings) } +/// A `vendor_requirements_constraints_link` warning for each sibling file +/// that reads a requirements file `wiring` wrote a vendor line into as `-c` +/// constraints (#740): the wheel path is a link, which pip 20.3.x and +/// 21.0.x refuse there. +async fn constraint_link_warnings( + project_root: &Path, + wiring: &[super::state::WiringRecord], + canon_name: &str, + version: &str, +) -> Vec { + let targets = requirements_record_files(wiring); + constraint_reader_warnings(project_root, &targets, canon_name, version, false).await +} + +/// The requirements files `wiring` holds vendor lines in. +fn requirements_record_files(wiring: &[super::state::WiringRecord]) -> Vec { + let mut files: Vec = wiring + .iter() + .filter(|r| r.kind == super::pypi_requirements::RECORD_KIND) + .map(|r| r.file.clone()) + .collect(); + files.sort(); + files.dedup(); + files +} + +/// The requirements files the wet run of `plan` writes vendor lines into, +/// for the dry run's [`constraint_reader_warnings`]: a fresh requirements +/// wiring's (directly, or as the exported sibling of a uv / Pipenv lock) +/// as the planner would write it, and a re-wire's as the superseded entry +/// recorded them. +async fn planned_requirements_targets( + project_root: &Path, + plan: &WiringPlan, + canon_name: &str, + version: &str, + rel_wheel: &str, +) -> Vec { + let fresh = + || super::pypi_requirements::planned_files(project_root, canon_name, version, rel_wheel); + let mut targets = match plan { + WiringPlan::Requirements | WiringPlan::Uv(_, true) => fresh().await, + WiringPlan::Pipenv(_, superseded, true, _) => { + let mut files = fresh().await; + if let Some(prev) = superseded { + files.extend(requirements_record_files(&prev.wiring)); + } + files + } + WiringPlan::RequirementsRewire(prev) => requirements_record_files(&prev.wiring), + WiringPlan::Supersede(superseded) => requirements_record_files(&superseded.prev.wiring), + _ => Vec::new(), + }; + targets.sort(); + targets.dedup(); + targets +} + +/// [`constraint_link_warnings`] for the requirements files `targets`; +/// `planned` words it for a dry run. +async fn constraint_reader_warnings( + project_root: &Path, + targets: &[String], + canon_name: &str, + version: &str, + planned: bool, +) -> Vec { + let view = super::lock_inventory::ProjectView::Disk(project_root); + let pins = if planned { "would pin" } else { "now pins" }; + super::pypi_requirements::constraint_readers_in(view, targets) + .await + .into_iter() + .map(|found| { + VendorWarning::new( + "vendor_requirements_constraints_link", + format!( + "{} reads {} as a constraints file (`-c`), and {} {pins} \ + {canon_name}=={version} to a vendored wheel path; {}", + found.reader, + found.target, + found.target, + super::pypi_requirements::CONSTRAINT_LINK_REMEDY + ), + ) + }) + .collect() +} + /// The flavor-guard refusals that name socket-patch's own wiring at another /// patch uuid of the release (alongside user sources the same codes cover): /// uv's `[tool.uv.sources]` path, a script lock / pylock `path` source, and diff --git a/crates/socket-patch-core/src/vendor/pypi_requirements.rs b/crates/socket-patch-core/src/vendor/pypi_requirements.rs index 0df16aeae..25c68c50d 100644 --- a/crates/socket-patch-core/src/vendor/pypi_requirements.rs +++ b/crates/socket-patch-core/src/vendor/pypi_requirements.rs @@ -1118,6 +1118,22 @@ async fn collect_requirements_files(root: &Path) -> Result, (&'stat Ok(out) } +/// The requirements files a fresh wiring of `canon_name==version` would +/// write (the dry run's view of [`wire_requirements`]). Empty when the +/// planner refuses: the wet run reports that refusal instead. +pub(super) async fn planned_files( + root: &Path, + canon_name: &str, + version: &str, + rel_wheel: &str, +) -> Vec { + let placeholder = "0".repeat(64); + plan_requirements(root, canon_name, version, rel_wheel, &placeholder) + .await + .map(|plan| plan.into_iter().map(|file| file.rel).collect()) + .unwrap_or_default() +} + /// Every requirements file the vendor planner may have written a pin into: /// the root `requirements.txt` plus each IN-ROOT `-r`/`--requirement` /// include it reaches (same walk as the planner, so a vendored pin hosted in @@ -1216,33 +1232,178 @@ async fn walk_requirements_tree( /// (`requirements/../x.txt` → `x.txt`; an escape keeps its `../`). The one /// include grammar behind the planner's walk and the lock inventory's. pub(crate) fn requirements_includes(rel: &str, content: &str) -> Vec { - let include_dir = match rel.rfind('/') { + nested_files(rel, content) + .into_iter() + .filter_map(|nested| match nested { + NestedFile::Requirements(target) => Some(target), + NestedFile::Constraints(_) => None, + }) + .collect() +} + +/// The `-c`/`--constraint` targets of the requirements file `rel`, +/// resolved as [`requirements_includes`] resolves `-r`. +fn requirements_constraints(rel: &str, content: &str) -> Vec { + nested_files(rel, content) + .into_iter() + .filter_map(|nested| match nested { + NestedFile::Constraints(target) => Some(target), + NestedFile::Requirements(_) => None, + }) + .collect() +} + +/// The nested files of the requirements file `rel` with `content`, in file +/// order, each resolved against `rel`'s directory and lexically normalized. +fn nested_files(rel: &str, content: &str) -> Vec { + let dir = match rel.rfind('/') { Some(i) => &rel[..i], None => "", }; + let resolve = |target: String| { + let joined = if dir.is_empty() { + target + } else { + format!("{dir}/{target}") + }; + crate::utils::relpath::normalize_rel_keeping_escapes(&joined) + }; logical_lines(content) .iter() - .filter_map(|ll| include_target(&ll.text)) - .map(|target| { - let joined = if include_dir.is_empty() { - target - } else { - format!("{include_dir}/{target}") - }; - crate::utils::relpath::normalize_rel_keeping_escapes(&joined) + .filter_map(|ll| nested_file(&ll.text)) + .map(|nested| match nested { + NestedFile::Requirements(target) => NestedFile::Requirements(resolve(target)), + NestedFile::Constraints(target) => NestedFile::Constraints(resolve(target)), }) .collect() } +/// A sibling requirements file that pip reads `target` from as constraints +/// (`-c target`), found by [`constraint_readers_in`]. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct ConstraintReader { + /// Root-relative path of the file with the `-c` line. + pub reader: String, + /// Root-relative path of the constraints file it names. + pub target: String, +} + +/// Whether `name` looks like a pip requirements file: `*.txt`, or a +/// pip-tools `*.in` input. +fn requirements_file_name(name: &str) -> bool { + let lower = name.to_ascii_lowercase(); + lower.ends_with(".txt") || lower.ends_with(".in") +} + +/// Top-level directories never searched for constraint readers: installed +/// packages and build output, not the project's own requirements. +const NOT_READER_DIRS: &[&str] = &[ + "node_modules", + "site-packages", + "venv", + "env", + "target", + "build", + "dist", + "__pycache__", +]; + +/// Every requirements file beside `targets` that names one of them as a +/// `-c` / `--constraint` file (#740). The writers turn a pin in a target +/// into a link (a hosted `@ ` or a vendored wheel path), which pip +/// 20.3.x and 21.0.x refuse in a constraints file ("Links are not allowed +/// as constraints"), so each reader found is an install that breaks there. +/// +/// Read: each `*.txt` / `*.in` file in the project root and in a root +/// directory whose name mentions `requirement` (`requirements/`), and each +/// one named like requirements (`requirements.txt`, `dev.in`) in the other +/// top-level directories (`docs/requirements.txt`, the Read the Docs +/// layout). Hidden directories and [`NOT_READER_DIRS`] are skipped, and a +/// file whose bytes never hold `-c` is not parsed. A file that cannot be +/// read or decoded is skipped (this only warns). A constraints file named +/// only on the pip command line (`pip install -c requirements.txt`) is not +/// visible here. +pub(crate) async fn constraint_readers_in( + view: crate::vendor::lock_inventory::ProjectView<'_>, + targets: &[String], +) -> Vec { + if targets.is_empty() { + return Vec::new(); + } + let mut candidates: Vec = Vec::new(); + let mut dirs: Vec<(String, bool)> = Vec::new(); + for entry in view.list_dir("").await.unwrap_or_default() { + let lower = entry.name.to_ascii_lowercase(); + if !entry.is_dir { + if requirements_file_name(&entry.name) { + candidates.push(entry.name); + } + } else if !entry.name.starts_with('.') && !NOT_READER_DIRS.contains(&lower.as_str()) { + dirs.push((entry.name, lower.contains("requirement"))); + } + } + for (dir, every_file) in dirs { + for entry in view.list_dir(&dir).await.unwrap_or_default() { + let lower = entry.name.to_ascii_lowercase(); + if !entry.is_dir + && requirements_file_name(&entry.name) + && (every_file || lower.contains("requirement") || lower.ends_with(".in")) + { + candidates.push(format!("{dir}/{}", entry.name)); + } + } + } + candidates.sort(); + let mut readers = Vec::new(); + for rel in candidates { + let Ok(bytes) = view.read_bytes(&rel).await else { + continue; + }; + // Every `-c` / `--constraint` spelling holds the bytes `-c`; a + // file without them (UTF-16 aside) is never parsed. + let utf16 = bytes.starts_with(&[0xFF, 0xFE]) || bytes.starts_with(&[0xFE, 0xFF]); + if !utf16 && !bytes.windows(2).any(|w| w == b"-c") { + continue; + } + let Ok(content) = decode_requirements(bytes) else { + continue; + }; + for target in requirements_constraints(&rel, &content) { + if target != rel && targets.contains(&target) { + let reader = ConstraintReader { + reader: rel.clone(), + target, + }; + if !readers.contains(&reader) { + readers.push(reader); + } + } + } + } + readers +} + +/// The remedy both writers name for a [`ConstraintReader`]. +pub(crate) const CONSTRAINT_LINK_REMEDY: &str = + "pip 20.3.x and 21.0.x refuse a link in a constraints file (\"Links are not allowed as \ + constraints\"), so installing with that file fails on those pips; use pip 21.1 or later \ + (or pip 20.2), or stop reading this file with `-c`"; + /// The `-r`/`--requirement` include target of a logical line, if any, /// read the way pip's `req_file.py` reads it: comment stripped, `${NAME}` /// expanded from the environment, then the options `shlex`-split, so /// `-r "dev reqs.txt"`, `-r dev\\ reqs.txt`, `--requirement="dev.txt"` and /// `-r ${REQDIR}/dev.txt` name the file pip opens (#994). +#[cfg(test)] fn include_target(text: &str) -> Option { include_target_with(text, |name| std::env::var(name).ok()) } +/// [`nested_file_with`] over the process environment. +fn nested_file(text: &str) -> Option { + nested_file_with(text, |name| std::env::var(name).ok()) +} + /// The long options pip's requirements-file parser knows that take a /// value (`SUPPORTED_OPTIONS` plus the per-requirement /// `SUPPORTED_OPTIONS_REQ`, and `--install-option` from older pips). @@ -1293,7 +1454,28 @@ fn resolve_long_option(name: &str) -> Option<&'static str> { /// requirement, never an include; otherwise the first `-r` value is the /// file pip follows (`opts.requirements[0]`). An option's value is /// consumed even when it looks like `-r`, as optparse does. +#[cfg(test)] fn include_target_with(text: &str, env: impl Fn(&str) -> Option) -> Option { + match nested_file_with(text, env)? { + NestedFile::Requirements(target) => Some(target), + NestedFile::Constraints(_) => None, + } +} + +/// The file an options line makes pip open next, and how pip reads it. +#[derive(Debug, PartialEq, Eq)] +enum NestedFile { + /// `-r` / `--requirement`: read as requirements. + Requirements(String), + /// `-c` / `--constraint`: read as constraints. + Constraints(String), +} + +/// The nested file of a logical line, read with [`include_target_with`]'s +/// option grammar. pip's `handle_option_line` follows ONE file per line: +/// the first `-r` value when the line has one, else the first `-c` value, +/// so `-c c.txt -r dev.txt` follows only `dev.txt`. +fn nested_file_with(text: &str, env: impl Fn(&str) -> Option) -> Option { let code = expand_env_vars(strip_comment(text), env); // pip's break_args_options: a leading word without `-` is a // requirement, whose per-requirement options never recurse. @@ -1304,6 +1486,7 @@ fn include_target_with(text: &str, env: impl Fn(&str) -> Option) -> Opti // is no file to follow. let mut words = shlex_split(&code)?.into_iter(); let mut target: Option = None; + let mut constraint: Option = None; while let Some(word) = words.next() { if word == "--" { break; @@ -1331,6 +1514,7 @@ fn include_target_with(text: &str, env: impl Fn(&str) -> Option) -> Opti match option { "--editable" => return None, "--requirement" if target.is_none() => target = value, + "--constraint" if constraint.is_none() => constraint = value, _ => {} } } else if let Some(short) = word.strip_prefix('-') { @@ -1352,13 +1536,19 @@ fn include_target_with(text: &str, env: impl Fn(&str) -> Option) -> Opti match flag { 'e' => return None, 'r' if target.is_none() => target = value, + 'c' if constraint.is_none() => constraint = value, _ => {} } } // A bare word among the options is an optparse positional that // pip ignores. } - target.filter(|t| !t.is_empty()) + match target { + Some(target) => (!target.is_empty()).then_some(NestedFile::Requirements(target)), + None => constraint + .filter(|c| !c.is_empty()) + .map(NestedFile::Constraints), + } } // The logical-line lexer lives in `utils::requirements` (shared with the @@ -1369,6 +1559,125 @@ mod tests { use super::*; use crate::vendor::state::VendorArtifact; + /// #740: an options line names one nested file, the first `-r` when it + /// has one, else the first `-c`, in every spelling pip's optparse takes. + #[test] + fn nested_file_reads_constraints_like_pip() { + let env = |_: &str| None; + let constraint = |t: &str| Some(NestedFile::Constraints(t.to_string())); + for (line, want) in [ + ("-c requirements.txt", constraint("requirements.txt")), + ("-crequirements.txt", constraint("requirements.txt")), + ( + "--constraint requirements.txt", + constraint("requirements.txt"), + ), + ( + "--constraint=requirements.txt", + constraint("requirements.txt"), + ), + // Only `--requirement=` reads past an empty attached value. + ( + "--constraint= -r dev.txt", + Some(NestedFile::Requirements("dev.txt".into())), + ), + ( + "--constr requirements.txt # pins", + constraint("requirements.txt"), + ), + ("-c a.txt -c b.txt", constraint("a.txt")), + ( + "-c a.txt -r b.txt", + Some(NestedFile::Requirements("b.txt".into())), + ), + ("-r b.txt", Some(NestedFile::Requirements("b.txt".into()))), + ("six==1.16.0", None), + ("-e . -c requirements.txt", None), + ("--constraint=", None), + ("# -c requirements.txt", None), + ] { + assert_eq!(nested_file_with(line, env), want, "{line:?}"); + } + } + + /// #740: the readers of a constraints file are the root and + /// `requirements*/` files, and the requirements-named files of other + /// top-level directories, whose `-c` resolves to it against the + /// reader's own directory. `-r` lines, other targets, the file itself, + /// deeper, hidden and dependency directories are not. + #[tokio::test] + async fn constraint_readers_resolve_against_the_reading_file() { + let tmp = tempfile::tempdir().unwrap(); + let root = tmp.path(); + for dir in [ + "requirements", + "Requirements-CI", + "docs", + "requirements/nested", + "node_modules/pkg", + ".tox", + ] { + tokio::fs::create_dir_all(root.join(dir)).await.unwrap(); + } + for (rel, content) in [ + ("requirements.txt", "six==1.16.0\n-c requirements.txt\n"), + ("requirements-dev.txt", "-c requirements.txt\nidna==3.7\n"), + ("dev.in", "-c ./requirements.txt\n"), + ("include.txt", "-r requirements.txt\n"), + ("other.txt", "-c constraints.txt\n"), + ( + "requirements/test.txt", + "--constraint ../requirements.txt\n", + ), + ("requirements/base.txt", "-c requirements.txt\n"), + ("Requirements-CI/ci.txt", "-c ../requirements/base.txt\n"), + ( + "requirements/nested/deep.txt", + "-c ../../requirements.txt\n", + ), + ("docs/requirements.txt", "-c ../requirements.txt\n"), + ("docs/notes.txt", "-c ../requirements.txt\n"), + ("node_modules/requirements.txt", "-c ../requirements.txt\n"), + (".tox/requirements.txt", "-c ../requirements.txt\n"), + ("setup.cfg", "-c requirements.txt\n"), + ] { + tokio::fs::write(root.join(rel), content).await.unwrap(); + } + let view = crate::vendor::lock_inventory::ProjectView::Disk(root); + let readers = |found: Vec| { + found + .into_iter() + .map(|r| format!("{} -> {}", r.reader, r.target)) + .collect::>() + }; + assert_eq!( + readers(constraint_readers_in(view, &["requirements.txt".into()]).await), + vec![ + "dev.in -> requirements.txt", + "docs/requirements.txt -> requirements.txt", + "requirements-dev.txt -> requirements.txt", + "requirements/test.txt -> requirements.txt", + ] + ); + assert_eq!( + readers( + constraint_readers_in( + view, + &["requirements.txt".into(), "requirements/base.txt".into()] + ) + .await + ), + vec![ + "Requirements-CI/ci.txt -> requirements/base.txt", + "dev.in -> requirements.txt", + "docs/requirements.txt -> requirements.txt", + "requirements-dev.txt -> requirements.txt", + "requirements/test.txt -> requirements.txt", + ] + ); + assert!(constraint_readers_in(view, &[]).await.is_empty()); + } + /// #1120: [`requirements_include_names`] decodes each file as pip does, /// so the includes of a UTF-16 root file (a Windows PowerShell 5.1 /// export) and of a PEP 263 include are named, not an `Err` that reads