Fix Poetry data-dir and placeholder model (#608, #640)

[Mikola Lysenko (mikolalysenko)]

Review follow-up on c9b240ad: the three original discovery findings and the CI-discovered sibling-environment regression are corrected. When an in-project environment is selected, the existing .venv plus venv inventory is preserved so a healthy copy cannot hide stale bytes from VEX verification. Verified with 151 repository tests, two actual-crawler tests covering five native fixtures, targeted Clippy, changed-region formatting, and independent review. The unchanged stale-copy/VEX CLI test now passes. Ready to merge as-is from this review. 335 successful checks, 6 skipped, and 8 successful workflows. Bugbot is clear on this commit; no unresolved review threads or new actionable findings.

LLM Description written by Claude Code:claude-opus-5-5

Fixes #608
Fixes #640

Shared root cause

socket-patch rebuilds Poetry's file locations without running Poetry, but two of Poetry's rules aren't modelled:

1. Config.process(): Poetry substitutes {key} in a config value only when key is a config setting that has a value, and leaves the text as-is otherwise. poetry_virtualenvs_path instead replaced a made-up {project-dir}, which Poetry keeps literally. It also ignored {data-dir} (a real setting since Poetry 2.1) and never processed placeholders inside cache-dir. For Poetry venv discovery expands a {project-dir} placeholder Poetry doesn't have, so agent mode misses the env, patches the global interpreter, and VEX attests not_affected #608 this means agent mode misses the env Poetry actually uses, patches a global copy instead, and VEX attests not_affected.
2. locations.data_dir(): $POETRY_HOME, or else the platformdirs user data dir pypoetry. This directory is both the {data-dir} value and the parent of the official installer's venv. Because nothing resolved it, scan -g never crawled Poetry's own venv (Global scan (-g) never crawls the venv that Poetry's official installer creates ($POETRY_HOME/venv), so patches for Poetry's own dependencies are never found, applied or rolled back #640).

Change

The change remains limited to the Python crawler and its regression tests; wrappers need no change.

• Resolve supported cache-dir and data-dir placeholders recursively, only when referenced, with bounded cycle detection. Unknown keys retain their version-specific literal/empty behavior. Explicit data-dir values also work on older Poetry releases that lack that default setting.
• Retain the compatible Poetry 1.1, 1.2–2.0 and current placement candidates. An existing parent directory for another project cannot hide this project's environment. Each root keeps its own activation record; active-shell, in-project and out-of-tree precedence is applied separately before collecting the environments.
• Resolve POETRY_DATA_DIR over config-file settings and POETRY_HOME over platform defaults. On macOS, preserve both current platformdirs' absolute XDG_DATA_HOME location and older supported versions' Library location for project environments.
• Preserve the existing .venv and venv inventory when Poetry selects an in-project environment. Active-shell and out-of-tree selections retain their precedence; hosted verification still inspects sibling copies before producing VEX.
• Inventory the official installer's <data dir>/venv separately, including both POETRY_HOME and its platform default. The macOS installer continues to use ~/Library/Application Support/pypoetry even when project discovery also checks XDG.

I checked the model against real Poetry 2.4.3 (Linux): poetry config data-dir resolves POETRY_DATA_DIR > config data-dir > POETRY_HOME > ~/.local/share/pypoetry, and poetry env info -p gives a literal {project-dir}/.envs/demo-…, <data-dir>/venvs/demo-… and <data-dir>/cache/virtualenvs/demo-…. The new code produces the same paths.

Per-issue regression tests (red on main, green with the fix)

• Poetry venv discovery expands a {project-dir} placeholder Poetry doesn't have, so agent mode misses the env, patches the global interpreter, and VEX attests not_affected #608: crawlers::python_crawler::tests::poetry_virtualenv_config_layers_and_templates, which now asserts {project-dir} stays literal (it previously asserted the wrong expansion). Before the fix: left: /home/dev/proj/.envs vs right: /home/dev/proj/{project-dir}/.envs.
• Poetry venv discovery expands a {project-dir} placeholder Poetry doesn't have, so agent mode misses the env, patches the global interpreter, and VEX attests not_affected #608 ({data-dir} in path and cache-dir, unknown keys, precedence): poetry_virtualenvs_path_mirrors_config_process, poetry_data_dir_defaults_to_xdg_data_home_on_linux (before the fix: /home/dev/proj/{data-dir}/venvs).
• Poetry venv discovery expands a {project-dir} placeholder Poetry doesn't have, so agent mode misses the env, patches the global interpreter, and VEX attests not_affected #608 (Poetry 1.1 empty substitution): poetry_virtualenvs_path_falls_back_to_poetry_1_1_placement.
• Global scan (-g) never crawls the venv that Poetry's official installer creates ($POETRY_HOME/venv), so patches for Poetry's own dependencies are never found, applied or rolled back #640: crawler_python_e2e::get_global_python_site_packages_discovers_poetry_installer_venv_{linux,under_xdg,under_poetry_home}, plus macOS and Windows variants gated by cfg. All 3 Linux tests failed on main and pass with the fix.

Original author verification

• cargo fmt --all -- --check already fails on main (045d7ec), and CI doesn't run it, so this PR doesn't reformat unrelated files. The new and changed code is rustfmt-formatted.
• cargo clippy --workspace --all-features -- -D warnings: clean on 66e508b.
• cargo test --workspace --all-features (on 5f7c5f5): 1712 passed. 3 failed in covgap_commands_vendor (*_state_write_failure_*). Those tests chmod a directory to 0555 to make writes fail, but the sandbox runs as uid 0, which ignores that. They're npm vendor tests and unrelated to this change; CI runs as non-root. On 66e508b, the python_crawler unit tests (63) and crawler_python_e2e (61) pass.
• e2e_vex_build poetry:: --ignored with real Poetry 2.4.3: 1 passed. poetry_hosted_fresh_install_then_manifestless_vex failed at its rollback step because the CLI's HTTP client couldn't reach https://pypi.org/pypi/six/1.16.0/json through the sandbox's TLS-intercepting proxy (curl reaches it). That's the hosted-unwind path, which this diff doesn't touch. CI will run the full Poetry matrix (1.0.10 to 2.4.3, Linux and macOS).

Follow-ups

Substitution of other configured settings, such as {installer.parallel}, remains outside the resolver's supported key set.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SnFb7SrwgNedWGzXdgZHR6

---

Note

Medium Risk
Changes path resolution and discovery precedence for Poetry/Pipenv/PDM-adjacent flows; mistakes could patch the wrong environment or miss packages, but scope is limited to the crawler with heavy new tests.

Overview
Reworks how the Python crawler infers Poetry virtualenv site-packages without invoking Poetry, fixing wrong paths (#608) and missing global scans of the official installer venv (#640).

Config and paths: Adds {data-dir} / POETRY_DATA_DIR, recursive placeholder handling aligned with Poetry’s Config.process() (including cache-dir), and drops the incorrect {project-dir} expansion. Discovery now unions multiple compatible roots (Poetry 1.1 vs 1.2–2.0 vs ≥2.1, macOS Library vs XDG) instead of picking one parent directory; each root keeps its own envs.toml activation and EnvManager-style precedence (active shell → in-project .venv → out-of-tree envs) before merging results.

Patch-target behavior: For in-project Poetry envs, results still include local venv alongside .venv so sibling copies stay visible for hosted VEX verification. Global crawling also checks <Poetry data dir>/venv for both POETRY_HOME and platform defaults.

Regression coverage is expanded in python_crawler unit tests and crawler_python_e2e (installer venv per OS / XDG / POETRY_HOME).

^{Reviewed by Cursor Bugbot for commit c9b240a. Configure here.}

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

[burn-down agent] Labeled Ready for review at 66e508bbad08799558dccc6e0028e18a78a266c1.

• CI: 335/335 check runs green on this head (6 skipped); 0 behind main, no conflicts.
• Bugbot: reviewed 66e508b, no new issues; no unresolved review threads.
• Reviewer focus: poetry_process() placeholder rules and the per-version (1.1 / 1.2–2.0 / current) placement fallback in python_crawler.rs. Linked issues Poetry venv discovery expands a {project-dir} placeholder Poetry doesn't have, so agent mode misses the env, patches the global interpreter, and VEX attests not_affected #608 and Global scan (-g) never crawls the venv that Poetry's official installer creates ($POETRY_HOME/venv), so patches for Poetry's own dependencies are never found, applied or rolled back #640 are still open.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Codex follow-up review of c9b240ad6d6977abeb3b88b410dec7016032ed6f: all reported findings are fixed; ready to merge as-is from this review.

The three original Poetry discovery findings remain corrected: compatible project environments are retained with their own activation records, nested cache/data placeholders resolve, and current plus older supported macOS data-directory layouts are covered separately from the official installer default.

This commit also fixes the regression found by CI in my preceding correction. When Poetry selects an in-project environment, discovery again retains both .venv and venv. A patched .venv therefore cannot hide stale bytes in its sibling when hosted verification decides whether VEX is valid. Active-shell and out-of-tree precedence remain intact. The existing healthy_interpreter_cannot_mask_a_stale_python_install assertions were not changed: the stale-copy case exits 1, warns and produces no VEX file.

Verified on the exact tested source: 151 repository tests passed (66 Python unit, 61 crawler end-to-end, 17 CLI environment and all 7 Poetry redirect CLI tests), plus two actual-crawler tests covering five native fixtures. The original three regressions and the follow-up inventory regression have recorded failing controls and passing corrected runs. Independent review, targeted Clippy, changed-region formatting and diff checks are clear; Clippy retains the existing macOS unused_variables allowance. The committed files match the tested hashes and merge cleanly with main 045d7ec7.

335 successful checks, 7 skipped, and 8 successful workflows (one additional label-triggered workflow skipped). Bugbot is clear on this commit; no unresolved review threads or new actionable findings. Ready for review has been restored. GitHub still requires the normal human approval before merge.

[Mikola Lysenko (mikolalysenko)]

BugBot review

Please review the Poetry discovery corrections on 4b6e369c4607c2ea20bfc4909e68e05b8a4144d8.

[cursor]

Stale Bugbot comment from a previous run.

[Mikola Lysenko (mikolalysenko)]

BugBot review

Please review the sibling-environment inventory correction on c9b240ad6d6977abeb3b88b410dec7016032ed6f.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit c9b240a. Configure here.}