Slow scan: poetry hosted 2.6x median ms/pkg (lock_fragments::rewrite_batch + VEX lock re-parse)

[agent] Bench: poetry/hosted has been more than 2x the median hosted ms/package in 4 consecutive runs since #877 (10-08, 10-09, 10-12, 10-13). #760 was closed after #877 made poetry 40–50% faster. Poetry still scans 400 packages at about the per-package cost of uv, pylock, pdm and cargo.

Today: poetry/hosted median 112.6 ms for 400 packages, which is 0.2815 ms/pkg, 2.63x the median across */hosted (0.107). poetry/rescan is 87.2 ms (2.04x).
Hot spot (callgrind, head 15c2c839, poetry/hosted): patch::redirect::poetry::rewrite_poetry takes 51% of Ir. Most of that is utils::poetry_lock::rewrite_poetry_lock_all → utils::lock_fragments::rewrite_batch (41%). vex::discover::pypi_locks::extract takes another 29%: it re-parses poetry.lock with toml_or_diag (16%) after the rewrite already parsed it.
Ideas: share one parsed lock between the batched rewrite and VEX discovery, and keep rewrite_batch from walking every fragment per patched package.

Not a regression: it's a standing slow-system entry on the ledger (Discussion #575). pdm (2.64x) has the same rewrite_batch + VEX extract profile, but its slow streak restarted today.

Runner: 4 vCPU, Intel(R) Xeon(R) Processor @ 2.10GHz (cloud sandbox), run 2026-10-13.

Generated by Claude Code

[agent] Triaged as priority:p1 (Poetry, PyPI family). Standing slow-system entry, not a duplicate: #1246 (pipenv) and #836 (uv/pylock) are separate per-format hot spots, though all three point at per-patch lock re-parsing.

Generated by Claude Code

[agent] Claiming this issue (shared root cause: the batched poetry.lock rewrite and lockfile VEX each re-scan or re-parse the whole lock per patched package). Branch: agent/fix-poetry-lock-batch-rescan. Claim-ID: 2026-10-13T09:21:01Z-4aa140

Generated by Claude Code

[agent] Draft PR: #1475

Generated by Claude Code

Speed up the batched Poetry and PDM lock rewrite

Find lock fragments with a small automaton

Tighten the lock rewrite counters' docs

[agent] #1475 is ready for review. It covers the rewrite half of this issue: one fragment-search pass over the original lock and one over the result instead of 2–3 per changed fragment, and no extra lock parse for the stale-install check. On poetry/hosted that takes the scan from 506M to 403M instructions (−20%), with byte-identical output, and PDM's batch gets the same fix.

The VEX half is still open: both discovery passes re-parse poetry.lock (~35% of the remaining instructions). That needs the shared uv/pylock/PDM VEX readers to accept a reused parse, so it's a follow-up slice and this issue stays open (the PR says Refs, not Fixes).

Generated by Claude Code

[agent] #1475 (the rewrite half) has merged. I'm releasing the claim so the remaining slice can be picked up: both VEX discovery passes still re-parse poetry.lock (vex::discover::pypi_locks::extract, about 35% of poetry/hosted instructions after #1475). Fixing that means letting the shared uv/pylock/PDM VEX readers reuse a parse instead of taking &DocumentMut.

Generated by Claude Code

[agent] Claiming this issue (remaining slice; shared root cause: hosted scan's VEX discovery re-parses poetry.lock in both its passes although the batched rewrite already parsed the same texts). Branch: agent/fix-poetry-vex-lock-reparse. Claim-ID: 2026-10-13T14:22:13Z-3898b9

Generated by Claude Code

[agent] Draft PR: #1486

Generated by Claude Code

Read lockfiles without copying them for VEX

Tidy read-only TOML parse after review

[agent] #1486 is ready for review. It covers part of the remaining slice: VEX discovery and the PyPI lock inventory now read lockfiles through toml_edit's read-only document instead of an editable copy, which takes poetry/hosted from 403M to 365M instructions (−9.6%) with byte-identical output. The lock is still parsed five times across two texts (about 46% of instructions), and sharing one parse per text is the next slice, so this issue stays open (the PR says Refs).

Generated by Claude Code

[agent] Bench: run 2026-10-14, main 313d5324. Poetry is no longer flagged slow.

The fixes for this issue landed between 15c2c839 and 313d5324: 5c33798 (#1475), the batched Poetry/PDM lock rewrite, and 690dc1c (#1486), which reads Python locks without copying them in VEX. Same-machine interleaved compare:

scenario	window	base	head	Δ wall [95% CI]	Δ CPU
`poetry/hosted`	day (`15c2c839`, 15 pairs)	216.7 ms	172.5 ms	-18.6% [-23.3, -13.6]	-16.1%
`poetry/rescan`	day	147.5 ms	146.0 ms	-1.7% [-9.7, +5.4]	-3.7%
`poetry/hosted`	week (`9c43dfc9`, 8 pairs)	306.5 ms	137.4 ms	-56.7% [-61.9, -45.2]	-54.7%
`pdm/hosted`	week	284.2 ms	154.0 ms	-45.1% [-50.0, -39.0]	-46.6%

In today's suite run, poetry/hosted is 1.55x the median hosted ms/pkg (0.380 vs 0.245), under the 2x slow threshold. Yesterday it was 2.63x. PDM is at 1.57x, also under. One run under the threshold isn't a trend, so I'm leaving the issue open for a human to close once it holds.

Runner: 4 vCPU Intel(R) Xeon(R) Processor @ 2.80GHz.

Generated by Claude Code

Navigation Menu

Repository navigation

Slow scan: poetry hosted 2.6x median ms/pkg (lock_fragments::rewrite_batch + VEX lock re-parse) #1473

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Slow scan: poetry hosted 2.6x median ms/pkg (lock_fragments::rewrite_batch + VEX lock re-parse) #1473

Description

Activity

mikolalysenko commented on Oct 13, 2026

mikolalysenko commented on Oct 13, 2026

mikolalysenko commented on Oct 13, 2026

mikolalysenko commented on Oct 13, 2026

mikolalysenko commented on Oct 13, 2026

mikolalysenko commented on Oct 13, 2026

mikolalysenko commented on Oct 13, 2026

mikolalysenko commented on Oct 13, 2026

mikolalysenko commented on Oct 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions