You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Flake: nightly poetry hosted rollback fails on one transient PyPI request (UpstreamClient has no retry) #1650
poetry::poetry_hosted_fresh_install_then_manifestless_vex panicked at crates/socket-patch-cli/tests/e2e_vex_build/poetry.rs:898: rollback exited 1 with hosted_restore_refused: cannot restore pkg:pypi/six@1.16.0 to its upstream registry entry: six==1.16.0: GET https://pypi.org/pypi/six/1.16.0/json: error sending request for url (...)
The other 3 poetry tests in the same job reached PyPI fine seconds earlier. The 10-19 nightly passed on the same code.
Frequency: 1 of the last 12 nightlies. It has not been seen on PR or merge_group runs (this leg is nightly-only).
Root cause (partly known) UpstreamClient (crates/socket-patch-core/src/patch/redirect/upstream/client.rs) sends each registry lookup once. get_json_authorized and download turn any transport error into a final GET {url}: {e} with no retry. The patch API client does retry dropped connections and 429/503 (api::retry, send_json_request). Because the message keeps only reqwest's top-level text, the log can't tell us the transport error class (DNS, reset, timeout).
Proposed fix (not started: the evidence was too thin this run)
Format registry-lookup errors with the cause chain, the same way api::client::network_error_detail does, so the next occurrence is classifiable.
If it recurs as a dropped connection or a 429/503, give UpstreamClient's GETs the same bounded ApiRetry policy that send_json_request uses (is_retryable_transport plus is_retryable_status, the shared retry window). A user's hosted rollback hits the same failure mode.
Filed by the hourly CI janitor; the job log expires, so the signature is quoted above.
[agent] Triaged (CI lane): p3, nightly-only flake (1 in 12 nightlies, not seen on PR or merge-queue runs), so it costs no PR/queue runner time. No open PR covers it yet. The proposed fix is in UpstreamClient (core), so it also affects users doing a hosted rollback. Not claimed this run because the CI-lane fixer already has 3 open PRs (#1641, #1642, #1647) waiting on review.
[agent] Re-verified (CI lane, 2026-10-22). Step 1 of the proposed fix is on main now: #1652 (bc307ce) formats registry-lookup errors with the transport cause chain (network_error_detail in crates/socket-patch-core/src/patch/redirect/upstream/client.rs, get_json_authorized and download). The regression test a_failed_lookup_keeps_the_transport_cause covers it.
Step 2, the bounded retry, still depends on the next occurrence's error class, as the issue says. I'm not claiming it until a nightly shows the flake again with the new message. If it shows a dropped connection or a 429/503, the fix is to give UpstreamClient's GETs the api::retry policy.
Activity
mikolalysenko commentedon Oct 21, 2026
[agent] Triaged (CI lane): p3, nightly-only flake (1 in 12 nightlies, not seen on PR or merge-queue runs), so it costs no PR/queue runner time. No open PR covers it yet. The proposed fix is in
UpstreamClient(core), so it also affects users doing a hosted rollback. Not claimed this run because the CI-lane fixer already has 3 open PRs (#1641, #1642, #1647) waiting on review.Generated by Claude Code
mikolalysenko commentedon Oct 22, 2026
[agent] Re-verified (CI lane, 2026-10-22). Step 1 of the proposed fix is on
mainnow: #1652 (bc307ce) formats registry-lookup errors with the transport cause chain (network_error_detailincrates/socket-patch-core/src/patch/redirect/upstream/client.rs,get_json_authorizedanddownload). The regression testa_failed_lookup_keeps_the_transport_causecovers it.Step 2, the bounded retry, still depends on the next occurrence's error class, as the issue says. I'm not claiming it until a nightly shows the flake again with the new message. If it shows a dropped connection or a 429/503, the fix is to give
UpstreamClient's GETs theapi::retrypolicy.Generated by Claude Code