Skip to content

Stop Maven e2e evictions on Central 429s - #1189

Merged
Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
ci-janitor/maven-warmup-mirror
Oct 9, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
ci-janitor/maven-warmup-mirror

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Maven Central flakes are evicting PRs from the merge queue. pr-1103 (#1103, which only changes go_sum_edit.rs) was evicted twice today by Maven legs:

  • 21:14 UTC, run 37843887676, job e2e (ubuntu-latest, e2e_redirect_maven_build, maven, 3.9.3). The fixture warm-up failed all 3 attempts (21 s total). Maven reported maven-dependency-plugin:3.6.1 as absent on repo.maven.apache.org, failing in 0.39 s per attempt. That version has been published for years. Panic at tests/maven_build_common/mod.rs:106 (the required-leg guard).
  • 22:38 UTC, run 37853387102, job e2e_vendor_maven_build, maven, 3.6.3. The install Maven step got six HTTP 429s from repo.maven.apache.org. This PR does not fix that one; Cut release-test wait time and recover Maven downloads #1166 does (see below).

Each eviction also rebuilt the queue entries behind it (#1058 and #1168 at 22:38).

Root cause

Central's CDN rate-limits shared runner IPs with 429s, and Maven reports a fast 429 or a negative-cached 404 as an absent artifact. The warm-up retried the same host 5 s and 10 s later with -U, so it hit the same limiter each time. From this agent's container, repo.maven.apache.org was still answering 429 for the exact maven-dependency-plugin-3.6.1.pom at about 22:55 UTC, while Google's Central mirror answered 200.

Fix

The first warm-up attempt still goes to Central. After a resolution failure, the retries (with -U) go through Google's official Maven Central mirror, maven-central.storage-download.googleapis.com/maven2, which runs on a separate CDN. The fallback settings.xml gives the mirror the id central. The local repository therefore records the same origin as a direct fetch, and every later step, which runs with the test's plain or socket-mirror settings, reuses the warmed repository unchanged. The first retry no longer waits 5 s, since the new host has no reason to be throttled; the second retry waits 5 s.

The change only affects the warm-up, which is the one step in these capstones that fetches from Central. No assertion changed. A non-resolution failure still fails right away, as before.

Proof

Every caller of warm_fixture was tested locally with Maven 3.9 and SOCKET_PATCH_MAVEN_E2E_REQUIRED=1. The first attempt was forced to fail with a resolution error by pointing it at a dead mirror (local-only hack, reverted). In each case the retry went through the Google mirror and the whole capstone passed:

test result
e2e_vendor_maven_build -- --ignored ok (retry 2/3 via mirror)
e2e_redirect_maven_build -- --ignored ok (retry 2/3 via mirror; trusted-checksum, tamper and resigned steps all passed after it)
e2e_vendor_jvm_build -- --ignored maven_reactor ok (retry 2/3 via mirror)
e2e_vendor_maven_build -- --ignored, final code, no hack ok (direct Central)

rustfmt --check is clean on the touched file. cargo clippy -p socket-patch-cli --all-targets -- -D warnings reports nothing in it; its existing findings in prebuilt_common, common and e2e_vendor_yarn_berry_build are unchanged on origin/main with local rustc 1.93.1.

Related

  • Cut release-test wait time and recover Maven downloads #1166 adds a pinned-SHA512 archive.apache.org fallback for the Maven install step, which fixes the 22:38 429 eviction. It touches only workflows and scripts, so it doesn't overlap this PR. Together they cover both places the Maven legs reach Central.
  • No tests were removed or moved.

🤖 Generated with Claude Code

https://claude.ai/code/session_016uTRZozqF1m3bgNxiAbspS


Generated by Claude Code

The Maven capstones' fixture warm-up retried Maven Central itself
after a resolution failure. Central's CDN rate-limits shared runner
IPs with 429s, which Maven reports as an absent artifact, so the
retries 5 and 10 seconds later fail the same way. That evicted
pr-1103 from the merge queue at 21:14 UTC (run 37843887676,
e2e_redirect_maven_build on Maven 3.9.3).

Retries now go through Google's official Maven Central mirror, a
separate CDN. The mirror keeps the id `central`, so the local
repository records the same origin as a direct fetch and every later
step, which runs with plain settings, reuses it unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016uTRZozqF1m3bgNxiAbspS
@mikolalysenko Mikola Lysenko (mikolalysenko) added the ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) label Oct 8, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit ac19c0d. Configure here.

@mikolalysenko
Mikola Lysenko (mikolalysenko) added this pull request to the merge queue Oct 8, 2026
Merged via the queue into main with commit eaaea02 Oct 9, 2026
244 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the ci-janitor/maven-warmup-mirror branch October 9, 2026 00:13
@mikolalysenko Mikola Lysenko (mikolalysenko) added Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review and removed Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review labels Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants