Repository navigation
Depot runners lost mid-job evicted 2 merge-queue entries in 27h #1660
Description
Two merge-queue entries were evicted in about 27 hours because a Depot runner (depot-ubuntu-24.04-4) stopped reporting partway through a job. GitHub then failed the job with "The self-hosted runner lost communication with the server". Both jobs normally finish in 1–5 minutes on the same label. The failed ones were declared lost about 10 minutes after the runner went quiet.
| When (UTC) | merge_group run | Job | Runner | Normal duration | Failed after |
|---|---|---|---|---|---|
| 2026-10-20 16:48 | 39460266425 (PR #1617) | e2e (ubuntu-latest, e2e_vex_build, pipenv) | depot-wkpbh4r80h | ~1 min | 10m00s |
| 2026-10-21 19:02 | 39657631762 (PR #1593) | e2e (ubuntu-latest, e2e_gradle_discovery_build …) | depot-v42fm5z0wd | ~5 min | 15m35s |
Both PRs merged after a rebuild, so each loss cost one full queue cycle (about 10 minutes plus the rebuild of every entry behind them). Neither job log could be downloaded, so I can't tell whether the runner died on its own or was starved for memory. The Gradle leg is the heaviest Linux e2e leg. The pipenv leg is light, which points away from memory pressure in that case.
This repo can't fix the cause; it's runner infrastructure. Next steps for a maintainer:
- Check the Depot dashboard for instance terminations or OOM kills at those times.
- If this keeps happening, consider larger runners for the e2e legs or a Depot support ticket.
The CI janitor will add new occurrences here.
Generated by Claude Code
Activity
mikolalysenko commentedon Oct 21, 2026
[agent] Triage (CI lane): this is runner infrastructure, so no change in this repo fixes it.
scripts/merge-queue-fail-fast.pyalready ignores runner-loss failures, which keeps a lost runner from cancelling sibling jobs. It cannot save the evicted entry, though. Labelled p3 and needs-human for the Depot dashboard and support-ticket checks above. The CI janitor keeps logging new occurrences here.Generated by Claude Code