Repository navigation
Recover an open issue left in-progress by a Run that was killed outright #710
Description
This was generated by AI during triage.
Split from #695.
Problem
A Run or Spec run killed outright (kill -9, an OOM kill, a power cut, a reboot) never ends its Claim. README.md says: "A process killed outright, with kill -9 or by a power cut, releases nothing: the issue stays in-progress for you to relabel." src/claim.rs ends a Claim only from inside the Run that made it. The Sweep only takes in-progress off closed issues. So an open issue whose Run died stays in-progress until the Day shift notices it. Its Worktree, local Issue branch and agent session are left behind too.
Recovery is possible today, but only by hand: a Continuation (ADR 0002) picks up from the Issue branch when someone reruns thirdshift on the issue.
What done looks like
A later Pass notices an open in-progress issue whose Run is no longer alive, and handles it without a human. It either continues the Run, resuming the agent session by id where the Harness allows, or ends the Claim the way a Failed run would.
Design questions
- Detecting a dead Run without a daemon. ADR 0009 rejected a long-running process. A Run may have been started on another machine. One option is to record the host and PID with the Claim, so a Pass on that host can check whether the process is alive. A Pass on another host would leave it alone.
- Continue or release? Does a dead Run's issue get continued automatically, or handed back to
ready-for-agent, or kept for the Day shift with a notification? - What to do with the leftovers: the Worktree, the local Issue branch, and unpushed commits.
- Interaction with related issues: make sending ctrl c on running implement result in switching in-progress label to ready-for-agent label #613 (Ctrl-C hands the issue back to
ready-for-agent) and Pickup runs continue a ready-for-agent issue the Day shift handed back after a failed Run #617 (Pickup continues an issue the Day shift handed back after a failed Run).
Background
The Temporal research in #694 (docs/research/temporal-for-thirdshift.md, §7) names crash recovery as the main thing Temporal would add. It recommends not adopting Temporal now.
Activity
JacobStephens2 commentedon Nov 10, 2026
Investigation summary
The recovery gap remains on selected Base commit 9eea88c. Keep #710 open for a recovery-policy decision and proposed split.
Evidence:
src/claim.rs:119ends a Claim through the running process. SIGKILL prevents that completion path. The current manual explicitly documents the resulting stranded Claim (docs/guide.md:102).src/pickup.rs:352sweeps closed issues; it does not inspect open Claims for dead owners.src/logs/command_record.rs:87).src/logs/process_identity.rscombines boot identity with process-start ticks, handling PID reuse and reboot. The log reader already classifies unfinished records whose process is gone asdied(src/logs/reading.rs:414). These provide reusable evidence, but do not associate exclusive recovery authority with the current Claim.src/reservation.rs:64); normal release happens in Drop with an exact lease. Open Notice a reservation whose owning process has died #1959 addresses orphan reservations, but releasing a reservation does not end a Claim or preserve retained work. ADR 0017 explicitly distinguishes these operations.src/worktree.rs:155), and existing Resume handling operates inside a live Run (src/session.rs:337). Cross-Run session recovery needs separate design.HeldRun::killexplicitly leaves descendants alive (tests/support/mod.rs:322); the killed-Run regression releases its surviving fake Harness later. Recovery must establish writer quiescence before allowing replacement work.Validation:
cargo test --test pickup_run a_killed_pickup_run_or_ticket_run_frees_its_slot_for_the_next_pass -- --exact --nocapturepassed (1 test). It confirms the Command slot is freed and the next Pass takes another Ready issue, rather than recovering the killed issue.scripts/affected-tests mainreported no changed files; Git status and diff remained clean.Related issues are complementary: #1901 explicitly excludes #710; #1959 covers reservations; #833 covers identifying the Claim's host. #613 and #617 are closed and address graceful interruption and subsequent hand-back, respectively.
Recommended next step: shape a staged Spec after the Day shift settles the questions below. Preserve ADR 0009's one-Pass model and ADR 0017's prohibition on timer-based reservation expiry. No GitHub writes or source/history changes were made.
Questionnaire
For the Day shift deciding #710: repository investigation confirms that killed Runs leave recovery work beyond the origin reservation covered by #1959. The following decisions are needed to make implementation actionable.
in-progressmarks only a live Run #1901/ADR 0029, with automatic Continuation deferred. This decides the resulting labels and retry behavior.