Run gets permanently stuck PARKED after an interruption, and stop_work cancel request never lands #790

Activity

1. proagentstore

   added

   bugSomething isn't workingSomething isn't working

   P0: high priority

   on Sep 10, 2026
2. serge-ivo

   added a commit that references this issue on Sep 10, 2026

   fix(runs): bound a park, enforce a cancel, retire a displaced run (#790)

   ...

   260da89
3. serge-ivo commented on Sep 10, 2026

   

   serge-ivo

   on Sep 10, 2026

   Contributor

   More actions

   Investigated and fixed. Landed on main as 260da89 — 10 files, +908/−48.

   All three symptoms are real and independent, but the mechanisms turned out to be somewhat different from the shapes they present as. Root causes first, since two of them are not where you would look.

   1. Stuck PARKED, self-reporting as not stalled

   Not a timer that resets — no timer ran at all. runHealth (lib/work-report.ts) short-circuits on the park before any elapsed-time test is reached:

   if (run.status !== "running") return "ended";
   if (run.waitingReason) return "waiting";        // ← unconditional, no bound
   const last = run.lastAliveAt ?? run.lastProgressAt ?? run.startedAt;
   return now - last > STALLED_AFTER_MS ? "stalled" : "working";

   That short-circuit is correct for what it was written for (#583): a platform_interrupt park is a journal replay in flight and has nothing ticking by design, so reading its silence as death would report a recovery as a failure. What it could not express is a park that never ends — so stalled was unreachable for a parked run at any age, not merely delayed.

   The park is written once, in the workflow's catch, and cleared by exactly one condition — recordIteration, and only when the iteration actually advances. Run fe53a0c1 never advanced past instruction 1, so nothing could ever have cleared it. health would have read waiting at 25 minutes and equally at 25 hours.

   The only backstop was the 3h sweepStaleRuns, and a parked run defeats even that: the pause tick refreshes last_alive_at on a timer on purpose (0127), precisely so a legitimate long park survives the cutoff.

   2. stop_work never lands

   requestCancel is a flag write, and that is by design — the cancel is cooperative so the in-flight step can settle its spend. The defect is on the reading side: the flag has exactly two readers, and both live inside the running workflow — the per-snapshot poll in capture() and the pause tick(). A workflow that has thrown for a journal replay, or died, executes neither.

   So the flag was written, requestCancel truthfully returned true (the row was running), stop_work truthfully said "requested", and check_work reported "a cancel has been requested" forever. Nothing outside the workflow ever converted the flag into a terminal row. No liveness check by stop_work would have helped — a cancel on a live run is supposed to be a flag write; what was missing was an independent enforcer for when nobody is home to read it.

   3. A new start re-engages the wedged run

   The function is claimSessionDriver (lib/coding-store.ts); there is no in-memory engine object holding a reference — this is a durable Workflow over D1, so the coupling is through rows and the tmux pane.

   claimSessionDriver takes a claim whose driver_at is older than STALE_DRIVER_MS (15m). A wedged run stops heartbeating (touchSessionDriver rides the same tick that has stopped), so after 15 minutes a new start legitimately steals the claim — and did so silently: the UPDATE matched, true came back, and the old agent_loop_runs row was left status = 'running' with nothing to close it. Because ensureActiveSession reuses a session that is still active, the new Pilot then attached to the same tmux pane the old engine was sitting in. That is what "the original stuck run's engine went back to working" is.

   The window was 2h45m wide: STALE_DRIVER_MS is 15m and STALE_RUN_MS is 3h.

   What landed in 260da89

   Migration 0150_loop_run_parked_since.sql — parked_since and cancel_requested_at (not a park counter; the bound needs an instant, not a count), plus two partial indexes following 0068's pattern.

   File	Change
   lib/work-report.ts	runHealth/isStalled live here. New parkOverrun() bounds the park; new PARK_LIMIT_MS + MAX_PARK_MS.
   lib/agent-loop-store.ts	recordLiveness stamps parked_since only when not already parked; recordIteration clears it in the same statement that clears waiting_reason; requestCancel stamps cancel_requested_at.
   lib/run-sweeper.ts	Two new passes — cancel enforcement and wedged-park reap — plus a shared closeRuns that all three passes now route through.
   lib/coding-store.ts	claimSessionDriver reads the holder before the steal and retires the displaced run.
   lib/status-domain.ts	needs_human promoted app? → app, with provenance.

   The park budget is per-reason, not flat — and that is the one design point worth arguing. A single cap would have been wrong: coding-session.ts records that a usage-limit park may legitimately last six hours, so a 4h cap would call a healthy engine_limit park stalled. That trades "nothing is ever reported" for "healthy runs are reported dead", which is the #459 direction and the worse of the two. So:

   • platform_interrupt → 15m (the incident's park; same clock as STALLED_AFTER_MS)
   • human → 30m (coding-pause.ts already gives up at 15m)
   • engine_limit → 8h
   • MAX_PARK_MS (4h) remains the fallback for a reason with no budget of its own

   A published waiting_until still in the future overrides the budget: when the platform has stated when a park ends, a guessed table has no business contradicting it. Once that deadline passes, the budget applies again.

   The cancel enforcer requires two conditions, not one: an old cancel_requested_at and a dead heartbeat. A run in a long engine turn keeps heartbeating from the capture loop and its own cooperative stop is still coming — killing it from outside would strand the budget reservation the in-flight step holds, which is exactly what cooperative cancellation exists to avoid. It closes as cancelled, not failed: the owner asked, and recording their own stop as a failure would be a worse lie than the stuck row.

   On STALE_DRIVER_MS vs STALE_RUN_MS: they should stay different. They answer different questions ("may I drive this session?" vs "is this run alive?"), and a claim must become takeable long before a run is declared dead, or a genuinely dead workflow would lock its repo for three hours. The gap was never the bug — nothing happening when it was crossed was. Retiring at the steal closes it regardless of the two numbers.

   Schema note: agent_loop_runs has no ended_at/error columns, so terminal writes use the existing finished_at + stop_reason (closed enum) + detail. A displaced run closes as interrupted/needs_human rather than failed — it may have pushed commits before it wedged, so somebody should look. That made retireDisplacedRuns the first literal writer of needs_human, which the status-domain guard caught on the first run.

   Tests

   26 new cases: claim-retires-displaced.test.ts (new, 185 lines, 10 cases), run-sweeper.test.ts (+159, 8 cases), work-report.test.ts (+132, 8 cases), work-stop.test.ts (+7, a type fix the tsconfig.test.json program caught).

   One existing assertion changed, deliberately: work-report.test.ts's "a park OUTRANKS a dead heartbeat" used a park an hour old — which is precisely the behaviour this issue reports as broken. Split in two, so #583's protection is still asserted within budget and the new bound is asserted past it.

   10623 of 10624 tests green. The one failure is environmental and pre-existing — migration-hygiene.test.ts runs its guard with --require-history and this checkout is shallow; CI checks out with fetch-depth: 0. (A second file, workers/host/src/admin-api-proxy.test.ts, fails to collect locally because it imports the gitignored pages.ts build output; CI builds it first.) Both reproduce with these changes stashed.

   Bearing on #788

   Worth stating plainly, since this issue raised it: the objective queue drains from a run's terminal path, so a run that never terminates never drains — entries stay pending rather than starting against a held lock. That is the safe direction, but it did mean a wedged run could park the whole queue behind it indefinitely. With all three fixes, every one of those limbo states now reaches a terminal row — by budget, by cancel enforcement, or by being displaced — so the queue drains rather than stalling silently.
4. serge-ivo commented on Sep 10, 2026

   

   serge-ivo

   on Sep 10, 2026

   Contributor

   More actions

   Closing — the fix is on main as 260da89 (10 files, +908/−48), CI green and the API Worker deploy succeeded, so migration 0150 has run against production.

   All three root causes are fixed and deployed:

   1. Stuck PARKED, never stalled — runHealth short-circuited on waitingReason before any elapsed-time test, so stalled was unreachable for a parked run at any age. Now bounded by parked_since (migration 0150) with per-reason budgets — flat would have called a legitimate 6-hour engine_limit park stalled.
   2. stop_work never landing — the cancel flag's only two readers live inside the running workflow, which a run mid-replay never reaches. An independent sweeper pass now enforces it, requiring both an old cancel_requested_at and a dead heartbeat so a live run's cooperative stop is never pre-empted.
   3. A new start re-engaging the old run — claimSessionDriver took a stale claim silently, leaving the displaced run running on a session the new Pilot then reused. The steal now retires it.

   26 new tests, each verified to go red against the defect it pins.

   One thing left explicit rather than implied: this was verified by test and by the deploy landing, not by reproducing a wedged run in production — provoking symptom 1 means engineering a platform interrupt mid-run. Reopen if the behaviour recurs.

   Full analysis in the comment above.
5. serge-ivo

   closed this as completedon Sep 10, 2026
6. serge-ivo

   added a commit that references this issue on Sep 19, 2026

   feat(console): what every instance is doing, in one call instead of o…

   ...

   d2d6216
7. serge-ivo

   added a commit that references this issue on Oct 16, 2026

   feat(coding): a question parks the run at zero compute and an answer …

   ...

   3a45f1c