A platform deploy kills in-flight coding runs, and the Pilot writes retryable:true for a reader that does not exist

Our own deploys kill in-flight coding runs, and the Pilot records `retryable:true` without retrying

Run b9d9c051-662c-47f6-a5ef-da0fab7aa6cd (instance bd43f4de-…, 2026-08-15) recorded two
terminal errors, both retryable: true, and was marked failed:

ts	class	phase	steps	message
00:25:20	`infra_transient`	`s9-waitidle`	2	Durable Object reset because its code was updated.
00:28:27	`provider_stall`	`s21-decide`	4	AI provider stopped sending mid-reply — 20s of silence

Both carry "retryable": true and "resumableRound": false.

1. The first cause is us

Durable Object reset because its code was updated is a platform deploy. It is not the
owner's network, not their machine, not the provider. Deploying the API worker evicts the DO and
kills whatever coding run was mid-flight.

$ gh run list --workflow=deploy-api.yml   → 8 successful API deploys in the 24h to 2026-08-15

Eight self-inflicted kill windows in one day, on a platform whose flagship agent runs for hours.
The owner experienced this as agents dying for no reason.

2. `retryable: true` has no consumer on this path, and the code says so

thinkWithAutoResume (lib/resumable-round.ts:170) is the #518 auto-retry — "on a retryable
failure that carried a round, run it ONCE more from that round." Its call sites:

$ grep -rn thinkWithAutoResume workers/api/src --include=*.ts | grep -v '\.test\.'
agent-do.ts:566              ← chat turns
coding-failure.ts:362        ← a comment, not a call

coding-failure.ts:362 states it plainly:

// The Pilot has no thinkWithAutoResume (#518) — this is expected to be false

So the Pilot classifies a failure as retryable, records that verdict, and then fails the
run. The verdict is written for a reader that does not exist. A chat turn dropped by the same
provider stall retries automatically; an hours-long autonomous run does not.

autoResumableRoundOf (:163) already encodes the safe-to-retry rule — retryable and carrying
a round — so the decision logic exists and is unused here.

3. One run, two deaths

Both rows above share runId b9d9c051. This is the double-filing #546 reports; noting it
because the classification has since improved (infra_transient / provider_stall rather than
unknown) while the double-write has not. #546 should be re-read against this, not re-derived.

What a fix must decide

An infra reset and a provider stall are both retryable but not alike:

DO reset from a deploy is ours, cheap to retry, and arguably should not end a run at all —
the workflow is durable and the session survives. Retrying is close to free.
Provider stall costs the owner's credit on every attempt. [bug] #442 stores a resumable round the user cannot reach — the gate is byte-equal text, the owner speaks, and the console clears the composer before sending #518's "exactly two attempts, not a
loop" reasoning applies unchanged: "a turn that retries until it succeeds spends someone else's
credit deciding when to stop."

Whether a deploy should defer while runs are in flight, or runs should simply survive a reset, is
the larger question and belongs in this issue's discussion rather than being assumed.

Acceptance criteria

A coding run killed by a DO code-update reset resumes rather than terminating, or the issue
records why terminating is correct and the owner is told the cause in the run's own detail —
today it reads as an unexplained failure.
The Pilot consumes the retryable verdict it already computes, reusing autoResumableRoundOf
rather than a second rule. Attempt count bounded and justified against [bug] #442 stores a resumable round the user cannot reach — the gate is byte-equal text, the owner speaks, and the console clears the composer before sending #518's reasoning.
Retry policy differs by failure class where the cost differs, and the difference is stated.
A test asserts a retryable:true coding failure is retried — the property [bug] #442 stores a resumable round the user cannot reach — the gate is byte-equal text, the owner speaks, and the console clears the composer before sending #518 was written to
prove is that the stored round is consumed, and the same standard applies here.
A guard that a failure class marked retryable has a consumer on every driver, stated with
its denominator per ADR 0002. A verdict computed and read by nobody is the defect this issue is
really about.

Verified vs inferred

Verified: both error rows with full context (retryable, resumableRound, failureClass,
phase, steps) read live from agent_trace; the 8 deploy-api runs; the complete call-site list
for thinkWithAutoResume; coding-failure.ts:362's comment verbatim; resumable-round.ts:163
and :170.
Inferred: that the DO reset was caused by one of those 8 deploys specifically. The reset
message names a code update and 8 code updates shipped that day, but the reset was not
correlated to a particular deploy by timestamp. If some other mechanism resets the DO, AC1's
framing changes and this should be re-checked before implementation.

The inferred correlation is now exact — and it was one of my own commits

The issue said the DO reset "was not correlated to a particular deploy by timestamp." It is now.

Time (UTC)	Event
00:21:53 → 00:24:23	`Deploy API Worker` for `4c0826b`, success
00:22:09	run `b9d9c051` starts
00:24:09	run records its `/tmp` scratch act (see #582)
00:25:19.958	`infra_transient` — "Durable Object reset because its code was updated."

56 seconds between the deploy finishing and the DO reset. 4c0826b is #571's commit — landed
by an agent working this backlog. We killed the owner's run with our own fix.

It was not one deploy, it was a storm

API deploys in the 45 minutes surrounding that run:

23:59:55 → 00:03:08  80eb42f
00:21:53 → 00:24:23  4c0826b   ← run starts 00:22:09, dies 00:25:19
00:25:59 → 00:28:49  f126db6
00:29:50 → 00:32:45  0172534
00:37:51 → 00:40:49  2fc3f90
00:42:01 → 00:44:17  50714da   (cancelled)
00:44:00 → 00:47:09  fcfbce0

Seven deploy runs in 48 minutes. The run existed entirely inside that window and had no chance of
surviving it. This is not a rare race — on an active development day, a long-running coding run is
close to guaranteed to be evicted.

The second failure (provider_stall, 00:28:27) falls inside f126db6's deploy window but is a
different class and is not claimed here as deploy-caused.

What this sharpens

AC1 asked whether a run should resume or terminate on a DO reset. The measured deploy cadence
answers it: terminating is not viable. A platform that ships 7 API deploys in 48 minutes cannot
also treat a code update as a run-ending event, or its flagship hours-long agent is unusable on any
day the team is working. Resumption is the requirement, not one of two options.

It also adds a case AC5's guard should cover: an infra_transient failure whose cause is our own
deploy is categorically different from one caused by the owner's network, and should not consume
the owner's retry budget or credit at all.

Verified vs inferred, updated

Verified now: all seven deploy runs with start/end timestamps and SHAs from
gh run list --workflow=deploy-api.yml; the run's own start and both error timestamps converted
from the epoch-ms values in agent_trace; the 56-second gap.
Still inferred: that 4c0826b's deploy is the specific eviction rather than a later
rollout step of the same release — the eviction moment itself is not logged, only its
consequence. The 56-second gap and the absence of any other candidate in that minute make it the
only plausible one, and nothing in the fix depends on which of the seven it was.

fix(coding): a run our own deploy interrupted is replayed, not buried…

Landed — the Pilot now consumes the verdict it was already writing

Commit 03762fd (CI green incl. test:e2e; API deploy 31876808933 succeeded, migration 0127
applied).

What the resume actually is

Rethrowing. An error escaping run() makes Cloudflare replay the instance from its journal, so
every step that already succeeded returns its recorded result and only the step that died runs
again. That is not a hoped-for mechanism — #546 measured it, when run 82739cb6 filed its own death
twice with an identical journalled runStartedAt and got further the second time.

So the missing piece was never the retry. It was that the driver tore the run down first: the
finally ends the session, releases the single-flight driver claim, closes the board card and posts
"Loop stopped". Skipping the teardown while resuming is the load-bearing half — without it the
replay carries on working inside a run every surface has already reported as failed. Guarded by
coding-resume.test.ts, which asserts each terminal writer sits inside the !resuming branch.

AC2 — `autoResumableRoundOf` could not be reused literally, and that is a finding

It gates on a stored ResumableRound, which only runAgentThink ever attaches. On a Pilot error it
returns null by construction — a "reuse" that would compile, pass, and retry nothing. That is
precisely why every Pilot failure records resumableRound: false; coding-failure.ts:364 was
right, and it was right for a reason that also makes the suggested fix a no-op.

What generalises is the predicate, so isRetryableFailure was lifted out of autoResumableRoundOf
and both drivers gate on one implementation. There is a test asserting both readers see it, and
asserting the Pilot's carrier is the classification (a DO reset arrives as a plain message with no
structural fields; classifyCodingFailure is the site that knows).

AC3 — retryable is not resumable, and the table is what keeps them apart

CODING_RESUME_POLICY is total over all 11 classes, each with a stated reason. Exactly one
resumes.

class	resume	why
`infra_transient`	yes	ours, and the journal makes the replay free
`workflow_internal`	no	CF already retried the attempt; a fresh run has no journal, and 2 of its 5 recorded occurrences had pushed to `origin main`
`provider_stall`	no	every attempt spends the owner's credit, and the Pilot has no stored round to resume from (#518)
8 others	no	credentials, rate limit, overrun, ceiling, runner, unknown — each with its own sentence

Your framing is what made the bound decidable: an infra_transient caused by our own deploy costs
the owner nothing, so three attempts are affordable where #518 allows two — and the difference is
stated rather than assumed. The counter is durable (agent_loop_runs.interruptions), because an
in-memory one would replay with the journal and reset on the very event it bounds. A run with no
loop-run row is not resumed: an unbounded replay is worse than a terminated run.

The owner is told, on both surfaces: a coding.run.interrupted trace event and a chat line naming
the cause and the attempt number. The run is marked waiting_reason: "platform_interrupt" so a
replay in flight reads as waiting, never as stalled — a resume has nothing ticking by design.

Left open

AC5's guard is stated over every failure CLASS of this driver, not over every DRIVER. The
chat, apply, browser and pipeline drivers each have their own failure vocabulary, and a
cross-driver guard needs a shared classification they do not yet have. That is a real piece of
work and I am not claiming it; say the word and I will file it.
Not yet observed in production. No deploy has interrupted an in-flight coding run since this
landed, so interruptions is still 0 on every row and the resume path has unit coverage only.
The next deploy that catches a live run is the test; if it fails I will fix it forward.
The double-filing ([bug] WorkflowInternalError, traced: it is not the objective failing and not terminal — one run filed its own death twice, and all 8 occurrences classify as unknown #546) is untouched — one run still records two deaths when it dies twice.
Correctly so: those are two attempts, and this change makes the first of them survivable rather
than merging the records.

fix(runs): a park states when it ends, because the instant was comput…

AC5 closed — `1105690`. The sweep found a driver that consumed nothing at all.

03762fd correctly declined to claim this: the guard was stated over every failure CLASS of one
driver, not over every DRIVER, because a cross-driver guard needs a shared classification and there
was none.

The finding: the classification was never coding-specific

Building the shared thing turned up what made it easy. classifyCodingFailure reads a provider
transport failure, a runner that went away, a Cloudflare per-invocation ceiling and an isolate
reset — every one a fact about the PLATFORM, and every one able to kill an apply, a browser
task, a pipeline or a chat loop identically. Only its first consumer was the Pilot. The name
recorded the consumer instead of the subject.

So codingResumePlan → driverResumePlan and CODING_RESUME_POLICY → DRIVER_RESUME_POLICY. Not
cosmetics: a decision function named after one driver is exactly how the next driver grows a second
copy of the rule instead of calling this one, which is the defect this issue is about.
classifyCodingFailure and CodingFailureClass keep their names deliberately — a shared vocabulary
whose name is a wart rather than a coupling, and renaming churns ~60 references for no property.

`agent-loop.ts` had no consumer at all

Five durable drivers, enumerated from workflows/ on disk. Four consumed the verdict in some form.
agent-loop.ts consumed nothing — its catch turned every death into stop = { reason: "failed" },
including our own deploy resetting the isolate, while its four peers all re-threw and resumed. At the
cadence this issue measured (seven API deploys in 48 minutes) that is a permanently closed run for a
cause that is ours.

Nobody had checked, because nobody had asked the question as a SET — the same reason
coding-session.ts sat at zero logEvent calls against 6/6/7/3 in its peers until
workflow-trace.test.ts asked.

It now reaches the same decision function the Pilot does, and it is bounded where the other
three are not, because it has an agent_loop_runs row and countInterruption can hold a durable
count. The replay is cheap for the Pilot's reason: act, decide, reserve and settle are all
step.do, so a replay returns each journalled result without re-executing or re-charging.

Skipping the budget settle while resuming is the load-bearing half, and the reason is money. The
finally's settle and the loop's own settle-${iteration} are DIFFERENT step ids, so settling on
the way out and then replaying into an un-journalled settle-${iteration} charges the shared tree
pool twice for one iteration — worse than the leak the surrounding comment already calls out.
Left alone, the replay settles it once through the step that was always meant to.

The guard, with its denominator (ADR 0002)

workflows/driver-failure.test.ts, 13 arms:

the driver set is read from workflows/ on disk — 7 source files, 5 drivers — and a
non-driver has to be NAMED with its reason, reusing the same two exemptions
workflow-trace.test.ts uses so two guards over one set cannot disagree about what the set is;
the registry must equal that set in both directions;
each entry's claimed consumer is checked against that file's source, so a registry cannot
credit a driver with a consumer it does not reach;
four further arms assert the new consumer consumes rather than references — the rethrow is
inside the resume branch, it precedes the terminal failed, the settle is gated, and only that
branch sets resuming. [bug] #442 stores a resumable round the user cannot reach — the gate is byte-equal text, the owner speaks, and the console clears the composer before sending #518's standard: a call whose verdict falls through to failed would
satisfy a registry and change nothing.

Verified red before green: deleting agent-loop's branch fails the guard on agent-loop
specifically. It also caught one of my own registry entries saying UNBOUNDED without saying why.

What this does NOT claim, deliberately

That every driver resumes. job-apply, browser-task and pipeline-run consume the verdict
through the narrower isTransientInfraError and rethrow UNBOUNDED, because none has a durable
run row to count against — the first two carry a task id, and pipeline_runs has no interruptions
column. The guard records that asymmetry with a stated reason per driver rather than smoothing it
over. Bounding those three is a migration and a separate decision; say the word and I will file
it.

Still not observed in production

interruptions is 0 across all 89 runs and waiting_reason null on all 89 — no deploy has
interrupted an in-flight run of either driver since 03762fd, so both resume paths still have unit
coverage only. That is unchanged from 03762fd's own caveat and I am not claiming better. The next
deploy that catches a live run is the test.

CI green on 1105690; API deploy 31880412727 succeeded.

fix(runs): a park states WHICH kind of deadline it has, so the handof…

fix(runs): a run the platform is about to replay stops being filed as…

fix(runs): bound a park, enforce a cancel, retire a displaced run (#790)

Navigation Menu

Repository navigation

A platform deploy kills in-flight coding runs, and the Pilot writes retryable:true for a reader that does not exist #583

Our own deploys kill in-flight coding runs, and the Pilot records `retryable:true` without retrying

1. The first cause is us

2. `retryable: true` has no consumer on this path, and the code says so

3. One run, two deaths

What a fix must decide

Acceptance criteria

Verified vs inferred

The inferred correlation is now exact — and it was one of my own commits

It was not one deploy, it was a storm

What this sharpens

Verified vs inferred, updated

Landed — the Pilot now consumes the verdict it was already writing

What the resume actually is

AC2 — `autoResumableRoundOf` could not be reused literally, and that is a finding

AC3 — retryable is not resumable, and the table is what keeps them apart

Left open

AC5 closed — `1105690`. The sweep found a driver that consumed nothing at all.

The finding: the classification was never coding-specific

`agent-loop.ts` had no consumer at all

The guard, with its denominator (ADR 0002)

What this does NOT claim, deliberately

Still not observed in production

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

A platform deploy kills in-flight coding runs, and the Pilot writes retryable:true for a reader that does not exist #583

Description

Our own deploys kill in-flight coding runs, and the Pilot records retryable:true without retrying

1. The first cause is us

2. retryable: true has no consumer on this path, and the code says so

3. One run, two deaths

What a fix must decide

Acceptance criteria

Verified vs inferred

Activity

serge-ivo commented on Aug 15, 2026

The inferred correlation is now exact — and it was one of my own commits

It was not one deploy, it was a storm

What this sharpens

Verified vs inferred, updated

serge-ivo commented on Aug 15, 2026

Landed — the Pilot now consumes the verdict it was already writing

What the resume actually is

AC2 — autoResumableRoundOf could not be reused literally, and that is a finding

AC3 — retryable is not resumable, and the table is what keeps them apart

Left open

serge-ivo commented on Aug 15, 2026

AC5 closed — 1105690. The sweep found a driver that consumed nothing at all.

The finding: the classification was never coding-specific

agent-loop.ts had no consumer at all

The guard, with its denominator (ADR 0002)

What this does NOT claim, deliberately

Still not observed in production

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Our own deploys kill in-flight coding runs, and the Pilot records `retryable:true` without retrying

2. `retryable: true` has no consumer on this path, and the code says so

AC2 — `autoResumableRoundOf` could not be reused literally, and that is a finding

AC5 closed — `1105690`. The sweep found a driver that consumed nothing at all.

`agent-loop.ts` had no consumer at all