Add a pause/resume capability for private instances (distinct from full cancel) #825

Activity

1. serge-ivo

   closed this as completedin 4f1a882on Sep 20, 2026
2. serge-ivo commented on Sep 20, 2026

   

   serge-ivo

   on Sep 20, 2026

   Contributor

   More actions

   Implemented in 4f1a8826 — the commit closes this issue

   A reversible paused state for a private instance, distinct from cancel: it blocks new runs, asks any run in flight to stop, and keeps the subscription, config, repos, documents, memory, timeline and run history exactly as they were. resume puts it back.

   The design was already recorded here, including what was missing

   'paused' has been in agent_instances.status's declared domain since 0002_instances.sql and had never had a writer. That was deliberate, and workers/api/src/lib/status-domain.ts carried the decision — MISSING CAPABILITY, not a dead word — naming the four pieces that had to land together, because "shipping the writer alone is what makes a half-working control" (#664): a writer, a resume path, a console control, and an admin instance filter. All four are in this commit.

   Two things had already been built for this, in advance, and needed no change:

   • lib/trigger-eligibility.ts gates every background dispatch on an ALLOWLIST (status = 'active') rather than status != 'canceled', saying why outright: "if pause ever acquires a writer, a paused instance must not be running cron work, and with an allowlist it silently already does not" (A cancelled instance's webhook triggers and inbound event deliveries still fire — the cron sweep was gated, the pump and the webhook route were not, and the console still has no surface to stop them #649). So the cron sweep, the agent-to-agent pump, delivery retries, the public webhook and the manual trigger run all started honouring a paused instance the moment this landed, with no new predicate anywhere.
   • lib/subscription-standing.ts + migration 0131 keep the shared subscriptions row standing for any sibling that is not canceled — paused included — so pausing cannot retire a subscription out from under a sibling instance.

   What landed

   Piece	Where
   The transitions + run refusal (pure)	workers/api/src/lib/instance-pause.ts
   POST /v1/instances/:instanceId/pause · POST /v1/instances/:instanceId/resume	workers/api/src/routes/instances-lifecycle.ts
   Run-admission gate	withPauseGate in lib/loop-drivers.ts
   MCP pause_instance · resume_instance (write, dry_run)	instance-tools/base.ts, server 0.1.46 → 0.1.47
   Console control	store/console/src/components/PauseCard.tsx, above the Danger zone
   Admin filter	paused restored to store/admin/src/pages/Instances.tsx

   One choke point blocks new runs. Every driver.start() in the tree takes its driver from loopDriverFor, so wrapping there covers all five entry points — POST /loop, the continue route, delegate_goal and start_work through the tool registry, the delegation helper, and the objective-queue drainer — and the sixth nobody has written yet. trigger-eligibility.ts's own header records what the per-call-site version of this cost on the background side.

   The admin filter came back because a tripwire fired. #598 deleted the paused option as "a capability advertised to a human" that did not exist, and list-query.test.ts derives the offered list from the server's provenance table in both directions — so gaining a writer made it demand the option back. That test failing is how this piece was found rather than remembered.

   Three decisions worth naming

   Interactive chat still works. Pause stops work the platform starts on its own, or that the owner starts as a run. A paused agent you cannot ask "what went wrong" is one you have to un-pause to diagnose — which defeats the case in this issue.

   Stopping a live run is a REQUEST, not a kill. There is no way to end a Workflow mid-step: requestCancel sets cancel_requested, each run stops at the top of its next iteration, and the sweeper enforces the ones that never read it. The route answers runsAskedToStop and the console says "asked to stop" — never "stopped".

   The two gates fail in opposite directions, deliberately. Background dispatch fails closed on a status it has not been taught, because it spends with nobody watching. The run gate fails open, because it sits in front of an owner pressing a button and a future status nobody has thought about yet must not silently make an agent unusable. Both directions are pinned by tests.

   Ordering: the status is written before runs are asked to stop, so a run finishing its iteration between the two writes finds the instance already paused and whatever it starts next is refused.

   Verified

   • 47 new tests — 12 pure transitions, 14 route tests driven end to end against the real migrated schema, 6 on the admission gate, 12 console, plus the contract and status-domain updates.
   • Mutation-checked, each turns tests red: removing the gate from loopDriverFor; not asking runs to stop; letting resume un-cancel a cancelled instance; making pause retire the subscription (i.e. become cancel); making the run gate fail closed on an unknown status.
   • Full suite 12,076/12,076; e2e 499/499; pnpm -r typecheck plus the three test-config typechecks (Three tsconfigs exclude test files and CI typechecks none of them, so a type-level guard in any worker test is inert #599); biome; docs:drift; all 13 check-*.mjs including check-surface-lock --require-history. Both routes documented in the OpenAPI spec.

   Not verified: no live pause against production. The runs stopped in tests are stopped through the real store, but no test drives a Workflow to the point of reading the flag — check_instance_loop is where that is watched in practice.