feat(workflows): let a synchronous run continue in the background after a wait timeout by waleedlatif1 · Pull Request #9464 · simstudioai/sim

waleedlatif1 · 2026-10-30T03:20:36Z

Summary

A synchronous workflows run could only be waited on until it settled or hit the sync execution limit. That includes the manual draft runs (--manual, --trigger, --mock-payload, --from-block/--source-run, --stop-after), and manual runs refuse --async. So a caller testing a long or stuck draft run had no way to get its run id back and decide whether to keep waiting, watch it, or cancel it. Published runs already had a way to do this: --async plus workflows runs wait.

This PR adds a wait bound to the synchronous run:

API: POST /api/v2/workflows/{id}/execute takes a new waitTimeoutSeconds field.
- If the run settles within the wait, the response is the same as today.
- If the run is still going when the wait runs out, the answer is 202 with the existing run receipt { runId, statusUrl }. On the NDJSON result transport this arrives as a { type: "detached", data: receipt } frame.
- 0 waits until the run settles. The field can't be combined with async or stream, and anonymous public-API callers get a 400.
- executionTimeoutSeconds is now also accepted alongside waitTimeoutSeconds.
Execution: the run keeps going in the server process exactly as it started:
- It uses the draft state the use case already captured, the same entry point, and the same pause persistence, so human-in-the-loop resume works unchanged.
- It is admitted like an async run: async rate bucket, async reservation and async timeout.
- It takes no request abort signal, so a dropped connection never cancels it. It also registers an in-process aborter, so workflows runs cancel reaches it without Redis as well as through the usual Redis signal.
- The admission ticket and the execution-id claim stay held until the run settles, not until the response ends.
- Nothing is enqueued and nothing runs twice. The response only stops waiting on the one execution promise.
CLI: sim workflows run ... --wait-timeout <seconds>, parsed like every other --wait-timeout.
- If the run outlasts the wait, the CLI reads it back once and reports it the way workflows runs wait reports an elapsed bound: the run record on stdout and exit 4 while the run is still moving, plus a stderr line naming workflows runs wait and workflows runs cancel.
- If the run happened to settle in between, the CLI reports that outcome with the runs-wait exit codes instead.
- --wait-timeout with --async or --follow is refused locally. A manual --async now points at --wait-timeout.
Without waitTimeoutSeconds / --wait-timeout, request handling, output and exit codes are unchanged.
Mothership capability: Sim now sends a workflow-run-wait-timeout client capability on chat turns, so the mothership worker only uses --wait-timeout on a Sim that supports it. Older Sims keep the held synchronous run. The protocol copy is synced from the mothership contract; the worker side is a companion change that is not yet pushed.

Why not `--async` for manual runs

An async manual run would have to carry the whole draft state, and for --from-block the source run's snapshot too, through the job queue payload. That would be a durable-payload change that queued jobs and rollbacks must tolerate. Async also refuses result shaping (--select-output, file inlining) and adds a queue hop to every draft test, though most finish in seconds. --wait-timeout keeps the fast path inline and returns a run id only when the run is actually long. Accepting --async for manual runs could be added later on top of this, but nothing here needs it.

Type of Change

Testing

Route tests (execute/route.test.ts). Each new test failed against the code before this change, and the cancel and disconnect guards were checked by reverting each one and watching its test fail:
- a run that settles within the wait answers as before
- a detachable run is admitted with the async bucket and async timeout
- the NDJSON detached frame arrives at the deadline and the run keeps going (ticket and claim held until it settles)
- the JSON path answers 202 with the receipt
- a client disconnect does not cancel the run
- an in-process runs cancel stops it
- the async, stream and anonymous combinations are refused
- executionTimeoutSeconds is accepted only on runs that can outlive the response
- a manual draft run is dispatched as sync-result-stream + detachable with no request signal
CLI tests (workflow-run-follow.test.ts): the request body, unchanged output when the run settles, exit 4 plus the run record when it detaches, a run that settled after detaching, a 202 on the JSON transport, rounding a fractional wait up and 0, and refused combinations. The workflows runs wait tests still pass after its loop was extracted into waitForRun.
bun run lint, type-check (apps/sim, sim-cli), check:audits, check:api-validation:strict, check:test-types, the sim-cli suite, and the apps/sim suites under app/api/v2/workflows, lib/workflows, lib/execution, lib/mothership/agent-cli, app/api/workflows.
Regenerated: generate:cli-api, generate:openapi, generate:cli-docs.

Reviewers: please focus on streamPendingRun (who closes the stream, and that only a non-detachable run is cancelled when the reader goes away) and on the admission and timeout class in executeWorkflowService.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…er a wait timeout A synchronous run, including a manual draft run (--manual, --trigger, --from-block, --stop-after), could only be waited on until it settled or hit the sync execution limit. Manual runs refuse async, so a caller had no way to get the run id back from a long or stuck draft run and decide to keep waiting, watch it, or cancel it. The v2 execute body takes waitTimeoutSeconds. Past it the response answers 202 with the run receipt (NDJSON: a `detached` frame) and the run keeps going in this process exactly as it started: same captured draft state, entry point and pause persistence. Such a run is admitted like an async one (async rate bucket, reservation and timeout), takes no request abort signal so a dropped connection never cancels it, and registers an in-process aborter so runs cancel reaches it without Redis. Without the field nothing changes. The CLI exposes it as `workflows run --wait-timeout <seconds>`. A run that outlasts it is read back once and reported the way `workflows runs wait` reports an elapsed bound: the run record on stdout and exit 4 while it is still moving, so `workflows runs wait <runId>` continues it.

vercel · 2026-10-30T03:20:40Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 30, 2026 4:07am UTC

waleedlatif1 · 2026-10-30T03:20:40Z

@greptile

waleedlatif1 · 2026-10-30T03:20:42Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-30T03:20:47Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 15 files

_{Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic}

greptile-apps · 2026-10-30T03:25:17Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new blocking issues were found.

Summary

Adds a bounded wait for synchronous workflow runs. Runs that outlast the wait continue in the background.

Synchronous workflow runs can continue after the caller stops waiting.
The workflow run command reports runs that outlast its wait.

Both earlier, unnumbered findings are fixed: receipts wait for the log row, and RunWait is private. No new actionable issues were found.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Synchronous run with wait timeout] --> B[Start one execution under async limits]
  B --> C{Run settles within wait?}
  C -->|Yes| D[Return ordinary result]
  C -->|No| E[Wait until run log exists]
  E --> F[Return receipt and keep execution running]
  F --> G[CLI reads run once]
  G --> H[Print run and report its current outcome]

_{Reviews (3) · Last reviewed commit: "feat(mothership): declare that workflows..." · Reviewed by Greptile}

…an be read A detach could fire before the run's log row existed, so the runs resource answered 404 for the receipt just handed out. The response now waits for the logging session to start before detaching, and the wait timeout docs say how each transport carries the receipt.

waleedlatif1 · 2026-10-30T03:32:52Z

@greptile

waleedlatif1 · 2026-10-30T03:32:53Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-30T03:32:59Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 17 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

…to the worker Sim now sends the workflow-run-wait-timeout client capability on chat turns, so the worker waits on a synchronous workflows run in slices only on a Sim whose CLI takes --wait-timeout there; an older Sim keeps its held run. The protocol copy is synced from the mothership contract that defines the token.

waleedlatif1 · 2026-10-30T04:07:51Z

@greptile

waleedlatif1 · 2026-10-30T04:07:52Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-30T04:07:59Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 20 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

…l controller (#9475) * test(mothership): drive a draft run past a wait slice through the real controller A live case for the worker's sliced synchronous `workflows run` (#9464, #9469): a draft with Wait blocks totalling 500 s runs through runCopilotLifecycle, the sim_cli handler, the embedded CLI and the v2 execute and runs routes against the worker's scripted draft-run probe. With workflow-run-wait-timeout declared it checks the detached first slice, the `workflows runs wait` slices, the output read-back, that no call holds the run and no slice row is failed, and that the model receives the run's output. Without the capability it checks one held call as before, and a model's own --wait-timeout 5 reaches the model as exit 4 with the running receipt. The controller fixture now pins direct Sim transport, since self-hosted defaults to checkpoint delivery and its run control went unanswered, and the fixture plan's execution timeout is per case so a run can outlast a slice. * test(mothership): keep the draft-run case inside the repo's audits Uses sleep from @sim/utils/helpers, guards the last wait slice instead of a non-null assertion, gives the probe's startup its own ten-second deadline like the neighbouring controller case, and tells knip the probe path lives in the worker checkout.

… worker's wait A run_workflow* call held the turn until its run settled, up to the execution limit, while a synchronous workflows run can already detach after --wait-timeout (#9464). Sim now declares workflow-tool-wait-timeout and honors the waitTimeoutMs a workflow tool frame names (synced from the mothership contract): - Server-run call: the handler gives the run its execution id up front and, once the wait passes, answers {status: running, executionId, workflowId}. The run goes on in-process; the tool execution keeps holding it, so its lease and a Stop through the call's abort signal still reach it. - Browser-run call: once the wait passes and a browser holds the call's execution claim, the call is settled background, the detach a browser sends on pagehide. The browser's run goes on, still bound to the call, so a Stop cancels it. - A background workflow completion now carries status: running beside its ids, the shape the worker waits on. A frame without waitTimeoutMs, or one with a malformed value, holds the call as before.

…ast a call A synchronous workflows run the worker waits on in slices detaches after its first --wait-timeout (#9464, mothership#872), and from then on no call's abort signal reaches it. Stop delivery cancelled only browser workflow runs bound to open calls, so a Stop mid-slice ended the chat while the draft run went on to completion. Driven through the real controller and the Stop route (live case in the next commit), the run was still running a minute after the Stop. Stop delivery now also cancels the runs this chat run's calls started and left running: a workflows run the CLI answered still running (exit 4 and its run record), and a workflow tool answered still running. A run the chat only waited on (workflows runs wait) is not one, since something else started it. The cancel is the same in-process abort and durable cancel flag a browser run gets; a detached run registers its in-process aborter. Autopsy: the slice tests on both sides covered waits that settle, lapse or lose an answer, and none stopped the chat mid-slice; the Stop tests seeded only open browser-run calls. stop-delivery.test.ts and the live declared-stop case now cover it.