Orphaned orchestration dispatches can indefinitely stall instances and exhaust a connected worker's capacity #304

Summary

If an orchestration dispatch exits without sending either a completion response or an abandonment request, the Python worker can remain alive and connected while the dispatch is no longer executing. The backend can continue retaining ownership of that unfinished delivery, indefinitely stalling the affected instance and consuming server-side dispatch capacity.

Enough orphaned deliveries can prevent the connection from receiving additional orchestration work even though the Python event loop and queue consumers are still running and local concurrency slots have been released.

This is a broader failure mode than the direct asyncio.CancelledError teardown case fixed by merged #303. Other execution exits, including grouped cancellation or another BaseException subclass, can still bypass both outcome paths. Related: #301 and #300.

Example trigger

Register and start an orchestrator whose dispatch-boundary finalizer raises an exception outside the existing Exception handlers:

from durabletask import task


class TeardownExit(BaseException):
    pass


def orchestrator(ctx: task.OrchestrationContext, _):
    try:
        yield ctx.call_activity("work")
    finally:
        raise TeardownExit("dispatch teardown exited unexpectedly")

This illustrates the missing-outcome path without requiring an additional async framework. Equivalent failures can arise from a BaseExceptionGroup containing cancellation, or another runtime's cancellation exception. The problem is not limited to this particular exception class or to generator teardown: the relevant condition is an accepted dispatch ending without a scheduler-visible outcome.

Observed Python behavior

• The exception escapes dispatch execution before the constructed response is returned or delivered.
• Neither CompleteOrchestratorTask nor AbandonTaskOrchestratorWorkItem is called.
• The per-work-item task still marks its local queue entry done and releases its semaphore in finally.
• The queue consumer and work-item stream can remain healthy. Subsequent locally queued work can execute, so this is not inherently a Python event-loop or thread-pool deadlock.
• This missing-outcome path does not itself initiate a reconnect.

Relevant public SDK paths at the reviewed follow-up commit: _execute_orchestrator, generator teardown, and work-item processing.

Consequences with Durable Task Scheduler

• Indefinitely stalled instance: A healthy, open worker connection can retain and renew ownership of the unacknowledged delivery independently of Python execution liveness. Waiting for a nominal work-item timeout is therefore not sufficient to guarantee redispatch.
• No progress for that instance: Additional events or task results can accumulate, but do not let another dispatch bypass the outstanding ownership for the same orchestration.
• No committed result from the lost response: New actions, custom status, and terminal outcomes in that response are not committed. Previously committed history survives. An instance can remain Pending if its initial dispatch never committed, or retain its preceding state instead of becoming Failed.
• Connection-level orchestration starvation: Orphaned deliveries retain backend dispatch capacity despite the released Python semaphore. One orphan need not exhaust the connection, but repeated occurrences can. The exact threshold is not simply the configured Python concurrency limit.
• Misleading worker health: The worker can remain connected and continue handling health traffic while orchestration delivery is starved. Activity and entity delivery have separate capacity and need not stop merely because orchestration delivery is exhausted.
• Repeated failures can re-orphan work: Recovering the connection alone does not resolve a deterministic execution failure encountered again on replay.

These backend consequences were established by source inspection; an end-to-end live-service reproduction has not been run. The Python-side continuation and missing abandonment were reproduced locally.

Recovery characteristics

Closing the affected work-item stream or restarting its worker allows the backend to release ownership and redispatch pending work once the disconnect is observed. If normal disconnect cleanup cannot run, recovery depends on backend ownership expiry after renewal stops. No bounded recovery time is established here for an otherwise healthy connection that continues retaining the orphan.

Merely waiting, adding another worker, sending an external event, or freeing local Python concurrency does not guarantee recovery of an already-owned orphaned instance. A graceful drain that leaves the old connection open is also not equivalent to closing it.

Expected behavior

An unexpected dispatch exit should not silently leave an instance owned indefinitely or consume enough delivery capacity to starve an otherwise healthy worker. There should be a defined, observable failure/recovery outcome for accepted work that is no longer executing.

This issue intentionally describes the failure conditions and consequences without prescribing an exception policy, ownership protocol, or implementation fix.