Python Durable Functions on DTS: oversized orchestrator completion fails the orchestration invisibly, and large-payload externalization can't be enabled #269

Activity

1. Bernd Verst (berndverst)

   added sub-issues on Oct 12, 2026

   • [Azure Functions/DTS] Surface oversized orchestrator completion failures in the app or host logs #288
   • [Azure Functions/DTS] Document orchestration completion-request size limits #290
   • [Azure Functions/DTS] Enable configurable large-payload externalization #289
   • [Azure Functions] Reduce non-Latin payload inflation in df_dumps #291
2. berndverst commented on Oct 12, 2026

   

   Bernd Verst (berndverst)

   on Oct 12, 2026

   Member

   More actions

   Addressing ask 3: large-payload support is now available in the released azure-functions-durable==2.0.0rc2, added by #270. You no longer need to implement the worker/client wiring or activity wrappers yourself.

   Upgrade and install the Blob Storage dependencies

   Your reported 2.0.0rc1 / durabletask==1.10.1 combination does not include this integration. The following pins a compatible released pair; include the same dependencies in your deployment's requirements.txt:

   azure-functions-durable==2.0.0rc2
   durabletask[azure-blob-payloads]==1.11.0
   aiohttp
   

   To install locally (the command is identical in Bash and PowerShell):

   python -m pip install --upgrade "azure-functions-durable==2.0.0rc2" "durabletask[azure-blob-payloads]==1.11.0" aiohttp

   Python 3.13+ is required; your Python 3.14 environment meets that requirement. The aiohttp dependency supports asynchronous Blob Storage operations, including orchestration execution.

   Configure the root DFApp at startup

   In your existing Function app module, configure your existing root DFApp before invocations:

   import os
   
   import azure.durable_functions as df
   from durabletask.extensions.azure_blob_payloads import (
       BlobPayloadStore,
       BlobPayloadStoreOptions,
   )
   
   app = df.DFApp()
   app.configure_large_payloads(
       payload_store=BlobPayloadStore(BlobPayloadStoreOptions(
           connection_string=os.environ["PAYLOAD_STORAGE_CONNECTION_STRING"],
           container_name="durable-payloads",
           threshold_bytes=256 * 1024,
       ))
   )

   Set PAYLOAD_STORAGE_CONNECTION_STRING in your Function App settings, or under Values in local.settings.json for local development. This is an application-defined setting read by the example, not a built-in Functions setting. Use storage credentials with the permissions needed to create/use the payload container, upload blobs, and download them. Keep credentials out of source control.

   This is explicit SDK configuration: selecting storageProvider.type: "azureManaged" does not enable it automatically. You may supply another PayloadStore instead of BlobPayloadStore. Automatically resolving an existing Functions storage connection is a separate follow-up in #271.

   What happens automatically

   The configured store is shared by Functions workers, synchronous/asynchronous durable clients, activities, entities, and registered blueprints. Serialized payloads above the threshold are uploaded to storage and replaced with references; the SDK resolves those references before application code consumes the payloads. This covers orchestration/activity inputs and outputs, custom status, external events, entity payloads, sub-orchestrations, and continue-as-new.

   For your fan-in case, the aggregated list passed to the next activity can be externalized even when each individual activity result is below the threshold. You do not need to manually upload the list or change the activity's input type.

   The JSON-quoting question in your proposal is handled by the provider: it emits Functions-compatible JSON-string references and recognizes/resolves them on input and activity completion. Do not add your own token serialization around activity results.

   A few operational considerations:

   • Configure once per Python worker process, with all scaled-out workers using the same backing storage and maintaining access across deployments. Keep the store open for the process lifetime.
   • The default maximum size of a single stored serialized payload is 10 MiB. If needed, add max_stored_payload_bytes=32 * 1024 * 1024 to BlobPayloadStoreOptions (or choose an appropriate cap). This is separate from the aggregate DTS completion-request limit; the reported 11.05 MB request is not necessarily one 11.05 MB payload. Externalization reduces eligible payloads but does not remove DTS request limits.
   • Retain blobs while any retained orchestration history or entity state references them. Purging an orchestration does not delete its payload blobs. Storage failures can fail durable invocations, so storage availability matters.
   • Configured Python clients hydrate payloads automatically. Host management HTTP endpoints and clients that lack equivalent store configuration may expose reference strings instead. A separately constructed DurableTaskSchedulerClient, such as the one used for your investigation, does not inherit the Functions app's process-wide configuration.

   Warning

   Review the rc2 release upgrade guidance before upgrading active instances: its core SDK dependency includes a replay-breaking fix for certain nested task races. Pin or lock the core SDK as well as the provider to control when behavior changes are adopted.

   The implementation's real-host end-to-end coverage used Azure Storage/Azurite, not DTS, so validate your DTS-backed workload in staging before production rollout. This addresses the externalization configuration in ask 3; it does not itself resolve the failure-observability or Unicode-escaping asks.

   Full configuration and retention guidance: Python Functions large-payload documentation.
3. berndverst commented on Oct 12, 2026

   

   Bernd Verst (berndverst)

   on Oct 12, 2026

   Member

   More actions

   (2) Re #288 - gRPC has a default request limit of 4MB (including all system request metadata / headers). The limit applies globally on all non-streaming calls.
   Maybe this documentation belongs in https://github.com/Azure/azure-functions-durable-extension primarily, and of course on the DTS Azure Docs. sophiatev can you take a look :)

   (1) #290 requires some coordination between the Python extension and https://github.com/Azure/azure-functions-durable-extension
   (4) #291 requires investigation beyond the Python SDK. In any case, changing this for inflight orchestrations will likely result in data loss.
4. andystaples commented on Oct 13, 2026

   

   andystaples

   on Oct 13, 2026

   Contributor

   More actions

   Hi Nacho García (@nachogarcia)
   Thanks for the reports! We are excited to hear back from anyone currently using v2 and its new features.
   As Bernd already said above, we're working on several follow-ups from this report, including the ensure_ascii change in df_dumps, but that one does require going through the functions python library which has a slower review and release schedule.
   Wanted to point out that we actually considered an alternate solution to this kind of problem - see #249 where I proposed allowing custom serializers (including the durabletask default one) in Functions v2. Unfortunately we couldn't make that one the default due to it breaking orchestration histories from v1, and the risk of customers with unpinned deps in requirements.txt silently breaking - but, I think it remains the more elegant solution to your problem and would give you even more flexibility on the shape and size of your specific payloads.
   If you'd be interested in that feature, a quick blurb there about your use case and the benefits you'd see would be helpful, so we can discuss it with the team.
5. nachogarcia commented on Oct 19, 2026

   

   Nacho García (nachogarcia)

   on Oct 19, 2026

   Author

   More actions

   Thanks Bernd Verst (@berndverst) and andystaples, and thanks for shipping the large-payload support in rc2 so quickly.

   Here's how we use Python Durable Functions, where we are now, and what we still need.

   How we use it

   • Azure Functions on Linux, Python 3.14, Durable Task Scheduler backend.
   • azure-functions==2.3.0, azure-functions-durable==2.0.0rc2, durabletask==1.12.0, extension bundle Microsoft.Azure.Functions.ExtensionBundle.Preview [4.46.0, 5.0.0).
   • Our main workload fans out to tens or hundreds of activities per orchestration with when_all, plus deadline races (when_any + create_timer) and sub-orchestrations.
   • Every orchestration is a saga with compensation in try/except. That's why failures the orchestrator never sees hurt us most: compensation never runs, and the work stays "in progress" for good.
   • Payloads are often non-Latin text, so ensure_ascii inflation shows up for us in practice.

   Where we are now

   • Large payloads: we use the host-side setting in host.json, not the SDK's configure_large_payloads. The docs describe it as .NET-isolated only, but it works for our Python app:

     "storageProvider": {
       "type": "azureManaged",
       "payloadStorageEnabled": true,
       "payloadStorageThresholdBytes": 131072
     }

     We also raised maxGrpcMessageSizeInBytes, so the Python client can start orchestrations with inputs over 4 MiB.

   • Two failure modes we've hit since then, both now worked around in our own code:

     1. An over-limit completion retries forever. A step that schedules many activities, each under the offload threshold, can still add up to more than the 4 MB gRPC limit. Nothing gets offloaded, because the threshold applies per payload. With payload storage on, the instance no longer goes to Failed. Instead CompleteTaskOrchestrationWorkItemAsync throws RpcResourceExhaustedException, the work item is abandoned and redelivered, and the replay builds the same request again. It retried hundreds of times across many hosts, and compensation never ran.
     2. A missing payload blob affects the whole host. Once an offloaded blob was gone while its instance was still running, every redelivery threw PayloadStorageException: The blob … was not found. The host it landed on then lost most of its other in-flight activity work items, so unrelated orchestrations failed too.

     Our workarounds: smaller activity inputs, an orchestrator-side check that refuses a step over the limit before dispatch (so it goes into compensation instead), and keeping offloaded payloads for longer.

   What we'd need

   1. A completion request over the limit should fail the instance, not retry forever. Ideally it would fail in a way the orchestrator can catch ([Azure Functions/DTS] Document orchestration completion-request size limits #290). At minimum it should be non-retriable. Replaying the same history rebuilds the same request, so retrying can never succeed.
   2. Offload based on the whole request, not just per payload. For example, offload the largest payloads until the request fits. As far as I can tell, configure_large_payloads also uses a per-payload threshold, so it would hit the same fan-out case.
   3. Keep one bad instance from affecting its host. An instance that can't load its payloads should fail on its own, without taking the other in-flight work items on that host with it.
   4. A way to stop instances that are stuck like this. Terminating them doesn't take effect, because the instance never finishes an episode, and purge refuses Running. A force-terminate or force-purge that doesn't need a successful replay would help a lot.
   5. Docs for the 4 MB request limit, and the fact that scheduled-activity inputs count toward it ([Azure Functions/DTS] Surface oversized orchestrator completion failures in the app or host logs #288).
   6. ensure_ascii ([Azure Functions] Reduce non-Latin payload inflation in df_dumps #291): understood that it can't change for in-flight orchestrations. An opt-in setting that only applies to new instances would work for us. I'll add our use case to Allow Azure Functions Durable 2.x apps to opt in to JsonDataConverter #249 too.

   A question about large payloads: should Python apps on DTS use the host-side payloadStorageEnabled or the SDK's configure_large_payloads? Is the host-side path supported for Python going forward, and is it safe to enable both at once? We're happy to test whichever you recommend against DTS, since your end-to-end coverage used Azurite.

   We can share instance IDs and host logs privately. There's also an open Azure support case: TrackingID#2609230050003654.
6. andystaples commented on Oct 22, 2026

   

   andystaples

   on Oct 22, 2026

   Last edited by andystaples

   Contributor

   More actions

   Hi Nacho García (@nachogarcia) , thanks again for the context! I’d recommend using the Python-side large-payload feature in 2.x. It gives us more flexibility to improve this behavior without waiting for extension-bundle updates, and potentially reduces memory usage on Functions plans as the Host side never loads the offloaded payloads. We haven’t yet established interoperability guarantees for enabling both offloading mechanisms together, but I would not expect it to work natively.

   On your specific asks:

   1. The full completion-failure solution requires coordination across the backend, extension, SDKs, and protobuf contract. We’re prototyping this now; see DTS oversized orchestration completion failures bypass Functions host error reporting Azure/azure-functions-durable-extension#3569 and its related PRs.

   2. I opened Proposal: Request-budget-aware large-payload externalization in the Python SDK #311 for request-budget-aware Python offloading: if a response remains oversized after normal per-payload offloading, continue externalizing eligible payloads that are individually below the configured threshold. This would address the many-medium-sized-inputs case without removing the underlying request limit. Separately - I'm curious about the actual size of the individual payloads in your fan-out case, and whether setting an extremely low externalization threshold that would capture all of them might work for your use case.

   3. With Python-managed offloading, payload downloads happen within individual Function invocations, so we expect failures to stay isolated to the affected work rather than disrupt the host’s shared receive stream. A missing activity payload should become an activity failure; a missing orchestration-history payload can fail that orchestration before its user code runs. I opened [Azure Functions] Add DTS coverage for isolation of SDK-managed missing-payload failures #312 to verify this end-to-end on DTS, and [AzureManaged] Isolate host-side payload hydration failures from unrelated in-flight work Azure/azure-functions-durable-extension#3582 separately tracks the reported host-side provider behavior.

   4. DTS already offers Force terminate through its dashboard, as discussed in the recent comment on Force-terminate operation for stuck orchestrations Azure/azure-functions-durable-extension#2081. Have you tried that specific action for the affected instance, rather than ordinary termination? Programmatic Functions support and/or equivalent adds for the other backends is still open, tracked in that older issue.

   5. The documentation now includes the aggregate completion-request limit, but still needs clarification around scheduled-activity inputs and why many individually small payloads can collectively exceed it.

   6. Opt-in Unicode serialization was added in [Azure Functions] Add opt-in Unicode JSON serialization #292 and is awaiting the next 2.x Functions SDK release. The API is app.configure_serialization(ensure_ascii=False), using azure-functions>=2.4.0b1. See the related notes about replay compatibility, review any byte-dependent hashing, IDs, signatures, or replay decisions, configure all workers consistently at startup, and validate the rollout in staging.