Proposal: Request-budget-aware large-payload externalization in the Python SDK #311

Summary

Extend Python SDK-managed large-payload externalization so an orchestration response can offload supported payloads that are individually below the configured per-payload threshold when their combined serialized response exceeds a size budget.

This is a draft feature proposal; the public API, budget calculation, and defaults need design review. It is scoped as a Python SDK enhancement using the existing payload-store/reference protocol, not as a change to the DTS transport limit.

Parent: #269, follow-up comment, ask 2.

Problem

Current externalization considers each payload independently. A fan-out episode can schedule tens or hundreds of activities whose inputs are each below threshold_bytes, while the orchestration response containing those inputs exceeds the aggregate completion-request limit. No individual payload qualifies for offloading, so enabling large-payload storage alone does not address this case.

The reporter currently uses host-side payloadStorageEnabled and encountered repeated oversized-completion retries. They also explicitly asked whether Python's configure_large_payloads() would have the same per-payload limitation. It currently does.

Lowering the per-payload threshold can mitigate the problem, but requires applications to anticipate fan-out size and externalizes more data even when the overall response would fit.

Proposed behavior

With SDK-managed payload storage and request-budget-aware externalization enabled:

1. Preserve the existing per-payload externalization policy.
2. Measure the resulting serialized orchestration response against a defined byte budget.
3. If it remains too large, select additional supported payload fields below the normal threshold, starting with the largest useful candidates, and replace them with existing payload-store references until the response fits or no further useful reduction is possible.
4. Recalculate the serialized size with actual references and protocol overhead rather than treating the sum of application payload lengths as the response size.
5. If the response cannot fit, report an explicit, actionable failure through the supported execution path. Do not silently drop actions, truncate application data, or claim that arbitrary fan-out can always be made to fit.

The exact configuration surface is intentionally unspecified. An explicit opt-in should preserve existing threshold-only behavior for applications that do not select this policy. Reuse the existing PayloadStore, Functions-compatible reference encoding, and hydration paths; no new protobuf fields or backend feature should be inherently necessary for the externalization mechanism.

Scope and transport boundaries

• Implement the reusable policy in the core durabletask payload machinery and make it usable by native workers and the azure-functions-durable orchestration response path, including synchronous/asynchronous execution where applicable.
• Verify that durabletask.azuremanaged continues to work with the shared core behavior.
• Python Functions applications must configure SDK-managed storage through DFApp.configure_large_payloads(payload_store=...) to benefit. Host-side payloadStorageEnabled alone does not enable Python SDK externalization; changing the host/provider implementation is separate work.
• A Python OrchestratorResponse is not the final DTS completion request. The Functions host constructs the downstream request, and native transports also have their own envelopes. Define the measured boundary and account for downstream expansion/overhead when selecting a budget. Do not present a Python response-size check as an exact guarantee of final DTS request size without establishing that relationship.
• This does not raise or remove the server's aggregate request limit, introduce new acknowledged-completion semantics, or make post-execution transport failures catchable inside orchestrator compensation code.

Acceptance criteria

• A response containing many individually subthreshold activity inputs can trigger additional offloading when it exceeds the selected aggregate budget.
• Responses already within budget retain normal per-payload behavior; applications not opting in retain existing behavior.
• Size checks use serialized bytes and cover Unicode, protobuf overhead, and the actual replacement references, not character counts or only summed payload sizes.
• Only supported durable payload fields are externalized. Existing references remain recognizable and are not uploaded again as application payloads; replay and retry behavior remain compatible.
• Storage-size limits and existing preflight guarantees remain respected. Storage failures and cancellation are surfaced without success-shaped fallbacks or loss of durable actions.
• An irreducibly oversized response, including one dominated by action/reference/metadata overhead, has a documented explicit failure outcome rather than an assertion that offloading always solves the limit.
• Regression coverage exercises the shared policy, the Python Functions response path, native/AzureManaged compatibility, threshold and budget boundaries, and round-trip hydration of newly offloaded subthreshold payloads.
• Documentation explains the aggregate-versus-per-payload distinction, the measured transport boundary, SDK-versus-host configuration, and limitations. Record user-visible changes in each affected package's changelog.

Design questions

• What is the appropriate public configuration surface and safe budget policy for native workers versus Functions-hosted responses?
• How should downstream host request expansion be accounted for without baking an unverified DTS-specific limit into a generic payload store?
• What supported failure path should be used when further externalization cannot bring the response within budget?

Related work

• #270 added Python Functions SDK-managed payload configuration; this proposal builds on it.
• #292 adds opt-in Unicode JSON serialization. Reduced escaping helps but does not solve aggregate fan-out limits.
• Azure/azure-functions-durable-extension#3569 tracks completion-failure observability. This proposal is an independent payload-size mitigation, not a replacement for that work.