[Azure Functions] Add DTS coverage for isolation of SDK-managed missing-payload failures #312

Summary

Add Functions-host/DTS integration coverage showing that missing blobs in Python SDK-managed payload storage fail only the affected invocation/work, without disrupting unrelated in-flight work on the same host.

This is a coverage and investigation issue, not a claim that SDK-only externalization currently reproduces the host-wide failure reported in #269, follow-up comment, ask 3.

Configuration boundary

Exercise the supported Python SDK-managed path:

• Configure the root app with DFApp.configure_large_payloads(payload_store=...).
• Explicitly disable host/provider-side payloadStorageEnabled, so the test does not accidentally exercise host-side hydration instead.
• Use a Functions host backed by DTS, with multiple unrelated instances and concurrent activity deliveries on that host.

The reporter currently uses host-side externalization. That path can hydrate payloads inside the AzureManaged provider's shared receive stream and has a distinct suspected blast-radius defect. It must not be treated as proof of the same defect in the Python SDK path.

Current evidence and expected behavior

In the Python Functions integration, orchestration/entity request hydration occurs within the individual Function invocation, before user execution. Activity hydration occurs inside the activity wrapper. DurableFunctionsWorker does not own a DTS work-item stream.

Ordinary invocation exceptions are already mapped by the extension to local failures:

• An orchestration invocation error produces a failed orchestration result.
• An activity invocation error produces a failed activity/task result, subject to the application's activity retry policy and observable by its orchestrator.
• An entity invocation error follows the existing failed-operation behavior; it must not be assumed to terminate the entity or unrelated callers.

Existing Python unit tests check error propagation, including missing activity references and entity-envelope hydration failures. They do not establish multi-instance DTS host isolation end-to-end. A Python storage exception also occurs before orchestrator user code when request hydration fails, so this issue must not promise that the orchestrator can catch its own hydration failure and run compensation.

Source references at the reviewed commits:

• Python orchestration/entity invocation-bound hydration
• Python activity invocation-bound payload handling
• Extension ordinary orchestration invocation failure handling
• Extension ordinary activity invocation failure handling

Proposed regression scenarios

Externalize data, then remove a specifically referenced blob while retaining unrelated valid payloads and keeping unrelated instances active. Cover missing orchestration input or replay-required history, missing activity input, and missing entity-operation payloads where supported. Arrange timing so the blob is actually needed by the target invocation; historical scheduled activity inputs that replay intentionally skips are not a valid proxy.

For each scenario, observe the affected work's outcome and concurrently verify that healthy activity deliveries and orchestrations continue to completion on the same host.

Acceptance criteria

• Exercise real missing-reference behavior through the Functions host and DTS transport, not only an exception from a directly invoked Python wrapper.
• Explicitly record that host-side payload storage is disabled, along with the Python package, extension, and AzureManaged provider versions used.
• Verify the appropriate affected orchestration/task/entity-operation failure is visible, and that a permanent missing blob does not create an unbounded provider-redelivery loop.
• Verify unrelated in-flight work remains healthy; capture correlated host diagnostics and delivery/connection behavior needed to investigate any collateral failure.
• Distinguish ordinary missing-blob exceptions from storage timeouts, worker exits, host shutdown, and other platform-abort paths. Preserve existing documented retry/cancellation behavior rather than silently introducing a new policy.
• If coverage exposes a Python integration bug, fix it minimally here. If it exposes extension/provider/runtime behavior, record a reproducible upstream issue with logs and the exact configuration rather than assuming the host-side interceptor root cause applies.
• State which environments were exercised. Azurite-only or emulator coverage must not be represented as hosted DTS validation.
• Document behavior changes if any are needed; test-only coverage does not require a user-facing changelog entry.

Related

• #269: customer report; its follow-up ask 3 currently describes host-side offloading.
• #270: Python Functions SDK-managed payload configuration.
• #297: closed native-worker completion-side payload-size/entity-lock cleanup work; not proof of SDK-only Functions missing-download isolation.
• The original host/provider-side shared-stream hydration failure is tracked separately in Azure/azure-functions-durable-extension#3582, not as the implementation target of this issue.