Preflight all externalized payload sizes before starting uploads #298

The synchronous and asynchronous payload helpers validate and upload one protobuf field at a time. A valid earlier field can therefore be uploaded and replaced by a token before a later field is rejected for a size that was already knowable before any storage I/O. If that response is never submitted, the earlier upload has no backend-recorded reference.

Current source evidence

Inspected current main at cb75760f250099d5b3b49e18ed03e10f8fe46d0d.

• The synchronous recursive walker visits fields and immediately calls _try_externalize_field, which checks only the current payload's maximum before uploading and mutating that field.
• The asynchronous walker has the same sequencing with await store.upload_async(...).
• The worker completion path calls externalization before submitting CompleteOrchestratorTask; there is no whole-message preflight or compensation for successful earlier uploads in that path.

This is not a claim that the business request partially succeeds. The problem is the avoidable storage side effect before a deterministically invalid message fails. A cleanup process based on backend-recorded references cannot discover a reference that was never submitted or recorded.

Requested regression

Add this case to the existing tests/durabletask/test_large_payload.py, using its real protobuf types, helper and FakePayloadStore:

store = FakePayloadStore(threshold_bytes=8, max_stored_payload_bytes=32)
response = pb.OrchestratorResponse(
    instanceId="size-preflight",
    actions=[
        pb.OrchestratorAction(
            id=1,
            scheduleTask=pb.ScheduleTaskAction(input=sv("a" * 20)),
        ),
        pb.OrchestratorAction(
            id=2,
            scheduleTask=pb.ScheduleTaskAction(input=sv("b" * 33)),
        ),
    ],
)
before = response.SerializeToString()

with pytest.raises(ValueError, match="exceeds the maximum"):
    externalize_payloads(response, store, instance_id="size-preflight")

assert len(store._blobs) == 0
assert response.SerializeToString() == before

The current source uploads the first input before inspecting the second input's maximum. Repeat the regression with externalize_payloads_async, plus an entity result with a valid earlier field and oversized later field.

Acceptance Criteria

• Collect and snapshot all relevant non-token payload fields before performing storage I/O, and validate their sizes according to the existing Python eligibility rules.
• A deterministic maximum-size violation causes zero uploads and leaves all original protobuf payload values unchanged.
• Apply the guarantee to both synchronous and asynchronous helpers, including nested/repeated messages, carryover events and entity payloads.
• Preserve known-token idempotency, empty/absent fields, the existing strict-greater threshold behavior, and async cancellation/error propagation.
• Preserve existing exception behavior for the helper; do not silently convert invalid input into a successful result.
• Add valid-first/oversized-later, UTF-8 boundary, already-tokenized and all-valid regression cases.
• Document the intentional sequencing change: deterministic invalid size is discovered before hypothetical earlier upload failures because no upload is attempted.
• Avoid introducing new abstract members or mandatory dependencies for custom payload stores.

Non-goals

This does not introduce a cross-blob transaction, recover existing orphan blobs, or guarantee zero orphans after actual I/O/cancellation/submission failures. Unconditionally deleting uploads after an uncertain backend response is unsafe because the backend may already have accepted the reference.

Carryover fields are already reached by Python's recursive protobuf traversal; this issue does not request a missing-field fix.

Related

• JavaScript zero-upload preflight fix.
• .NET tracking issue 820.
• Python issue 297 tracks permanent worker failure completion and entity-lock cleanup. Those acceptance criteria do not cover validating the entire message before any upload; this issue tracks the separate storage-side-effect guarantee.