Repository navigation
Conversation
- Download asks for uncompressed content ('Accept-Encoding: identity').
'Content-Length' and 'Range' of compressed response are related to
compressed content, but 'iter_content' returns decoded content, so
the size check failed and resume requested wrong range.
- When server compresses content anyway, size isn't checked against
decoded content and failed download restarts instead of resuming.
Incomplete compressed content is detected by received raw bytes,
'urllib3' 1.x does not fail on it.
- Response without 'Content-Length' (chunked) doesn't fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The question is if we do actually use gzip anywhere, I'd say we can handle this when it becomes an issue. |
And possibly relevant to note is: 🤔
But it's been untested against a real S3. |
There was a problem hiding this comment.
🟡 Changes recommended
Resumed downloads can silently report truncated files as successful, and encoded-response retries can retain a stale Range header that prevents a proper restart.
2 open issues
What changed in this PR
This PR updates the shared download implementation used by download_file and download_file_to_stream to handle compressed responses and responses without a known size.
Changes:
- Requests uncompressed content and restarts failed compressed downloads rather than resuming them.
- Handles missing
Content-Lengthand checks compressed response completeness using raw bytes received. - Adds local HTTP-server tests for compression, interrupted compressed transfers, and chunked responses.
| File | Description |
|---|---|
tests/test_download_content_encoding.py |
Adds four regression tests for compression and chunked downloads. |
ayon_api/server_api.py |
Updates download headers, retry handling, and response-size validation. |
Review effort: Balanced
| if is_encoded and content_length: | ||
| # 'urllib3' 1.x does not fail when compressed | ||
| # content is incomplete, compare received bytes | ||
| received = getattr(response.raw, "tell", None) | ||
| if ( | ||
| received is not None | ||
| and received() < int(content_length) | ||
| ): | ||
| raise requests.exceptions.ConnectionError( | ||
| f"Downloaded {received()} out of" | ||
| f" {content_length} compressed bytes" | ||
| f" from '{url}'." | ||
| ) |
There was a problem hiding this comment.
Fixed in c4a8ddb. Received raw bytes are now compared with Content-Length of every response, not only compressed ones. Covered by test_download_resumed_response_is_complete (chunked response cut, then a resumed 206 that ends early), which failed on urllib3 1.26 before the fix.
| stream.seek(0) | ||
| stream.truncate() | ||
| progress.reset_transferred() | ||
| offset = 0 |
There was a problem hiding this comment.
Fixed in c4a8ddb. Range is removed at the start of every attempt and set again only when resuming. Covered by test_download_restart_does_not_keep_range (resumed response comes back compressed and is cut, the next request has no Range).
- Received raw bytes are compared with 'Content-Length' of every response, not only compressed. A resumed download of chunked response has unknown total size, and 'urllib3' 1.x accepted a resumed response that ended early. - 'Range' header is removed at start of every attempt, so a restart after compressed response does not keep 'Range' of previous attempt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>


Changelog Description
download_fileanddownload_file_to_streamwork when the server (or the host it redirects to) sends compressed content (Content-Encoding: gzip) or noContent-Length(chunked transfer).Additional info
Two problems in
_download_file_to_stream:requestsasks forgzipby default.Content-Lengthis then the compressed size, butiter_contentreturns decoded bytes. The size check failed (Downloaded 16384 out of 420 bytes), and the retry sent aRangebased on the decoded size, which the server answered with 416, or with a wrong part of the compressed content.Content-Length. A chunked response failed withKeyError: 'content-length'.Changes:
Accept-Encoding: identity, so the server sends the file as it is, and size andRangematch the written bytes.Range. TheRangeheader is removed at the start of every attempt, so a restart doesn't keep the previous attempt'sRange.Content-Lengthleaves the content size unknown instead of failing. The size is also no longer taken from a206response, where it's only the size of the remaining part.Content-Lengthof every response.urllib31.26 accepts a response that ends early, which the decoded size check can't catch when the total size is unknown (compressed content, or a resumed download of a chunked response).Testing notes:
pytest tests/test_download_content_encoding.py, with a local server that compresses content when accepted, always compresses, cuts the first compressed response in half, sends chunked content, cuts a chunked response and then a resumed one, or answers a resumed request with compressed content that is cut. All tests pass on requests 2.27.1 + urllib3 1.26 and requests 2.32.3 + urllib3 2.6, and fail ondevelop.python automated_api.pymakes no changes.