Repository navigation
Monitoring download progress. #1208
Description
For clients that would like to display something like a progress bar for downloads, there's an issue in that while you're streaming data from iter_content(), you can't properly know how far you have progressed through the download because we'll often be performing gzip decompression. The Content-Length header gives the length of the raw data. (And we really don't want to be streaming from iter_raw() because we really do want the uncompressed bytes, not the bytes on the wire.)
See also for example, this question related to requests... https://stackoverflow.com/questions/21778885/python3-progress-bar-and-download-with-gzip
It might make sense to provide an API for monitoring the number of bytes that have been consumed by the underlying iter_raw calls, for example something like... response.num_bytes_downloaded.
with open("download.txt", "wb") as download_file:
with httpx.stream('http://www.example.org') as response:
content_length = response.headers.get("Content-Length")
for chunk in response.iter_content():
download_file.write(chunk)
if content_length is not None:
percent_complete = 100.0 * (response.num_bytes_downloaded / content_length)
print(f"{percent_complete:.2f}%")Perhaps we ought to provide a response.percent_complete that wraps this up nicely, returning either a 0.0 <= float <= 100.0 or None if the Content-Length is unknown, so...
with open("download.txt", "wb") as download_file:
with httpx.stream('http://www.example.org') as response:
for chunk in response.iter_content():
download_file.write(chunk)
if response.percent_complete is not None:
print(f"{response.percent_complete:.2f}%")
Activity
lovelydinosaur commentedon Aug 21, 2020
For
requestsit looks like this interface is possibly available viarequest.raw.tell()... https://urllib3.readthedocs.io/en/latest/reference/#urllib3.response.HTTPResponse.telllovelydinosaur commentedon Aug 21, 2020
Any API decision here might be influenced by if we want to expose the raw stream instance on the response or not.
florimondmanca commentedon Sep 1, 2020
Regardless of the API we decide on, one way to challenge it and make sure it's right enough for most use cases is to see if it would integrate well with tqdm...
tqdmsupports a manual.update(<increment>)-based usage that was specifically designed for this kind of "read from a known-length stream" use cases.The subtle thing here is that we can't just naively pass
.update(len(chunk))sincechunkis decoded data, while we want to do.update(<raw chunk size>).Since
tqdmexpects an increment, I guess one sensible thing to do would be to also add aresponse.last_raw_chunk_sizeattribute (or dropresponse.num_bytes_downloadedentirely, since we can accumulate last raw chunk sizes to get the total number of downloaded bytes anyway — but computing the diff manually is cumbersome).For example:
Example in-progress output (obtained by using
progress.update(len(chunk)), just for demo purposes):32%|█████████████████████████ | 317352/1000431 [00:00<00:00, 922202.25it/s]And here's Tom's
percent_completeexample using this alternativeresponse.last_raw_chunk_sizeAPI:Added last_raw_chunk_size to the Response object (encode#1208)
Added example with progress bar (encode#1208)
Add progress to streaming download (#1268)
cdeler commentedon Sep 10, 2020
@tomchristie
can we treat this issue as closed after #1268 ?
lovelydinosaur commentedon Sep 10, 2020
Yup! 👍👍👍