Skip to content

Storage: download_to_filename hangs if network disconnects in the middle #5909

Closed

Description

@logmackenzie

I have an issue where I am downloading large files (a few GB) from Google Cloud Storage using the download_to_filename method. My application needs to be able to transition into and out of offline mode nicely so during testing I was disconnecting from the network and found that if I disconnect in the middle of download_to_filename, the application just hangs and the function never returns.

Example code:

from google.cloud import storage

client = storage.Client('project-id')
bucket = client.get_bucket('bucket-name')
blob = storage.blob.Blob('Large-file.txt', bucket)
blob.download_to_filename(blob.name)

If I disconnect from the network just before calling blob.download_to_filename, then I get a ConnectionError from requests which is what I would expect. But if I disconnect from the network after blob.download_to_filename starts, the function just hangs. There is no timeout or exception raised.

I am using Python 3.6.1 on Windows 10 with these package versions:

google-api-core==1.3.0
google-auth==1.5.1
google-cloud-core==0.28.1
google-cloud-storage==1.10.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3

I have not been able to find a good work-around to this, but in my opinion it would make sense for download_to_filename to take a timeout argument and raise an exception if it fails.

I realize that it does resume nicely after network connection is re-established, but in my application the state of those files can change while offline so that does not help me. So maybe having a timeout that by default is None and would maintain the existing behavior, but provides the option to force the function to return within a reasonable amount of time.

I also posted a question to StackOverflow about a possible solution (https://stackoverflow.com/questions/52239860/download-to-filename-hangs-if-network-disconnects-in-the-middle), but this is more of a request for an improvement to the API.

Also, I am not sure if this issue also applies to the upload_from_filename or not, but that might also require a similar modification.

Activity

  1. added
    type: questionRequest for information or clarification. Not an issue.
    api: storageIssues related to the Cloud Storage API.
    on Sep 10, 2018
  2. tseaver commented on Sep 10, 2018

    @tseaver
    Contributor
  3. pulltab commented on Sep 26, 2018

    @pulltab

    +1, I am experiencing a similar issue when uploading files as well.

    Being able to set a timeout on such operations would help me recover more gracefully.

  4. tseaver commented on Oct 18, 2018

    @tseaver
    Contributor

    We might need to handle this in google-resumable-media. @dhermes WDYT?

  5. dhermes commented on Oct 18, 2018

    @dhermes
    Contributor

    @tseaver Sounds reasonable. Any good ideas on reproducing this?

  6. tseaver commented on Oct 22, 2018

    @tseaver
    Contributor

    @dhermes maybe monkey-patch requests.request with something which doesn't return?

  7. added
    help wantedWe'd love to have community involvement on this issue.
    on Feb 15, 2019
  8. fgrzadkowski commented on Mar 18, 2019

    @fgrzadkowski

    @tseaver @dhermes We experience the same issue when uploading and our thread hangs forever. In our setup every few seconds we upload a relatively small file to a bucket (below 100 kB). And we depend on those files to exist.

    What's the recommended workaround for this issue? Currently it makes our code not production-grade because of this problem...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: storageIssues related to the Cloud Storage API.help wantedWe'd love to have community involvement on this issue.type: questionRequest for information or clarification. Not an issue.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions