This repository was archived by the owner on Mar 31, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 31, 2026. It is now read-only.
Upload of large files times out. #40
Description
OS 10.14.4
Python 3.8.0
Google_cloud_storage: 1.20.0
I am getting a connection error due to the following code:
blob = bucket.blob(file)
blob.upload_from_filename(file)
This is certainly a fault of google.client.storage for the following reasons:
- I can upload a 60 meg file to google drive with no problem. Basically takes about 6 minutes.
- When I try to upload a 20 meg file I get the aforementioned error.
- I've uploaded about 70,000 files so far with this code, most between 10 and 60 megs with no problem.
I think I've had this problem before and it happened when my upload speed was between 1 and 2 mbps which is what my upload speed is now. When my upload speed is above 10 mbps I do not have this problem.
Still, I should be able to upload an 18 meg file in less than 18 seconds, so I don't see why I'm getting a connection error.
Here is the full traceback:
blob.upload_from_filename(file)
File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 1246, in upload_from_filename
self.upload_from_file(
File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 1195, in upload_from_file
created_json = self._do_upload(
File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 1105, in _do_upload
response = self._do_resumable_upload(
File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/cloud/storage/blob.py", line 1053, in _do_resumable_upload
response = upload.transmit_next_chunk(transport)
File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/resumable_media/requests/upload.py", line 419, in transmit_next_chunk
response = _helpers.http_request(
File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/resumable_media/requests/_helpers.py", line 116, in http_request
return _helpers.wait_and_retry(func, RequestsMixin._get_status_code, retry_strategy)
File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/resumable_media/_helpers.py", line 150, in wait_and_retry
response = func()
File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/google/auth/transport/requests.py", line 207, in request
response = super(AuthorizedSession, self).request(
File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/requests/sessions.py", line 512, in request
resp = self.send(prep, **send_kwargs)
File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/requests/sessions.py", line 622, in send
r = adapter.send(request, **kwargs)
File "/Users/kylefoley/codes/venv/lib/python3.8/site-packages/requests/adapters.py", line 495, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out'))
Reacted by Caleb Hattingh, Bartek Pacia, Taylor Archibald, Maksim Kalmykov and Brandon Segal
Activity
kylefoley76 commentedon Feb 3, 2020
I've got some more precise numbers. My connection speed is now about 2 mbps. I can upload a file of 10 mb but nothing larger, otherwise I get the connectionerror. Also, I can upload files of at least 60 mb with Google Drive, so it's not like you need a high upload speed to upload a large file.
IlyaFaer commentedon Jun 15, 2020
@kylefoley76, please, take a look at this comment, it seems to be the info that can help in your case
[-]storage: connectionerror[/-][+]Upload of large files times out.[/+]tseaver commentedon Aug 17, 2020
PR #185 added explicit
timeoutarguments to theBlobmethods making API calls. You should be able to pass a longer timeout to resolve this issue.