This repository was archived by the owner on Mar 31, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 31, 2026. It is now read-only.
blob.download_as_text followed by blob.upload_from_string raises a CRC32C validation error #487
Description
from google.cloud import storage
BUCKET_NAME="crwilcox"
BLOB_NAME="blob.txt"
storage_client = storage.Client()
bucket = storage_client.bucket(BUCKET_NAME)
blob = bucket.blob(BLOB_NAME)
blob_text = blob.download_as_text()
blob_text += "edit"
blob.upload_from_string(blob_text)
Traceback (most recent call last):
File "/home/crwilcox/workspace/python-playground/venv39/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2488, in upload_from_file
created_json = self._do_upload(
File "/home/crwilcox/workspace/python-playground/venv39/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2290, in _do_upload
response = self._do_multipart_upload(
File "/home/crwilcox/workspace/python-playground/venv39/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 1823, in _do_multipart_upload
response = upload.transmit(
File "/home/crwilcox/workspace/python-playground/venv39/lib/python3.9/site-packages/google/resumable_media/requests/upload.py", line 149, in transmit
self._process_response(response)
File "/home/crwilcox/workspace/python-playground/venv39/lib/python3.9/site-packages/google/resumable_media/_upload.py", line 116, in _process_response
_helpers.require_status_code(response, (http_client.OK,), self._get_status_code)
File "/home/crwilcox/workspace/python-playground/venv39/lib/python3.9/site-packages/google/resumable_media/_helpers.py", line 99, in require_status_code
raise common.InvalidResponse(
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/crwilcox/workspace/python-playground/bucket_blob_test.py", line 11, in <module>
blob.upload_from_string(blob_text)
File "/home/crwilcox/workspace/python-playground/venv39/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2761, in upload_from_string
self.upload_from_file(
File "/home/crwilcox/workspace/python-playground/venv39/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 2505, in upload_from_file
_raise_from_invalid_response(exc)
File "/home/crwilcox/workspace/python-playground/venv39/lib/python3.9/site-packages/google/cloud/storage/blob.py", line 4271, in _raise_from_invalid_response
raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.BadRequest: 400 POST https://storage.googleapis.com/upload/storage/v1/b/crwilcox/o?uploadType=multipart: {
"error": {
"code": 400,
"message": "Provided CRC32C \"9+rL+w==\" doesn't match calculated CRC32C \"ueIORg==\".",
"errors": [
{
"message": "Provided CRC32C \"9+rL+w==\" doesn't match calculated CRC32C \"ueIORg==\".",
"domain": "global",
"reason": "invalid"
}
]
}
}
: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>)
Possibly helpful, recreating the blob resolves the issue:
blob = bucket.blob(BLOB_NAME)
blob_text = blob.download_as_text()
blob = bucket.blob(BLOB_NAME)
blob_text += "edit"
blob.upload_from_string(blob_text)
Versions:
google-api-core==1.30.0
google-auth==1.32.1
google-cloud-core==1.7.1
google-cloud-storage==1.40.0
google-crc32c==1.1.2
google-resumable-media==1.3.1
googleapis-common-protos==1.53.0
Activity
crwilcox commentedon Jul 7, 2021
I suspect what is happening:
on download, the metadata for crc32c is placed in the blob, on 'upload_from_string' with new info, that metadata hasn't been updated, so on put there is a content/checksum mismatch.
andrewsg commentedon Jul 22, 2021
Diagnosed this to a flaw in the update-from-header strategy added in #204 where properties are being set through the front door, which updates Blob._changes[], rather than directly setting the property in Blob._properties[]. I think modifying that part of the code to directly set the property will fix this issue and also resolve many confusing customer complaints we've received over the past year as well. Thanks so much for your detailed report @crwilcox, couldn't have done it without you. PR coming soon.