This repository was archived by the owner on Mar 31, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 31, 2026. It is now read-only.
Batch context manager throws only the last exception occurred and no response is returned #793
Description
Thanks for stopping by to let us know something could be better!
Environment details
- MacOS
- Python version:
3.8.9 - pip version:
22.0.4 google-cloud-storageversion:2.3.0
Steps to reproduce
- Send Batch delete request for non existent objects in a bucket
Code example
storage_client = storage.Client.from_service_account_json('key.json')
bucket_obj_1 = storage_client.bucket("bucket")
with storage_client.batch():
bucket_obj_1.delete_blobs(blobs=["1.txt", "2.txt", "3.txt"])Stack trace
Traceback (most recent call last):
File "****", line 72, in <module>
bucket_obj_1.delete_blobs(blobs=["1.txt", "2.txt", "3.txt"])
File "****/lib/python3.8/site-packages/google/cloud/storage/batch.py", line 288, in __exit__
self.finish()
File "****/lib/python3.8/site-packages/google/cloud/storage/batch.py", line 274, in finish
self._finish_futures(responses)
File "****/lib/python3.8/site-packages/google/cloud/storage/batch.py", line 250, in _finish_futures
raise exceptions.from_http_response(exception_args)
google.api_core.exceptions.NotFound: 404 BATCH contentid://None: No such object: bucket/3.txt
In the above example, the batch request has been made with 3 blobs. However, the exception message only shows details about the last error occurred and this is evident in the implementation of the Batch context manager as mentioned in the below code snippet.
def _finish_futures(self, responses):
"""Apply all the batch responses to the futures created.
:type responses: list of (headers, payload) tuples.
:param responses: List of headers and payloads from each response in
the batch.
:raises: :class:`ValueError` if no requests have been deferred.
"""
# If a bad status occurs, we track it, but don't raise an exception
# until all futures have been populated.
exception_args = None
if len(self._target_objects) != len(responses): # pragma: NO COVER
raise ValueError("Expected a response for every request.")
for target_object, subresponse in zip(self._target_objects, responses):
if not 200 <= subresponse.status_code < 300:
exception_args = exception_args or subresponse
elif target_object is not None:
try:
target_object._properties = subresponse.json()
except ValueError:
target_object._properties = subresponse.content
if exception_args is not None:
raise exceptions.from_http_response(exception_args)
Since, each blob exception need to be handled individually, it is becoming difficult to handle exceptions. Any suggestions on how to workaround this issue?
Also, is there a way to retrieve the response objects of individual blobs so that the caller can process and handle it accordingly
Activity
[-]Batch context manager throws only the last exception occurred[/-][+]Batch context manager throws only the last exception occurred and no response is returned[/+]andrewsg commentedon May 31, 2022
Thank you for your detailed report. I've looked through the source code and as of yet I haven't identified a way to fix this without breaking backwards compatibility. I've added this issue to our list of items to address with a future major version change.
In the meantime, if it suits your use case, I would recommend looking into concurrent individual requests using the
concurrent.futuresstandard library module instead of batch requests. For instance, with a list of blob objects to delete asblobs:I haven't tried this exact code yet, but it's modified from code that works. Please let you know if it gives you any trouble. Thank you again.
andrewsg commentedon Jun 1, 2022
Reopening this as a feature request to add a flag to batch "finish" to change behavior in a non-backwards-incompatible way.
ranjanp75 commentedon Jun 2, 2022
@andrewsg I have an approach to implement this. Why not store the responses in the Batch context manager so that its accessible after the requests are done and disable the validation.
andrewsg commentedon Jun 7, 2022
@ranjanp75 I haven't yet found a way to make that approach discoverable and user-friendly, but I will take a serious look at it when we look at this request. Thanks!