Repository navigation
copy_blob calls have started to fail in system tests (400 no file found in request) #2289
Description
@thobrla Any idea why these requests would start to become flaky?
Some recent examples:
- https://travis-ci.org/GoogleCloudPlatform/google-cloud-python/builds/158783443
- https://travis-ci.org/GoogleCloudPlatform/google-cloud-python/builds/158777834
- https://travis-ci.org/GoogleCloudPlatform/google-cloud-python/builds/158764459
- https://travis-ci.org/GoogleCloudPlatform/google-cloud-python/builds/158717892
- https://travis-ci.org/GoogleCloudPlatform/google-cloud-python/builds/158645485
Activity
Adding retries for failed copy_blob() calls in system tests.
Adding retries for failed copy_blob() calls in system tests.
thobrla commentedon Sep 23, 2016
Do you have a example of the full request / response for this error case? I'm not aware of anything that would lead to a 400 like this. Possibly the request had a connection drop and was only partially sent, but I would expect that to be quite rare.
dhermes commentedon Sep 23, 2016
@thobrla The whole response was just
No file found in request.(not a JSON or HTML response, just text) with a 400 status code (I don't have any of the other headers, sorry).The request is a POST with no body to
dhermes commentedon Sep 23, 2016
@thobrla It happened consistently for several days (see the Travis failures linked above). I checked some recent test runs and it doesn't seem to be triggering anymore, so that's good.
thobrla commentedon Sep 23, 2016
Thanks. I wonder if there was a bug in the service during this time. Please let me know if it resurfaces.
dhermes commentedon Sep 23, 2016
Will do. We may not detect it, right now we are retrying on the failure up to 4 times.
CONTEXT: We have been sprinkling lots of retries in our system tests to get them to go green. Talking to 14 Google services at once (in 2 different Python environments) essentially guarantees a failure if no retries are used.
thobrla commentedon Sep 23, 2016
That's understandable but 400 is not intended to be a retryable error, so you really shouldn't retry a 400 as in theory it will never succeed.
dhermes commentedon Sep 23, 2016
@thobrla Indeed. We added a special error predicate check to make sure it wasn't any old 400, but it was the specific 400 we're discussing here.
chore(main): release 3.38.0 (#2289)
refactor: add agg_ops.ProductOp and NuniqueOp to the sqlglot compiler (…