Repository navigation
requests.exceptions.ConnectionError retry #176
Description
Hi,
Could we add ConnectionError in list of retry so the client libraries which depend over api-core don't need to add.
"/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/bigquery/client.py", line 561, in create_table api_response = self._call_api( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/bigquery/client.py", line 574, in _call_api return call() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func return retry_target( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/api_core/retry.py", line 184, in retry_target return target() File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 472, in api_request response = self._make_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 336, in _make_request return self._do_request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/cloud/_http.py", line 374, in _do_request return self.http.request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/google/auth/transport/requests.py", line 464, in request response = super(AuthorizedSession, self).request( File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/layers/google.python.pip/pip/lib/python3.8/site-packages/requests/adapters.py", line 498, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
I have reached out to individually to storage,bigquery and auth sdk page to report about this issue and they have acknowledged it googleapis/google-auth-library-python#739, googleapis/google-auth-library-python#737, googleapis/python-storage#414
Activity
busunkim96 commentedon Apr 29, 2021
Hi,
I don't think we should retry at every level that shows up in the stack trace (that might result in retrying more times than is desirable). I think addressing googleapis/google-auth-library-python#737 should be sufficient. If it turns out that isn't enough, we can revisit tweaking the rtries in
google-api-coreand/orgoogle-cloud-core.parthea commentedon May 1, 2021
I'm planning to add
to the list of errors that can be retried here as we have had success with this configuration in the BigQuery client and it follows the spirit of client side retries in AIP-4221. Googlers see b/182321177 for internal discussion.
feat: retry google.auth TransportError and requests ConnectionError (#…
nitishxp commentedon May 4, 2021
@parthea Could you also add https://github.com/googleapis/python-bigquery/blob/6a48e80bc7d347f381b181f4cf81fef105d0ad0d/google/cloud/bigquery/retry.py#L26 in the list of retry
busunkim96 commentedon May 4, 2021
@nitishxp
ConnectionErrorwas added, it's just referred to asrequests.exceptions.ConnectionErrorhere.nitishxp commentedon May 4, 2021
@busunkim96 ConnectionError is specfic to python https://docs.python.org/3/library/exceptions.html#ConnectionError which is an OS error big query sdk https://github.com/googleapis/python-bigquery/blob/6a48e80bc7d347f381b181f4cf81fef105d0ad0d/google/cloud/bigquery/retry.py#L26 has added os connection error in retry along with requests.ConnectionError
busunkim96 commentedon May 4, 2021
Ah I see. Do you have a stack trace that shows an un-retried
ConnectionError? I'm wondering if that's specific to BIgQuery.nitishxp commentedon May 4, 2021
@busunkim96 Naah , But i have faced this issue in Cloud function when there r too many instances of CF running and same instance is used for another request