Skip to content
This repository was archived by the owner on Mar 31, 2026. It is now read-only.

[Prototype] Retries for Storage - #197

Closed
frankyn wants to merge 3 commits into
masterfrom
retries-enabled
Closed

frankyn wants to merge 3 commits into
masterfrom
retries-enabled

Conversation

@frankyn

@frankyn frankyn commented Jun 27, 2020 •

Copy link
Copy Markdown
Contributor

Prototype based on BigQuery retry implementation in Python: https://github.com/googleapis/python-bigquery/blob/master/google/cloud/bigquery/retry.py

Work towards resolving: #108

Remaining:

  • Idempotent operations are retried only
  • Reset Connection Approval
  • Unit Tests
  • Review

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jun 27, 2020
@frankyn frankyn changed the title prototype retries for python prototype retries for Storage Jun 27, 2020
@frankyn frankyn added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 27, 2020
@frankyn
frankyn requested review from andrewsg and tritone June 29, 2020 16:07
@frankyn frankyn changed the title prototype retries for Storage [Prototype] Retries for Storage Jun 29, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 29, 2020

@andrewsg andrewsg left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM so far



def _call_api(self, client, retry, **kwargs):
call = functools.partial(client._connection.api_request, **kwargs)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this because the interface for the retry function is not expected to pass in any arguments?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copied it over mainly:
At the moment no, I'm not making it configurable for the first pass and only introducing it to help support/unblock customer friction when using the library.

@tritone tritone left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice first pass! I have some thoughts, let me know if you want to discuss more.

import urllib3

from google.api_core import exceptions
from google.api_core import retry

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neat, so this is what we told customers to use as a workaround already right?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, where are configs around number of attempts or timeouts? Are these values hard-coded into google.api_core.retry?

reason = exc.errors[0]["reason"]

return reason in _RETRYABLE_REASONS
else:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for else here right? And the nested if statements and uses of isinstance look pretty ugly, I'm sure there's a nicer way to do this.

return isinstance(exc, _UNSTRUCTURED_RETRYABLE_TYPES)
reason = exc.errors[0]["reason"]

return reason in _RETRYABLE_REASONS

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you not have access to response codes here? Seems kind of a strange workaround to be doing string matching for a 503 for example.

return True
return False

_DEFAULT_RETRY = retry.Retry(predicate=_should_retry)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would be curious to look at other examples of where people have written these predicates.

return params


def _call_api(self, client, retry, **kwargs):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should retry be a kwarg? Do callers explicitly have to pass in None if they don't want a retry? Would be good to clarify here.

@simonvanderveldt

Copy link
Copy Markdown

Is there any chance of this being merged soon? We'd very much like to have this functionality available in the library, we currently have to wrap everything in retries ourselves.

@busunkim96 busunkim96 closed this Jul 31, 2020
@simonvanderveldt

simonvanderveldt commented Jul 31, 2020 •

Copy link
Copy Markdown

@busunkim96 Why was this closed? Is there a new PR somewhere else?

@andrewsg

Copy link
Copy Markdown
Contributor

I'll be working on a new one.

@busunkim96 busunkim96 reopened this Jul 31, 2020
@frankyn frankyn added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Aug 26, 2020
@ahill00

ahill00 commented Sep 11, 2020

Copy link
Copy Markdown

Is there any update on this?

We recently encountered no retries for google.cloud.storage.blob.Blob.reload https://googleapis.dev/python/storage/latest/blobs.html?#google.cloud.storage.blob.Blob.reload and it would be nice to have this functionality.

@frankyn

frankyn commented Sep 11, 2020

Copy link
Copy Markdown
Contributor Author

Closing this PR as @andrewsg is working on a new PR now. PR should be available soon. Thank you for your patience.

@frankyn frankyn closed this Sep 11, 2020
@tseaver
tseaver deleted the retries-enabled branch August 24, 2021 18:08
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

cla: yes This human has signed the Contributor License Agreement. do not merge Indicates a pull request not ready for merge, due to either quality or timing.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants