Skip to content
This repository was archived by the owner on Jun 8, 2026. It is now read-only.
This repository was archived by the owner on Jun 8, 2026. It is now read-only.

samples.samples.backup_sample_test: test_create_backup failed #320

Closed

Description

@flaky-bot

Note: #264 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.


commit: 7bddb81
buildURL: Build Status, Sponge
status: failed

Test output
target = functools.partial(>)
predicate = .if_exception_type_predicate at 0x7f1d178f5488>
sleep_generator = 
deadline = 1200, on_error = None
def retry_target(target, predicate, sleep_generator, deadline, on_error=None):
    """Call a function and retry if it fails.

    This is the lowest-level retry helper. Generally, you'll use the
    higher-level retry helper :class:`Retry`.

    Args:
        target(Callable): The function to call and retry. This must be a
            nullary function - apply arguments with `functools.partial`.
        predicate (Callable[Exception]): A callable used to determine if an
            exception raised by the target should be considered retryable.
            It should return True to retry or False otherwise.
        sleep_generator (Iterable[float]): An infinite iterator that determines
            how long to sleep between retries.
        deadline (float): How long to keep retrying the target. The last sleep
            period is shortened as necessary, so that the last retry runs at
            ``deadline`` (and not considerably beyond it).
        on_error (Callable[Exception]): A function to call while processing a
            retryable exception.  Any error raised by this function will *not*
            be caught.

    Returns:
        Any: the return value of the target function.

    Raises:
        google.api_core.RetryError: If the deadline is exceeded while retrying.
        ValueError: If the sleep generator stops yielding values.
        Exception: If the target raises a method that isn't retryable.
    """
    if deadline is not None:
        deadline_datetime = datetime_helpers.utcnow() + datetime.timedelta(
            seconds=deadline
        )
    else:
        deadline_datetime = None

    last_exc = None

    for sleep in sleep_generator:
        try:
          return target()

.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py:184:


self = <google.api_core.operation.Operation object at 0x7f1d17417e10>
retry = <google.api_core.retry.Retry object at 0x7f1d178f3940>

def _done_or_raise(self, retry=DEFAULT_RETRY):
    """Check if the future is done and raise if it's not."""
    kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry}

    if not self.done(**kwargs):
      raise _OperationNotComplete()

E google.api_core.future.polling._OperationNotComplete

.nox/py-3-6/lib/python3.6/site-packages/google/api_core/future/polling.py:86: _OperationNotComplete

The above exception was the direct cause of the following exception:

self = <google.api_core.operation.Operation object at 0x7f1d17417e10>
timeout = 1200, retry = <google.api_core.retry.Retry object at 0x7f1d178f3940>

def _blocking_poll(self, timeout=None, retry=DEFAULT_RETRY):
    """Poll and wait for the Future to be resolved.

    Args:
        timeout (int):
            How long (in seconds) to wait for the operation to complete.
            If None, wait indefinitely.
    """
    if self._result_set:
        return

    retry_ = self._retry.with_deadline(timeout)

    try:
        kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry}
      retry_(self._done_or_raise)(**kwargs)

.nox/py-3-6/lib/python3.6/site-packages/google/api_core/future/polling.py:107:


args = (), kwargs = {}
target = functools.partial(<bound method PollingFuture._done_or_raise of <google.api_core.operation.Operation object at 0x7f1d17417e10>>)
sleep_generator = <generator object exponential_sleep_generator at 0x7f1d1748d728>

@general_helpers.wraps(func)
def retry_wrapped_func(*args, **kwargs):
    """A wrapper that calls target function with retry."""
    target = functools.partial(func, *args, **kwargs)
    sleep_generator = exponential_sleep_generator(
        self._initial, self._maximum, multiplier=self._multiplier
    )
    return retry_target(
        target,
        self._predicate,
        sleep_generator,
        self._deadline,
      on_error=on_error,
    )

.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py:286:


target = functools.partial(<bound method PollingFuture._done_or_raise of <google.api_core.operation.Operation object at 0x7f1d17417e10>>)
predicate = <function if_exception_type..if_exception_type_predicate at 0x7f1d178f5488>
sleep_generator = <generator object exponential_sleep_generator at 0x7f1d1748d728>
deadline = 1200, on_error = None

def retry_target(target, predicate, sleep_generator, deadline, on_error=None):
    """Call a function and retry if it fails.

    This is the lowest-level retry helper. Generally, you'll use the
    higher-level retry helper :class:`Retry`.

    Args:
        target(Callable): The function to call and retry. This must be a
            nullary function - apply arguments with `functools.partial`.
        predicate (Callable[Exception]): A callable used to determine if an
            exception raised by the target should be considered retryable.
            It should return True to retry or False otherwise.
        sleep_generator (Iterable[float]): An infinite iterator that determines
            how long to sleep between retries.
        deadline (float): How long to keep retrying the target. The last sleep
            period is shortened as necessary, so that the last retry runs at
            ``deadline`` (and not considerably beyond it).
        on_error (Callable[Exception]): A function to call while processing a
            retryable exception.  Any error raised by this function will *not*
            be caught.

    Returns:
        Any: the return value of the target function.

    Raises:
        google.api_core.RetryError: If the deadline is exceeded while retrying.
        ValueError: If the sleep generator stops yielding values.
        Exception: If the target raises a method that isn't retryable.
    """
    if deadline is not None:
        deadline_datetime = datetime_helpers.utcnow() + datetime.timedelta(
            seconds=deadline
        )
    else:
        deadline_datetime = None

    last_exc = None

    for sleep in sleep_generator:
        try:
            return target()

        # pylint: disable=broad-except
        # This function explicitly must deal with broad exceptions.
        except Exception as exc:
            if not predicate(exc):
                raise
            last_exc = exc
            if on_error is not None:
                on_error(exc)

        now = datetime_helpers.utcnow()

        if deadline_datetime is not None:
            if deadline_datetime <= now:
                six.raise_from(
                    exceptions.RetryError(
                        "Deadline of {:.1f}s exceeded while calling {}".format(
                            deadline, target
                        ),
                        last_exc,
                    ),
                  last_exc,
                )

.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py:206:


value = None, from_value = _OperationNotComplete()

???
E google.api_core.exceptions.RetryError: Deadline of 1200.0s exceeded while calling functools.partial(<bound method PollingFuture._done_or_raise of <google.api_core.operation.Operation object at 0x7f1d17417e10>>), last exception:

:3: RetryError

During handling of the above exception, another exception occurred:

capsys = <_pytest.capture.CaptureFixture object at 0x7f1d1748fa90>
database = <google.cloud.spanner_v1.database.Database object at 0x7f1d174d4cc0>

def test_create_backup(capsys, database):
    version_time = None
    with database.snapshot() as snapshot:
        results = snapshot.execute_sql("SELECT CURRENT_TIMESTAMP()")
        version_time = list(results)[0][0]
  backup_sample.create_backup(INSTANCE_ID, DATABASE_ID, BACKUP_ID, version_time)

backup_sample_test.py:75:


backup_sample.py:41: in create_backup
operation.result(1200)
.nox/py-3-6/lib/python3.6/site-packages/google/api_core/future/polling.py:129: in result
self._blocking_poll(timeout=timeout, **kwargs)


self = <google.api_core.operation.Operation object at 0x7f1d17417e10>
timeout = 1200, retry = <google.api_core.retry.Retry object at 0x7f1d178f3940>

def _blocking_poll(self, timeout=None, retry=DEFAULT_RETRY):
    """Poll and wait for the Future to be resolved.

    Args:
        timeout (int):
            How long (in seconds) to wait for the operation to complete.
            If None, wait indefinitely.
    """
    if self._result_set:
        return

    retry_ = self._retry.with_deadline(timeout)

    try:
        kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry}
        retry_(self._done_or_raise)(**kwargs)
    except exceptions.RetryError:
        raise concurrent.futures.TimeoutError(
          "Operation did not complete within the designated " "timeout."
        )

E concurrent.futures._base.TimeoutError: Operation did not complete within the designated timeout.

.nox/py-3-6/lib/python3.6/site-packages/google/api_core/future/polling.py:110: TimeoutError

Activity

  1. added
    flakybot: issueAn issue filed by the Flaky Bot. Should not be added manually.
    priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    on Apr 26, 2021
  2. added
    api: spannerIssues related to the googleapis/python-spanner API.
    samplesIssues that are directly related to samples.
    on Apr 26, 2021
  3. flaky-bot commented on Apr 27, 2021

    @flaky-bot
    Author

    Looks like this issue is flaky. 😟

    I'm going to leave this open and stop commenting.

    A human should fix and close this.


    When run at the same commit (7bddb81), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

  4. added
    flakybot: flakyTells the Flaky Bot not to close or comment on this issue.
    on Apr 27, 2021
  5. larkee commented on Apr 29, 2021

    @larkee
    Contributor

    This is a known flake. The backup taking >20 minutes does not happen frequently (last flake related to timeout was #157 which was 6 months ago) so I am closing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: spannerIssues related to the googleapis/python-spanner API.flakybot: flakyTells the Flaky Bot not to close or comment on this issue.flakybot: issueAn issue filed by the Flaky Bot. Should not be added manually.priority: p1Important issue which blocks shipping the next release. Will be fixed prior to next release.samplesIssues that are directly related to samples.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions