Repository navigation
samples.samples.autocommit_test: test_enable_autocommit_mode failed #435
Description
Note: #282 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.
commit: 2487800
buildURL: Build Status, Sponge
status: failed
Test output
target = functools.partial(>) predicate = .if_exception_type_predicate at 0x7f8555698d08> sleep_generator = deadline = 120, on_error = Nonedef retry_target(target, predicate, sleep_generator, deadline, on_error=None): """Call a function and retry if it fails. This is the lowest-level retry helper. Generally, you'll use the higher-level retry helper :class:`Retry`. Args: target(Callable): The function to call and retry. This must be a nullary function - apply arguments with `functools.partial`. predicate (Callable[Exception]): A callable used to determine if an exception raised by the target should be considered retryable. It should return True to retry or False otherwise. sleep_generator (Iterable[float]): An infinite iterator that determines how long to sleep between retries. deadline (float): How long to keep retrying the target. The last sleep period is shortened as necessary, so that the last retry runs at ``deadline`` (and not considerably beyond it). on_error (Callable[Exception]): A function to call while processing a retryable exception. Any error raised by this function will *not* be caught. Returns: Any: the return value of the target function. Raises: google.api_core.RetryError: If the deadline is exceeded while retrying. ValueError: If the sleep generator stops yielding values. Exception: If the target raises a method that isn't retryable. """ if deadline is not None: deadline_datetime = datetime_helpers.utcnow() + datetime.timedelta( seconds=deadline ) else: deadline_datetime = None last_exc = None for sleep in sleep_generator: try:return target().nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py:189:
self = <google.api_core.operation.Operation object at 0x7f85552f0780>
retry = <google.api_core.retry.Retry object at 0x7f85556ae7b8>def _done_or_raise(self, retry=DEFAULT_RETRY): """Check if the future is done and raise if it's not.""" kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry} if not self.done(**kwargs):raise _OperationNotComplete()E google.api_core.future.polling._OperationNotComplete
.nox/py-3-6/lib/python3.6/site-packages/google/api_core/future/polling.py:87: _OperationNotComplete
The above exception was the direct cause of the following exception:
self = <google.api_core.operation.Operation object at 0x7f85552f0780>
timeout = 120, retry = <google.api_core.retry.Retry object at 0x7f85556ae7b8>def _blocking_poll(self, timeout=None, retry=DEFAULT_RETRY): """Poll and wait for the Future to be resolved. Args: timeout (int): How long (in seconds) to wait for the operation to complete. If None, wait indefinitely. """ if self._result_set: return retry_ = self._retry.with_deadline(timeout) try: kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry}retry_(self._done_or_raise)(**kwargs).nox/py-3-6/lib/python3.6/site-packages/google/api_core/future/polling.py:108:
args = (), kwargs = {}
target = functools.partial(<bound method PollingFuture._done_or_raise of <google.api_core.operation.Operation object at 0x7f85552f0780>>)
sleep_generator = <generator object exponential_sleep_generator at 0x7f8555216a98>@general_helpers.wraps(func) def retry_wrapped_func(*args, **kwargs): """A wrapper that calls target function with retry.""" target = functools.partial(func, *args, **kwargs) sleep_generator = exponential_sleep_generator( self._initial, self._maximum, multiplier=self._multiplier ) return retry_target( target, self._predicate, sleep_generator, self._deadline,on_error=on_error,).nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py:291:
target = functools.partial(<bound method PollingFuture._done_or_raise of <google.api_core.operation.Operation object at 0x7f85552f0780>>)
predicate = <function if_exception_type..if_exception_type_predicate at 0x7f8555698d08>
sleep_generator = <generator object exponential_sleep_generator at 0x7f8555216a98>
deadline = 120, on_error = Nonedef retry_target(target, predicate, sleep_generator, deadline, on_error=None): """Call a function and retry if it fails. This is the lowest-level retry helper. Generally, you'll use the higher-level retry helper :class:`Retry`. Args: target(Callable): The function to call and retry. This must be a nullary function - apply arguments with `functools.partial`. predicate (Callable[Exception]): A callable used to determine if an exception raised by the target should be considered retryable. It should return True to retry or False otherwise. sleep_generator (Iterable[float]): An infinite iterator that determines how long to sleep between retries. deadline (float): How long to keep retrying the target. The last sleep period is shortened as necessary, so that the last retry runs at ``deadline`` (and not considerably beyond it). on_error (Callable[Exception]): A function to call while processing a retryable exception. Any error raised by this function will *not* be caught. Returns: Any: the return value of the target function. Raises: google.api_core.RetryError: If the deadline is exceeded while retrying. ValueError: If the sleep generator stops yielding values. Exception: If the target raises a method that isn't retryable. """ if deadline is not None: deadline_datetime = datetime_helpers.utcnow() + datetime.timedelta( seconds=deadline ) else: deadline_datetime = None last_exc = None for sleep in sleep_generator: try: return target() # pylint: disable=broad-except # This function explicitly must deal with broad exceptions. except Exception as exc: if not predicate(exc): raise last_exc = exc if on_error is not None: on_error(exc) now = datetime_helpers.utcnow() if deadline_datetime is not None: if deadline_datetime <= now: six.raise_from( exceptions.RetryError( "Deadline of {:.1f}s exceeded while calling {}".format( deadline, target ), last_exc, ),last_exc,).nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py:211:
value = None, from_value = _OperationNotComplete()
???
E google.api_core.exceptions.RetryError: Deadline of 120.0s exceeded while calling functools.partial(<bound method PollingFuture._done_or_raise of <google.api_core.operation.Operation object at 0x7f85552f0780>>), last exception::3: RetryError
During handling of the above exception, another exception occurred:
@pytest.fixture(scope="module") def spanner_instance(): spanner_client = spanner.Client() config_name = f"{spanner_client.project_name}/instanceConfigs/regional-us-central1" instance = spanner_client.instance(INSTANCE_ID, config_name) op = instance.create()op.result(120) # block until completionautocommit_test.py:38:
.nox/py-3-6/lib/python3.6/site-packages/google/api_core/future/polling.py:130: in result
self._blocking_poll(timeout=timeout, **kwargs)
self = <google.api_core.operation.Operation object at 0x7f85552f0780>
timeout = 120, retry = <google.api_core.retry.Retry object at 0x7f85556ae7b8>def _blocking_poll(self, timeout=None, retry=DEFAULT_RETRY): """Poll and wait for the Future to be resolved. Args: timeout (int): How long (in seconds) to wait for the operation to complete. If None, wait indefinitely. """ if self._result_set: return retry_ = self._retry.with_deadline(timeout) try: kwargs = {} if retry is DEFAULT_RETRY else {"retry": retry} retry_(self._done_or_raise)(**kwargs) except exceptions.RetryError: raise concurrent.futures.TimeoutError("Operation did not complete within the designated " "timeout.")E concurrent.futures._base.TimeoutError: Operation did not complete within the designated timeout.
.nox/py-3-6/lib/python3.6/site-packages/google/api_core/future/polling.py:111: TimeoutError
Activity
flaky-bot commentedon Jul 29, 2021
Looks like this issue is flaky. 😟
I'm going to leave this open and stop commenting.
A human should fix and close this.
When run at the same commit (2487800), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).
tests: harden old instance cleanup against NotFound
tests: harden old instance cleanup against NotFound (#471)