Skip to content

fix(bigquery): expand retry allowlist and propagate structured errors (#14579) - #32

Open
lqiu96 wants to merge 3 commits into
mainfrom
fix/bq-retry-allowlist-structured-errors
Open

lqiu96 wants to merge 3 commits into
mainfrom
fix/bq-retry-allowlist-structured-errors

Conversation

@lqiu96

@lqiu96 lqiu96 commented Oct 22, 2026 •

Copy link
Copy Markdown
Owner

This pull request addresses the BigQuery exception handling and retry allowlist gaps identified in issue googleapis#14579.

Summary of Changes

  • Transport Exception Allowlist: Adds SocketTimeoutException and EOFException to DEFAULT_BIGQUERY_EXCEPTION_HANDLER in BigQueryBaseService. SocketTimeoutException is a subclass of InterruptedIOException rather than SocketException in the JDK hierarchy, and EOFException is a direct subclass of IOException. Both represent transient network failures that were previously omitted from the default handler.
  • DirectRetryingExecutor Workaround: Adds BigQueryDirectRetryingExecutor in BigQueryRetryHelper to catch SocketTimeoutException separately before InterruptedIOException. GAX DirectRetryingExecutor catches InterruptedIOException and calls Thread.currentThread().interrupt() before retryingFuture.setAttemptFuture(...), which causes BasicRetryingFuture to see the thread as interrupted and abort the retry loop with an InterruptedException instead of evaluating shouldRetry.
  • Structured BigQuery Error Reasons: Expands BigQueryException.RETRYABLE_ERRORS with the six core BigQuery SLA and error troubleshooting guide reasons (rateLimitExceeded, userRateLimitExceeded, jobRateLimitExceeded, backendError, internalError, and badGateway) and HTTP status codes 429, 500, 502, 503, and 504.
  • Retryability Evaluation: Overrides BigQueryException.isRetryable() to check RETRYABLE_ERRORS (including HttpResponseException status codes when getCode() is UNKNOWN_CODE) as well as SocketTimeoutException and EOFException causes.
  • Centralized Retry Evaluation: Updates BigQueryRetryAlgorithm.shouldRetryBasedOnResult to evaluate BigQueryException.isRetryable() and removes the 16 per-call-site BigQueryRetryHelper.maybeWrapForHttpRetry wrappers in BigQueryImpl.
  • Query Polling Retryability: Updates Job.waitForQueryResults to check ((BigQueryException) prevThrowable).isRetryable() when polling query results.
  • Unit Tests: Adds unit tests in BigQueryExceptionTest, BigQueryImplTest, and JobTest.

Interaction with SkipExceptionTranslation and maybeWrapForHttpRetry

  1. Why SkipExceptionTranslation was introduced (PR Exception in thread "main" java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly configured googleapis/google-cloud-java#3668):
    Originally, HttpBigQueryRpc caught IOException inside every RPC method and threw new BigQueryException(ioException) upfront. Because ExceptionHandler.shouldRetry only checks the top-level thrown exception class (ex.getClass()) against its retriableExceptions set and does not inspect ex.getCause(), wrapping an UnknownHostException into a BigQueryException hid the UnknownHostException class from DEFAULT_BIGQUERY_EXCEPTION_HANDLER. PR Exception in thread "main" java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly configured googleapis/google-cloud-java#3668 added the *SkipExceptionTranslation methods on BigQueryRpc so that the callable inside BigQueryRetryHelper.runWithRetries throws raw IOException during the retry loop, and translates to BigQueryException only after retries are exhausted.

  2. Why SkipExceptionTranslation broke BigQueryException.RETRYABLE_ERRORS (PR fix(bigquery): retry cancel job and routine creation on transient HTTP 5xx errors googleapis/google-cloud-java#13416):
    A side effect of PR Exception in thread "main" java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly configured googleapis/google-cloud-java#3668 was that GoogleJsonResponseException and HttpResponseException (which are subclasses of IOException, not BaseServiceException) were no longer translated into BigQueryException before DEFAULT_BIGQUERY_EXCEPTION_HANDLER evaluated them. Because BaseService.EXCEPTION_HANDLER_INTERCEPTOR only checks if (exception instanceof BaseServiceException), raw HTTP 500, 502, 503, and 504 responses bypassed BigQueryException.RETRYABLE_ERRORS and were only retried if their error message happened to match a regex in BigQueryRetryConfig. PR fix(bigquery): retry cancel job and routine creation on transient HTTP 5xx errors googleapis/google-cloud-java#13416 added BigQueryRetryHelper.maybeWrapForHttpRetry on 16 call sites in BigQueryImpl to translate IOException into BigQueryException when resultAlgorithm == DEFAULT_BIGQUERY_EXCEPTION_HANDLER.

  3. How BigQueryRetryAlgorithm.shouldRetryBasedOnResult preserves both behaviors:

    • BigQueryRetryAlgorithm.shouldRetryBasedOnResult evaluates getResultAlgorithm().shouldRetry(previousThrowable, previousResponse) first on the raw previousThrowable.
    • For UnknownHostException, SocketException, SocketTimeoutException, and EOFException (as well as any custom ExceptionHandler or ResultRetryAlgorithm configured via BigQueryOptions.setResultRetryAlgorithm), getResultAlgorithm().shouldRetry(...) sees the raw IOException directly, preserving PR Exception in thread "main" java.lang.IllegalArgumentException: Jetty ALPN/NPN has not been properly configured googleapis/google-cloud-java#3668 (SkipExceptionTranslation) and custom user retry configurations.
    • Only when getResultAlgorithm().shouldRetry(...) returns false and the caller is using DEFAULT_BIGQUERY_EXCEPTION_HANDLER does BigQueryRetryAlgorithm evaluate new BigQueryException((IOException) previousThrowable).isRetryable() as a read-only predicate (without mutating previousThrowable) to check HTTP status codes (429, 500, 502, 503, 504) and structured JSON error reasons (rateLimitExceeded, backendError, etc.) across all *SkipExceptionTranslation RPCs uniformly.

@lqiu96
lqiu96 force-pushed the fix/bq-retry-allowlist-structured-errors branch from 68db3f1 to b5bf72e Compare October 22, 2026 22:00
@lqiu96
lqiu96 force-pushed the fix/bq-retry-allowlist-structured-errors branch from 23c0272 to 3796a34 Compare October 23, 2026 04:12
@lqiu96
lqiu96 force-pushed the fix/bq-retry-allowlist-structured-errors branch from 3796a34 to 9a4dcd9 Compare October 23, 2026 04:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant