Skip to content

fix: start socket_timeout clock when executor work begins (#1278) - #1294

Open
hippi345 wants to merge 2 commits into
aws:mainfrom
hippi345:fix/1278-timeout-starts-on-run
Open

hippi345 wants to merge 2 commits into
aws:mainfrom
hippi345:fix/1278-timeout-starts-on-run

Conversation

@hippi345

@hippi345 hippi345 commented Oct 17, 2026 •

Copy link
Copy Markdown

Description

Fixes #1278.

socket_timeout is enforced by submitting the underlying driver call to a shared ThreadPoolExecutor (DriverDialectExecutor from services_container.get_thread_pool) and calling future.result(timeout=socket_timeout). That clock used to start at submit(), so time spent queued behind other in-flight calls counted as query time. When the process-wide pool is saturated (default size min(32, cpu_count + 4)), a fast query could raise QueryTimeoutError without ever being slow on the database.

This PR starts the timeout clock when a worker actually begins the call. Queue delay is not charged against socket_timeout.

Queue-bound expiry (correctness): if the call is still queued when the queue-wait bound expires, the future is cancel()led and a lock/abandoned flag prevents the worker from starting func after the caller has given up. If cancel() returns False (the worker started in that window), the call is treated as started and the existing abort/drain path runs. That closes the hole where the caller already had a timeout and the submitted work later ran on the same connection.

Queue-wait bound: max(timeout * 2, timeout + 5) seconds, not max(timeout * 10, 60). A 5s socket_timeout can wait at most 10s for a worker (instead of 60s), which is enough extra time to get scheduled under contention without amplifying pile-ups during an outage. A 0.2s timeout still has 5.2s to get a worker. No new connection property: the bound is derived from socket_timeout so it stays in the existing timeout API.

No pool-size property. get_thread_pool is unchanged.

Maintainer options (this PR is open to switching approach):

  • (a) Start the clock at run start (this PR). Smallest change. socket_timeout matches actual driver/socket time. Trade-offs: a saturated pool can delay a caller until a worker is free (or until the queue-wait bound); the default pool can still be a hidden concurrency ceiling smaller than the application's connection pool.
  • (b) Configurable DriverDialectExecutor pool size. Add a wrapper property (e.g. driver_dialect_executor_max_workers) plus docs. Lets operators size the pool to expected DB concurrency. Trade-off: does not by itself stop queue time from counting as query time if the pool is still too small.
  • (c) Both. Clock starts at run start, and the pool size is configurable. Most complete; more API/docs surface.

Happy to switch to (b) or (c) if that is the preferred design.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

This change was prepared with AI assistance and reviewed by the author.

Queue time in the shared DriverDialectExecutor no longer counts against
socket_timeout, so a fast query waiting behind a saturated pool does not
raise QueryTimeoutError. A call that actually runs too long still times out.
On queue-wait expiry, cancel the future and block the worker from
starting the call after the caller gives up. If cancel loses the race,
run abort/drain. Bound queue wait by max(timeout*2, timeout+5).
@hippi345
hippi345 marked this pull request as ready for review October 17, 2026 17:26

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

socket_timeout includes queueing time in a shared, unconfigurable, process-wide thread pool, causing misleading QueryTimeoutErrors on unrelated queries

1 participant