Skip to content

fix(aio): failover and read/write splitting use the plugin service's host list provider - #1371

Draft
JuanLeee wants to merge 2 commits into
mainfrom
fix/integration-test-failures-2
Draft

JuanLeee wants to merge 2 commits into
mainfrom
fix/integration-test-failures-2

Conversation

@JuanLeee

@JuanLeee JuanLeee commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Description

Since #1292 (a2ac1ada, 2026-10-28, after the 3.1.0 release), the first async connection to an Aurora instance endpoint, an IP or a custom domain starts on AsyncStaticHostListProvider, built from the guessed dialect, and the plugin service swaps in the Aurora provider once the dialect is confirmed. AsyncFailoverPlugin and AsyncReadWriteSplittingPlugin keep the provider they were built with, so on that connection writer failover can keep reconnecting to the old writer, as every gdb_failover writer-failover test on main does, and set_read_only(True) stays on the writer. The replacement provider also never gets the probe its topology monitor needs for panic mode. Two older async gaps sit on the failover path: an error raised while the wrapper creates the driver cursor reaches the application raw, and reader failover marks every reader it couldn't reach as unavailable for 5 minutes, for every async connection in the process. The Integration Tests workflow on main has failed on the async failover and read/write splitting tests since that commit (run 40751906508) and still does at the base of this branch (run 43391980255); the integration conftest clears the endpoint dialect cache before each test, so every test takes the first-connection path.

This PR makes the async plugins read the plugin service's provider, as the sync plugins already do, subscribes async failover to cursor creation as sync failover is, and drops the reader marks as sync failover v2 and JDBC's failover2 do. It also removes three timing and state dependencies from the async integration tests.

  1. AsyncFailoverPlugin and AsyncReadWriteSplittingPlugin stored the provider build_async_plugins handed them and refreshed through it, in the writer loop AsyncRetryUtil.get_writer_connection and in read/write splitting's switches. For a connection whose guessed dialect is RDS_PG, RDS_MYSQL, PG or MYSQL, that provider is the static one and returns only the seed host. While the topology still named the old writer, writer failover kept reconnecting to it and ended with "[Failover] Unable to establish SQL connection to the writer instance.", which fails the gdb_failover writer-failover tests in every job of run 43391980255, and set_read_only(True) logged "[ReadWriteSplittingPlugin] A reader instance was requested via set_read_only, but there are no readers in the host list" and stayed on the writer. Both classes now read _host_list_provider from plugin_service.host_list_provider through a read-only property, and AsyncGdbFailoverPlugin and AsyncGdbReadWriteSplittingPlugin inherit it. Cluster endpoints were unaffected, because their guess is already Aurora.
  2. AsyncPluginServiceImpl._update_host_list_provider copied the panic-mode probe onto the replacement provider only when the previous provider had one, and AsyncStaticHostListProvider never does. The Aurora provider that replaced a static one therefore built its topology monitor without a probe, so _should_panic returned False and the monitor never searched the hosts for the new writer once its own monitoring connection was gone. The service now builds the probe with build_probe_host when the previous provider has none and sets it before the replacement's first force_refresh_host_list, as the wrapper already does for the provider it builds first.
  3. AsyncFailoverPlugin._SUBSCRIBED didn't list Connection.cursor. The async wrapper creates the driver cursor on the first cursor operation and sends that call through the plugin pipeline, so on a broken connection the driver's error reached the application and failover never started. Run 43391980255 shows it on Aurora PostgreSQL: test_fail_from_reader_to_writer_async failed with psycopg.OperationalError "the connection is closed", raised from _resolve_target_cursor. CONNECTION_CURSOR is now subscribed, so the error starts failover and the caller gets FailoverSuccessError, or TransactionResolutionUnknownError inside a transaction. The gap dates from feat: asyncio support (async wrapper, plugins, and SQLAlchemy dialects) #1257 and is in 3.1.0.
  4. AsyncFailoverPlugin._failover_reader_impl marked each reader candidate it couldn't open UNAVAILABLE through plugin_service.set_availability. The mark goes into AsyncPluginServiceImpl._host_availability_expiring_cache, a class-level cache shared by every async connection in the process for 5 minutes, and the default random selector picks only available hosts, so a reader that was down for a moment during a failover stayed out of selection afterwards. A failed candidate is now only dropped from the current pass; the failed current host is still marked UNAVAILABLE at the start of _do_failover. The mark dates from feat: asyncio support (async wrapper, plugins, and SQLAlchemy dialects) #1257, but in 3.1.0, and again from refactor: revert the alias removal and async topology move #1328 (2408eea8, 2026-10-26) until refactor: identify hosts by host id and host-and-port instead of aliases #1297 (20d619b3, 2026-11-03), marks were written by alias and read by URL, so it changes host selection only on main.
  5. Four async writer-failover tests compared the connection's instance with the RDS API's writer right after failover, although their own comment says the control plane can lag "by tens of seconds to minutes". They now wait with retry_until(lambda: aurora_utility.is_db_instance_writer(...)) first, as their sync twins do, and keep both checks.
  6. The integration conftest now clears the async host-availability cache before each test, as it already clears the sync one, so a mark one test leaves (item 4) can't reach the next.
  7. Two async srw tests now point srw_read_endpoint and srw_write_endpoint at instance endpoints, because the srw docs say a cluster endpoint may connect to the previous writer right after a failover. The autocommit test turns srw_verify_new_connections off, so srw uses its endpoints as given. The incorrect-reader-endpoint test keeps verification on, and in one Aurora run its writer cluster endpoint, used as the read endpoint, resolved to a reader. Both tests keep their asserts.

Behaviour changes

  1. A driver error raised while the async wrapper creates a cursor on a broken connection now starts failover, so the application gets FailoverSuccessError, TransactionResolutionUnknownError or FailoverFailedError instead of the driver's error. This differs from 3.1.0.
  2. Async reader failover no longer marks the reader candidates it couldn't reach as unavailable, so they stay in host selection.
  3. When the plugin service replaces a static provider, the new provider's topology monitor now opens a probe connection to each host in panic mode, as a provider the wrapper builds directly already does.

Not in scope

The first three are separate parity changes, so they're left for later.

  1. Async failover still subscribes to a hand-written method list rather than the driver dialect's whole network-bound set, as sync failover v2 does, so errors from Cursor.callproc, the autocommit setter or set_read_only still don't start failover.
  2. Async still marks hosts UNAVAILABLE in three places sync failover v2 doesn't: the original writer in the reader pass, and the topology writer and each reader in AsyncRetryUtil.
  3. The sync twins of the two srw tests still use cluster endpoints.
  4. UsingTheFailover2Plugin.md still says async injects the AsyncHostListProvider at plugin construction, which is now only the fallback; the page needs its own docs update.
  5. retry_until sleeps synchronously, so the four async failover tests block their event loop for up to 5 minutes while they wait; the tests have no async retry helper.

Tests

The 5 new unit tests (3 for failover: cursor creation, writer convergence and reader candidates; 1 for the panic-mode probe; 1 for read/write splitting) fail on main and pass with this change. With the head's four changed unit test files on main's source they give 5 failed and 186 passed, none on an import, and 191 passed at the head, on Python 3.12. pytest tests/unit -Werror passes 2565 tests on Python 3.10 and 3.12, with mypy, flake8 and isort clean.

The repo's Gradle integration harness ran on Python 3.12 against fresh Aurora clusters (3 instances) in us-east-2. At this tree test-pg-aurora-async had 98 passed and 0 failed, and test-mysql-aurora-async had 128 passed and 0 failed with 2 setup errors in the Django and SQLAlchemy IAM tests, because the lanes run without IAM ([IamAuthPlugin] Property "user" is None or empty). The two async least-connections tests passed on 5 instances, and test-docker had 120 passed. test-pg-aurora and test-mysql-aurora ran on the tree before item 7, which touches only an async test module: 134 passed with 1 custom endpoint failure that also fails on main, and 154 passed with the same 2 IAM errors. TestReadWriteSplittingAsync on Aurora PostgreSQL passed 54 of 54 twice before item 7, and at a commit without item 4 and item 6, test_failover_reader_to_writer__switch_read_only_async[plugin_config0-failover_v2,host_monitoring_v2] failed both times. Python 3.11, 3.13 and 3.14, Aurora with 1 or 2 instances (or 5 outside the least-connections tests), the IAM and telemetry tests, Multi-AZ and Blue/Green weren't run locally. The Integration Tests workflow on this branch (run 43642588024) passed 3 of its 8 LTS jobs (3.14 mysql, 3.11 pg and 3.14 pg), where main's run 43391980255 at the base passed none. Its 9 failed cases are 8 distinct tests. Seven are sync tests this change doesn't touch (test_pooled_connection__failover in two configs, test_failover_to_new_writer__switch_read_only, a custom endpoint test and three TestAuroraFailover writer-failover tests), and each also failed in a recent main run (40422400519, 43184367058 or 43391980255). The one async failure, test_pooled_connection__failover_in_transaction_async[plugin_config0] on 3.12 pg with the writer id unchanged after failover (assert '<id>' != '<id>'), also failed in run 43391980255, which had 32 distinct async failures. No CHANGELOG entry, message key or docs change.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

… host list provider, and failover covers cursor creation

The async failover and read/write splitting plugins kept the static provider the wrapper
built from the guessed dialect, so after the plugin service swapped in the Aurora provider
they saw only the seed host: no readers, and writer failover that never reached the new
writer. They now read the service's current provider, as sync does. Async failover also
subscribes to Connection.cursor, as sync does, so a connection closed while a cursor is
created starts failover. The replacement provider is armed for panic-mode writer discovery
before its first refresh when the previous one had no probe. Async reader failover now only
drops a reader it cannot reach from the pass, as sync v2 and JDBC do, without marking it down.
The four async writer-failover tests compared the connection with the RDS API's
writer right after the data-plane check, though their own comment says the control
plane lags. They now retry it with retry_until, as test_aurora_failover.py does.
The container conftest clears the async plugin service's host-availability cache before
each test, as tests/unit/conftest.py does, so a host one test marked unavailable is no
longer unavailable in the next. Two async simple read/write splitting tests now set
instance endpoints. The autocommit test turns verification off, so srw follows them
unchecked, and the docs say cluster endpoints may route to the wrong instance. The
incorrect-reader-endpoint test points both at the writer instance, because the writer
cluster endpoint it used as the read endpoint can resolve to a reader after a failover.
DbApiMethod.INIT_HOST_PROVIDER.method_name,
DbApiMethod.CONNECT.method_name,
DbApiMethod.FORCE_CONNECT.method_name,
DbApiMethod.CONNECTION_CURSOR.method_name,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this required?

# which can lag by tens of seconds to minutes post-failover.
assert await query_host_role_async(conn) == HostRole.WRITER
# RDS API lags behind the writer election, so we retry the check.
assert retry_until(lambda: aurora_utility.is_db_instance_writer(current_connection_id))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need this retry_util? I believe the RDS team pushed a fix that should have addressed the RDS API lag issue that we have been seeing?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants