Repository navigation
Conversation
… host list provider, and failover covers cursor creation The async failover and read/write splitting plugins kept the static provider the wrapper built from the guessed dialect, so after the plugin service swapped in the Aurora provider they saw only the seed host: no readers, and writer failover that never reached the new writer. They now read the service's current provider, as sync does. Async failover also subscribes to Connection.cursor, as sync does, so a connection closed while a cursor is created starts failover. The replacement provider is armed for panic-mode writer discovery before its first refresh when the previous one had no probe. Async reader failover now only drops a reader it cannot reach from the pass, as sync v2 and JDBC do, without marking it down.
The four async writer-failover tests compared the connection with the RDS API's writer right after the data-plane check, though their own comment says the control plane lags. They now retry it with retry_until, as test_aurora_failover.py does. The container conftest clears the async plugin service's host-availability cache before each test, as tests/unit/conftest.py does, so a host one test marked unavailable is no longer unavailable in the next. Two async simple read/write splitting tests now set instance endpoints. The autocommit test turns verification off, so srw follows them unchecked, and the docs say cluster endpoints may route to the wrong instance. The incorrect-reader-endpoint test points both at the writer instance, because the writer cluster endpoint it used as the read endpoint can resolve to a reader after a failover.
karenc-bq
reviewed
Nov 14, 2026
| DbApiMethod.INIT_HOST_PROVIDER.method_name, | ||
| DbApiMethod.CONNECT.method_name, | ||
| DbApiMethod.FORCE_CONNECT.method_name, | ||
| DbApiMethod.CONNECTION_CURSOR.method_name, |
karenc-bq
approved these changes
Nov 14, 2026
karenc-bq
reviewed
Nov 14, 2026
| # which can lag by tens of seconds to minutes post-failover. | ||
| assert await query_host_role_async(conn) == HostRole.WRITER | ||
| # RDS API lags behind the writer election, so we retry the check. | ||
| assert retry_until(lambda: aurora_utility.is_db_instance_writer(current_connection_id)) |
Contributor
There was a problem hiding this comment.
Do we still need this retry_util? I believe the RDS team pushed a fix that should have addressed the RDS API lag issue that we have been seeing?
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Since #1292 (
a2ac1ada, 2026-10-28, after the 3.1.0 release), the first async connection to an Aurora instance endpoint, an IP or a custom domain starts onAsyncStaticHostListProvider, built from the guessed dialect, and the plugin service swaps in the Aurora provider once the dialect is confirmed.AsyncFailoverPluginandAsyncReadWriteSplittingPluginkeep the provider they were built with, so on that connection writer failover can keep reconnecting to the old writer, as everygdb_failoverwriter-failover test onmaindoes, andset_read_only(True)stays on the writer. The replacement provider also never gets the probe its topology monitor needs for panic mode. Two older async gaps sit on the failover path: an error raised while the wrapper creates the driver cursor reaches the application raw, and reader failover marks every reader it couldn't reach as unavailable for 5 minutes, for every async connection in the process. The Integration Tests workflow onmainhas failed on the async failover and read/write splitting tests since that commit (run 40751906508) and still does at the base of this branch (run 43391980255); the integration conftest clears the endpoint dialect cache before each test, so every test takes the first-connection path.This PR makes the async plugins read the plugin service's provider, as the sync plugins already do, subscribes async failover to cursor creation as sync failover is, and drops the reader marks as sync failover v2 and JDBC's failover2 do. It also removes three timing and state dependencies from the async integration tests.
AsyncFailoverPluginandAsyncReadWriteSplittingPluginstored the providerbuild_async_pluginshanded them and refreshed through it, in the writer loopAsyncRetryUtil.get_writer_connectionand in read/write splitting's switches. For a connection whose guessed dialect isRDS_PG,RDS_MYSQL,PGorMYSQL, that provider is the static one and returns only the seed host. While the topology still named the old writer, writer failover kept reconnecting to it and ended with "[Failover] Unable to establish SQL connection to the writer instance.", which fails thegdb_failoverwriter-failover tests in every job of run 43391980255, andset_read_only(True)logged "[ReadWriteSplittingPlugin] A reader instance was requested via set_read_only, but there are no readers in the host list" and stayed on the writer. Both classes now read_host_list_providerfromplugin_service.host_list_providerthrough a read-only property, andAsyncGdbFailoverPluginandAsyncGdbReadWriteSplittingPlugininherit it. Cluster endpoints were unaffected, because their guess is already Aurora.AsyncPluginServiceImpl._update_host_list_providercopied the panic-mode probe onto the replacement provider only when the previous provider had one, andAsyncStaticHostListProvidernever does. The Aurora provider that replaced a static one therefore built its topology monitor without a probe, so_should_panicreturnedFalseand the monitor never searched the hosts for the new writer once its own monitoring connection was gone. The service now builds the probe withbuild_probe_hostwhen the previous provider has none and sets it before the replacement's firstforce_refresh_host_list, as the wrapper already does for the provider it builds first.AsyncFailoverPlugin._SUBSCRIBEDdidn't listConnection.cursor. The async wrapper creates the driver cursor on the first cursor operation and sends that call through the plugin pipeline, so on a broken connection the driver's error reached the application and failover never started. Run 43391980255 shows it on Aurora PostgreSQL:test_fail_from_reader_to_writer_asyncfailed withpsycopg.OperationalError"the connection is closed", raised from_resolve_target_cursor.CONNECTION_CURSORis now subscribed, so the error starts failover and the caller getsFailoverSuccessError, orTransactionResolutionUnknownErrorinside a transaction. The gap dates from feat: asyncio support (async wrapper, plugins, and SQLAlchemy dialects) #1257 and is in 3.1.0.AsyncFailoverPlugin._failover_reader_implmarked each reader candidate it couldn't openUNAVAILABLEthroughplugin_service.set_availability. The mark goes intoAsyncPluginServiceImpl._host_availability_expiring_cache, a class-level cache shared by every async connection in the process for 5 minutes, and the default random selector picks only available hosts, so a reader that was down for a moment during a failover stayed out of selection afterwards. A failed candidate is now only dropped from the current pass; the failed current host is still markedUNAVAILABLEat the start of_do_failover. The mark dates from feat: asyncio support (async wrapper, plugins, and SQLAlchemy dialects) #1257, but in 3.1.0, and again from refactor: revert the alias removal and async topology move #1328 (2408eea8, 2026-10-26) until refactor: identify hosts by host id and host-and-port instead of aliases #1297 (20d619b3, 2026-11-03), marks were written by alias and read by URL, so it changes host selection only onmain.retry_until(lambda: aurora_utility.is_db_instance_writer(...))first, as their sync twins do, and keep both checks.srwtests now pointsrw_read_endpointandsrw_write_endpointat instance endpoints, because thesrwdocs say a cluster endpoint may connect to the previous writer right after a failover. The autocommit test turnssrw_verify_new_connectionsoff, sosrwuses its endpoints as given. The incorrect-reader-endpoint test keeps verification on, and in one Aurora run its writer cluster endpoint, used as the read endpoint, resolved to a reader. Both tests keep their asserts.Behaviour changes
FailoverSuccessError,TransactionResolutionUnknownErrororFailoverFailedErrorinstead of the driver's error. This differs from 3.1.0.Not in scope
The first three are separate parity changes, so they're left for later.
Cursor.callproc, theautocommitsetter orset_read_onlystill don't start failover.UNAVAILABLEin three places sync failover v2 doesn't: the original writer in the reader pass, and the topology writer and each reader inAsyncRetryUtil.srwtests still use cluster endpoints.UsingTheFailover2Plugin.mdstill says async injects theAsyncHostListProviderat plugin construction, which is now only the fallback; the page needs its own docs update.retry_untilsleeps synchronously, so the four async failover tests block their event loop for up to 5 minutes while they wait; the tests have no async retry helper.Tests
The 5 new unit tests (3 for failover: cursor creation, writer convergence and reader candidates; 1 for the panic-mode probe; 1 for read/write splitting) fail on
mainand pass with this change. With the head's four changed unit test files onmain's source they give 5 failed and 186 passed, none on an import, and 191 passed at the head, on Python 3.12.pytest tests/unit -Werrorpasses 2565 tests on Python 3.10 and 3.12, with mypy, flake8 and isort clean.The repo's Gradle integration harness ran on Python 3.12 against fresh Aurora clusters (3 instances) in us-east-2. At this tree
test-pg-aurora-asynchad 98 passed and 0 failed, andtest-mysql-aurora-asynchad 128 passed and 0 failed with 2 setup errors in the Django and SQLAlchemy IAM tests, because the lanes run without IAM ([IamAuthPlugin] Property "user" is None or empty). The two async least-connections tests passed on 5 instances, andtest-dockerhad 120 passed.test-pg-auroraandtest-mysql-auroraran on the tree before item 7, which touches only an async test module: 134 passed with 1 custom endpoint failure that also fails onmain, and 154 passed with the same 2 IAM errors.TestReadWriteSplittingAsyncon Aurora PostgreSQL passed 54 of 54 twice before item 7, and at a commit without item 4 and item 6,test_failover_reader_to_writer__switch_read_only_async[plugin_config0-failover_v2,host_monitoring_v2]failed both times. Python 3.11, 3.13 and 3.14, Aurora with 1 or 2 instances (or 5 outside the least-connections tests), the IAM and telemetry tests, Multi-AZ and Blue/Green weren't run locally. The Integration Tests workflow on this branch (run 43642588024) passed 3 of its 8 LTS jobs (3.14 mysql, 3.11 pg and 3.14 pg), wheremain's run 43391980255 at the base passed none. Its 9 failed cases are 8 distinct tests. Seven are sync tests this change doesn't touch (test_pooled_connection__failoverin two configs,test_failover_to_new_writer__switch_read_only, a custom endpoint test and threeTestAuroraFailoverwriter-failover tests), and each also failed in a recentmainrun (40422400519, 43184367058 or 43391980255). The one async failure,test_pooled_connection__failover_in_transaction_async[plugin_config0]on 3.12 pg with the writer id unchanged after failover (assert '<id>' != '<id>'), also failed in run 43391980255, which had 32 distinct async failures. No CHANGELOG entry, message key or docs change.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.