Repository navigation
fix: select blue/green routings from the latest status and match them by host and port - #1360
Merged
Merged
Conversation
) Add unit tests that reproduce the two defects reported in #1352. test_blue_green_switchover_loops.py suspends a sync execute or connect while the switchover is IN_PROGRESS and then moves the status on. On main the call keeps re-selecting the stale suspend routing and never returns, even with a short bg_connect_timeout_ms; an async test checks that the aio loop still resumes. test_blue_green_routing_match.py and test_aio_blue_green_routing_match.py build routings with the endpoints the status provider builds. On main none of them match, because is_match compares with HostInfo.url and its trailing slash, so a connect during PREPARATION or POST goes to the host name instead of the substituted IP. Guards check that another host, port or role still doesn't match. The 8 reproduction tests are strict xfails, so the suite stays green on main. The fix for #1352 must remove the markers and update test_substitute_connect_routing_is_match_by_url, which asserts the current is_match behaviour.
… by host and port The sync connect and execute loops chose the next routing from the status captured before the loop, so a call suspended during IN_PROGRESS spun forever once the phase moved on. They now re-read the status, recompute the role and select from it, as the asyncio loops do. BaseRouting.is_match compared the endpoint with HostInfo.url, which ends in "/", so no routing matched. It now compares get_host_and_port().lower() in both engines. With routings matching, ADFS and Okta count as IAM in use for SubstituteConnectRouting, and the corresponding-host lookups ignore case. Fixes #1352.
JuanLeee
marked this pull request as ready for review
November 12, 2026 17:04
karenc-bq
approved these changes
Nov 12, 2026
The set now holds the iam, federated_auth and okta plugin codes, all of which authenticate with an IAM token, so name it for that.
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
After a Blue/Green switchover, a call that was suspended during
IN_PROGRESSnever returns:BlueGreenPlugin.executeandBlueGreenPlugin.connectspin at full CPU with no sleep, andbg_connect_timeout_msdoesn't end them. A new connection also never reaches the green host's IP address during POST, because no routing that names an endpoint ever matches the host it was built for.This PR makes the sync retry loops pick the next routing from the latest status, as the asyncio loops already do, and makes
BaseRouting.is_matchcompare the host and port in both engines. Two places thatis_matchmakes reachable for the first time needed changes too.The sync connect and execute loops in
BlueGreenPluginreadlatest_statusafter each routing, but chose the next routing fromself._bg_status, the status captured before the loop. Once the phase leftIN_PROGRESS,SuspendExecuteRoutingandSuspendConnectRoutingreturned at once, the stale table handed back the same routing, and the call repeated forever. The loops now store the latest status inself._bg_status, recompute the role from it, leave the loop when the host has no role, and select from the latest routings. The asyncio loops already did this.BaseRouting.is_matchcompared the routing's endpoint withhost_info.url.lower(), andHostInfo.urlends in/, so no routing built withhost,host:port,iporip:portmatched.SubstituteConnectRoutingand the other routings the status provider builds never applied, so a connect during PREPARATION or POST went to the old DNS name.is_matchnow compareshost_info.get_host_and_port().lower(), in both engines.With
is_matchworking,SubstituteConnectRoutingsends a connect to an IP address, and thefederated_authandoktaplugins reject an IP as the token host ("Invalid IAM host"). The sync routing only treatedIamAuthPluginas IAM in use, and the asyncio_IAM_PLUGIN_CODESheld onlyiam. Both now also count the federated and Okta plugins, so the routing sets the IAM host to the green host's name for them as it does foriam. The asyncio set is renamed_AUTH_PLUGIN_CODES.SuspendUntilCorrespondingHostFoundConnectRoutinglooked upcorresponding_hostswithhost_info.hostas given, but the keys are lower case, so a host written in upper case waited outbg_connect_timeout_msfor a host that was already known. The four lookups (two per engine) now usehost_info.host.lower().Behaviour changes
When every IAM host fails to log in at the green IP during POST,
SubstituteConnectRoutingraisesInProgressCantOpenConnection. Before this PR the routing never matched, so the connect went by name. Users ofiam, ADFS (federated_auth) and Okta (okta) can see this error. JDBC returns no connection here (fix: small fix for blue/green connect routing aws-advanced-jdbc-wrapper#1872), but with these loops that picks the same routing again from an unchanged status and repeats without pause (22,978 to 27,152 connect attempts in about 1 s when tried), and that is the defect this PR removes. So the raise stays.Without IAM, a login error from the connect to the substituted IP now fails the call; before, the connect went by name. JDBC catches it and returns no connection, which loops the same way (33,219 to 35,914 attempts in about 1 s when tried), so the error is raised here. With password or Secrets Manager auth, connecting by name would have raised the same error.
The sync loops recompute the role from the latest status, so a host whose role changes while the call waits gets the routings of its new role. This matches the asyncio loops and the issue; JDBC keeps the role it computed before the loop.
Python now differs from JDBC in two ways: ADFS and Okta count as IAM, and the corresponding-host lookup ignores case.
Not in scope
The comment in the asyncio connect loop in
aio/blue_green_plugin.pystill says re-selecting from the latest status avoids a busy loop that "the literal sync re-selection is prone to", which this PR makes untrue. The module docstring there still saysis_plugin_in_use(IamAuthPlugin)is approximated from thepluginsproperty. Both are left for a follow-up, since a comment-only change here wouldn't be covered by a test.A new asyncio connection keeps its own Blue/Green status store, so a second
AsyncAwsWrapperConnectionnever sees the status. That's a separate defect, left for its own issue.With
sslmode=verify-full(or MySQLssl_verify_identity), a connect to the substituted IP fails the certificate's host-name check, which is not a login error, so such connects may now raise during PREPARATION and POST. JDBC substitutes the IP the same way. This wasn't tested because it needs a server.Tests
The 8 new unit test cases (4 sync and 4 async: 2 each for the IAM host with ADFS and Okta, 1 each for a known upper-case host that must not wait, and 1 each for an upper-case host found on a later status) fail on the previous code. The 8
xfail(strict=True)markers on the existing reproduction tests are removed, andtest_substitute_connect_routing_is_match_by_urlbecomestest_substitute_connect_routing_is_match_by_host_and_port, because it asserted a match against"h:5432/".On the tests-only commit the reproduction tests fail for both defects while their guards pass; with this change they all pass. The same holds in a local Docker run against MySQL with a simulated switchover, for the sync code with the pure-Python and C-extension drivers. The asyncio upper-case and IP-routing fixes are covered by the unit tests only, because a second asyncio connection doesn't see the Blue/Green status (see Not in scope). Reverting any one of the 11 product hunks makes at least one of these tests fail.
mypy,flake8,isortand the unit tests pass on Python 3.12.15 and 3.10.22.On a real Aurora MySQL Blue/Green switchover (73 s), both clients running this change, with the pure-Python and C-extension drivers, wrote again 0.07 and 0.05 s after
SWITCHOVER_COMPLETED, after 3.70 and 3.49 s of downtime, and their first new connection went to the new writer's IP. A client without the fix never wrote again in the 370 s observed. A second switchover with the client order reversed gave 3.47 to 3.50 s for three clients running this change.No CHANGELOG entry, message key or docs change.
Fixes #1352.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.