Skip to content

fix: select blue/green routings from the latest status and match them by host and port - #1360

Merged
JuanLeee merged 4 commits into
mainfrom
fix/bg-1352-switchover-hang
Nov 13, 2026
Merged

JuanLeee merged 4 commits into
mainfrom
fix/bg-1352-switchover-hang

Conversation

@JuanLeee

@JuanLeee JuanLeee commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Description

After a Blue/Green switchover, a call that was suspended during IN_PROGRESS never returns: BlueGreenPlugin.execute and BlueGreenPlugin.connect spin at full CPU with no sleep, and bg_connect_timeout_ms doesn't end them. A new connection also never reaches the green host's IP address during POST, because no routing that names an endpoint ever matches the host it was built for.

This PR makes the sync retry loops pick the next routing from the latest status, as the asyncio loops already do, and makes BaseRouting.is_match compare the host and port in both engines. Two places that is_match makes reachable for the first time needed changes too.

  1. The sync connect and execute loops in BlueGreenPlugin read latest_status after each routing, but chose the next routing from self._bg_status, the status captured before the loop. Once the phase left IN_PROGRESS, SuspendExecuteRouting and SuspendConnectRouting returned at once, the stale table handed back the same routing, and the call repeated forever. The loops now store the latest status in self._bg_status, recompute the role from it, leave the loop when the host has no role, and select from the latest routings. The asyncio loops already did this.

  2. BaseRouting.is_match compared the routing's endpoint with host_info.url.lower(), and HostInfo.url ends in /, so no routing built with host, host:port, ip or ip:port matched. SubstituteConnectRouting and the other routings the status provider builds never applied, so a connect during PREPARATION or POST went to the old DNS name. is_match now compares host_info.get_host_and_port().lower(), in both engines.

  3. With is_match working, SubstituteConnectRouting sends a connect to an IP address, and the federated_auth and okta plugins reject an IP as the token host ("Invalid IAM host"). The sync routing only treated IamAuthPlugin as IAM in use, and the asyncio _IAM_PLUGIN_CODES held only iam. Both now also count the federated and Okta plugins, so the routing sets the IAM host to the green host's name for them as it does for iam. The asyncio set is renamed _AUTH_PLUGIN_CODES.

  4. SuspendUntilCorrespondingHostFoundConnectRouting looked up corresponding_hosts with host_info.host as given, but the keys are lower case, so a host written in upper case waited out bg_connect_timeout_ms for a host that was already known. The four lookups (two per engine) now use host_info.host.lower().

Behaviour changes

  1. When every IAM host fails to log in at the green IP during POST, SubstituteConnectRouting raises InProgressCantOpenConnection. Before this PR the routing never matched, so the connect went by name. Users of iam, ADFS (federated_auth) and Okta (okta) can see this error. JDBC returns no connection here (fix: small fix for blue/green connect routing aws-advanced-jdbc-wrapper#1872), but with these loops that picks the same routing again from an unchanged status and repeats without pause (22,978 to 27,152 connect attempts in about 1 s when tried), and that is the defect this PR removes. So the raise stays.

  2. Without IAM, a login error from the connect to the substituted IP now fails the call; before, the connect went by name. JDBC catches it and returns no connection, which loops the same way (33,219 to 35,914 attempts in about 1 s when tried), so the error is raised here. With password or Secrets Manager auth, connecting by name would have raised the same error.

  3. The sync loops recompute the role from the latest status, so a host whose role changes while the call waits gets the routings of its new role. This matches the asyncio loops and the issue; JDBC keeps the role it computed before the loop.

  4. Python now differs from JDBC in two ways: ADFS and Okta count as IAM, and the corresponding-host lookup ignores case.

Not in scope

  1. The comment in the asyncio connect loop in aio/blue_green_plugin.py still says re-selecting from the latest status avoids a busy loop that "the literal sync re-selection is prone to", which this PR makes untrue. The module docstring there still says is_plugin_in_use(IamAuthPlugin) is approximated from the plugins property. Both are left for a follow-up, since a comment-only change here wouldn't be covered by a test.

  2. A new asyncio connection keeps its own Blue/Green status store, so a second AsyncAwsWrapperConnection never sees the status. That's a separate defect, left for its own issue.

  3. With sslmode=verify-full (or MySQL ssl_verify_identity), a connect to the substituted IP fails the certificate's host-name check, which is not a login error, so such connects may now raise during PREPARATION and POST. JDBC substitutes the IP the same way. This wasn't tested because it needs a server.

Tests

The 8 new unit test cases (4 sync and 4 async: 2 each for the IAM host with ADFS and Okta, 1 each for a known upper-case host that must not wait, and 1 each for an upper-case host found on a later status) fail on the previous code. The 8 xfail(strict=True) markers on the existing reproduction tests are removed, and test_substitute_connect_routing_is_match_by_url becomes test_substitute_connect_routing_is_match_by_host_and_port, because it asserted a match against "h:5432/".

On the tests-only commit the reproduction tests fail for both defects while their guards pass; with this change they all pass. The same holds in a local Docker run against MySQL with a simulated switchover, for the sync code with the pure-Python and C-extension drivers. The asyncio upper-case and IP-routing fixes are covered by the unit tests only, because a second asyncio connection doesn't see the Blue/Green status (see Not in scope). Reverting any one of the 11 product hunks makes at least one of these tests fail. mypy, flake8, isort and the unit tests pass on Python 3.12.15 and 3.10.22.

On a real Aurora MySQL Blue/Green switchover (73 s), both clients running this change, with the pure-Python and C-extension drivers, wrote again 0.07 and 0.05 s after SWITCHOVER_COMPLETED, after 3.70 and 3.49 s of downtime, and their first new connection went to the new writer's IP. A client without the fix never wrote again in the 370 s observed. A second switchover with the client order reversed gave 3.47 to 3.50 s for three clients running this change.

No CHANGELOG entry, message key or docs change.

Fixes #1352.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

)

Add unit tests that reproduce the two defects reported in #1352.
test_blue_green_switchover_loops.py suspends a sync execute or connect while the switchover is IN_PROGRESS and then moves the status on.
On main the call keeps re-selecting the stale suspend routing and never returns, even with a short bg_connect_timeout_ms; an async test checks that the aio loop still resumes.
test_blue_green_routing_match.py and test_aio_blue_green_routing_match.py build routings with the endpoints the status provider builds.
On main none of them match, because is_match compares with HostInfo.url and its trailing slash, so a connect during PREPARATION or POST goes to the host name instead of the substituted IP.
Guards check that another host, port or role still doesn't match.
The 8 reproduction tests are strict xfails, so the suite stays green on main.
The fix for #1352 must remove the markers and update test_substitute_connect_routing_is_match_by_url, which asserts the current is_match behaviour.
… by host and port

The sync connect and execute loops chose the next routing from the status captured before the loop, so a call suspended during IN_PROGRESS spun forever once the phase moved on. They now re-read the status, recompute the role and select from it, as the asyncio loops do.

BaseRouting.is_match compared the endpoint with HostInfo.url, which ends in "/", so no routing matched. It now compares get_host_and_port().lower() in both engines.

With routings matching, ADFS and Okta count as IAM in use for SubstituteConnectRouting, and the corresponding-host lookups ignore case.

Fixes #1352.
@JuanLeee
JuanLeee marked this pull request as ready for review November 12, 2026 17:04
The set now holds the iam, federated_auth and okta plugin codes, all of
which authenticate with an IAM token, so name it for that.
@JuanLeee
JuanLeee merged commit 98a5906 into main Nov 13, 2026
9 checks passed
@JuanLeee
JuanLeee deleted the fix/bg-1352-switchover-hang branch November 13, 2026 08:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Blue/Green plugin: call hangs after switchover; IP substitution never applied

2 participants