Repository navigation
Do not deadlock when instant DDL completes the migration - #1796
d-bytebase wants to merge 2 commits into
Conversation
A migration that `--attempt-instant-ddl` completes never returns. Three
goroutines form a cycle:
- the changelog listener parks on the unbuffered `GhostTableMigrated`
send, which the instant-DDL path returns without ever reading;
- `notifyListeners` holds `listenersMutex` across that synchronous
call, so `shouldDecodeRowsEvent` — called on the binlog parser's
goroutine since github#1699 — stalls on the same mutex;
- `BinlogSyncer.Close`, which `finalCleanup` calls, waits for that
parser goroutine.
`Migrate()` therefore blocks until the migration context is cancelled.
`notifyListeners` now collects the matching listeners under the mutex and
invokes them after releasing it, and `ghostTableMigrated` gets a one-slot
buffer, since the changelog carries exactly one such state per migration.
`--attempt-instant-ddl` defaults to off and the short-circuit had no test,
which is why this went unnoticed; it was found by a downstream consumer
that enables the flag for every migration.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Instant-migration cleanup can leak replication connections because the cutover flag remains unset, while the deadlock regression test can leave a blocked worker holding locks and connections that disrupt subsequent tests.
2 open issues
What changed in this PR
Fixes a deadlock that prevents gh-ost from returning after a successful instant-DDL migration.
Changes:
- Invoke matching binlog listeners after releasing the listener mutex.
- Buffer the migration-completion signal so an unread notification does not block the listener.
- Add an integration test that checks migration completion within a timeout and verifies the schema change.
| File | Description |
|---|---|
go/logic/streamer.go |
Separates listener lookup under the mutex from callback execution. |
go/logic/migrator.go |
Adds a one-slot buffer to the migration-completion channel. |
go/logic/migrator_test.go |
Adds regression coverage for instant-DDL migration completion. |
Review effort: Max (auto)
Two follow-ups from review. `canStopStreaming` reads `CutOverCompleteFlag`, which the instant-DDL path left unset. `StreamEvents` therefore treated the close that `finalCleanup` performs as an unexpected error and reconnected after its sleep, leaking a replication connection and goroutine past the point where `Migrate()` had already returned nil. Set the flag once the instant DDL succeeds, before `finalCleanup`, as the revert path already does. `TestMigrateInstantDDLReturns` now cancels the migration context and waits for the worker before failing. `Require().Fail` unwinds only the test goroutine, so on a regression `Migrate()` would stay blocked and hold its connections and locks on the test table for the rest of the suite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ritojoao
left a comment
There was a problem hiding this comment.
Thanks for the detailed write-up, it made the cycle easy to follow.
I hit this while working on #1773 / #1805, so I can confirm it from the outside:
- On
master(3b904f2), a test that runsMigrate()withAttemptInstantDDLand anADD COLUMNnever returns. The package timed out after 560s with the stack parked inMigrate(). - With this branch checked out, the same test returns in well under a second. The whole
TestMigratorsuite passes (go test -count=1 -run TestMigrator ./go/logic, about 112s, testcontainersmysql:8.0.42), andgofmt -l,go vet ./go/logicandgo build ./...are clean.
A few notes from reading the diff, none blocking:
notifyListeners: droppinglistener := listeneris safe here, sincego.modis on Go 1.25 and the loop variable is per iteration. Taking a snapshot of the matched listeners also looks safe, becauseAddListeneris only called before streaming starts.- The one-slot buffer on
ghostTableMigratedmatches the singleGhostTableMigratedwrite ininitiateApplier. SettingCutOverCompleteFlagon the instant path, as the revert path already does, also looks right to me. - Out of scope for this PR, but nearby: on the instant path,
if err := mgtr.finalCleanup(); err != nil { return nil }swallows the cleanup error. It may be worth a follow-up.
LGTM from my side. #1805 builds on this one: it drops the _gho table that the instant path leaves behind, and its test can only run once this lands.

Related issue: #1795
Description
A migration that
--attempt-instant-ddlcompletes never returns. The ALTER isapplied,
Success! table ... migrated instantlyis logged, and thengh-osthangs until it is killed.
Migrate()short-circuits on that path — on success it callsfinalCleanup(),runs the
OnSuccesshook and returns, so it never reads<-mgtr.ghostTableMigrated(migrator.go#L534-L552).
Three goroutines then form a cycle:
The changelog listener parks on the
GhostTableMigratedsend inonChangelogStateEvent.ghostTableMigratedis unbuffered and this pathnever reads it. The send is wrapped in
base.SendWithContext, which onlyunblocks on context cancellation — and nothing cancels the context here,
precisely because
Migrate()has not returned. The listener runs fromnotifyListeners, which heldlistenersMutexacross that synchronous call,so the mutex stayed held for as long as the send was parked.
The binlog parser goroutine (go-mysql's
BinlogSyncer.onStream) callsEventsStreamer.shouldDecodeRowsEventthroughRowsEventDecodeFuncandblocks taking that same
listenersMutex.BinlogSyncer.Close(), reached fromfinalCleanup()→EventsStreamer.Close(), ends inb.wg.Wait()and waits for that parsergoroutine, which is never coming back.
Step 2 is what made the cycle reachable.
shouldDecodeRowsEventwas introducedby #1699 ("Don't parse binlog events of unrelated tables", first released in
v1.1.11); it put
listenersMutexon the binlog parser's goroutine, so a listenerparked while holding that mutex now stalls the parser directly, and with it
BinlogSyncer.Close().--attempt-instant-ddldefaults to off and the short-circuit had no testcoverage, which is presumably why this has gone unnoticed. It was found by a
downstream consumer that passes the flag for every migration, where every ALTER
MySQL could apply instantly hung.
The change
go/logic/streamer.go:notifyListenerscollects the matching listeners underlistenersMutexand invokes them after releasing it. A synchronous listener canblock indefinitely; the mutex is only needed for the lookup.
go/logic/migrator.go:ghostTableMigratedgets a one-slot buffer. Thechangelog carries exactly one
GhostTableMigratedstate per migration — it iswritten once, in
initiateApplier— so one slot is enough, and the send nolonger parks when nobody reads it.
Either of those two on its own clears the hang. Both are kept because they fix
different halves of it: the buffer stops the listener parking at all, and scoping
the mutex stops any blocked listener from stalling the binlog parser and
BinlogSyncer.Close().go/logic/migrator.go: the instant-DDL path also setsCutOverCompleteFlagbefore
finalCleanup(), as the revert path already does.canStopStreamingreads that flag, so without it
StreamEventstreated the close thatfinalCleanupperforms as an unexpected error and reconnected after its sleep,leaking a replication connection and goroutine past the point where
Migrate()had already returned nil. This only became reachable once the migration stopped
hanging; it was caught in review.
Test
TestMigrator/TestMigrateInstantDDLReturnssetsAttemptInstantDDLwith anADD COLUMNalter and fails ifMigrate()has not returned within 60s, thenasserts the column is actually there. On timeout it cancels the migration context
and waits for the worker first, so a regression does not leave a blocked
Migrate()holding connections and locks for the rest of the suite.With the two production changes reverted and only the test applied:
The subtest takes 60.11s rather than the full 60s + 30s bound, i.e. the cancel
unwinds the cycle and the worker returns in ~0.1s.
With the full change:
go test -count=1 -v -race -covermode=atomic ./go/...passes in all six packageswith no data races, and
script/cibuildis clean.script/cibuildreturns with no formatting errors, build errors or unit test errors.