GTID master failover recovery + integration test (GTID failover 3/3)#453
Open
driv3r wants to merge 2 commits into
Open
GTID master failover recovery + integration test (GTID failover 3/3)#453driv3r wants to merge 2 commits into
driv3r wants to merge 2 commits into
Conversation
When the source master fails over, the binlog streamer previously died: a
non-timeout GetEvent error went straight to ErrorHandler.Fatal, and
go-mysql's syncer only reconnects to the same host. This adds opt-in,
GTID-only recovery that reconnects the whole run to the promoted writer.
Built on SourceRuntime: recovery is expressed as an in-process resume onto
a new source session rather than hand-repointing each consumer.
- SourceReconnector is the single seam between the streamer ("I lost the
source; here is the GTID set I must not fall behind") and the Ferry,
which owns the SourceRuntime. ferrySourceReconnector resolves the new
writer, then SourceRuntime.ReplaceValidated opens one connection,
validates it, and only publishes it to consumers if valid. Because every
source consumer already reads from the SourceRuntime, that single swap
repoints the entire run.
- validateFailoverTarget fails closed: the candidate must be a writer
(not read_only), have gtid_mode=ON, and its executed set must contain
everything already applied downstream. The applied set is the committed
streamed set plus any in-flight transaction's GTID (tracked from
GTIDEvent/XIDEvent), since in-flight rows may reach listeners before the
commit advances the streamed set. If a cutover stop target is already
recorded, the candidate must also contain it, or streaming would hang
waiting for GTIDs the new writer will never produce. Fail-closed
everywhere: a candidate that fails validation is never published, and an
undecodable in-flight GTID aborts recovery rather than validating against
an under-approximated set.
- On success the streamer rebuilds its binlog syncer against the promoted
writer and restarts StartSyncGTID from the last resumable set (replaying
the interrupted transaction). DB/DBConfig swaps are under connMu;
FlushAndStop reads currentDB() so cutover records the stop coordinate
from the new writer. On recovery exhaustion the Run loop fatals and
returns.
Config.MasterFailoverRecovery is validated (requires gtid mode + a
resolver) and wired onto the source streamer only; the target verifier
streams from the stable target and does not fail over.
Adds an end-to-end Ruby integration test that proves master failover recovery works, plus the fix that surfaced while writing it. Fix: bound go-mysql's same-host reconnect retries when failover recovery is enabled (MaxReconnectAttempts, default 3). Without this the binlog syncer retries the dead host forever and GetEvent only ever returns context timeouts, so our reconnect-to-a-new-host recovery never triggers. With failover disabled the behavior is unchanged (infinite same-host retries, as before). Tunable via MasterFailoverRecoveryConfig.SyncerMaxReconnectAttempts. Integration test (GTID-only; skips in file_position mode): - integrationferry wires MasterFailoverRecovery from GHOSTFERRY_FAILOVER_MASTER_PORT with a static resolver pointing at the promoted server. - A third MySQL (mysql-3, 29293, already defined in the compose files) is started/waited-for by start-mysql.sh and used as the promotion target. - FailoverHelper sets mysql-3 up as a GTID replica of the source (aligning gtid_purged so it replicates the real seed + writes), promotes it, and stops the source container to force a genuine unrecoverable disconnect. - The test fails the source over at cutover entry (row copy done, streamer live, no stop target yet), waits for the streamer to log a successful recovery, then asserts the promoted master and target are identical and that recovery was logged. Teardown restores the source container. db_helper gains an opt-in future_master_db (port 29293) that existing tests, which start only two servers, do not touch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Implements opt-in, GTID-only master failover recovery: when the source MySQL master fails over, Ghostferry reconnects the whole run to the promoted writer instead of dying. Includes an end-to-end Ruby integration test that proves it works.
PR 3 of 3 of the GTID failover rework:
SourceRuntime(Introduce SourceRuntime to own the source connection (GTID failover 1/3) #451).SourceRuntime(Migrate source consumers to SourceRuntime (GTID failover 2/3) #452).How
Commit 1 —
Add GTID master failover recoveryFailover is expressed as an in-process resume onto a new source session, not a hand-repoint of each consumer:
SourceReconnectoris the seam between the streamer ("I lost the source; here's the GTID set I must not fall behind") and the Ferry, which owns theSourceRuntime.ferrySourceReconnectorresolves the new writer, thenSourceRuntime.ReplaceValidatedopens one connection, validates it, and only publishes it if valid. Because every source consumer already reads from theSourceRuntime(PR Migrate source consumers to SourceRuntime (GTID failover 2/3) #452), that single swap repoints the entire run.validateFailoverTarget): the candidate must be a writer (not@@read_only), have@@GLOBAL.gtid_mode = ON, and its executed set must contain everything already applied downstream — the committed streamed set plus any in-flight transaction's GTID (tracked fromGTIDEvent/XIDEvent). If a cutover stop target is recorded, the candidate must contain it too, or streaming would hang. An undecodable in-flight GTID aborts recovery rather than validating against an under-approximated set. A candidate that fails validation is never published to consumers.StartSyncGTIDfrom the last resumable set (replaying the interrupted transaction). DB/DBConfig swaps are underconnMu;FlushAndStopreadscurrentDB()so cutover records the stop coordinate from the new writer. On recovery exhaustion the Run loop fatals and returns.Config.MasterFailoverRecoveryis validated (requires gtid mode + resolver) and wired onto the source streamer only.Commit 2 —
Add master failover integration test and bound syncer reconnectsMaxReconnectAttempts, default 3). Without this the syncer retries the dead host forever andGetEventonly ever returns context timeouts, so recovery never triggers. Behavior is unchanged when failover is disabled. Tunable viaSyncerMaxReconnectAttempts.file_positionmode, and skips if nodocker/podmanis available to stop the source container): setsmysql-3(already in the compose files) up as a GTID replica of the source, promotes it, and stops the source container to force a genuine unrecoverable disconnect. It fails over at cutover entry, waits for the streamer to log a successful recovery, then asserts the promoted master and target are identical.Testing
ReplaceValidatedpublish/reject, reconnector resolver-error/nil-candidate paths, in-flight GTID tracking, config validation.-raceclean.test/gosuite green in both modes; existing Ruby integration tests unaffected.go build+gofmtclean.