You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
fix(replication): prevent excessive wait times after deploy (fixes#131) (#137)
* fix(replication): prevent excessive wait times after deploy (fixes#131)
Two issues caused multi-minute delays on 'deploy replication' in v2.4.0:
1. wait_until_wsrep_ready spun 120s per node on non-Galera MySQL
(commit a2caf94 added wait_wsrep_after_start but didn't guard
against vanilla MySQL where wsrep_ready does not exist)
2. wait_until_replica_ready defaulted to max_attempts=60 (60s per
replica), while the original reporter found 5s was sufficient
Fixes:
- wait_until_wsrep_ready: check if the wsrep_ready status variable
exists before polling; return immediately on non-Galera nodes
- wait_until_replica_ready: reduce default max_attempts from 60 to
20 (≤20s total wait per replica)
- Update call sites in init_slaves.gotxt and init_slaves_84.gotxt
Tests:
- TestInitSlavesTemplates_IncludeReplicaReadyWait: also check
timeout warning message; fix pre-existing missing template fields
- TestWaitWsrepAfterStart_DoesNotBlockNonGalera: verifies the
wait_wsrep_after_start template renders and uses || true
- TestSbInclude_WaitUntilWsrepReady_DetectsNonGalera: verifies
the wsrep existence guard exists in sb_include
- TestSbInclude_WaitUntilReplicaReady_BoundedWait: verifies
max_attempts default is 20 (≤30s budget)
* feat(ci): add time-bounded deploy test for issue #131 regression
Adds test/deploy-time-budget.sh, a standalone bash test that:
- Deploys replication (1 slave) with real MySQL binaries
- Measures wall-clock time against MAX_DEPLOY_SECONDS (45s)
- Verifies the replica serves queries immediately after deploy
(catches the #131 race condition where deploy returned before
replica was ready)
- Checks for server errors in the error log
Adds a deploy-time-budget CI job in integration_tests.yml that:
- Downloads and unpacks MySQL 8.4.4
- Runs the time-budget test with MAX_DEPLOY_SECONDS=45
- Fails if deploy exceeds the budget (catches regressions like
the 120s wsrep spin or 60s replica wait from v2.4.0)
* Fix wsrep detection for fresh Galera nodes and fix deploy-time-budget -n flag
- sb_include.gotxt: connect as root (no password) via socket for wsrep
detection guard, so it works before msandbox user exists
- deploy-time-budget.sh: use -n 2 (total nodes, not slaves) for
replication with 1 slave
* fix: use root socket for wsrep polling, not /use
wait_until_wsrep_ready now uses root via socket for both detection
and polling, since msandbox user does not exist before load_grants.
This eliminates the 120s wasted polling loop per node on fresh
Galera/PXC nodes while still correctly waiting for wsrep_ready=ON
before grants/DML.
* fix tests: template flavor checks must account for shell-level if/else
The start/stop/use templates use shell if blocks (not Go template
{{if}}) so both branches' literal strings appear in rendered output.
Tests now verify the correct default AND the presence of the flavor
conditional, rather than asserting the absence of the other flavor's
binary string.
* fix: harden wait_until_wsrep_ready connect and client resolution
- Retry root socket connect before Galera detection so a not-yet-ready
server is not mistaken for non-Galera (empty query output).
- Resolve client as mariadb/mysql by FLAVOR (CLIENT_BASEDIR then BASEDIR).
- Match wsrep_ready ON specifically; keep a modest connect budget (30s).
* fix: address CodeRabbit findings on deploy-time-budget CI
- workflow: persist-credentials: false on the executable-download job
so a compromised MySQL binary cannot read the job token (artipacked).
- deploy-time-budget.sh: send version-discovery diagnostics to stderr
and branch on exit status so a failed discovery hits the skip path
instead of leaking text into VERSION.
- Require MySQL >= 8.4 (was >= 8.0) to match the stated #131 target.
- Clean stale rsandbox_* fixtures before deploying so a leftover
ready topology cannot make the test pass spuriously.
* fix: gate wsrep non-Galera short-circuit on probe exit status
Only treat an empty wsrep_ready probe as "non-Galera" when the SHOW
STATUS query itself succeeds. A failed probe now falls through to the
polling loop instead of bypassing the wsrep_ready wait (CodeRabbit).
* fix: replica-ready probe must authenticate (clear leaked NOPASSWORD)
wait_until_replica_ready connects as the sandbox user, but init_slaves
exports NOPASSWORD=1 for the root steps and that leaked into the probe:
'use' then drops the password, so SELECT 1 fails with Access denied even
once the user exists, and the poll burned its full ~20s budget on every
replication deploy. Clear NOPASSWORD for the probe so it authenticates
and returns as soon as the replica is actually ready (~1-2s).
Copy file name to clipboardExpand all lines: sandbox/templates/single/sb_include.gotxt
+68-7Lines changed: 68 additions & 7 deletions
Original file line number
Diff line number
Diff line change
@@ -70,13 +70,70 @@ function check_output
70
70
# Wait until wsrep_ready is ON for Galera/PXC nodes.
71
71
# This prevents grants and user DML from running before the node
72
72
# has joined the cluster.
73
+
# On non-Galera servers the status variable does not exist and
74
+
# this function returns immediately (no delay) once the server accepts
75
+
# connections.
76
+
# Detection and polling connect as root via socket (no password),
77
+
# because the sandbox user (msandbox) may not exist yet on fresh
78
+
# nodes (load_grants has not run).
79
+
# Connect failures are NOT treated as non-Galera: the pid file can
80
+
# appear before the server accepts connections, so we retry until
81
+
# root can connect, then check whether wsrep_ready exists.
73
82
function wait_until_wsrep_ready
74
83
{
75
-
local use_cmd=${1:-$SBDIR/use}
76
-
local max_attempts=${2:-60}
77
-
local sleep_sec=${3:-2}
84
+
local max_attempts=${1:-60}
85
+
local sleep_sec=${2:-2}
86
+
local mysql_cmd=""
87
+
local clients
88
+
if [ "$FLAVOR" = "mariadb" ]; then
89
+
clients="mariadb mysql"
90
+
else
91
+
clients="mysql mariadb"
92
+
fi
93
+
local d c
94
+
for d in "$CLIENT_BASEDIR/bin" "$BASEDIR/bin"; do
95
+
for c in $clients; do
96
+
if [ -x "$d/$c" ]; then
97
+
mysql_cmd="$d/$c"
98
+
break 2
99
+
fi
100
+
done
101
+
done
102
+
if [ -z "$mysql_cmd" ]; then
103
+
echo "WARNING: no mysql/mariadb client found under $CLIENT_BASEDIR/bin or $BASEDIR/bin" >&2
104
+
return 1
105
+
fi
106
+
# Wait until root can connect via socket. Empty query output before
107
+
# connect succeeds must not be treated as "non-Galera".
108
+
# Keep this budget separate and modest: pid-file can appear slightly
109
+
# before the server accepts connections, but not for minutes.
110
+
local connect_attempts=30
111
+
local connect_sleep=1
112
+
local connected=0
113
+
local i
114
+
for i in $(seq 1 $connect_attempts); do
115
+
if $mysql_cmd --no-defaults -S "$SOCKET_FILE" -u root -BN -e "SELECT 1" >/dev/null 2>&1; then
116
+
connected=1
117
+
break
118
+
fi
119
+
sleep $connect_sleep
120
+
done
121
+
if [ "$connected" -ne 1 ]; then
122
+
echo "WARNING: cannot connect as root via socket after $((connect_attempts * connect_sleep))s" >&2
123
+
return 1
124
+
fi
125
+
# Connected: a SHOW STATUS that succeeds with no rows means wsrep_ready
126
+
# does not exist (non-Galera) -> short-circuit. Only the successful-empty
127
+
# case may bypass the wait: if the probe itself fails, fall through to the
128
+
# polling loop so a transient error cannot skip the wsrep_ready gate.
129
+
local wsrep_check
130
+
if wsrep_check=$($mysql_cmd --no-defaults -S "$SOCKET_FILE" -u root -BN -e "SHOW STATUS LIKE 'wsrep_ready';" 2>/dev/null); then
131
+
if [ -z "$wsrep_check" ]; then
132
+
return 0
133
+
fi
134
+
fi
78
135
for i in $(seq 1 $max_attempts); do
79
-
if $use_cmd -BN -e "SHOW STATUS LIKE 'wsrep_ready';" 2>/dev/null | grep -qi 'ON'; then
136
+
if $mysql_cmd --no-defaults -S "$SOCKET_FILE" -u root -BN -e "SHOW STATUS LIKE 'wsrep_ready';" 2>/dev/null | grep -qiE 'wsrep_ready[[:space:]]+ON'; then
80
137
return 0
81
138
fi
82
139
sleep $sleep_sec
@@ -90,14 +147,18 @@ function wait_until_wsrep_ready
90
147
# initialization to prevent the race where the first client command on a
91
148
# just-started replica fails with "Access denied" or connection errors.
0 commit comments