fix(undo): unpin the idle-peer recycle floor and let the cleaner keep pace#57
Merged
Conversation
added 2 commits
July 15, 2026 19:07
…p pace TT lane (S3 pre-queue baseline forensics, 2026-07-15 leg-1 stop). Two arms, both mechanism, no tuning: H1 idle-unconstrained sentinel. A provably idle node (zero active transactions AND zero held snapshots, both proven in one ProcArrayLock scan) publishes UINT64_MAX as its undo-horizon report instead of its Lamport clock sample. The fold's time-axis min ignores the sentinel structurally, so a lone writer's recycle floor becomes its own local horizon (ms-scale lag) instead of trailing the idle peers' report cadence by 1-2 LMON ticks. At 12k tps that lag exceeded the whole TT slot pool (undo_segments_max_per_instance x 48 = 12,288 slots ~= 1s of allocation), so every pooled slot stayed younger than the floor: zero recyclable inventory, pool camped at the hard cap (11.3k x 53R9E), the "retention horizon pinned" oscillation. The sentinel is capability-gated per connection (new HELLO bit 0x100); an old receiver keeps getting the conservative clock sample, so the mixed-version monotonicity latch can never trip. A sentinel report still owes capability/presence/stability/epoch/freshness proof -- an idle peer that stops reporting stalls the fold exactly as before. A peer waking mid-report-window self-fences through the BELOW_HORIZON read_scn admissibility gate (fail-closed; the wire observe heals the next snapshot): rule 8.A polarity unchanged. Handler-side the sentinel stays OUTSIDE the same-epoch monotone base, so the first real report after waking compares against the last REAL accepted value. H2 pressure-driven continuous recycle. A cleaner pass that exhausts its segment batch while still making progress re-runs immediately instead of sleeping the recycle interval. Pure cadence: the "who may be recycled" decision layer is untouched; a pinned pass (zero progress) or a drained backlog falls back to the interval/stall-retry waits, so the loop cannot busy-spin. Default-GUC honesty (pinned by t/403 L10): the wake-race self-heal above covers snapshot ADMISSIBILITY only. A remote xid whose TT slot was lapped during the storm is NOT self-healing under the default cluster.crossnode_runtime_visibility = off -- node1's read fails closed 53R97 (cluster TT status unknown) with the verdict wire untouched. With the GUC on, the origin-verdict wire (cluster_runtime_visibility.c) resolves the same read through its COMMITTED_BELOW_HORIZON arm. Both behaviours are pinned as a RED->GREEN twin on one driver in t/403 L10. Evidence: t/403 RED on the pre-fix binary with this exact harness -- floor lag 4,122 slots vs a 768-slot pool, 12,352 TT hard-cap failures, the pinned oscillation in the log; GREEN post-fix -- lag_max ~200, zero hard-cap, recycling advances mid-phase, node1's sentinels observable, and a live snapshot on the peer still clamps the floor (protection leg). New TAP t/403 (2-node lone-writer storm + idle peer + wake legs), fold unit rows U17/U17b, undo dump key horizon_idle_sentinel_sent_count (t/214 71->72), HELLO cap ripple (test_cluster_ic), nightly shard stage7-s3-wave 401-403 (also closes the t/401/402 shard hole). Gates: unit clean 178/178, t/017+214+370+402+403 PASS, cluster_regress 13/13, PG 219/219, format/headers/scn-cmp/ges-mode/clog all 0. Spec: spec-5.22e-undo-cluster-retention-horizon.md, spec-3.12, spec-3.13.
…403 L10)
One driver, no cleanout bypass: a node0 xid whose TT slot is provably
lapped (>=2 pool rollovers, hard-floor assert) is read from node1
twice. RED twin under the default cluster.crossnode_runtime_visibility
= off: the read MUST fail closed 53R97 ("cluster TT status unknown")
with node1's rtvis_verdict_wire_count provably flat -- a succeeding
read is a hard fail, never retried, so the GREEN result cannot come
from an overlay / hint / delayed-cleanout side door. GREEN twin after
flipping ONLY the GUC (PGC_SUSET reload, both nodes): the same read
returns the exact terminal 64|2080 and node1's verdict counters move
(wire=64, below_horizon=64) -- every row resolved through the
origin-verdict COMMITTED_BELOW_HORIZON arm
(cluster_runtime_visibility.c).
cluster.tt_status_hint_emit_mode is disabled (PGC_SIGHUP) before the
fixture write so the V4 hint wire cannot install the terminal status
into node1's TT overlay at commit time: this deterministically
manufactures the "hint never arrived" state the resolve path exists
for (the outbound hint ring drops under storm load anyway) -- it
forces the mechanism under test, it does not weaken any assertion.
Known structural limit (registered follow-up, not this leg): with
cluster.xid_striping off (this rig) classify_ref's recycled-ref
derivation branch is unreachable -- only the fresh-ref AUTHORITATIVE
ask is served; a striping-on variant leg needs a PGC_POSTMASTER
re-provision.
Gates: t/403 33/33 x3 independent runs; L1-L9 assertions unweakened;
test-only diff.
Spec: spec-5.22e-undo-cluster-retention-horizon.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Evidence
Test plan
Spec: spec-5.22e-undo-cluster-retention-horizon.md