[Bugfix][Store] Retry HA stale-handle cleanup oplog persist on transient backpressure - #27
Closed
Connor-Matthew wants to merge 0 commit into
Closed
[Bugfix][Store] Retry HA stale-handle cleanup oplog persist on transient backpressure#27Connor-Matthew wants to merge 0 commit into
Connor-Matthew wants to merge 0 commit into
Conversation
Connor-Matthew
changed the base branch from
main
to
upstream-pr/store-retry-ha-cleanup-oplog
August 14, 2026 10:19
Connor-Matthew
force-pushed
the
store/retry-ha-cleanup-oplog
branch
from
August 14, 2026 10:30
2c3c6d5 to
e58e966
Compare
Connor-Matthew
changed the base branch from
upstream-pr/store-retry-ha-cleanup-oplog
to
main
August 14, 2026 10:30
Connor-Matthew
force-pushed
the
store/retry-ha-cleanup-oplog
branch
from
August 14, 2026 12:35
e58e966 to
9e554a5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
In HA mode (
enable_ha_ && enable_oplog_),ClearInvalidHandlespersistsstale-handle cleanup through the oplog writer before removing metadata.
Today a single transient failure makes the persist call give up
immediately:
TASK_PENDING_LIMIT_EXCEEDED— the batch writer's slots are momentarilyfull while it seals the current batch; capacity frees up within
milliseconds.
UNAVAILABLE_IN_CURRENT_STATUS— the writer is briefly rejecting writeswhile its
write_batchretries against the KV backend.When that happens, the stale handles / last-replica metadata survive until
the next cleanup sweep, so invalid client handles linger longer than
necessary and (for the last-replica path) removed objects stay visible in
the metadata map despite the client being gone.
This PR wraps both oplog persist calls in
ClearInvalidHandleswith abounded exponential-backoff retry (
RetryOplogPersist):TASK_PENDING_LIMIT_EXCEEDED: retry up to 10 attempts, backoffmin(2^attempt, 16ms)— the writer frees slots imminently, waiting paysoff.
UNAVAILABLE_IN_CURRENT_STATUS: retry up to 5 attempts with the samebackoff — recovery depends on the KV backend, so bail out earlier
instead of spinning on a persistent outage.
Control flow is unchanged: if the persist still fails after retries,
the key is skipped exactly as before and cleanup is retried on the next
sweep. No fallback semantics are added or removed.
Module
mooncake-transfer-engine)mooncake-store)mooncake-reshard)mooncake-ep)mooncake-pg)mooncake-integration)mooncake-p2p-store)mooncake-wheel)mooncake-common)mooncake-rl)Type of Change
How Has This Been Tested?
Test commands:
Test results:
The change is a pure retry wrapper around existing calls: the success path
is byte-identical to before, and the failure path after exhausting retries
returns the same error the caller already handles (skip key, retry next
sweep). Formatting verified with
clang-format20.1.8 per.clang-format(clean, no violations on touched lines).Checklist
./scripts/code_format.sh(clang-format 20.1.8, no violations on touched lines)
pass — toolchain unavailable on the submitter's macOS host; relying
on CI
no behavior contract change
depends on the oplog writer's internal backpressure state; open to
adding a unit test if reviewers suggest a good injection point
(+59/-4 LOC)
AI Assistance Disclosure
Ported and adapted with AI assistance (Proma Agent): locating the upstream
call sites, adapting the retry helper to current
main(the originalcommits target a fork whose
ClearInvalidHandlescontrol flow hasdiverged), and formatting checks. The human submitter has reviewed every
changed line and can defend the change end-to-end.