fix(ramp): evaluate chop convergence when a failure tightens hi, not only on a pass - #27
Open
davidrichardson wants to merge 1 commit into
Open
fix(ramp): evaluate chop convergence when a failure tightens hi, not only on a pass#27davidrichardson wants to merge 1 commit into
davidrichardson wants to merge 1 commit into
Conversation
…only on a pass pollChop had two exits. The failure path set hi = currentRate and returned beginDrain(...) immediately; the convergence check lived further down on the pass path, after lo = currentRate. So the stop condition was only ever evaluated after a hold SUCCEEDED -- and a stop condition reachable only when candidates succeed cannot stop a search whose candidates all fail, which is precisely the runaway case it exists to prevent. report12's encrypt arm: lo held clean at 140,000, then ELEVEN consecutive candidates failed while chop bisected [140,000, 150,000] down to 4.88 msg/s wide. The bracket had been inside the 5% tolerance since the second failure. Cost 9m57s and eleven drain cycles of a 20m19s discovery to move lo by 4.88 msg/s, i.e. the confirmed rate from a would-be 133,000 to 133,004.64 -- 0.003%. Stopping at candidate 2 saves ~9m29s, 47%; the confirmation hold has to run either way. Not a seeding failure: achievedRatio was 0.98-0.996 throughout, so the producer kept up and rampSeedFromAchievedRate correctly declined to seed. What breached was the backlog level -- rampMaxBacklogSeconds 0.5 x 140,000 ~= 70,000 against an observed 70,814-99,415. Every rate in 140,009-150,000 sits in a band where the verdict is near-random, so this is bisecting into noise, and the convergence tolerance is the guard that was wired to the wrong branch. direct and transparent in the same run were untouched (14 holds, 1 drain each), which is what makes it specific to a marginal band. Convergence by failure routes through beginDrain(confirmRate()) rather than straight to the hold: the candidate that just failed left a backlog, and a confirmation hold that inherits one judges the previous candidate's overshoot rather than its own rate -- Finding 17. A test pins that separately from the stop condition itself. Two tests, both red first. The fixture needed care: FakeSystem cannot express this at all, since it keeps received equal to published, so backlog is always zero and the only thing that can breach is the 5%-tolerant producer ratio -- candidates within 5% of capacity pass and failure runs stay short. Added FakeConsumerLimitedSystem, where the producer keeps up and the consumers drain at a ceiling, which is the shape the real arm failed with. Same asymmetry as production: the grind belongs to BACKLOG, and THROUGHPUT's rampMinThroughputRatio slack hides it. That asymmetry also explains a broken existing test rather than churn. seedingReachesTheSameCeilingInFewerHoldsThanBlindBisection asserted seeded < bisecting at capacity 841,000. On that fixture the fix moves exactly one cell -- blind at 841,000, 256 -> 211 polls -- which is enough to overtake seeded's 226. A sweep across seven capacities shows seeding was never a general time win (2 wins, 3 losses, 2 ties), so Finding 6's "~12%" was an artifact of measuring at the one capacity this test also happened to pick. Both mechanisms shorten the same thing, a run of failing candidates. Renamed to seedingAndBlindBisectionAgreeOnTheCeiling, asserting the property that survives; the seed still buys accurate placement of lo, which is a correctness argument rather than a speed one. Whether it earns its complexity is flagged open, not decided. Docs: RATE_FINDING.md's Confirm step said convergence happens "once a candidate holds clean", which was the bug restated as intent -- now states both routes and the drain requirement. Finding 6 marked superseded, Finding 20 records the run and the sweep. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
pollChop()has two exits. The failure path setshi = currentRateand returnsbeginDrain(...)immediately; the convergence check lives further down on the pass path,after
lo = currentRate. So the stop condition was only ever evaluated after a holdsucceeded — and a stop condition reachable only when candidates succeed cannot stop a
search whose candidates all fail, which is precisely the runaway case it exists to prevent.
Evidence
An AKS gateway
encryptarm on the shipped defaults (120s holds, 5% tolerance).loheldclean at 140,000 msg/s, then eleven consecutive candidates failed while chop bisected
[140,000, 150,000]down to 4.88 msg/s wide:(hi-lo)/loafter9m57s and eleven drain cycles of a 20m19s discovery, to move
loby 4.88 msg/s — theconfirmed rate went from a would-be 133,000 to 133,004.64, i.e. 0.003%. Stopping at
candidate 2 saves ~9m29s (47%); the confirmation hold has to run either way.
Not a seeding failure:
achievedRatiowas 0.98–0.996 throughout, so the producer kept up andrampSeedFromAchievedRatecorrectly declined to seed. What breached was the backlog level(
rampMaxBacklogSeconds: 0.5× 140,000 ≈ 70,000 against an observed 70,814–99,415). Everyrate in 140,009–150,000 sits in a band where the verdict is near-random — bisecting into
noise, with the convergence tolerance wired to the wrong branch.
directandtransparentin the same run were untouched (14 holds, 1 drain each), which is what makes this specific to
a marginal band rather than general.
The fix
Evaluate convergence when a failure tightens
hias well, extracted toconverged()andused on both paths. Convergence-by-failure routes through
beginDrain(confirmRate())ratherthan straight to the hold: the candidate that just failed left a backlog, and a confirmation
hold that inherits one judges the previous candidate's overshoot rather than its own rate.
A separate test pins that.
Tests
Two, both red on their assertion first. The fixture needed care —
FakeSystemcannot expressthis at all, since it keeps received equal to published, so backlog is always zero and the
only thing that can breach is the 5%-tolerant producer ratio; candidates within 5% of capacity
pass and failure runs stay short. Added
FakeConsumerLimitedSystem(producer keeps up,consumers drain at a ceiling), which is the shape the real arm failed with. Same asymmetry as
production: the grind belongs to
BACKLOG, andTHROUGHPUT's slack hides it.One existing test changed, and why it isn't churn
seedingReachesTheSameCeilingInFewerHoldsThanBlindBisectionassertedseeded < bisectingatcapacity 841,000. On that fixture this fix moves exactly one cell — blind 256 → 211 polls —
enough to overtake seeded's 226. A sweep across seven capacities shows seeding was never a
general time win (2 wins, 3 losses, 2 ties), so the trial log's "~12%" was an artifact of
measuring at the one capacity that test also happened to pick. Both mechanisms shorten the
same thing: a run of failing candidates.
Renamed to
seedingAndBlindBisectionAgreeOnTheCeiling, asserting the property that survives.rampSeedFromAchievedRateis unchanged and still on by default — it buys accurate placementof
lo, a correctness argument rather than a speed one. Whether it still earns its complexityis flagged open in the trial log, not decided here.
Docs
RATE_FINDING.md's Confirm step said convergence happens "once a candidate holds clean" —the bug restated as intent. Now states both routes and the drain requirement. Trial-log
Finding 6 marked superseded; Finding 20 records the run and the sweep.
Verification
123 tests, spotless, checkstyle, spotbugs — BUILD SUCCESS, re-run on this branch's base(the commit was cherry-picked onto the squashed mainline, so the base differs from where it
was authored).
🤖 Generated with Claude Code