Skip to content

fix(ramp): evaluate chop convergence when a failure tightens hi, not only on a pass - #27

Open
davidrichardson wants to merge 1 commit into
conduktor-gatewayfrom
dmr/chop-convergence-fix
Open

fix(ramp): evaluate chop convergence when a failure tightens hi, not only on a pass#27
davidrichardson wants to merge 1 commit into
conduktor-gatewayfrom
dmr/chop-convergence-fix

Conversation

@davidrichardson

Copy link
Copy Markdown

The bug

pollChop() has two exits. The failure path sets hi = currentRate and returns
beginDrain(...) immediately; the convergence check lives further down on the pass path,
after lo = currentRate. So the stop condition was only ever evaluated after a hold
succeeded
— and a stop condition reachable only when candidates succeed cannot stop a
search whose candidates all fail, which is precisely the runaway case it exists to prevent.

Evidence

An AKS gateway encrypt arm on the shipped defaults (120s holds, 5% tolerance). lo held
clean at 140,000 msg/s, then eleven consecutive candidates failed while chop bisected
[140,000, 150,000] down to 4.88 msg/s wide:

# candidate verdict (hi-lo)/lo after
1 150,000 exceeded 0.0714
2 145,000 exceeded 0.0357 — inside the tolerance
11 140,009.765625 exceeded 0.00007
140,004.8828125 clean 0.00003

9m57s and eleven drain cycles of a 20m19s discovery, to move lo by 4.88 msg/s — the
confirmed rate went from a would-be 133,000 to 133,004.64, i.e. 0.003%. Stopping at
candidate 2 saves ~9m29s (47%); the confirmation hold has to run either way.

Not a seeding failure: achievedRatio was 0.98–0.996 throughout, so the producer kept up and
rampSeedFromAchievedRate correctly declined to seed. What breached was the backlog level
(rampMaxBacklogSeconds: 0.5 × 140,000 ≈ 70,000 against an observed 70,814–99,415). Every
rate in 140,009–150,000 sits in a band where the verdict is near-random — bisecting into
noise, with the convergence tolerance wired to the wrong branch. direct and transparent
in the same run were untouched (14 holds, 1 drain each), which is what makes this specific to
a marginal band rather than general.

The fix

Evaluate convergence when a failure tightens hi as well, extracted to converged() and
used on both paths. Convergence-by-failure routes through beginDrain(confirmRate()) rather
than straight to the hold: the candidate that just failed left a backlog, and a confirmation
hold that inherits one judges the previous candidate's overshoot rather than its own rate.
A separate test pins that.

Tests

Two, both red on their assertion first. The fixture needed care — FakeSystem cannot express
this at all, since it keeps received equal to published, so backlog is always zero and the
only thing that can breach is the 5%-tolerant producer ratio; candidates within 5% of capacity
pass and failure runs stay short. Added FakeConsumerLimitedSystem (producer keeps up,
consumers drain at a ceiling), which is the shape the real arm failed with. Same asymmetry as
production: the grind belongs to BACKLOG, and THROUGHPUT's slack hides it.

One existing test changed, and why it isn't churn

seedingReachesTheSameCeilingInFewerHoldsThanBlindBisection asserted seeded < bisecting at
capacity 841,000. On that fixture this fix moves exactly one cell — blind 256 → 211 polls —
enough to overtake seeded's 226. A sweep across seven capacities shows seeding was never a
general time win (2 wins, 3 losses, 2 ties), so the trial log's "~12%" was an artifact of
measuring at the one capacity that test also happened to pick. Both mechanisms shorten the
same thing: a run of failing candidates.

Renamed to seedingAndBlindBisectionAgreeOnTheCeiling, asserting the property that survives.
rampSeedFromAchievedRate is unchanged and still on by default — it buys accurate placement
of lo, a correctness argument rather than a speed one. Whether it still earns its complexity
is flagged open in the trial log, not decided here.

Docs

RATE_FINDING.md's Confirm step said convergence happens "once a candidate holds clean" —
the bug restated as intent. Now states both routes and the drain requirement. Trial-log
Finding 6 marked superseded; Finding 20 records the run and the sweep.

Verification

123 tests, spotless, checkstyle, spotbugs — BUILD SUCCESS, re-run on this branch's base
(the commit was cherry-picked onto the squashed mainline, so the base differs from where it
was authored).

🤖 Generated with Claude Code

…only on a pass

pollChop had two exits. The failure path set hi = currentRate and returned
beginDrain(...) immediately; the convergence check lived further down on the pass
path, after lo = currentRate. So the stop condition was only ever evaluated after a
hold SUCCEEDED -- and a stop condition reachable only when candidates succeed cannot
stop a search whose candidates all fail, which is precisely the runaway case it exists
to prevent.

report12's encrypt arm: lo held clean at 140,000, then ELEVEN consecutive candidates
failed while chop bisected [140,000, 150,000] down to 4.88 msg/s wide. The bracket had
been inside the 5% tolerance since the second failure. Cost 9m57s and eleven drain
cycles of a 20m19s discovery to move lo by 4.88 msg/s, i.e. the confirmed rate from a
would-be 133,000 to 133,004.64 -- 0.003%. Stopping at candidate 2 saves ~9m29s, 47%;
the confirmation hold has to run either way.

Not a seeding failure: achievedRatio was 0.98-0.996 throughout, so the producer kept up
and rampSeedFromAchievedRate correctly declined to seed. What breached was the backlog
level -- rampMaxBacklogSeconds 0.5 x 140,000 ~= 70,000 against an observed
70,814-99,415. Every rate in 140,009-150,000 sits in a band where the verdict is
near-random, so this is bisecting into noise, and the convergence tolerance is the guard
that was wired to the wrong branch. direct and transparent in the same run were
untouched (14 holds, 1 drain each), which is what makes it specific to a marginal band.

Convergence by failure routes through beginDrain(confirmRate()) rather than straight to
the hold: the candidate that just failed left a backlog, and a confirmation hold that
inherits one judges the previous candidate's overshoot rather than its own rate --
Finding 17. A test pins that separately from the stop condition itself.

Two tests, both red first. The fixture needed care: FakeSystem cannot express this at
all, since it keeps received equal to published, so backlog is always zero and the only
thing that can breach is the 5%-tolerant producer ratio -- candidates within 5% of
capacity pass and failure runs stay short. Added FakeConsumerLimitedSystem, where the
producer keeps up and the consumers drain at a ceiling, which is the shape the real arm
failed with. Same asymmetry as production: the grind belongs to BACKLOG, and
THROUGHPUT's rampMinThroughputRatio slack hides it.

That asymmetry also explains a broken existing test rather than churn.
seedingReachesTheSameCeilingInFewerHoldsThanBlindBisection asserted seeded < bisecting
at capacity 841,000. On that fixture the fix moves exactly one cell -- blind at 841,000,
256 -> 211 polls -- which is enough to overtake seeded's 226. A sweep across seven
capacities shows seeding was never a general time win (2 wins, 3 losses, 2 ties), so
Finding 6's "~12%" was an artifact of measuring at the one capacity this test also
happened to pick. Both mechanisms shorten the same thing, a run of failing candidates.
Renamed to seedingAndBlindBisectionAgreeOnTheCeiling, asserting the property that
survives; the seed still buys accurate placement of lo, which is a correctness argument
rather than a speed one. Whether it earns its complexity is flagged open, not decided.

Docs: RATE_FINDING.md's Confirm step said convergence happens "once a candidate holds
clean", which was the bug restated as intent -- now states both routes and the drain
requirement. Finding 6 marked superseded, Finding 20 records the run and the sweep.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant