Skip to content

fix(auth): floor quota cooldown at the escalating ladder - #198

Open
warelik wants to merge 12 commits into
kaitranntt:mainfrom
warelik:fix/quota-backoff-hint-floor
Open

fix(auth): floor quota cooldown at the escalating ladder#198
warelik wants to merge 12 commits into
kaitranntt:mainfrom
warelik:fix/quota-backoff-hint-floor

Conversation

@warelik

@warelik warelik commented Aug 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Gemini and Antigravity can answer HTTP 429 for a fully exhausted daily quota with a RetryInfo hint of only ~479ms (observed: 479417207ns). Taking that hint verbatim returned the dead credential to the pool ~500ms later AND pinned BackoffLevel forever because every retry recomputed the ladder and then discarded it in favor of the sub-second hint.

This change floors the quota cooldown deadline at the escalating ladder calculation. A provider hint may still push the recovery deadline further out, but can never pull it in below the ladder step — except for the two cases the provider itself marks as non-exhaustion (see the follow-up rounds below).

Code changes

Before

next, backoffLevel = quotaCooldownAfterFailure(state.Quota, now)
if result.RetryAfter != nil {
    next = now.Add(*result.RetryAfter)
}

After

next, backoffLevel = quotaCooldownAfterFailure(state.Quota, now)
if result.RetryAfter != nil {
    // A provider hint can be sub-second even when the quota is exhausted for
    // the whole day, so never let it undercut the escalating quota ladder.
    if hinted := now.Add(*result.RetryAfter); hinted.After(next) {
        next = hinted
    }
}

Blast radius

quotaCooldownAfterFailure is package-private with exactly two call sites:

  • MarkResult in sdk/cliproxy/auth/conductor_cooldown.go
  • applyAuthFailureState in sdk/cliproxy/auth/conductor_cooldown.go

Both call sites are updated identically to respect the ladder floor.

Tests & Verification

Added two unit tests in sdk/cliproxy/auth/cooldown_backoff_test.go:

  • TestMarkResultSubSecondQuotaHintStillEscalates
  • TestApplyAuthFailureStateSubSecondQuotaHintStillEscalates

Mandatory reverse bite-check failure

Reverting the fix produced the expected test failure:

=== RUN   TestMarkResultSubSecondQuotaHintStillEscalates
    cooldown_backoff_test.go:354: expected BackoffLevel 4 after hinted post-window failure, got 3
--- FAIL: TestMarkResultSubSecondQuotaHintStillEscalates (0.00s)
=== RUN   TestApplyAuthFailureStateSubSecondQuotaHintStillEscalates
    cooldown_backoff_test.go:372: expected BackoffLevel 1 after the first hinted failure, got 0
--- FAIL: TestApplyAuthFailureStateSubSecondQuotaHintStillEscalates (0.00s)
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.430s
FAIL

Passing tests with fix applied

=== RUN   TestMarkResultSubSecondQuotaHintStillEscalates
--- PASS: TestMarkResultSubSecondQuotaHintStillEscalates (0.00s)
=== RUN   TestApplyAuthFailureStateSubSecondQuotaHintStillEscalates
--- PASS: TestApplyAuthFailureStateSubSecondQuotaHintStillEscalates (0.00s)
PASS
ok  	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.348s

Follow-up review round 1 — zero-delay retries (165e9326)

Review pointed out that an unconditional floor also swallowed RetryAfter: 0, which upstream
uses to mean "retry immediately on the same credential". The gate now exempts a non-positive
hint (conductor_cooldown.go:829 in MarkResult, :1966 in applyAuthFailureState).

Tests: TestMarkResultZeroRetryAfterDoesNotApplyLadderFloor,
TestApplyAuthFailureStateZeroRetryAfterDoesNotApplyLadderFloor.

Reverse bite-check — dropping the *retryAfter <= 0 clause from both gates:

--- FAIL: TestMarkResultZeroRetryAfterDoesNotApplyLadderFloor (0.00s)
    cooldown_backoff_test.go:424: expected BackoffLevel to remain 0 for zero RetryAfter, got 1
--- FAIL: TestApplyAuthFailureStateZeroRetryAfterDoesNotApplyLadderFloor (0.00s)
    cooldown_backoff_test.go:439: expected BackoffLevel 0 for zero RetryAfter, got 1
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.319s
FAIL

Follow-up review round 2 — transient rate limits (c6ec29ef)

Review then pointed out that the floor still applied to 429s the executor had already
classified as a short-lived rate limit rather than an exhausted quota, parking a usable
credential for a whole ladder step.

classifyAntigravity429 (internal/runtime/executor/antigravity_executor_credits.go:203) already
distinguishes RATE_LIMIT_EXCEEDED from QUOTA_EXHAUSTED, but the classification never reached
the conductor: newAntigravityStatusErr built a bare statusErr{code, msg, retryAfter} and only
status, text and hint were carried across.

  • statusErr gains transientRateLimit bool and func (e statusErr) TransientRateLimit() bool
    (internal/runtime/executor/openai_compat_executor.go:1031). Every other executor leaves it
    false, so their behaviour is unchanged.
  • newAntigravityStatusErr sets it for 429s only:
    err.transientRateLimit = classifyAntigravity429(body) == antigravity429RateLimited
    (antigravity_executor_credits.go:344). Deliberately narrow: QUOTA_EXHAUSTED, soft bodies
    and anything unclassified keep the floor and keep escalating.
  • isTransientRateLimitError (sdk/cliproxy/auth/conductor_cooldown.go:1459) recovers the flag
    with errors.As over an interface{ TransientRateLimit() bool }, mirroring how
    retryAfterFromError already recovers the hint. The execution paths set
    result.TransientRateLimit next to the existing result.RetryAfter assignment.

The flag lives on Result, not on auth.Error, and is threaded as the sixth parameter of
applyAuthFailureState: sdk/cliproxy/auth/errors_compat_test.go
TestErrorLegacyUnkeyedLiteralCompatibility constructs Error with an unkeyed literal, so any
new field on that struct breaks go vet. The seven pre-existing applyAuthFailureState call
sites in cooldown_backoff_test.go were updated mechanically with a trailing , false; no
assertion was changed.

Tests: TestMarkResultTransientRateLimitKeepsProviderHint,
TestApplyAuthFailureStateTransientRateLimitKeepsProviderHint (its second half feeds the same
sub-second hint through an exhausted-quota error and asserts the floor and BackoffLevel == 1
still apply), TestIsTransientRateLimitErrorDetectsWrappedProviderClassification,
TestNewAntigravityStatusErrMarksTransientRateLimit.

Reverse bite-check — reverting both gates to *retryAfter <= 0:

=== RUN   TestMarkResultTransientRateLimitKeepsProviderHint
    cooldown_backoff_test.go:482: expected BackoffLevel to stay 0 for a transient rate limit, got 1
--- FAIL: TestMarkResultTransientRateLimitKeepsProviderHint (0.00s)
=== RUN   TestApplyAuthFailureStateTransientRateLimitKeepsProviderHint
    cooldown_backoff_test.go:497: expected BackoffLevel to stay 0 for a transient rate limit, got 1
--- FAIL: TestApplyAuthFailureStateTransientRateLimitKeepsProviderHint (0.00s)
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.422s
FAIL

Reverse bite-check — discarding the executor classification:

=== RUN   TestNewAntigravityStatusErrMarksTransientRateLimit
    antigravity_executor_credits_test.go:249: expected a RATE_LIMIT_EXCEEDED 429 with a sub-second hint to be marked transient
--- FAIL: TestNewAntigravityStatusErrMarksTransientRateLimit (0.00s)
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/runtime/executor	0.666s
FAIL

go build ./..., go vet ./sdk/cliproxy/... ./internal/runtime/executor/..., gofmt -l and
go test ./sdk/cliproxy/auth/... ./internal/runtime/executor/... are clean.

Follow-up review round 3 — streaming and token-count paths (8c07697e)

Review found the classification was still only wired into the non-stream execution path.

  • executeStreamWithModelPool built every failure result from retryAfterFromError alone. All
    five results now set the flag next to the hint: sdk/cliproxy/auth/conductor_stream.go:275,
    :354, :374, :386, :401. The mid-stream result in wrapStreamResult (:131) is left
    alone on purpose — it sets no hint today, so the ladder still applies there and adding one
    would be a behavioural change beyond this review.
  • Antigravity CountTokens hand-rolled statusErr twice. Both now go through
    newAntigravityStatusErr (internal/runtime/executor/antigravity_executor_tokens.go:167 and
    :172), which does the same helps.ParseRetryDelay work and additionally applies
    classifyAntigravity429.

Tests: TestExecuteStreamKeepsProviderHintForTransientRateLimit (new file
sdk/cliproxy/auth/conductor_stream_classification_test.go, drives a real Manager.ExecuteStream
against a stub executor whose error reports 429 + 479417207ns + transient) and
TestAntigravityCountTokensClassifiesTransient429 (real CountTokens against an httptest
server serving a structured RATE_LIMIT_EXCEEDED 429).

Reverse bite-check — dropping the new line from the executor-error stream branch:

--- FAIL: TestExecuteStreamKeepsProviderHintForTransientRateLimit (0.00s)
    conductor_stream_classification_test.go:54: expected BackoffLevel to stay 0 for a transient rate limit, got 1
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.433s
FAIL

Reverse bite-check — restoring the hand-rolled token-count statusErr:

--- FAIL: TestAntigravityCountTokensClassifiesTransient429 (0.00s)
    antigravity_executor_credits_test.go:858: expected a RATE_LIMIT_EXCEEDED token-count 429 to be marked transient
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/runtime/executor	0.666s
FAIL

Upstream counterpart: router-for-me/CLIProxyAPI#5130

Review follow-up: short-cooldown 429s are transient

The Antigravity executor raises a synthetic 429 while an auth sits in a short cooldown. That cooldown is a local, self-imposed pause of at most a few minutes, but the error carried only a positive retryAfter hint and no classification, so isTransientRateLimitError() (sdk/cliproxy/auth/conductor_cooldown.go:1459) returned false and applyAuthFailureState() (sdk/cliproxy/auth/conductor_cooldown.go:914) treated it as an exhausted upstream quota, escalating BackoffLevel toward the 30 minute ceiling.

All three cooldown short-circuits now set transientRateLimit: true — internal/runtime/executor/antigravity_executor_execute.go:36 (Execute), :268 (executeClaudeNonStream) and internal/runtime/executor/antigravity_executor_stream.go:35 (ExecuteStream) — because the classification is consumed independently on each path (conductor_execution.go:381, conductor_stream.go:275/354/374/386/401, conductor_home_execution.go:181).

Test: TestAntigravityShortCooldownErrorIsTransient in internal/runtime/executor/antigravity_executor_cooldown_transient_test.go.

Reverse bite-check — dropping transientRateLimit: true from the three literals:

--- FAIL: TestAntigravityShortCooldownErrorIsTransient (0.00s)
    --- FAIL: TestAntigravityShortCooldownErrorIsTransient/execute (0.00s)
        antigravity_executor_cooldown_transient_test.go:81: expected the synthetic short-cooldown 429 to be transient so the conductor rotates instead of escalating backoff
    --- FAIL: TestAntigravityShortCooldownErrorIsTransient/execute-claude (0.00s)
        antigravity_executor_cooldown_transient_test.go:81: expected the synthetic short-cooldown 429 to be transient so the conductor rotates instead of escalating backoff
    --- FAIL: TestAntigravityShortCooldownErrorIsTransient/execute-stream (0.00s)
        antigravity_executor_cooldown_transient_test.go:81: expected the synthetic short-cooldown 429 to be transient so the conductor rotates instead of escalating backoff
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/runtime/executor	0.671s
FAIL

Review follow-up: ordinary Claude 429s are transient

Ordinary (non-unified) Claude 429s reached claudeRateLimitError wrapping a statusErr with no transientRateLimit flag (internal/runtime/executor/claude_executor_request.go:295-308), so isTransientRateLimitError() returned false and applyAuthFailureState() treated an ordinary model-level throttle as exhausted quota, escalating BackoffLevel toward the 30 minute ceiling. The ordinary path in classifyClaudeUpstreamError now sets transientRateLimit = true; the unified 5h/7d rejection path is untouched and stays on the quota ladder.

Tests: TestClassifyClaudeUpstreamError_OrdinaryRateLimitIsTransient and TestClassifyClaudeUpstreamError_UnifiedRejectionNotTransient (the negative pin for the unified path).

Reverse bite-check — dropping err.transientRateLimit = true:

--- FAIL: TestClassifyClaudeUpstreamError_OrdinaryRateLimitIsTransient (0.00s)
    claude_executor_beta_policy_test.go:306: ordinary Claude 429 = {"type":"error","error":{"type":"rate_limit_error","message":"Number of requests has exceeded your rate limit."}}, want a transient rate limit
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/runtime/executor	0.695s
FAIL

Review follow-up: transient 429s without a hint bypass the quota ladder

The ladder bypass previously required a parseable retryAfter hint, so a transient 429 with no hint (e.g. an ordinary Claude throttle without reset headers) fell into quotaCooldownAfterFailure and advanced BackoffLevel anyway. Both copies of the logic — MarkResult's per-model state (sdk/cliproxy/auth/conductor_cooldown.go:829) and applyAuthFailureState's credential state (:1917) — now bypass the ladder for any transient 429, keeping the hint verbatim when present and falling back to nextTransientErrorRetryAfter (~60s) otherwise.

Test: TestManager_MarkResult_Transient429WithoutHintBypassesQuotaLadder.

Reverse bite-check — reverting conductor_cooldown.go:

--- FAIL: TestManager_MarkResult_Transient429WithoutHintBypassesQuotaLadder (0.00s)
    conductor_overrides_test.go:836: expected credential quota ladder to stay at level 0 for a transient 429 without hint, got 1
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.610s
FAIL

Review follow-up: transient fallback respects disabled cooldowns

With transientErrorCooldownSeconds < 0 the transient fallback returns a zero time, but both 429 branches still recorded Unavailable=true / Quota.Exceeded=true with an empty NextRecoverAt, which availabilityBlock reads as an indefinite park. The transient-429 handling in MarkResult (sdk/cliproxy/auth/conductor_cooldown.go:861) and applyAuthFailureState (:1952) now skips the marking when the transient fallback yields a zero time, preserving any pre-existing quota block.

Test: TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldown.

Reverse bite-check — reverting conductor_cooldown.go:

--- FAIL: TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldown (0.00s)
    conductor_overrides_test.go:899: expected the credential quota state to stay clear with transient cooldowns disabled
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.435s
FAIL

Review follow-up: disabled-cooldown skip restores availability fields

The transient-cooldown-off skip restored the status/quota fields but left auth.Unavailable=true (set at the top of applyAuthFailureState) and auth.NextRetryAfter untouched, so the credential stayed blocked anyway. The prior availability fields are now captured and restored (sdk/cliproxy/auth/conductor_cooldown.go:2027).

Test: TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldownAuthLevel (auth-level Result drives applyAuthFailureState).

Reverse bite-check — reverting conductor_cooldown.go:

--- FAIL: TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldownAuthLevel (0.00s)
    conductor_overrides_test.go:952: expected the credential to stay available with transient cooldowns disabled
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.426s
FAIL

Review follow-up: reasoned RATE_LIMIT_EXCEEDED without RetryInfo is transient

RATE_LIMIT_EXCEEDED without a RetryInfo detail downgrades to SoftRetry, and only the RateLimited category was marked transient, so a plain per-minute throttle was read as exhausted quota. newAntigravityStatusErr now marks the soft rate limit transient when the classification comes from the ErrorInfo reason (antigravity_executor_credits.go:350); the bare "too many requests" message heuristic still stays on the quota ladder.

Test: new case in TestNewAntigravityStatusErrMarksTransientRateLimit.

Reverse bite-check — reverting antigravity_executor_credits.go:

--- FAIL: TestNewAntigravityStatusErrMarksTransientRateLimit (0.00s)
    antigravity_executor_credits_test.go:290: expected a RATE_LIMIT_EXCEEDED 429 without RetryInfo to be marked transient
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/runtime/executor	0.788s
FAIL

Tooling note

The jbcontext CLI is installed on this machine but its stored session cannot be decrypted (the OS keychain is not accessible), so jbcontext search could not run. The equivalent semantic search, review and blast-radius passes were performed with local code-intelligence tooling instead.

W ARELIK W ARELIK
fix(auth): floor quota cooldown at the escalating ladder
df719cc 
Providers answering 429 for an exhausted daily quota can attach a RetryInfo hint far shorter than the real recovery window. Gemini and Antigravity were observed returning 479417207ns while the key stayed dead for the rest of the day.

Both quota paths took that hint verbatim, so an exhausted credential returned to the pool half a second later and BackoffLevel never advanced past its current step: every retry recomputed the same level and immediately overwrote the deadline with the sub-second hint.

Compute the escalating ladder first and let a provider hint only push the deadline further out, never pull it in. A genuine long hint still wins; a sub-second one can no longer undercut the ladder.

Covered by TestMarkResultSubSecondQuotaHintStillEscalates and TestApplyAuthFailureStateSubSecondQuotaHintStillEscalates in sdk/cliproxy/auth/cooldown_backoff_test.go.
@warelik warelik mentioned this pull request Aug 21, 2026
Open
W ARELIK added 4 commits August 21, 2026 07:28
W ARELIK W ARELIK
fix(auth): preserve zero-delay cooldown for non-quota 429 retries
165e932 
Do not apply the escalating quota ladder floor when a 429 response explicitly specifies a zero or non-positive RetryAfter duration (e.g. transient websocket connection limit errors). Escalating quota cooldown remains gated to positive retry hints and default quota exhaustion.
W ARELIK W ARELIK
fix(auth): keep provider hint for transient 429s
c6ec29e 
decideAntigravity429 classifies a RATE_LIMIT_EXCEEDED 429 whose retry
hint is shorter than three seconds as an instant retry on the same
credential rather than an exhausted quota. The unconditional ladder
floor still replaced that hint with a quota cooldown step, parking a
still-usable credential for up to the full ladder window.

Carry the executor classification through statusErr and Result so the
ladder floor only applies to 429s that were not decisively classified
as a short-lived rate limit. Exhausted quota and unclassified bodies
keep the floor and keep escalating exactly as before.
W ARELIK W ARELIK
fix(auth): carry 429 classification through stream and token count
8c07697 
The transient rate-limit flag only reached the conductor from the
non-stream execution path. Streaming failures built their results from
retryAfterFromError alone, and the Antigravity token-count path built
statusErr by hand, so both still floored a provider-classified
short-lived 429 at the quota ladder and parked a usable credential.

Set TransientRateLimit next to every RetryAfter assignment in the
streaming pool, and build the token-count errors through
newAntigravityStatusErr so they inherit the same classification.
W ARELIK W ARELIK
fix(antigravity): mark short-cooldown 429s as transient rate limits
d9edde2 
The Antigravity executor raises a synthetic 429 while an auth sits in a
short cooldown. That cooldown is a local, self-imposed pause of at most a
few minutes, but the error carried only a positive retryAfter hint and no
classification, so isTransientRateLimitError() returned false and
MarkResult()/applyAuthFailureState() read it as an exhausted upstream
quota. BackoffLevel then escalated toward the 30 minute ceiling and parked
an account that was never throttled upstream.

Set transientRateLimit on all three cooldown short-circuits (Execute,
executeClaudeNonStream, ExecuteStream) so the conductor rotates to the next
auth instead of escalating backoff.

Covered by TestAntigravityShortCooldownErrorIsTransient, which asserts the
classification on all three entry points.
@warelik

warelik commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

Mirrored the upstream review follow-up from router-for-me/CLIProxyAPI#5130 as d9edde2: the synthetic short-cooldown 429 raised by the Antigravity executor is now marked transientRateLimit: true on all three short-circuits (antigravity_executor_execute.go:36 and :267, antigravity_executor_stream.go:35), so the conductor rotates to the next auth instead of escalating BackoffLevel toward the 30 minute ceiling.

Test: TestAntigravityShortCooldownErrorIsTransient. Reverse bite-check reproduced here verbatim:

--- FAIL: TestAntigravityShortCooldownErrorIsTransient (0.00s)
    --- FAIL: TestAntigravityShortCooldownErrorIsTransient/execute (0.00s)
        antigravity_executor_cooldown_transient_test.go:81: expected the synthetic short-cooldown 429 to be transient so the conductor rotates instead of escalating backoff
    --- FAIL: TestAntigravityShortCooldownErrorIsTransient/execute-claude (0.00s)
        antigravity_executor_cooldown_transient_test.go:81: expected the synthetic short-cooldown 429 to be transient so the conductor rotates instead of escalating backoff
    --- FAIL: TestAntigravityShortCooldownErrorIsTransient/execute-stream (0.00s)
        antigravity_executor_cooldown_transient_test.go:81: expected the synthetic short-cooldown 429 to be transient so the conductor rotates instead of escalating backoff
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/runtime/executor	0.776s
FAIL

W ARELIK W ARELIK
fix(claude): classify ordinary 429s as transient rate limits
193a4ae 
classifyClaudeUpstreamError built ordinary (non-unified) Claude 429s as
claudeRateLimitError wrapping a statusErr with no transientRateLimit flag,
so isTransientRateLimitError() returned false and MarkResult() treated an
ordinary model-level throttle as exhausted quota, escalating BackoffLevel
toward the 30 minute ceiling and parking a credential that was only
briefly throttled.

Mark the ordinary path transient. Unified 5h/7d rejections keep the quota
ladder untouched.

Covered by TestClassifyClaudeUpstreamError_OrdinaryRateLimitIsTransient
and TestClassifyClaudeUpstreamError_UnifiedRejectionNotTransient.
@warelik

warelik commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

Mirrored the second upstream review follow-up from router-for-me/CLIProxyAPI#5130 as 193a4ae: ordinary (non-unified) Claude 429s from classifyClaudeUpstreamError are now marked transientRateLimit (internal/runtime/executor/claude_executor_request.go), so the conductor rotates instead of escalating BackoffLevel toward the 30 minute ceiling on a plain model-level throttle. Unified 5h/7d rejections stay on the quota ladder.

Tests: TestClassifyClaudeUpstreamError_OrdinaryRateLimitIsTransient and TestClassifyClaudeUpstreamError_UnifiedRejectionNotTransient. Reverse bite-check reproduced here verbatim:

--- FAIL: TestClassifyClaudeUpstreamError_OrdinaryRateLimitIsTransient (0.00s)
    claude_executor_beta_policy_test.go:306: ordinary Claude 429 = {"type":"error","error":{"type":"rate_limit_error","message":"Number of requests has exceeded your rate limit."}}, want a transient rate limit
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/runtime/executor	0.782s
FAIL

W ARELIK W ARELIK
fix(auth): bypass the quota ladder for transient 429s without a hint
48444c8 
Both 429 cooldown branches (MarkResult's per-model state and
applyAuthFailureState's credential state) only kept a short-lived rate
limit out of the quota ladder when a parseable retryAfter hint was
present. A transient 429 with no hint fell through to
quotaCooldownAfterFailure and advanced BackoffLevel toward the 30 minute
ceiling, parking a credential that was only briefly throttled.

Transient 429s now bypass the ladder regardless of hint presence: the
provider-supplied retryAfter is kept verbatim when present, and without a
hint the cooldown falls back to nextTransientErrorRetryAfter (the standard
~60s transient-error cooldown).

Covered by TestManager_MarkResult_Transient429WithoutHintBypassesQuotaLadder,
which asserts both the per-model and credential quota ladders stay at
level 0 and both NextRetryAfter values land at the transient cooldown.
@warelik

warelik commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

Mirrored the third upstream review follow-up from router-for-me/CLIProxyAPI#5130 as 48444c8: transient 429s without a parseable hint now bypass the quota ladder in both copies of the cooldown logic (MarkResult per-model state and applyAuthFailureState credential state), falling back to nextTransientErrorRetryAfter (~60s) instead of advancing BackoffLevel.

Test: TestManager_MarkResult_Transient429WithoutHintBypassesQuotaLadder. Reverse bite-check reproduced here verbatim:

--- FAIL: TestManager_MarkResult_Transient429WithoutHintBypassesQuotaLadder (0.00s)
    conductor_overrides_test.go:993: expected credential quota ladder to stay at level 0 for a transient 429 without hint, got 1
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.851s
FAIL

W ARELIK W ARELIK
fix(auth): keep credentials available when transient cooldowns are off
7d7fe41 
The transient-429 fallback (nextTransientErrorRetryAfter) returns a zero
time when transient cooldowns are disabled (transientErrorCooldownSeconds
< 0). Both 429 cooldown branches still stored that zero alongside
Unavailable=true and Quota.Exceeded=true with an empty NextRecoverAt, and
availabilityBlock read the zero-time quota block as an indefinite park.

When the transient fallback yields a zero time the 429 handling now leaves
the model and the credential available: no quota mark, no suspension, no
retry time. A pre-existing quota block is preserved in both paths.

Covered by
TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldown.
@warelik

warelik commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

Mirrored the fourth upstream review follow-up from router-for-me/CLIProxyAPI#5130 as 7d7fe41: when transient cooldowns are disabled, the transient-429 zero-time fallback no longer gets stored as an indefinite quota block — both 429 branches skip the marking and preserve any pre-existing quota state.

Test: TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldown. Reverse bite-check reproduced here verbatim:

--- FAIL: TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldown (0.00s)
    conductor_overrides_test.go:1056: expected the credential quota state to stay clear with transient cooldowns disabled
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.475s
FAIL

W ARELIK added 2 commits August 21, 2026 10:18
W ARELIK W ARELIK
fix(auth): restore availability fields when skipping a disabled trans…
9c1b9f2 
…ient cooldown

The transient-cooldown-off skip in applyAuthFailureState restored the
status message and quota fields but left auth.Unavailable=true (set at the
top of the function) and auth.NextRetryAfter untouched, so a credential
hit by a transient 429 without a hint stayed indefinitely blocked anyway.

Capture the prior availability fields and restore them in the skip.

Covered by
TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldownAuthLevel,
which drives an auth-level Result (empty Model) through
applyAuthFailureState.
W ARELIK W ARELIK
fix(antigravity): treat reasoned RATE_LIMIT_EXCEEDED without RetryInf…
6143d45 
…o as transient

When Google returns RESOURCE_EXHAUSTED with an ErrorInfo reason of
RATE_LIMIT_EXCEEDED but omits the RetryInfo detail, the decision table
downgrades to SoftRetry and the resulting error reported
TransientRateLimit() == false. The conductor then read a plain per-minute
throttle as exhausted quota and escalated BackoffLevel toward the 30
minute ceiling instead of rotating.

newAntigravityStatusErr now also marks the soft rate limit transient, but
only when the classification came from the ErrorInfo reason: the bare
"too many requests" message heuristic stays on the quota ladder.

Covered by a new case in TestNewAntigravityStatusErrMarksTransientRateLimit.
@warelik

warelik commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

Mirrored two more upstream review follow-ups from router-for-me/CLIProxyAPI#5130:

  • 9c1b9f2: the disabled-transient-cooldown skip in applyAuthFailureState now restores auth.Unavailable and auth.NextRetryAfter too — otherwise the credential stayed indefinitely blocked. Test TestManager_MarkResult_Transient429WithoutHintRespectsDisabledCooldownAuthLevel; reverse bite-check: conductor_overrides_test.go:1109: expected the credential to stay available with transient cooldowns disabled.
  • 6143d45: a reasoned RATE_LIMIT_EXCEEDED 429 without RetryInfo is now marked transient (the bare too many requests message heuristic still stays on the quota ladder). Test case added to TestNewAntigravityStatusErrMarksTransientRateLimit; reverse bite-check: antigravity_executor_credits_test.go:290: expected a RATE_LIMIT_EXCEEDED 429 without RetryInfo to be marked transient.

@warelik

warelik commented Aug 21, 2026

Copy link
Copy Markdown
Contributor Author

The build failure on the latest run is a flake in an unrelated test: TestAntigravityConcurrentRequestsReusePooledConnections opened 9 connections where at most 8 were expected. That test is a connection-pool concurrency probe (scheduling-sensitive under CI load) and none of the commits in this PR touch transport pooling. Local verification: 5/5 consecutive passes on this exact HEAD (go test -count=1 -run TestAntigravityConcurrentRequestsReusePooledConnections ./internal/runtime/executor/).

I don't have admin rights to rerun the job — a maintainer re-run of the failed job should turn it green. Happy to make the probe less timing-sensitive in a follow-up if it keeps flaking.

W ARELIK added 2 commits August 21, 2026 10:27
W ARELIK W ARELIK
test(antigravity): tolerate one stray dial per wave boundary in pooli…
4b73e20 
…ng test

TestAntigravityConcurrentRequestsReusePooledConnections failed in CI with 9
distinct connections for 3 waves of 8, even though pooling works: a wave
boundary can cost one extra dial when an idle connection is retired at
exactly the wrong moment. Allow one stray dial per later wave. The
MaxIdleConnsPerHost=2 regression still fails loudly: it would open roughly
totalConns - 2*(waves-1) distinct connections (20 here), far above the new
allowance of 10.

Verified with 10 consecutive local runs on this commit: 10/10 pass.
W ARELIK W ARELIK
test(qoder): raise usage-record wait timeout to 30s for CI jitter
31ae940 
TestExecuteStream_PublishesUsageRecordFromStreamUsage timed out in CI at
its 5s cap while waiting for the asynchronously published usage record.
The cap only bounds the failure case — a matching record returns
immediately — so raising it to 30s costs nothing on success and absorbs
scheduler starvation on loaded runners.

Verified with 3/3 consecutive local runs on this commit.
This was referenced Aug 21, 2026
Open
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@warelik