Skip to content

fix(auth): keep failure bookkeeping when cooling is disabled - #200

Open
warelik wants to merge 2 commits into
kaitranntt:mainfrom
warelik:fix/disable-cooling-records-backoff
Open

fix(auth): keep failure bookkeeping when cooling is disabled#200
warelik wants to merge 2 commits into
kaitranntt:mainfrom
warelik:fix/disable-cooling-records-backoff

Conversation

@warelik

@warelik warelik commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR fixes a bug where disable-cooling: true erased all evidence of 429/quota and rate limit failures instead of merely skipping the blackout window. It preserves failure bookkeeping (BackoffLevel increment and UpdatedAt timestamp) under disableCooling while maintaining the flag's contract that credentials stay immediately usable without blackout or suspension. It also adds per-attempt failure logging to MarkResult.

Production symptom

Under disable-cooling: true, when rate limits were encountered:

  • The proxy walked all 12 candidates and looped every ~2 minutes against rate-limited credentials.
  • A grep for cooldown|quota|backoff|suspend in the server log returned zero lines because failures were completely unrecorded and unlogged.

Root cause and lines that erased state

  1. In sdk/cliproxy/auth/conductor_cooldown.go, under MarkResult case 429, quotaCooldownAfterFailure was skipped when disableCooling was true, preventing BackoffLevel from incrementing.
  2. In updateAggregatedAvailability, when state.Quota.Exceeded was false (reset by disableCooling), auth.Quota.BackoffLevel was unconditionally reset to 0.
  3. In applyAuthFailureState, 429 handling did not advance auth.Quota.BackoffLevel when disableCooling was true.
  4. MarkResult did not emit any log line when an execution attempt failed, making repeated failure loops invisible.

Changes

  1. Failure bookkeeping under disableCooling:
    • In MarkResult (case 429) and applyAuthFailureState, advance BackoffLevel via nextQuotaCooldown(prevLevel, false) when disableCooling is true.
    • In updateAggregatedAvailability, aggregate maxBackoffLevel across model states and assign auth.Quota.BackoffLevel = maxBackoffLevel even when Quota.Exceeded is false.
    • Record the failure timestamp on state.UpdatedAt and auth.UpdatedAt.
  2. Contract preservation:
    • The contract of disable-cooling: true is strictly preserved: Unavailable remains false, Quota.Exceeded remains false, NextRetryAfter remains zero, NextRecoverAt remains zero, no model suspension is triggered, and the credential remains immediately usable without blackout.
  3. Per-attempt failure logging:
    • Added structured single-line warning log in MarkResult on every failed attempt using the house logging format:
      auth-cooldown: attempt failed | auth=<id> status=<status> class=<class> cooldown=<duration|skipped> backoff=<level> disable_cooling=<bool>

Sample log line

time="2026-08-21T07:13:27+03:00" level=warning msg="auth-cooldown: attempt failed | auth=auth-1 status=429 class=quota cooldown=skipped backoff=1 disable_cooling=true"

Tests

  • TestDisableCoolingRecordsBackoffAndTimestampWhileStayingUsable
  • TestDisableCoolingDisabledKeepsStandardCooldownBehavior
  • TestMarkResultPerAttemptFailureLogging

Verbatim bite-check failure

Reverting the BackoffLevel increment under disableCooling produced the expected failure:

=== RUN   TestDisableCoolingRecordsBackoffAndTimestampWhileStayingUsable
time="2026-08-21T07:14:01+03:00" level=warning msg="auth-cooldown: attempt failed | auth=auth-disable-cooling-records status=429 class=quota cooldown=skipped backoff=0 disable_cooling=true"
    cooldown_backoff_test.go:354: expected BackoffLevel=1 after first failure, got 0
--- FAIL: TestDisableCoolingRecordsBackoffAndTimestampWhileStayingUsable (0.00s)
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.457s
FAIL

Verbatim green test output

=== RUN   TestDisableCoolingRecordsBackoffAndTimestampWhileStayingUsable
time="2026-08-21T07:13:27+03:00" level=warning msg="auth-cooldown: attempt failed | auth=auth-disable-cooling-records status=429 class=quota cooldown=skipped backoff=1 disable_cooling=true"
time="2026-08-21T07:13:27+03:00" level=warning msg="auth-cooldown: attempt failed | auth=auth-disable-cooling-records status=429 class=quota cooldown=skipped backoff=2 disable_cooling=true"
--- PASS: TestDisableCoolingRecordsBackoffAndTimestampWhileStayingUsable (0.00s)
=== RUN   TestDisableCoolingDisabledKeepsStandardCooldownBehavior
time="2026-08-21T07:13:27+03:00" level=warning msg="auth-cooldown: attempt failed | auth=auth-normal-cooling status=429 class=quota cooldown=1s backoff=1 disable_cooling=false"
--- PASS: TestDisableCoolingDisabledKeepsStandardCooldownBehavior (0.00s)
=== RUN   TestMarkResultPerAttemptFailureLogging
time="2026-08-21T07:13:27+03:00" level=warning msg="auth-cooldown: attempt failed | auth=auth-log-test status=429 class=quota cooldown=skipped backoff=1 disable_cooling=true"
--- PASS: TestMarkResultPerAttemptFailureLogging (0.00s)
PASS
ok  	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	10.310s

Tooling note

The jbcontext CLI is installed on this machine but its stored session cannot be decrypted (the OS keychain is not accessible), so jbcontext search could not run. The equivalent semantic search, review and blast-radius passes were performed with local code-intelligence tooling instead.

W ARELIK added 2 commits August 21, 2026 07:14
Record BackoffLevel increments and last-failure timestamp even when disable-cooling is true, while keeping credentials immediately usable without blackout or suspension.

Add per-attempt failure logging to MarkResult to surface failure details.
The per-attempt failure logging test replaced the whole logrus hook map with an empty one during cleanup, deleting every hook the test process had already installed. That made later tests order-dependent and could silently disable process-wide hooks such as log forwarding. Deep-clone the pre-test hook map before AddHook and restore it in cleanup, matching setupTestLoggerHook in conductor_warn_logging_test.go.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant