fix(auth): keep failure bookkeeping when cooling is disabled - #200
Open
warelik wants to merge 2 commits into
Open
fix(auth): keep failure bookkeeping when cooling is disabled#200warelik wants to merge 2 commits into
warelik wants to merge 2 commits into
Conversation
added 2 commits
August 21, 2026 07:14
Record BackoffLevel increments and last-failure timestamp even when disable-cooling is true, while keeping credentials immediately usable without blackout or suspension. Add per-attempt failure logging to MarkResult to surface failure details.
The per-attempt failure logging test replaced the whole logrus hook map with an empty one during cleanup, deleting every hook the test process had already installed. That made later tests order-dependent and could silently disable process-wide hooks such as log forwarding. Deep-clone the pre-test hook map before AddHook and restore it in cleanup, matching setupTestLoggerHook in conductor_warn_logging_test.go.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes a bug where
disable-cooling: trueerased all evidence of 429/quota and rate limit failures instead of merely skipping the blackout window. It preserves failure bookkeeping (BackoffLevelincrement andUpdatedAttimestamp) underdisableCoolingwhile maintaining the flag's contract that credentials stay immediately usable without blackout or suspension. It also adds per-attempt failure logging toMarkResult.Production symptom
Under
disable-cooling: true, when rate limits were encountered:grepforcooldown|quota|backoff|suspendin the server log returned zero lines because failures were completely unrecorded and unlogged.Root cause and lines that erased state
sdk/cliproxy/auth/conductor_cooldown.go, underMarkResultcase 429,quotaCooldownAfterFailurewas skipped whendisableCoolingwas true, preventingBackoffLevelfrom incrementing.updateAggregatedAvailability, whenstate.Quota.Exceededwas false (reset bydisableCooling),auth.Quota.BackoffLevelwas unconditionally reset to 0.applyAuthFailureState, 429 handling did not advanceauth.Quota.BackoffLevelwhendisableCoolingwas true.MarkResultdid not emit any log line when an execution attempt failed, making repeated failure loops invisible.Changes
MarkResult(case 429) andapplyAuthFailureState, advanceBackoffLevelvianextQuotaCooldown(prevLevel, false)whendisableCoolingis true.updateAggregatedAvailability, aggregatemaxBackoffLevelacross model states and assignauth.Quota.BackoffLevel = maxBackoffLeveleven whenQuota.Exceededis false.state.UpdatedAtandauth.UpdatedAt.disable-cooling: trueis strictly preserved:Unavailableremainsfalse,Quota.Exceededremainsfalse,NextRetryAfterremains zero,NextRecoverAtremains zero, no model suspension is triggered, and the credential remains immediately usable without blackout.MarkResulton every failed attempt using the house logging format:auth-cooldown: attempt failed | auth=<id> status=<status> class=<class> cooldown=<duration|skipped> backoff=<level> disable_cooling=<bool>Sample log line
Tests
TestDisableCoolingRecordsBackoffAndTimestampWhileStayingUsableTestDisableCoolingDisabledKeepsStandardCooldownBehaviorTestMarkResultPerAttemptFailureLoggingVerbatim bite-check failure
Reverting the
BackoffLevelincrement underdisableCoolingproduced the expected failure:Verbatim green test output
Tooling note
The
jbcontextCLI is installed on this machine but its stored session cannot be decrypted (the OS keychain is not accessible), sojbcontext searchcould not run. The equivalent semantic search, review and blast-radius passes were performed with local code-intelligence tooling instead.