Skip to content

fix(auth): scope model suspension reasons to prevent stale sibling resume - #188

Open
warelik wants to merge 2 commits into
kaitranntt:mainfrom
warelik:fix/model-suspension-reason-scoping
Open

fix(auth): scope model suspension reasons to prevent stale sibling resume#188
warelik wants to merge 2 commits into
kaitranntt:mainfrom
warelik:fix/model-suspension-reason-scoping

Conversation

@warelik

@warelik warelik commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Problem

When multiple models share a single credential, model failure and recovery handling in MarkResult suffered from two severe defects:

  1. TOCTOU Race & Stale Sibling Resumption: MarkResult checked GetClientModelSuspensionReason and compared it to "invalid_api_key" before calling ResumeClientModel. Under concurrent requests, the suspension reason could change between check and resume. Furthermore, unrelated sibling model successes could erroneously clear model-specific suspensions (e.g. not_found, quota, payment_required), putting dead endpoints back into active rotation and preventing proper failover.
  2. Permanent Suspension of model_not_supported: MarkResult suspends unsupported models with reason model_not_supported and a 12-hour retry window. When reason-scoping was introduced, model_not_supported was omitted from the resumable per-model set, causing models to remain permanently suspended even after subsequent successful requests on the same model.

Fix

  • internal/registry/model_registry.go:749: Added SuspendClientModelReplacingReasons to allow model-specific reasons to atomically overwrite credential-wide ones.
  • internal/registry/model_registry.go:828: Added ResumeClientModelIfReason to atomically inspect the recorded suspension reason and remove the suspension under a single lock only if the reason matches the permitted set.
  • sdk/cliproxy/auth/conductor_cooldown.go:29: Defined resumableCooldownReasons (including model_not_supported, not_found, quota, invalid_api_key, invalid_grant, unauthorized, payment_required) and credentialWideCooldownReasons (invalid_api_key).
  • sdk/cliproxy/auth/conductor_cooldown.go:990: Updated MarkResult to atomically resume sibling models only for credential-wide reasons (ResumeClientModelIfReason), resume own model for all valid per-model reasons, and overwrite credential-wide suspensions when a specific failure occurs (SuspendClientModelReplacingReasons).

Tests

  • TestResumeClientModelIfReason_RacePreservesNewerSuspension: Pins atomic reason checking in registry, verifying that a newer suspension reason is preserved and not accidentally cleared.
  • TestSuspendClientModelReplacingReasons: Pins reason replacement rules when suspending clients.
  • TestManager_ModelNotSupportedSuspensionResumesOnOwnSuccess: Pins that model_not_supported suspensions are cleared upon own-model success.
  • TestManager_ModelSpecificResumableSiblingSuspensionSurvivesSiblingSuccess: Pins that model-specific suspensions (not_found, quota) survive sibling model success.
  • TestManager_CredentialWideSiblingSuspensionResumesOnSiblingSuccess: Pins that credential-wide suspensions (invalid_api_key) resume when a sibling succeeds.
  • TestManager_ModelSpecificFailureOverwritesCredentialWideSuspension: Pins that a model-specific failure replaces a stale credential-wide suspension.

Reverse bite-check

Reverting model_not_supported from resumableCooldownReasons in sdk/cliproxy/auth/conductor_cooldown.go causes TestManager_ModelNotSupportedSuspensionResumesOnOwnSuccess to fail immediately:

=== RUN   TestManager_ModelNotSupportedSuspensionResumesOnOwnSuccess
    conductor_availability_test.go:372: registry model count for model after success = 0, want 1
--- FAIL: TestManager_ModelNotSupportedSuspensionResumesOnOwnSuccess (0.00s)
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	0.461s
FAIL

Reverting reason verification in ResumeClientModelIfReason (internal/registry/model_registry.go) so it unconditionally clears suspensions causes TestResumeClientModelIfReason_RacePreservesNewerSuspension to fail:

=== RUN   TestResumeClientModelIfReason_RacePreservesNewerSuspension
    model_registry_resume_reason_test.go:29: ResumeClientModelIfReason resumed a model whose current suspension should remain, want no-op
--- FAIL: TestResumeClientModelIfReason_RacePreservesNewerSuspension (0.00s)
FAIL
FAIL	github.com/router-for-me/CLIProxyAPI/v7/internal/registry	0.386s
FAIL

Verification

TMPDIR=/Users/warelik/.cache/gotmp GOCACHE=/Users/warelik/.cache/gocache go build ./...
TMPDIR=/Users/warelik/.cache/gotmp GOCACHE=/Users/warelik/.cache/gocache go vet ./internal/registry/... ./sdk/cliproxy/auth/...
gofmt -l internal/registry/model_registry.go internal/registry/model_registry_resume_reason_test.go sdk/cliproxy/auth/conductor_availability_test.go sdk/cliproxy/auth/conductor_cooldown.go
TMPDIR=/Users/warelik/.cache/gotmp GOCACHE=/Users/warelik/.cache/gocache go test -v ./internal/registry/... ./sdk/cliproxy/auth/...

Output:

ok  	github.com/router-for-me/CLIProxyAPI/v7/internal/registry	0.309s
ok  	github.com/router-for-me/CLIProxyAPI/v7/sdk/cliproxy/auth	11.377s

Tooling note

The jbcontext semantic-search CLI is installed but non-functional in this environment
(jbcontext search fails with the OS keychain is not accessible). The equivalent
review passes were performed with the repository's own tooling and manual inspection
instead; this is disclosed for transparency about how the change was reviewed.

W ARELIK added 2 commits August 20, 2026 11:54
Add model_not_supported to resumableCooldownReasons so a model suspended with model_not_supported resumes when that same model succeeds.

The failure sets a 12-hour temporary suspension whose registry counterpart would otherwise never clear even after the cooldown expires and the model serves requests successfully.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant