test(auth): e2e failover doctrine harness

[warelik]

Summary

Adds sdk/cliproxy/auth/e2e_failover_doctrine_test.go, a scripted local integration harness that drives the real Manager/scheduler/conductor stack through the P1 failover doctrine scenarios.

Scenarios covered:

1. Sub-second Retry-After 429 escalation/rotation (t.SKIP - fix(auth): floor quota cooldown at the escalating ladder #198)
2. Single transient 503 cooldown default (actual: 60 s legacy, fix(auth): lower transient error cooldown default to 10s #205)
3. Empty completion rotation, non-stream and stream
4. In-stream provider error in 200 SSE bootstrap (t.SKIP - fix(auth): rotate on in-stream provider errors during bootstrap #195)
5. All-but-one accounts dead still serves
6. Session affinity stays healthy after transient blip
7. API-key model alias sibling discovery (t.SKIP - fix(auth): rotate API-key model pools for all configured providers #208)

Skipped tests document the current main violations and are linked to the open PRs that fix them, so the suite remains green while exposing the gaps.

Full report: /Users/warelik/Developer/AIRouters/reports/e2e-failover.md

Test plan

• go test -count=1 ./sdk/cliproxy/auth passes
• go build -o /tmp/cli-proxy-api ./cmd/server succeeds

[warelik]

Review: changes requested

The harness itself is solid: every referenced symbol matches current main (NewManager, SetRetryConfig, SessionAffinitySelector, SetTransientErrorCooldownSeconds, the global registry, internalconfig Gemini types), the fake satisfies ProviderExecutor, and I traced each active scenario through the conductor:

• TestSingleTransient503Cooldown: 503 -> applyAuthFailureState sets Unavailable + NextRetryAfter = now + transientErrorCooldown (60 s, conductor_cooldown.go:2017-2020); maxWait=0 short-circuits shouldRetryAfterError, so no minute-long sleep. OK.
• Empty-completion rotation (both variants): isEmptyCompletionPayload on the non-stream path (conductor_execution.go:462) and isTerminalEmpty on the stream bootstrap path both exist on main and rotate as scripted. OK.
• TestAllButOneDeadStillServes and TestAffinityStaysHealthyAfterTransientBlip: rotation and session-stickiness paths check out; the global cooldown override is restored; no interaction with the package parallel tests (they resume after sequential tests finish). OK.

But two of the three skipped doctrine tests are scripted in ways that cannot pass once their fix PRs (#195, #208) merge and the skips are removed, which undercuts the stated purpose of the suite. Details inline; both fixes are small (per-model error keying in the fake; flip one assertion).

Non-blocking nit: the affinity test restores the global with a hardcoded SetTransientErrorCooldownSeconds(0); the package convention (cooldown_backoff_test.go, connection_lifecycle_cooldown_test.go) saves and restores the previous value via t.Cleanup, which stays correct if the default ever changes.

[warelik]

Both skipped-test scripting issues are fixed:

1. TestAliasedAccountDiscoveredWhenSiblingsDie now uses exec.failFirstN = 1 and exec.executePayloads[auth.ID]. doctrineExecutor.Execute was reordered so the first sibling call fails with 429 and the second returns the configured payload (commit 2bd76fa).

2. TestInStreamProviderErrorDuringBootstrap now asserts err == nil and reads fallback content from stream.Chunks; ids are sorted so the error auth is the deterministic first pick (commit cee2b13).

Non-blocking nit: TestAffinityStaysHealthyAfterTransientBlip now saves the previous transient cooldown and restores it via t.Cleanup (commit 7f282be).

go test -count=1 ./... passes.

[warelik]

Review: approved

Re-review of 7f282be after the previous round. All three findings are fixed correctly:

1. In-stream bootstrap test (#195): the assertion is now consistent with in-call rotation — success + fallback content + cooled first auth + fallback tried. The new sort.Strings(ids) determinism claim checks out: availableAuthsFromPriorityBuckets sorts candidates by ID (selector.go:349-351), and a fresh RoundRobin cursor starts at index 0, so ids[0] is provably the first pick.

2. Aliased sibling test (#208): failFirstN = 1 + reordered payload check in the fake means the first sibling attempt fails with a bare 429 and the second sibling returns the candidates payload. Traced through executeMixedOnce: the 429 is not credential-scoped, so the models loop continues to the second sibling within the same auth — both models are recorded, models[0] != models[1]. The fake reorder (payloads now below firstExecuteEmpty/failFirstN) does not regress the other scenarios: no other test sets executePayloads, and executeErrs still wins first for the 503/Retry-After scenarios.

3. Affinity test: save/load/restore of transientErrorCooldownSeconds via t.Cleanup now matches the package convention.

The active (non-skipped) scenarios were re-traced against current main and still hold: 503 -> 60 s transient cooldown with Unavailable set; empty-completion rotation in both modes; last-live-auth serving; affinity stickiness with the 1 s cooldown override.