fix(auth): throttle token-exchange retries with 429 + Retry-After - #20947
fix(auth): throttle token-exchange retries with 429 + Retry-After#20947toufali wants to merge 1 commit into
Conversation
98eb8b1 to
3f165c0
Compare
There was a problem hiding this comment.
Pull request overview
This PR strengthens throttling and client backoff behavior around OAuth token exchange in fxa-auth-server, by adding a token-hash keyed rate limit for the RFC 8693 token-exchange flow and ensuring 429 Too Many Requests responses reliably include a standards-compliant Retry-After header (seconds) while keeping the response body retryAfter aligned to milliseconds (customs v2 / fxa-settings expectations).
Changes:
- Add a new rate-limit “blockOn” dimension (
token) and apply it totokenExchangeso repeated retries on the samesubject_tokenare throttled early (before DB reads). - Normalize legacy customs
retryAfterto milliseconds in the response payload, and ensureRetry-Afteris always present in whole seconds (including a server-level backstop). - Extend tests/docs/config rules to cover the new token-hash throttling behavior and header/payload semantics.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/fxa-auth-server/test/remote/oauth_token_route.in.spec.ts | Adds a customs.checkToken mock for remote oauth token route tests. |
| packages/fxa-auth-server/test/mocks.js | Extends customs mock surface to include checkToken. |
| packages/fxa-auth-server/lib/server.js | Adds an onPreResponse backstop to set Retry-After for 429 responses when missing. |
| packages/fxa-auth-server/lib/server.in.spec.ts | Updates/extends integration tests to validate Retry-After behavior and defaulting. |
| packages/fxa-auth-server/lib/routes/oauth/token.spec.ts | Adds tests asserting token-exchange calls customs.checkToken with the subject token hash and short-circuits DB reads when throttled. |
| packages/fxa-auth-server/lib/routes/oauth/token.js | Calls customs.checkToken for token-exchange requests before DB reads. |
| packages/fxa-auth-server/lib/routes/oauth/index.js | Wires customs dependency through to the token route. |
| packages/fxa-auth-server/lib/routes/index.js | Threads customs through the route construction chain. |
| packages/fxa-auth-server/lib/customs.spec.ts | Updates tests for legacy retryAfter ms normalization and adds coverage for checkToken. |
| packages/fxa-auth-server/lib/customs.js | Implements checkToken (v2-only), normalizes legacy customs retryAfter seconds → ms, and passes token through to rate-limit v2 checks. |
| packages/fxa-auth-server/docs/swagger/auth-server-api.ts | Documents that rate-limiting 429 payload retryAfter is milliseconds while Retry-After header remains seconds. |
| packages/fxa-auth-server/config/rate-limit-rules.txt | Adds tokenExchange rules for both token and ip dimensions. |
| packages/fxa-auth-server/config/index.spec.ts | Adds a test ensuring shipped rate-limit rules parse and include expected tokenExchange entries. |
| libs/accounts/rate-limit/src/lib/rate-limit.ts | Adds token support to ban lookup checks. |
| libs/accounts/rate-limit/src/lib/rate-limit.spec.ts | Adds/updates tests for token bans, required options, and accepted blockOn values. |
| libs/accounts/rate-limit/src/lib/models.ts | Extends BlockOn to include token. |
| libs/accounts/rate-limit/src/lib/config.ts | Extends rule parsing validation/error message to accept token. |
| libs/accounts/rate-limit/README.md | Documents token as a valid blockOn dimension and its intended usage. |
| libs/accounts/errors/src/util.ts | Adds helpers for default retry-after ms normalization and header value conversion. |
| libs/accounts/errors/src/index.spec.ts | Updates tests to validate ms semantics and whole-second Retry-After rounding/defaulting. |
| libs/accounts/errors/src/app-error.ts | Changes tooManyRequests to normalize retryAfter in ms and emit whole-second Retry-After values. |
3f165c0 to
82ead2c
Compare
|
@toufali I haven't looked at these changes but just confirming, did you check that the client code (Mobile being the most important, and Desktop will use this later but does not currently) will handle this as expected? The ticket says it will, but that was also me pointing Claude at the codebase to see what it'd say. Good to just double check the findings there and/or ping Jonathan Almeida (mobile) / Mark Hammond (desktop, but he did some of this work for mobile) about it if it's not clear. Relay on iOS uses the token exchange too but since we currently always grant that maybe it doesn't matter there. (Android and iOS share a Rust layer where the actual token exchange call to us happens too but not sure where the handling would be.) Thanks! |
| extraData, | ||
| { | ||
| 'retry-after': retryAfter.toString(), | ||
| 'retry-after': retryAfterHeaderValue(retryAfter), |
There was a problem hiding this comment.
It looks like profile server has it's own error class. It's using seconds. Whould probably update this to use ms to be consistent? See,
fxa/packages/fxa-profile-server/lib/error.js
Line 187 in 82ead2c
| it('handles customs block', async () => { | ||
| customs.checkIpOnly = jest.fn(async () => { | ||
| throw error.tooManyRequests(100, 'foo'); | ||
| throw error.tooManyRequests(100000, 'foo'); |
There was a problem hiding this comment.
This made check for other instances. I found a spot where maybe we missed converting to ms. See
dschom
left a comment
There was a problem hiding this comment.
Looks good to me! Maybe do one more search for 'AppError.tooManyRequests' and make sure ms are being used in all cases. I onlty found a couple inconsistentcy in tests files.
Because
/v1/oauth/tokenwere answered identically every time, giving the client no signal to back off.retryAfterin the body only, so clients that back off on theRetry-Afterheader saw nothing.This pull request
subject_tokenhash, ahead of the grant's db reads.tokenblock key to@fxa/accounts/rate-limit, for actions checked before the account behind a credential is known.Retry-Afterin whole seconds, with anonPreResponsebackstop so every 429 carries the header.retryAfterto milliseconds so the payload matches customs v2 and fxa-settings.Issue that this pull request solves
Closes: FXA-14081
Checklist
Put an
xin the boxes that applyHow to review (Optional)
lib/routes/oauth/token.js,lib/customs.js,lib/server.js,config/rate-limit-rules.txt.token.js→customs.js→ errors lib →server.js→ tests.retryAfterbody value on the legacy customs path changes from seconds to ms.Retry-Afteris now correct on both.Other information (Optional)
Follow-ups, not addressed here:
ip_uidis missing fromfindBansin the rate-limit lib, so anip_uid : banrule would never fire.