Skip to content

fix(auth): throttle token-exchange retries with 429 + Retry-After - #20947

Open
toufali wants to merge 1 commit into
mainfrom
auth-retry-after
Open

fix(auth): throttle token-exchange retries with 429 + Retry-After#20947
toufali wants to merge 1 commit into
mainfrom
auth-retry-after

Conversation

@toufali

@toufali toufali commented Jul 30, 2026

Copy link
Copy Markdown
Member

Because

  • Repeated rejected token-exchange requests on /v1/oauth/token were answered identically every time, giving the client no signal to back off.
  • 429s carried retryAfter in the body only, so clients that back off on the Retry-After header saw nothing.
  • Customs v2 reports that wait in milliseconds, where the header requires seconds.

This pull request

  • Rate limits the token-exchange grant on the subject_token hash, ahead of the grant's db reads.
  • Adds a token block key to @fxa/accounts/rate-limit, for actions checked before the account behind a credential is known.
  • Sends Retry-After in whole seconds, with an onPreResponse backstop so every 429 carries the header.
  • Normalizes the legacy customs retryAfter to milliseconds so the payload matches customs v2 and fxa-settings.
  • Folds in FXA-14217, which was cancelled in favour of landing it here.

Issue that this pull request solves

Closes: FXA-14081

Checklist

Put an x in the boxes that apply

  • My commit is GPG signed.
  • If applicable, I have modified or added tests which pass locally.
  • I have added necessary documentation (if appropriate).
  • I have verified that my changes render correctly in RTL (if appropriate).
  • I have manually reviewed all AI generated code.

How to review (Optional)

  • Key files: lib/routes/oauth/token.js, lib/customs.js, lib/server.js, config/rate-limit-rules.txt.
  • Suggested order: rules file → token.jscustoms.js → errors lib → server.js → tests.
  • A successful exchange revokes the subject token, so a repeat with the same token is a retry of a rejection — which is why counting every attempt is equivalent to counting rejections.
  • Riskiest part: the retryAfter body value on the legacy customs path changes from seconds to ms.
  • Customs v2 already reported ms and is the path that actually serves 429s, so there is no client-visible change; Retry-After is now correct on both.

Other information (Optional)

Follow-ups, not addressed here:

  • Token blocks self-expire after 15 minutes and are not searchable or clearable through admin tooling.
  • The throttle counts every exchange attempt, so repeated backend failures on the same token consume its budget.
  • Pre-existing: ip_uid is missing from findBans in the rate-limit lib, so an ip_uid : ban rule would never fire.

@toufali
toufali force-pushed the auth-retry-after branch from 98eb8b1 to 3f165c0 Compare July 30, 2026 21:44
@toufali
toufali marked this pull request as ready for review July 30, 2026 21:47
@toufali
toufali requested a review from a team as a code owner July 30, 2026 21:47
Copilot AI review requested due to automatic review settings July 30, 2026 21:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR strengthens throttling and client backoff behavior around OAuth token exchange in fxa-auth-server, by adding a token-hash keyed rate limit for the RFC 8693 token-exchange flow and ensuring 429 Too Many Requests responses reliably include a standards-compliant Retry-After header (seconds) while keeping the response body retryAfter aligned to milliseconds (customs v2 / fxa-settings expectations).

Changes:

  • Add a new rate-limit “blockOn” dimension (token) and apply it to tokenExchange so repeated retries on the same subject_token are throttled early (before DB reads).
  • Normalize legacy customs retryAfter to milliseconds in the response payload, and ensure Retry-After is always present in whole seconds (including a server-level backstop).
  • Extend tests/docs/config rules to cover the new token-hash throttling behavior and header/payload semantics.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
packages/fxa-auth-server/test/remote/oauth_token_route.in.spec.ts Adds a customs.checkToken mock for remote oauth token route tests.
packages/fxa-auth-server/test/mocks.js Extends customs mock surface to include checkToken.
packages/fxa-auth-server/lib/server.js Adds an onPreResponse backstop to set Retry-After for 429 responses when missing.
packages/fxa-auth-server/lib/server.in.spec.ts Updates/extends integration tests to validate Retry-After behavior and defaulting.
packages/fxa-auth-server/lib/routes/oauth/token.spec.ts Adds tests asserting token-exchange calls customs.checkToken with the subject token hash and short-circuits DB reads when throttled.
packages/fxa-auth-server/lib/routes/oauth/token.js Calls customs.checkToken for token-exchange requests before DB reads.
packages/fxa-auth-server/lib/routes/oauth/index.js Wires customs dependency through to the token route.
packages/fxa-auth-server/lib/routes/index.js Threads customs through the route construction chain.
packages/fxa-auth-server/lib/customs.spec.ts Updates tests for legacy retryAfter ms normalization and adds coverage for checkToken.
packages/fxa-auth-server/lib/customs.js Implements checkToken (v2-only), normalizes legacy customs retryAfter seconds → ms, and passes token through to rate-limit v2 checks.
packages/fxa-auth-server/docs/swagger/auth-server-api.ts Documents that rate-limiting 429 payload retryAfter is milliseconds while Retry-After header remains seconds.
packages/fxa-auth-server/config/rate-limit-rules.txt Adds tokenExchange rules for both token and ip dimensions.
packages/fxa-auth-server/config/index.spec.ts Adds a test ensuring shipped rate-limit rules parse and include expected tokenExchange entries.
libs/accounts/rate-limit/src/lib/rate-limit.ts Adds token support to ban lookup checks.
libs/accounts/rate-limit/src/lib/rate-limit.spec.ts Adds/updates tests for token bans, required options, and accepted blockOn values.
libs/accounts/rate-limit/src/lib/models.ts Extends BlockOn to include token.
libs/accounts/rate-limit/src/lib/config.ts Extends rule parsing validation/error message to accept token.
libs/accounts/rate-limit/README.md Documents token as a valid blockOn dimension and its intended usage.
libs/accounts/errors/src/util.ts Adds helpers for default retry-after ms normalization and header value conversion.
libs/accounts/errors/src/index.spec.ts Updates tests to validate ms semantics and whole-second Retry-After rounding/defaulting.
libs/accounts/errors/src/app-error.ts Changes tooManyRequests to normalize retryAfter in ms and emit whole-second Retry-After values.

Comment thread libs/accounts/errors/src/app-error.ts
Comment thread libs/accounts/rate-limit/src/lib/rate-limit.ts
Comment thread packages/fxa-auth-server/docs/swagger/auth-server-api.ts Outdated
@toufali
toufali force-pushed the auth-retry-after branch from 3f165c0 to 82ead2c Compare July 31, 2026 17:31
@LZoog

LZoog commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

@toufali I haven't looked at these changes but just confirming, did you check that the client code (Mobile being the most important, and Desktop will use this later but does not currently) will handle this as expected? The ticket says it will, but that was also me pointing Claude at the codebase to see what it'd say.

Good to just double check the findings there and/or ping Jonathan Almeida (mobile) / Mark Hammond (desktop, but he did some of this work for mobile) about it if it's not clear. Relay on iOS uses the token exchange too but since we currently always grant that maybe it doesn't matter there. (Android and iOS share a Rust layer where the actual token exchange call to us happens too but not sure where the handling would be.) Thanks!

@dschom
dschom self-requested a review August 4, 2026 18:01
extraData,
{
'retry-after': retryAfter.toString(),
'retry-after': retryAfterHeaderValue(retryAfter),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like profile server has it's own error class. It's using seconds. Whould probably update this to use ms to be consistent? See,

'retry-after': retryAfter,

it('handles customs block', async () => {
customs.checkIpOnly = jest.fn(async () => {
throw error.tooManyRequests(100, 'foo');
throw error.tooManyRequests(100000, 'foo');

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This made check for other instances. I found a spot where maybe we missed converting to ms. See

let result = AppError.tooManyRequests(900, 'in 15 minutes');

@dschom dschom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Maybe do one more search for 'AppError.tooManyRequests' and make sure ms are being used in all cases. I onlty found a couple inconsistentcy in tests files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants