Skip to content

fix(payment): remove dead GlueX provider + make LiFi support lookup non-blocking - #223

Merged
fichiokaku merged 4 commits into
developfrom
fix/arbitrary-token-cache-stampede
Jul 31, 2026
Merged

fix(payment): remove dead GlueX provider + make LiFi support lookup non-blocking#223
fichiokaku merged 4 commits into
developfrom
fix/arbitrary-token-cache-stampede

Conversation

@fichiokaku

@fichiokaku fichiokaku commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Problem

~hourly clusters of HTTP 502 on mee-node under real traffic (~1.7% 5xx), wall-clock-aligned to container-start + N h. Root cause (evidence in comments): isArbitraryPaymentTokensSupported (on the /quote and /v1/info health-check path) did live GlueX+LiFi calls behind a per-chain 1h cache that expired for all chains in the same minute. getGlueXChainInfo wasn't memoised, so the expiry burst = ~900 GlueX GETs; GlueX rate-limited; /v1/info went slow; the LB pulled the single backend out of rotation → 502 for all traffic.

Change (per review: GlueX is dead, remove it)

  • Remove the GlueX provider entirely: providers/gluex, its config (GLUEX_API_KEY, GLUEX_PARTNER_UNIQUE_ID), gluexRouterGasLimit, the "gluex" provider-type member, and ChainsService GlueX methods. LiFi is now the sole arbitrary-payment-token provider.
  • Harden the remaining LiFi lookup so this class of stall can't recur: stale-while-revalidate (never block /quote or /v1/info on the upstream), single-flight per chain, and per-chain TTL jitter (no synchronised expiry).

Notes

  • Behaviourally null if GlueX is truly defunct (no chains report gluex support today).
  • Type-clean (tsc --noEmit) + biome-clean. The pre-existing rpc-manager.service.ts:140 build error on develop is unrelated to this change.
  • Follow-up worth discussing separately: /v1/info (health-critical) arguably shouldn't depend on any third-party API.

…ache

isArbitraryPaymentTokensSupported is on the quote hot path. Its per-chain
1h cache was filled for all chains at container start, so every chain's
entry expired in the same minute; the requests that hit the expiry blocked
on a synchronised burst of ~46 GlueX/LiFi calls, stalling the HTTP server
and producing ~hourly 502 spikes (observed ~1.7% 5xx on a node under real
load).

Fix: serve the cached value immediately and refresh in the background when
stale (never block a quote); guard so a burst triggers at most one refresh
per chain; add 0-10min random TTL jitter so entries no longer expire in
lockstep. Cold-cache lookups still fetch synchronously.
@fichiokaku
fichiokaku requested a review from vr16x July 31, 2026 08:21
The GlueX liquidity list is identical for every chain but was fetched
fresh on every per-chain support check, so one payment-token lookup =
23 GlueX GETs, and a burst of quotes multiplied it (observed 943 GlueX
fetch failures in a single expiry window). Memoise the list (5min TTL)
+ single-flight so concurrent callers share one upstream request.
@fichiokaku

Copy link
Copy Markdown
Collaborator Author

Why it 5xxes the server (checked the logs — good call to ask)

They're HTTP 502, not 500 (657 502s, zero app-500s over 3h on the affected node). So the app isn't throwing — the HTTP server goes briefly unresponsive and the LB returns 502. Mechanism, evidence-backed:

  1. getGlueXChainInfo() was not memoised — every per-chain support check did a fresh GET /liquidity. One payment-token lookup = 23 GlueX GETs; a burst of quotes at cache-expiry multiplied it. Logs show 943 "Failed to fetch gluex token support" clustered in the ~5-min expiry window (08:33–08:38) — GlueX rate-limiting/timing-out under the self-inflicted burst. Not 46 calls — ~900+.
  2. The per-chain 1h result cache was the only throttle, and all chains were filled at startup → synchronised hourly expiry removes it in lockstep.
  3. The 502 to end users: the LB backend health check hits /v1/info (5s timeout ×3). /v1/info's health computation runs the t-s-d check → getSupportedPaymentTokens() → the same GlueX path. During the burst /v1/info goes slow → LB marks the single backend UNHEALTHY → 502 for ALL traffic until it recovers. That's why it takes out everything, not just quotes, and why it's a 5xx on the server itself rather than a caught downstream error.

So you were right that 46 downstream calls shouldn't 5xx the server — they don't directly. The damage is (a) no memoisation turning it into ~900 calls, and (b) the health endpoint sharing the external dependency and pulling the backend from rotation.

Fix in this PR now covers all three

  • stale-while-revalidate on isArbitraryPaymentTokensSupported → neither /quote nor /v1/info ever blocks on GlueX/LiFi (kills the 502 amplifier).
  • memoise + single-flight getGlueXChainInfo (5min TTL) → burst collapses to one upstream call (kills the 943→~1).
  • per-chain TTL jitter → no more lockstep expiry.

Follow-up worth considering separately: /v1/info (health-critical) probably shouldn't transitively depend on third-party APIs at all.

Per review (Venkatesh): GlueX is defunct, so rather than caching around its
calls we remove the provider entirely. Deletes providers/gluex, its config
(GLUEX_API_KEY/GLUEX_PARTNER_UNIQUE_ID), the gluexRouterGasLimit, the
provider-type union member, and the GlueX chain-support methods in
ChainsService. LiFi becomes the sole arbitrary-payment-token provider.

isArbitraryPaymentTokensSupported keeps stale-while-revalidate + per-chain
TTL jitter + single-flight (now LiFi-only): it sits on the /v1/info
health-check path, so it must never block on the upstream, else a slow
provider pulls the single backend out of LB rotation (the ~hourly 502
root cause).
@fichiokaku fichiokaku changed the title fix(chains): stop hourly 502 stalls from arbitrary-payment-token cache stampede fix(payment): remove dead GlueX provider + make LiFi support lookup non-blocking Jul 31, 2026
@fichiokaku

Copy link
Copy Markdown
Collaborator Author

Pivoted per your call, @venkatesh — instead of caching around GlueX, removed the provider entirely (it's defunct). LiFi is now the only arbitrary-payment-token provider. Kept stale-while-revalidate + jitter on the LiFi lookup specifically because it's on the /v1/info health-check path — that shared dependency was the actual 502 amplifier (a slow upstream pulled the single backend from LB rotation), so it shouldn't block regardless of which provider it is. Diff is net-clean in Files changed; earlier commits (jitter, gluex memoisation) are superseded by the removal.

@fichiokaku
fichiokaku merged commit 1e29942 into develop Jul 31, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants