fix(payment): remove dead GlueX provider + make LiFi support lookup non-blocking - #223
Conversation
…ache isArbitraryPaymentTokensSupported is on the quote hot path. Its per-chain 1h cache was filled for all chains at container start, so every chain's entry expired in the same minute; the requests that hit the expiry blocked on a synchronised burst of ~46 GlueX/LiFi calls, stalling the HTTP server and producing ~hourly 502 spikes (observed ~1.7% 5xx on a node under real load). Fix: serve the cached value immediately and refresh in the background when stale (never block a quote); guard so a burst triggers at most one refresh per chain; add 0-10min random TTL jitter so entries no longer expire in lockstep. Cold-cache lookups still fetch synchronously.
The GlueX liquidity list is identical for every chain but was fetched fresh on every per-chain support check, so one payment-token lookup = 23 GlueX GETs, and a burst of quotes multiplied it (observed 943 GlueX fetch failures in a single expiry window). Memoise the list (5min TTL) + single-flight so concurrent callers share one upstream request.
Why it 5xxes the server (checked the logs — good call to ask)They're HTTP 502, not 500 (657 502s, zero app-500s over 3h on the affected node). So the app isn't throwing — the HTTP server goes briefly unresponsive and the LB returns 502. Mechanism, evidence-backed:
So you were right that 46 downstream calls shouldn't 5xx the server — they don't directly. The damage is (a) no memoisation turning it into ~900 calls, and (b) the health endpoint sharing the external dependency and pulling the backend from rotation. Fix in this PR now covers all three
Follow-up worth considering separately: |
Per review (Venkatesh): GlueX is defunct, so rather than caching around its calls we remove the provider entirely. Deletes providers/gluex, its config (GLUEX_API_KEY/GLUEX_PARTNER_UNIQUE_ID), the gluexRouterGasLimit, the provider-type union member, and the GlueX chain-support methods in ChainsService. LiFi becomes the sole arbitrary-payment-token provider. isArbitraryPaymentTokensSupported keeps stale-while-revalidate + per-chain TTL jitter + single-flight (now LiFi-only): it sits on the /v1/info health-check path, so it must never block on the upstream, else a slow provider pulls the single backend out of LB rotation (the ~hourly 502 root cause).
|
Pivoted per your call, @venkatesh — instead of caching around GlueX, removed the provider entirely (it's defunct). LiFi is now the only arbitrary-payment-token provider. Kept stale-while-revalidate + jitter on the LiFi lookup specifically because it's on the |
Problem
~hourly clusters of HTTP 502 on mee-node under real traffic (~1.7% 5xx), wall-clock-aligned to container-start + N h. Root cause (evidence in comments):
isArbitraryPaymentTokensSupported(on the/quoteand/v1/infohealth-check path) did live GlueX+LiFi calls behind a per-chain 1h cache that expired for all chains in the same minute.getGlueXChainInfowasn't memoised, so the expiry burst = ~900 GlueX GETs; GlueX rate-limited;/v1/infowent slow; the LB pulled the single backend out of rotation → 502 for all traffic.Change (per review: GlueX is dead, remove it)
providers/gluex, its config (GLUEX_API_KEY,GLUEX_PARTNER_UNIQUE_ID),gluexRouterGasLimit, the"gluex"provider-type member, andChainsServiceGlueX methods. LiFi is now the sole arbitrary-payment-token provider./quoteor/v1/infoon the upstream), single-flight per chain, and per-chain TTL jitter (no synchronised expiry).Notes
tsc --noEmit) + biome-clean. The pre-existingrpc-manager.service.ts:140build error on develop is unrelated to this change./v1/info(health-critical) arguably shouldn't depend on any third-party API.