Skip to content

Fix Redis crash loop from unhandled socket errors#18

Open
UsamaSadiq wants to merge 2 commits into
foss-sandboxfrom
usama/fix-redis-crash-loop
Open

Fix Redis crash loop from unhandled socket errors#18
UsamaSadiq wants to merge 2 commits into
foss-sandboxfrom
usama/fix-redis-crash-loop

Conversation

@UsamaSadiq

@UsamaSadiq UsamaSadiq commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Description

Twenty pods crash-loop on the FOSS sandbox because the node-redis clients (session storage and cache storage) lack error handlers. When Memorystore (Valkey) drops idle TLS connections, node-redis emits an error event with no listener, which Node treats as fatal (exit 1). The .connect().catch() in session storage also re-throws, producing an unhandled rejection that breaks built-in auto-reconnect.

This patch fixes both the session-storage (connect-redis) and cache-storage (cache-manager-redis-yet) Redis clients:

  • Adds .on('error', ...) handler to both clients to prevent unhandled error crashes
  • Replaces re-throwing .catch() with log-only handler so auto-reconnect works (session client)
  • Switches cache storage from redisStore (which creates its own client without error handling) to redisInsStore with a pre-built client that has a proper error handler attached
  • Adds pingInterval: 30s to keep idle connections alive and avoid Memorystore reaping
  • Adds socket.keepAlive + keepAliveInitialDelay for TCP-level keepalive
  • Adds socket.reconnectStrategy with exponential backoff capped at 5s

The ioredis clients (BullMQ, pub/sub) are not affected -- they have built-in reconnect and error handling.

Testing

  • Rebuild Twenty Docker image and redeploy to sandbox
  • Confirm pod stays up past the previous crash window (~1-2 min after boot)
  • Wait 10+ minutes or kill the Redis connection from Memorystore side -- pod should log the error and reconnect, not crash
  • Check logs for Session Redis client error: / Cache Redis client error: lines (expected, non-fatal) instead of SocketClosedUnexpectedlyError stack traces

node-redis session and cache clients crash the process when
Memorystore drops idle TLS connections. Add error handlers,
pingInterval, TCP keepalive, and reconnect strategy to both
the session-storage and cache-storage Redis clients.
@UsamaSadiq UsamaSadiq force-pushed the usama/fix-redis-crash-loop branch from 089c3b9 to 8645ab6 Compare July 9, 2026 12:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant