Skip to content

feat: add REDIS_OP_TIMEOUT for per-operation Redis timeout#1188

Open
hoangcn95 wants to merge 3 commits into
envoyproxy:mainfrom
hoangcn95:feat/redis-op-timeout
Open

feat: add REDIS_OP_TIMEOUT for per-operation Redis timeout#1188
hoangcn95 wants to merge 3 commits into
envoyproxy:mainfrom
hoangcn95:feat/redis-op-timeout

Conversation

@hoangcn95

@hoangcn95 hoangcn95 commented Jul 13, 2026

Copy link
Copy Markdown

Description

REDIS_TIMEOUT only bounds the dial. Once connected, radix v4 has no read/write timeout — a command is bounded only by the caller's context, and the hot rate-limit path passes the inbound gRPC context, which usually carries no deadline. When Redis stalls, calls park indefinitely, goroutines pile up, and the pod OOMs.

This PR adds two new settings, REDIS_OP_TIMEOUT and REDIS_PERSECOND_OP_TIMEOUT, both disabled by default. When enabled, each Redis command or pipeline gets its context wrapped with a deadline, bounding per-operation latency independent of the caller's context.

Variable Description Default
REDIS_OP_TIMEOUT Bounds a single Redis command or pipeline using context.WithTimeout. Set to 0 to disable. 0
REDIS_PERSECOND_OP_TIMEOUT Same, for the per-second pool. 0

Architecture

flowchart LR
    A[gRPC request] --> B[DoLimit]
    B --> C[PipeDo / DoCmd]
    C --> D{opTimeout > 0?}
    D -->|yes| E[context.WithTimeout deadline]
    D -->|no| F[caller ctx, no deadline]
    E --> G[radix client]
    F --> G
    G --> H[(Redis)]

    style E fill:#2a9d5c,color:#fff
    style C fill:#2563eb,color:#fff
Loading

Legend: 🟢 new — 🔵 rewired

Test Result

Added unit tests in src/redis/driver_impl_test.go using the existing recordingRedisClient / testAction fakes (whitebox tests already in that package):

  • TestPipeDoReturnsWithinOpTimeoutWhenRedisHangs / TestDoCmdReturnsWithinOpTimeoutWhenRedisHangs: simulate a Redis that never responds (blockDelay: time.Hour) and assert that with opTimeout = 30ms, both PipeDo and DoCmd return a context.DeadlineExceeded error in well under 500ms instead of blocking indefinitely.
  • TestPipeDoAppliesOpTimeoutDeadlineToContext / TestDoCmdAppliesOpTimeoutDeadlineToContext: assert the context handed to the underlying radix client carries a deadline when opTimeout > 0.
  • TestPipeDoWithoutOpTimeoutLeavesContextUnbounded / TestDoCmdWithoutOpTimeoutLeavesContextUnbounded: assert no deadline is added when opTimeout == 0, preserving current behavior exactly.

Settings parsing is covered in src/settings/settings_test.go (TestRedisOpTimeout_Default, TestRedisOpTimeout_Configured), mirroring the style of the settings tests added in #1018.

All of the above pass under go test -race, along with the full existing suite (./...).

Load test

  • TBD

Peter Nguyen added 3 commits July 13, 2026 16:24
Signed-off-by: Peter Nguyen <petnguyen@axon.com>
Clarify that a timed-out op fails the request (counted in redis_error,
returned as a gRPC error) rather than silently degrading, so operators
know what to expect on dashboards after enabling this timeout.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant