Add token bucket rate limiting for burst control

[hemna]

Summary

Adds an optional token bucket algorithm alongside the existing sliding window for controlling request bursts. This prevents database lock contention caused by bursty clients (e.g., Gardener CSI driver firing multiple volume creates in parallel).

Configuration

Append ,burst=N to any existing rate limit string:

rates:
  default:
    volumes/volume:
      - action: write
        limit: 100r/m,burst=3

This allows at most 3 concurrent requests while maintaining 100 requests/minute sustained throughput. Requests beyond the burst capacity are delayed or rejected with 429.

Changes

File	Change
rate_limit/lua/redis_token_bucket.lua	New Lua script (Redis Hash-based token bucket)
rate_limit/units.py	parse_token_bucket_rate_limit() parser
rate_limit/backend.py	Load both scripts; route on burst=; new __rate_limit_token_bucket()
rate_limit/tests/test_token_bucket.py	8 parsing + 6 integration tests
TOKEN_BUCKET_SPEC.md	Design spec
README.md	Document both algorithms
docs/configure.md	Token bucket configuration guide

Backward Compatibility

• Existing Xr/Yt configs (no burst=) continue using the sliding window unchanged
• Redis state uses different data structures (Hash vs Sorted Set) — no key collisions
• Mixed configurations (some endpoints sliding window, others token bucket) work correctly

Testing

All 14 new tests pass (8 unit + 6 Redis integration). Existing test suite passes with no regressions.

Context

See TOKEN_BUCKET_SPEC.md for full design rationale and the Cinder API exhaustion investigation.

[hemna]

Background: Cinder API Exhaustion Investigation

This PR addresses a production issue identified during investigation of Cinder API exhaustion in our OpenStack regions.

Root Cause

Gardener's Kubernetes CSI driver issues volume create requests in parallel (4+ concurrent requests per project). Each POST /volumes triggers a SELECT quota_usages ... FOR UPDATE query in Cinder's database layer. When multiple requests arrive simultaneously for the same project:

1. All requests hit the quota_usages table concurrently
2. InnoDB row-level locks queue up on the same project's quota row
3. Lock wait times cascade as each transaction holds the lock while completing the full volume create workflow
4. ProxySQL's connection pool is exhausted waiting for locked connections to return
5. Cinder API pods restart due to connection timeout failures

Why the Sliding Window Doesn't Help

The existing sliding window algorithm (redis_sliding_window.lua) counts requests over a time window but cannot limit concurrency. A project with a 100r/m limit can fire all 100 requests in the first millisecond — they all pass because the window count hasn't been exceeded yet. The damage is done before the rate limit activates.

How the Token Bucket Solves This

With 100r/m,burst=3, at most 3 requests pass simultaneously. The 4th request is delayed ~600ms until a token refills. This ensures max 3 concurrent FOR UPDATE queries per project, eliminating the lock contention cascade while preserving the same 100r/m sustained throughput.

Deployment Plan

1. Merge this PR
2. Update Cinder helm-chart template (_ratelimit.yaml.tpl) with limit: 100r/m,burst=3
3. Roll out to eu-de-2 first, monitor slow query log and ProxySQL connection usage
4. Validate that Gardener volume creates succeed without pod restarts

[joker-at-work]

This article states "The database is paused while a script runs." - did you investigate if we lose parallelism and/or performance because of that?

[hemna]

@joker-at-work Good catch — the article's claim is technically correct: Redis Lua scripts block the server during execution (Redis docs confirm this: "While executing the script, all server activities are blocked during its entire runtime").

However, this PR doesn't introduce that property — the existing sliding window algorithm (redis_sliding_window.lua) already runs as a Lua script with the same blocking semantics. We're swapping one Lua script for another (when burst=N is configured), not adding scripting where there wasn't any.

I ran a benchmark on Redis 7 to compare both scripts directly:

Single-call latency (1000 iterations, fresh keys)

Algorithm	min	p50	p95	p99	max	mean
sliding_window	161µs	300µs	719µs	1519µs	8548µs	385µs
token_bucket	152µs	223µs	496µs	734µs	1709µs	263µs
plain GET (baseline)	—	219µs	404µs	622µs	—	—

Throughput

Scenario	sliding_window	token_bucket
Sequential (5000 calls, 100 keys)	4,193 calls/sec	4,328 calls/sec
Concurrent (10 threads × 500)	14,538 calls/sec	13,150 calls/sec

Why the token bucket is actually slightly faster

The token bucket script issues fewer Redis commands per execution:

• redis_sliding_window.lua: 6 commands (zremrangebyscore, zrangebyscore, zadd, expire, zadd, expire)
• redis_token_bucket.lua: 5 commands (hmget, hset, expire, hset, expire) — and Hash ops are O(1) vs Sorted Set's O(log N)

Practical impact

The p50 of 223µs means Redis can handle ~4,500 token bucket evaluations per second on a single core, sequential. Cinder's API traffic is on the order of tens to low-hundreds of req/sec per region — three orders of magnitude below where script-blocking would become a bottleneck.

If we ever did approach that limit, the mitigation is well-known: shard the Redis instance per service (Cinder has its own Redis sub-chart) or move to Redis Cluster — both are independent of the algorithm choice.

---

Update: Detailed analysis of the linked article

I read through the full article. The "database is paused" line is presented rhetorically — the author defers concrete performance numbers to Part 2. The only quantitative ceiling claim is in their forward reference:

"Redis clusters on average hardware can only handle around 100K requests / second / counter anyway."

100K req/sec/counter is ~1000× our actual Cinder API load.

The article makes several other claims worth examining against this PR:

Article concern	Applies to this PR?	Notes
Lua blocks Redis	Yes, but pre-existing	Sliding window already does this. Token bucket is faster (5 ops vs 6, Hash O(1) vs ZSET O(log N)).
100K req/sec/counter ceiling	Theoretically yes	Cinder traffic is ~100 req/sec. We're 1000× below the ceiling.
"Lost EXPIRE" bug	No	Both scripts call expire on every write path.
Memory exhaustion on rejected requests	No (this PR improves it)	The sliding window adds ZSET entries even on reject (the article calls this out as a defect in the ClassDojo implementation, which our sliding window is based on). The new token bucket only updates a fixed-size Hash — no unbounded growth.
Synchronized windows problem	No	Token bucket has no fixed window boundary.
Race conditions in non-Lua solutions	No	Single Lua script = atomic, which is precisely why we use it.

Notably, the author's own conclusion supports adopting a token bucket over a sliding window:

"The first red flag was that we didn't even consider implementing the gold standard token bucket algorithm..."

"...even with the hopefully useful O(N) memory overhead of the 'sliding log window', you're still not getting any advantage over a token bucket; you just chose it because it's what can be conveniently implemented on top of Redis"

So the article's own logic is: if you must use Redis, use a token bucket via Lua — which is exactly what this PR enables. Beyond the burst-control benefit for Cinder, this PR also fixes the "memory exhaustion on rejected requests" defect the article identifies in the existing sliding window implementation.