Skip to content

fix(ratelimit): set counter TTL atomically on every request - #73

Merged
samcm merged 1 commit into
masterfrom
fix/ratelimit-ttl
Sep 30, 2026
Merged

samcm merged 1 commit into
masterfrom
fix/ratelimit-ttl

Conversation

@samcm

@samcm samcm commented Sep 30, 2026

Copy link
Copy Markdown
Member

Rate limit counters are now updated with INCR, EXPIRE NX and TTL in one MULTI transaction on every request, instead of setting EXPIRE only when INCR returns 1. A counter that lost its TTL, for example when an EXPIRE was not yet replicated at a Redis failover, never regained one and blocked that IP permanently. In production 60 of 73 keys had no TTL, including the node address kubelet probes come from, so lab pods on that node failed every health check.

EXPIRE only ran when INCR returned 1, so a counter that lost its TTL (an
EXPIRE not yet replicated when Redis failed over) never got one again and
blocked its IP forever. 60 of 73 production keys had no TTL, including the
node address kubelet probes come from, which failed every probe on that node.
@samcm
samcm requested a review from mattevans as a code owner September 30, 2026 06:25

@redpandabot redpandabot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR replaces the INCR + count==1-only EXPIRE + separate TTL sequence with a single TxPipelined (MULTI/EXEC) containing INCR, EXPIRE NX, and TTL, so a counter that lost its TTL gets healed on every request. The pipeline ordering is correct (TTL runs after EXPIRE NX, so it reads the restored TTL), the ttl <= 0 fallback to window keeps resetAt sane if EXPIRE NX fails, the error path keeps the existing fail_open/fail_closed handling, and the Allow signature and caller (internal/middleware/ratelimit.go:67) are unchanged. The new regression test matches the described production failure mode, including precondition, denial-while-over-limit, TTL restoration, and reset after expiry. No real problems found.


Reviewed 2 changed file(s) @ 2b7a0e24 — no blocking issues found.
"Make it work, make it right, make it fast." — Kent Beck

@samcm
samcm merged commit 570e400 into master Sep 30, 2026
3 checks passed
@samcm
samcm deleted the fix/ratelimit-ttl branch September 30, 2026 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant