1. Introduction — Why Rate Limiting Matters

A rate limiter is a control mechanism that restricts the number of requests a client can make to a service within a defined time window. Rate limiting is one of the most fundamental building blocks of distributed systems, yet it is frequently underestimated in terms of complexity and importance. Every major technology company on the planet relies on rate limiting to protect their infrastructure, enforce business policies, and ensure fair resource allocation across millions of users.

Consider the scale of modern internet platforms. Twitter allows 300 tweets per three hours for unauthenticated users. GitHub permits 5,000 API requests per hour for authenticated accounts and 60 per hour for unauthenticated requests. Stripe enforces 100 read operations per second on its API. Google Cloud Platform imposes per-user quotas on every single API endpoint. Without rate limiting, a single malicious or poorly-behaved client could exhaust the resources of an entire platform, causing cascading failures that affect millions of legitimate users.

Rate limiting serves several critical purposes in a production system. First, it prevents abuse and denial-of-service attacks by capping the traffic any single source can generate. Second, it protects backend services from being overwhelmed during traffic spikes or flash crowds. Third, it enforces API quotas tied to business tiers, allowing companies to monetize access to their services by offering free, pro, and enterprise plans with different request allowances. Fourth, it reduces operational costs by throttling expensive operations such as complex database joins or machine learning inference calls. Fifth, it maintains quality of service during peak periods by ensuring no single consumer degrades performance for everyone else.

A rate limiter can be deployed at multiple layers in a system. At the API Gateway level, it acts as middleware that intercepts every inbound request before it reaches any backend service. As a sidecar microservice, it runs alongside application pods and makes local decisions. Embedded in the application layer, it becomes a library that each service calls directly. Each deployment model has distinct trade-offs in terms of latency, consistency, and operational complexity. This guide focuses on the most common and scalable approach: a centralized rate limiter service backed by Redis Cluster, consumed by API Gateways via gRPC, and capable of handling over one million decisions per second across global data centers.

Throughout this guide, we will dissect every aspect of designing a production-grade distributed rate limiter. We begin with requirements gathering, move through capacity estimation and algorithm selection, dive deep into Redis-backed distributed state management, walk through a complete C# implementation, and conclude with reliability engineering, observability patterns, and real-world interview questions. By the end, you will have a comprehensive blueprint that you can adapt to your specific infrastructure and business needs.

2. Functional Requirements

Before writing any code, we must precisely define what the rate limiter needs to do. Functional requirements determine the behavior of the system from the client's perspective. Gathering these requirements early prevents costly redesigns later.

Per-client request limiting. The system must limit the number of requests each client can make within a configurable time window. The client identity can be an API key, user ID, IP address, or any arbitrary string that the caller provides. This is the most fundamental requirement and must work for all endpoint types.

Per-endpoint rate limits. Different API endpoints have different cost profiles. A search endpoint that triggers a full-text index scan should have a lower limit than a simple status check endpoint. The rate limiter must support independent limits for each endpoint path, not just per-client.

Configurable time windows. Limits must be enforceable per second, per minute, per hour, or per day. The window size must be configurable per rule without requiring code changes or redeployments.

Burst allowance. Many legitimate use cases involve short bursts of traffic. A web application might load 50 resources on a single page. The rate limiter must support a bucket size that exceeds the steady-state rate, allowing temporary bursts while still enforcing the average over time.

Response headers. Every response must include standard rate limit headers: X-RateLimit-Limit (the maximum allowed), X-RateLimit-Remaining (how many requests are left), and X-RateLimit-Reset (the Unix timestamp when the window resets). When a request is denied, the response must include a Retry-After header indicating the number of seconds until the client can retry.

API key tiers. Rate limits must vary by subscription tier. A free-tier user might get 100 requests per minute, a pro-tier user gets 1,000, and an enterprise user gets 10,000. The tier information is resolved from the API key and applied automatically.

Distributed enforcement. When running behind a load balancer across multiple rate limiter instances, the limit must be enforced globally, not per node. If the limit is 100 per minute, a client must not be able to send 100 requests to each of 10 nodes for a total of 1,000.

Whitelist support. Certain IP addresses or API keys (internal services, health checks, monitoring probes) should bypass rate limiting entirely. The whitelist must be configurable at runtime.

Health check exclusion. Load balancer health checks and internal service-to-service pings must never count toward rate limits. These are typically identified by specific user agents or source IP ranges.

Real-time analytics. The system must expose metrics on rate limit violations, per-client usage patterns, and current capacity utilization. These must be queryable in near-real-time.

Admin API for runtime configuration. Limit values, window sizes, and whitelist entries must be adjustable through an administrative API without requiring service restarts or deployments.

Separate read and write limits. Read operations are typically cheaper than writes. The rate limiter should support independent limits for GET and POST/PUT/DELETE operations on the same endpoint.

3. Non-Functional Requirements

Non-functional requirements define the quality attributes of the system: how fast it must be, how reliable it must be, and how it behaves under stress. For a rate limiter, these constraints are particularly demanding because the rate limiter sits on the critical path of every single request.

Availability: 99.999%. The rate limiter must never become the bottleneck that causes platform-wide outages. A five-nines availability target means less than 5.26 minutes of downtime per year. This requires redundant instances across multiple availability zones, automatic failover, and a graceful degradation strategy when dependencies fail.

Latency: sub-millisecond p99. Rate limit decisions must complete in less than one millisecond at the 99th percentile. If the rate limiter adds 10ms of latency to every request, it doubles the response time of a fast API endpoint. The decision must be faster than the network hop to the backend service.

Throughput: 1M+ decisions per second. The system must handle over one million rate limit decisions per second at peak. This requires horizontal scaling of the rate limiter service and efficient use of Redis pipelining and Lua scripting.

Consistency: strong for enforcement, eventual for analytics. When two requests from the same client arrive at different rate limiter nodes simultaneously, both must see the same counter to make a consistent decision. This requires a centralized state store (Redis) with atomic operations. Analytics data can tolerate eventual consistency.

Fault tolerance: no single point of failure. The failure of any single Redis node, rate limiter instance, or network link must not cause the entire rate limiting system to fail. The system must degrade gracefully, falling back to local in-memory counters if the distributed store is unavailable.

Horizontal scalability. Adding more rate limiter instances must linearly increase throughput without requiring changes to the Redis cluster or configuration. The system must support auto-scaling based on traffic patterns.

Memory efficiency. With potentially millions of active rate limit keys (one per API key, per IP, per endpoint), the system must store counters compactly. Each key should require no more than 100 bytes of storage.

Audit trail. Every rate limit decision (allow or deny) must be logged for compliance and debugging purposes. Logs must include the client identity, endpoint, decision, current count, and timestamp.

Zero-downtime deployment. New versions of the rate limiter must be deployable without dropping any rate limit decisions or resetting counters. Rolling deployments with health checks are mandatory.

4. Capacity Estimation and Back-of-Envelope Math

Before designing the architecture, we must estimate the scale of the system to make informed decisions about infrastructure, data storage, and network capacity. These estimates drive every subsequent design choice.

Request volume. Assume the platform handles 1 billion API requests per day. The average requests per second is 1B divided by 86,400, which equals approximately 11,574 RPS. Peak traffic is typically 3x to 5x the average, giving us a peak RPS of 35,000 to 58,000. Let us design for 60,000 peak RPS with headroom.

Unique rate limit keys. Each API key, user ID, or IP address that sends requests becomes a rate limit key. With 100 million registered API keys and 100 million unique IP addresses, we could theoretically have 200 million keys. However, not all keys are active simultaneously. In any given 1-minute window, we expect approximately 10 million active keys. Each key requires two counters for the sliding window algorithm (current window and previous window).

Memory per key. A single rate limit key with two counters, a window identifier, and a TTL uses approximately 100 bytes in Redis. This includes the key name (e.g., rl:user:abc123:60), the counter value (8 bytes), the window timestamp (8 bytes), and Redis overhead (key encoding, expiration metadata). For 10 million active keys at 100 bytes each, the total memory requirement is 1 GB. With Redis Cluster replication factor of 3, total memory across the cluster is 3 GB.

Redis operations per second. Each rate limit decision requires one Redis operation (the Lua script with INCR and GET). At 60,000 peak RPS, we need 60,000 Redis operations per second. A single Redis instance can handle approximately 100,000 operations per second for simple Lua scripts. This means a single Redis shard can handle peak load, but we should use multiple shards for redundancy and to distribute the load.

Network bandwidth. Each rate limit request and response is approximately 200 bytes (key, counters, decision). At 60,000 RPS, the total bandwidth is 12 MB/s. This is negligible for modern data center networks.

Rate limiter service instances. Each rate limiter instance can handle approximately 500,000 decisions per second with local caching. For 60,000 peak RPS, two instances provide sufficient capacity with 8x headroom. In practice, we deploy at least 10 instances across three availability zones for high availability.

Capacity Estimation
═══════════════════════════════════════════════════════════════
Daily requests:            1,000,000,000
Average RPS:               11,574
Peak RPS (5x):             57,870 → Design for 60,000

Active rate limit keys:    10,000,000 per minute
Memory per key:            100 bytes (2 counters + overhead)
Total active memory:       1 GB
Redis replication (3x):    3 GB

Redis ops per second:      60,000 (1 Lua script per decision)
Single Redis capacity:     100,000 ops/s → 1 shard sufficient

Network bandwidth:         60,000 * 200 bytes = 12 MB/s
Rate limiter instances:    10 (across 3 AZs)
Decisions per instance:    6,000 (with headroom)
═══════════════════════════════════════════════════════════════

5. High-Level Architecture

The rate limiter operates as middleware in the request path between external clients and backend services. When a client sends an HTTP request, the API Gateway intercepts it, extracts the rate limiting key, and queries the rate limiter service before forwarding the request to any backend. This ensures that abusive traffic is rejected at the edge, consuming minimal backend resources.

The architecture consists of four primary components. The API Gateway is the entry point for all external traffic. It extracts rate limit keys from the request (API key from the Authorization header, client IP from the X-Forwarded-For header, user ID from the JWT token) and sends a synchronous gRPC call to the rate limiter service. The Gateway enforces the decision: if allowed, it forwards the request to the appropriate backend service; if denied, it immediately returns an HTTP 429 Too Many Requests response with appropriate headers.

The Rate Limiter Service is a stateless, horizontally scalable microservice. It receives decision requests from the Gateway, applies the configured algorithm, checks and updates counters in Redis, and returns the decision. Because the service is stateless, it can be deployed behind a load balancer with automatic scaling. Each instance maintains a small local in-memory cache for hot keys to reduce Redis round trips.

Redis Cluster serves as the distributed state store. It holds all rate limit counters with atomic Lua scripts that ensure consistency. Redis is configured with multiple shards for scalability and replicas for high availability. Each shard handles a portion of the key space based on hash slots.

The Configuration Service (etcd or Consul) stores all rate limiting rules: limits, window sizes, tier definitions, and whitelist entries. The rate limiter watches this service for changes and updates its local configuration cache in real time, enabling runtime adjustments without restarts.

Request Path

Every request follows a deterministic path. The client sends an HTTP request to the API Gateway. The Gateway middleware extracts the rate limit key and constructs a protobuf message containing the key, the configured limit, the window size, and the algorithm type. It sends this via a unary gRPC call to the Rate Limiter Service, which executes a Redis Lua script to atomically check and increment the counter. The script returns the decision (ALLOW or DENY), the remaining quota, and the reset timestamp. The Gateway sets the appropriate response headers and either forwards the request or returns a 429.

Configuration Path

Administrators modify rate limiting rules through the Admin Dashboard, which writes to etcd. The Rate Limiter Service watches etcd for changes and updates its local in-memory configuration cache within 100 milliseconds. This means a limit change takes effect almost immediately across all instances without any deployment or restart.

6. Rate Limiting Algorithms — Deep Comparison

The choice of algorithm is the single most important design decision in a rate limiter. Each algorithm has distinct characteristics in terms of accuracy, memory usage, burst handling, and implementation complexity. We will examine five widely-used algorithms and determine which is best suited for a distributed system.

6.1 Token Bucket

The Token Bucket algorithm models rate limiting as a bucket that holds tokens. Tokens are added to the bucket at a fixed rate (e.g., 10 tokens per second) up to a maximum bucket size. Each incoming request consumes one token. If the bucket has tokens, the request is allowed and one token is removed. If the bucket is empty, the request is denied. The bucket size determines burst tolerance: a bucket size of 50 with a refill rate of 10/s allows a burst of 50 requests in one instant, after which the steady-state rate of 10/s applies.

Token Bucket is used by Amazon Web Services for API Gateway throttling and by Stripe for API rate limiting. It is memory-efficient (two values per key: token count and last refill timestamp) and naturally supports bursts. However, it requires calculating the number of tokens to add based on elapsed time since the last request, which can introduce minor accuracy issues with clock skew.

6.2 Leaky Bucket

The Leaky Bucket algorithm models requests as water flowing into a bucket with a hole at the bottom. Requests fill the bucket, and they drain at a fixed rate. If the bucket is full, new requests are dropped. This produces a perfectly smooth output rate regardless of input burstiness. NGINX uses this algorithm in its rate limiting module.

The main advantage is output smoothing. The main disadvantage is that legitimate bursts are rejected, which frustrates users who load multiple resources simultaneously. It also requires maintaining a queue, which adds memory overhead.

6.3 Fixed Window Counter

The Fixed Window Counter divides time into fixed intervals (e.g., each minute starting at :00, :01, :02) and counts requests within each interval. At the start of each interval, the counter resets to zero. This is the simplest algorithm to implement and understand.

The critical flaw is the boundary problem. If a client sends 100 requests at 11:59:59 and 100 requests at 12:00:01, they have sent 200 requests in 2 seconds, but each fixed window only sees 100. The effective rate is double the intended limit at window boundaries. This makes Fixed Window unsuitable for production systems without additional mitigations.

6.4 Sliding Window Log

The Sliding Window Log stores the timestamp of every request in a sorted set (Redis Sorted Set). To determine if a request is allowed, it removes all timestamps older than now - window_size and counts the remaining entries. If the count is below the limit, the request is allowed and its timestamp is added.

This algorithm is perfectly accurate because it uses a true sliding window. However, it is memory-intensive: storing individual timestamps for a key receiving 10,000 requests per minute requires 10,000 entries in the sorted set, each approximately 50 bytes, totaling 500 KB per key. For millions of keys, this is prohibitive.

6.5 Sliding Window Counter (Recommended)

The Sliding Window Counter algorithm approximates a true sliding window using two fixed-window counters: the current window and the previous window. The effective count is computed as a weighted sum: effective = current_count + previous_count * overlap_ratio, where overlap_ratio = (window_size - elapsed_in_current_window) / window_size.

This approach eliminates the boundary spike problem of Fixed Window Counter while using only two integers per key (versus a sorted set for Sliding Window Log). It is the algorithm used by Cloudflare, Envoy, and most production rate limiters. It trades a small amount of accuracy for dramatic memory savings.

7. Sliding Window Counter — The Recommended Algorithm

The Sliding Window Counter is the most practical algorithm for distributed rate limiting. It provides a strong balance between accuracy and resource efficiency. In this section, we walk through the mathematics, the Redis implementation, and the edge cases that trip up engineers in production.

7.1 How It Works

Time is divided into fixed windows of size W (e.g., 60 seconds). Each window is identified by window_id = floor(timestamp / W). For each rate limit key, we maintain two counters: the count for the current window and the count for the previous window. When a request arrives at time T, we compute the effective count as follows:

Let current_window_id = floor(T / W). Let previous_window_id = current_window_id - 1. Let elapsed = T - (current_window_id * W), which is the number of seconds elapsed in the current window. Let overlap_ratio = (W - elapsed) / W, which represents what fraction of the previous window overlaps with the sliding window ending at T.

The effective count is: effective_count = current_count + previous_count * overlap_ratio. If effective_count < limit, allow the request and increment the current window counter. If effective_count >= limit, deny the request.

7.2 Worked Example

Suppose the limit is 100 requests per 60-second window. At time 120 seconds, the current window is 2 (from 120 to 180), and the previous window is 1 (from 60 to 120). The previous window had 80 requests. The current window has 20 requests so far. The elapsed time in the current window is 0 seconds (we are at the very start), so the overlap ratio is (60 - 0) / 60 = 1.0. The effective count is 20 + 80 * 1.0 = 100. The next request would be denied because the effective count equals the limit.

Now consider time 150 seconds. The current window is still 2. The previous window is still 1 (80 requests). The elapsed time is 30 seconds, so the overlap ratio is (60 - 30) / 60 = 0.5. If the current window has 40 requests, the effective count is 40 + 80 * 0.5 = 80. A new request would be allowed, bringing the effective count to 81.

7.3 Redis Lua Implementation

The entire check-and-increment logic runs as a single Redis Lua script, which executes atomically on a single Redis shard. This eliminates race conditions: no two concurrent requests can read the counter before either increments it.

Redis Lua
-- Sliding Window Counter: atomic check + increment
-- KEYS[1] = rate limit key prefix
-- ARGV[1] = window size in seconds
-- ARGV[2] = max requests allowed
-- ARGV[3] = current unix timestamp

local key = KEYS[1]
local window = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
local now = tonumber(ARGV[3])

local current_window = math.floor(now / window)
local previous_window = current_window - 1

-- Build Redis keys for current and previous windows
local current_key = key .. ":cw:" .. current_window
local previous_key = key .. ":cw:" .. previous_window

-- Atomically increment current window counter
local current_count = redis.call('INCR', current_key)

-- Set TTL on current key to auto-expire (2x window for safety)
redis.call('EXPIRE', current_key, window * 2)

-- Read previous window count (does not need to be atomic)
local previous_count = tonumber(redis.call('GET', previous_key) or '0')

-- Calculate overlap ratio
local elapsed = now - (current_window * window)
local weight = (window - elapsed) / window

-- Weighted effective count
local effective_count = current_count + (previous_count * weight)

-- Decision
if effective_count > limit then
    -- Over limit: decrement the counter we just incremented
    redis.call('DECR', current_key)
    local retry_after = window - elapsed
    return {0, 0, retry_after}
end

-- Under limit: return remaining quota and reset time
local remaining = math.floor(limit - effective_count)
local reset_after = window - elapsed
return {1, remaining, reset_after}

7.4 Edge Cases

Clock skew. If rate limiter nodes have significantly different system clocks, the window calculation can produce inconsistent results. Mitigation: synchronize all nodes with NTP and accept small drift (under 1 second). Use monotonic clocks for elapsed-time calculations.

Window boundary. At the exact boundary between two windows, the previous window becomes the current window's predecessor. The overlap ratio drops to nearly 1.0, which means the previous window's full count contributes to the effective count. This is correct behavior but can surprise engineers who expect the counter to reset at boundaries.

Negative counters. If the DECR after a denied request brings the counter below zero (due to a race between the INCR and DECR across retries), use a Redis check: if redis.call('GET', current_key) == 0 then redis.call('DEL', current_key) end.

8. Token Bucket Algorithm — Implementation Details

While the Sliding Window Counter is recommended for most use cases, the Token Bucket algorithm is the better choice when burst handling is a first-class requirement. It is used by Amazon API Gateway and Stripe. In this section, we implement a production-grade Token Bucket in Redis with Lua scripting.

8.1 Algorithm Logic

The bucket has two properties: capacity (maximum tokens) and refill rate (tokens added per second). Each request consumes one token. When a request arrives, we first calculate how many tokens to add based on the time elapsed since the last request: tokens_to_add = elapsed * refill_rate. The new token count is min(capacity, current_tokens + tokens_to_add). If the new count is greater than zero, allow the request and decrement by one. Otherwise, deny.

Redis Lua
-- Token Bucket: atomic check + consume
-- KEYS[1] = bucket key
-- ARGV[1] = bucket capacity (burst size)
-- ARGV[2] = refill rate (tokens per second)
-- ARGV[3] = current unix timestamp (milliseconds)

local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now_ms = tonumber(ARGV[3])

-- Get current bucket state
local bucket = redis.call('HMGET', key, 'tokens', 'last_refill_ms')
local tokens = tonumber(bucket[1]) or capacity
local last_refill = tonumber(bucket[2]) or now_ms

-- Calculate refill
local elapsed_ms = math.max(0, now_ms - last_refill)
local tokens_to_add = (elapsed_ms / 1000) * refill_rate
tokens = math.min(capacity, tokens + tokens_to_add)

-- Decision
if tokens >= 1 then
    tokens = tokens - 1
    redis.call('HMSET', key, 'tokens', tokens, 'last_refill_ms', now_ms)
    redis.call('EXPIRE', key, math.ceil(capacity / refill_rate) * 2)
    return {1, math.floor(tokens), 0}
else
    -- Calculate wait time for next token
    local wait_ms = math.ceil((1 - tokens) / refill_rate * 1000)
    redis.call('HMSET', key, 'tokens', tokens, 'last_refill_ms', now_ms)
    redis.call('EXPIRE', key, math.ceil(capacity / refill_rate) * 2)
    return {0, 0, wait_ms}
end

8.2 When to Choose Token Bucket

Choose Token Bucket when your API has legitimate bursty workloads that should be accommodated. For example, a web page that loads 30 resources simultaneously should not be rate-limited to 1 request per second. The Token Bucket allows this burst (if bucket capacity is 30) while maintaining a steady-state limit over time. Choose Sliding Window Counter when your primary concern is a hard cap on requests per time period with minimal memory overhead.

9. Distributed Architecture with Redis Cluster

Redis Cluster is the backbone of the distributed rate limiter. It provides the centralized state store that all rate limiter instances share, enabling global enforcement of limits. Understanding how to properly configure and operate Redis Cluster for rate limiting is critical to system reliability.

9.1 Key Sharding Strategy

Redis Cluster partitions data across shards using hash slots. There are 16,384 hash slots, and each shard owns a portion of them. When a rate limiter instance issues a command for a key, Redis computes CRC16(key) % 16384 to determine which shard owns that key. All operations for the same key always go to the same shard, which is essential for atomicity.

The rate limit key format should be designed for even distribution. A good pattern is rl:{type}:{id}:{window_seconds}, for example rl:user:abc123:60 or rl:ip:192.168.1.1:3600. The hash of this key determines the shard, so different user IDs and IP addresses naturally distribute across all shards.

9.2 Cluster Configuration

For a system handling 60,000 peak RPS, we recommend 6 Redis shards with 2 replicas each, for a total of 18 Redis instances. This provides enough capacity (each shard handles 10,000+ ops/s), redundancy (each shard can lose one replica without data loss), and memory distribution (approximately 170 MB per instance with 1 GB total data).

Redis Cluster requires a minimum of 3 master nodes for cluster consensus. With 6 masters and 12 replicas, we exceed this requirement comfortably. The cluster uses gossip protocol for node discovery and failure detection. When a master fails, one of its replicas is promoted automatically within 10-30 seconds.

9.3 Connection Management

Each rate limiter instance maintains a connection pool to the Redis Cluster. The pool size should match the number of concurrent requests the instance handles. For a typical 4-core instance handling 6,000 decisions per second, a pool of 50 connections is sufficient. Use the StackExchange.Redis client for .NET, which automatically handles cluster topology changes, reconnections, and MOVED/ASK redirects.

9.4 Cross-Region Replication

For globally distributed systems, each region runs its own Redis Cluster. Regional rate limits (e.g., per-user per-minute) are enforced locally. Global rate limits (e.g., per-user per-day across all regions) require cross-region synchronization. Options include Redis CRDT (Conflict-Free Replicated Data Types) for active-active replication, or a central global Redis cluster with regional read replicas. Redis CRDT provides eventual convergence with sub-second replication lag, which is acceptable for most rate limiting use cases.

10. Atomic Operations with Redis Lua Scripts

The atomicity of rate limit decisions depends entirely on Redis Lua scripting. In this section, we explore why Lua scripts are the correct choice, how to debug them, and the performance characteristics that matter in production.

10.1 Why Lua Scripts

Redis executes Lua scripts atomically: while a script is running, no other client command is processed. This is not true atomicity in the distributed systems sense (it does not span multiple Redis shards), but it is sufficient for rate limiting because all operations for a single key go to a single shard. The Lua script can read the current count, compute the effective count, decide whether to allow or deny, and increment the counter, all within a single uninterruptible execution.

Without Lua, you would need to use WATCH/MULTI/EXEC transactions, which are more complex, less performant, and prone to optimistic locking failures under high contention. Lua scripts execute in approximately 0.1ms for simple operations, compared to 0.5ms for a WATCH/MULTI/EXEC sequence.

10.2 Script Loading and SHA

Redis supports loading Lua scripts via the SCRIPT LOAD command, which returns a SHA1 hash. Subsequent calls can use EVALSHA with the hash instead of the full script text, reducing network overhead. In production, load all Lua scripts at rate limiter startup and cache the SHA hashes. This saves approximately 200 bytes per request (the script text) and avoids Redis recompiling the script on each invocation.

C#
// Load Lua scripts at startup and cache SHA hashes
public class RateLimitScripts
{
    private readonly IDatabase _redis;
    private string _slidingWindowSha;
    private string _tokenBucketSha;

    public RateLimitScripts(IConnectionMultiplexer redis)
    {
        _redis = redis.GetDatabase();
    }

    public async Task InitializeAsync()
    {
        string slidingWindowScript = await File.ReadAllTextAsync("Scripts/sliding_window.lua");
        _slidingWindowSha = await _redis.ScriptLoadAsync(slidingWindowScript);

        string tokenBucketScript = await File.ReadAllTextAsync("Scripts/token_bucket.lua");
        _tokenBucketSha = await _redis.ScriptLoadAsync(tokenBucketScript);
    }

    public async Task<RateLimitResult> CheckSlidingWindowAsync(
        string key, int windowSeconds, int limit, long timestamp)
    {
        var result = await _redis.ScriptEvaluateAsync(
            _slidingWindowSha,
            new RedisKey[] { key },
            new RedisValue[] { windowSeconds, limit, timestamp });

        var values = (RedisValue[])result;
        bool allowed = (int)values[0] == 1;
        int remaining = (int)values[1];
        int retryAfter = (int)values[2];

        return new RateLimitResult
        {
            Allowed = allowed,
            Remaining = remaining,
            RetryAfter = retryAfter
        };
    }
}

10.3 Performance Characteristics

A well-optimized Redis Lua script for rate limiting executes in under 0.1ms. This includes the INCR operation (O(1)), the GET operation (O(1)), the EXPIRE operation (O(1)), and the arithmetic computations. At this latency, a single Redis shard can handle 100,000+ decisions per second. The bottleneck is typically network round-trip time (approximately 0.2ms within the same data center) rather than script execution time.

To maximize throughput, use Redis pipelining when checking multiple rate limits for the same request (e.g., per-IP limit and per-API-key limit). Pipelining sends multiple commands in a single network round trip, reducing total latency from N * RTT to 1 * RTT.

11. C# Implementation — Production-Ready Rate Limiter

In this section, we build a complete rate limiter middleware in C# using ASP.NET Core, StackExchange.Redis, and the Sliding Window Counter algorithm. This implementation includes local caching for hot keys, circuit breaker fallback, and proper response header management.

11.1 Data Models

C#
public class RateLimitRule
{
    public string Key { get; set; }         // e.g., "api:search"
    public int MaxRequests { get; set; }     // e.g., 100
    public int WindowSeconds { get; set; }   // e.g., 60
    public RateLimitAlgorithm Algorithm { get; set; }
    public bool IsWhitelisted { get; set; }
}

public enum RateLimitAlgorithm
{
    SlidingWindowCounter,
    TokenBucket,
    FixedWindow
}

public class RateLimitResult
{
    public bool Allowed { get; set; }
    public int Limit { get; set; }
    public int Remaining { get; set; }
    public int ResetSeconds { get; set; }
    public int RetryAfterSeconds { get; set; }

    public void ApplyToHeaders(HttpResponse response)
    {
        response.Headers["X-RateLimit-Limit"] = Limit.ToString();
        response.Headers["X-RateLimit-Remaining"] = Remaining.ToString();
        response.Headers["X-RateLimit-Reset"] =
            DateTimeOffset.UtcNow.AddSeconds(ResetSeconds)
                .ToUnixTimeSeconds().ToString();
        if (!Allowed)
        {
            response.Headers["Retry-After"] = RetryAfterSeconds.ToString();
        }
    }
}

public class LocalCacheEntry
{
    public int Count { get; set; }
    public DateTime WindowStart { get; set; }
    public DateTime LastUpdated { get; set; }
}

11.2 Core Rate Limiter Service

C#
using StackExchange.Redis;
using System.Collections.Concurrent;

public class DistributedRateLimiter : IDisposable
{
    private readonly IConnectionMultiplexer _redis;
    private readonly IDatabase _db;
    private readonly ConcurrentDictionary<string, LocalCacheEntry> _localCache;
    private readonly TimeSpan _localCacheTtl = TimeSpan.FromMilliseconds(100);
    private readonly ILogger<DistributedRateLimiter> _logger;

    // Circuit breaker state
    private int _consecutiveFailures = 0;
    private bool _circuitOpen = false;
    private DateTime _circuitOpenedAt = DateTime.MinValue;
    private const int FailureThreshold = 5;
    private const int CircuitOpenDurationSeconds = 30;

    // Lua script SHA hashes
    private string _slidingWindowSha;

    public DistributedRateLimiter(
        IConnectionMultiplexer redis,
        ILogger<DistributedRateLimiter> logger)
    {
        _redis = redis;
        _db = redis.GetDatabase();
        _localCache = new ConcurrentDictionary<string, LocalCacheEntry>();
        _logger = logger;
    }

    public async Task InitializeAsync()
    {
        string script = @"local key=KEYS[1]
local window=tonumber(ARGV[1])
local limit=tonumber(ARGV[2])
local now=tonumber(ARGV[3])
local cw=math.floor(now/window)
local pw=cw-1
local ck=key..':cw:'..cw
local pk=key..':cw:'..pw
local cc=redis.call('INCR',ck)
redis.call('EXPIRE',ck,window*2)
local pc=tonumber(redis.call('GET',pk) or '0')
local el=now-(cw*window)
local w=(window-el)/window
local eff=cc+(pc*w)
if eff>limit then
  redis.call('DECR',ck)
  return {0,0,math.ceil(window-el)}
end
return {1,math.floor(limit-eff),0}";

        _slidingWindowSha = (string)await _db.ScriptLoadAsync(script);
        _logger.LogInformation("Rate limiter initialized with Lua script SHA: {Sha}",
            _slidingWindowSha);
    }

    public async Task<RateLimitResult> CheckAsync(
        RateLimitRule rule, string clientId)
    {
        // Check whitelist
        if (rule.IsWhitelisted)
        {
            return new RateLimitResult
            {
                Allowed = true,
                Limit = rule.MaxRequests,
                Remaining = rule.MaxRequests,
                ResetSeconds = rule.WindowSeconds
            };
        }

        // Build the full rate limit key
        string cacheKey = $"{rule.Key}:{clientId}:{rule.WindowSeconds}";
        long now = DateTimeOffset.UtcNow.ToUnixTimeSeconds();

        // Check local cache for hot keys
        if (_localCache.TryGetValue(cacheKey, out var cached) &&
            DateTime.UtcNow - cached.LastUpdated < _localCacheTtl)
        {
            // Serve from local cache (estimated count)
            if (cached.Count < rule.MaxRequests)
            {
                cached.Count++;
                return new RateLimitResult
                {
                    Allowed = true,
                    Limit = rule.MaxRequests,
                    Remaining = Math.Max(0, rule.MaxRequests - cached.Count),
                    ResetSeconds = rule.WindowSeconds
                };
            }
        }

        // Check circuit breaker
        if (_circuitOpen)
        {
            if (DateTime.UtcNow - _circuitOpenedAt <
                TimeSpan.FromSeconds(CircuitOpenDurationSeconds))
            {
                // Circuit is open: fail-open with local counter
                return CheckLocalFallback(cacheKey, rule);
            }
            // Half-open: try Redis
            _circuitOpen = false;
            _consecutiveFailures = 0;
        }

        try
        {
            var result = await _db.ScriptEvaluateAsync(
                _slidingWindowSha,
                new RedisKey[] { cacheKey },
                new RedisValue[]
                {
                    rule.WindowSeconds,
                    rule.MaxRequests,
                    now
                });

            var values = (RedisValue[])result;
            bool allowed = (int)values[0] == 1;
            int remaining = (int)values[1];
            int retryAfter = (int)values[2];

            // Reset failure counter on success
            Interlocked.Exchange(ref _consecutiveFailures, 0);

            // Update local cache
            _localCache[cacheKey] = new LocalCacheEntry
            {
                Count = rule.MaxRequests - remaining,
                WindowStart = DateTime.UtcNow,
                LastUpdated = DateTime.UtcNow
            };

            return new RateLimitResult
            {
                Allowed = allowed,
                Limit = rule.MaxRequests,
                Remaining = remaining,
                ResetSeconds = retryAfter,
                RetryAfterSeconds = retryAfter
            };
        }
        catch (Exception ex)
        {
            _logger.LogWarning(ex,
                "Redis call failed for key {Key}, falling back to local",
                cacheKey);

            int failures = Interlocked.Increment(ref _consecutiveFailures);
            if (failures >= FailureThreshold)
            {
                _circuitOpen = true;
                _circuitOpenedAt = DateTime.UtcNow;
                _logger.LogError(
                    "Circuit breaker tripped after {Failures} failures",
                    failures);
            }

            return CheckLocalFallback(cacheKey, rule);
        }
    }

    private RateLimitResult CheckLocalFallback(
        string cacheKey, RateLimitRule rule)
    {
        // Local in-memory fallback: not globally accurate,
        // but preserves availability
        var entry = _localCache.GetOrAdd(cacheKey,
            _ => new LocalCacheEntry
            {
                Count = 0,
                WindowStart = DateTime.UtcNow,
                LastUpdated = DateTime.UtcNow
            });

        // Reset if window expired
        if (DateTime.UtcNow - entry.WindowStart >
            TimeSpan.FromSeconds(rule.WindowSeconds))
        {
            entry.Count = 0;
            entry.WindowStart = DateTime.UtcNow;
        }

        entry.Count++;
        entry.LastUpdated = DateTime.UtcNow;

        bool allowed = entry.Count <= rule.MaxRequests;
        return new RateLimitResult
        {
            Allowed = allowed,
            Limit = rule.MaxRequests,
            Remaining = Math.Max(0, rule.MaxRequests - entry.Count),
            ResetSeconds = rule.WindowSeconds -
                (int)(DateTime.UtcNow - entry.WindowStart).TotalSeconds,
            RetryAfterSeconds = allowed ? 0 : rule.WindowSeconds -
                (int)(DateTime.UtcNow - entry.WindowStart).TotalSeconds
        };
    }

    public void Dispose()
    {
        _redis?.Dispose();
    }
}

11.3 ASP.NET Core Middleware

C#
public class RateLimitMiddleware
{
    private readonly RequestDelegate _next;
    private readonly DistributedRateLimiter _limiter;
    private readonly RateLimitConfig _config;

    public RateLimitMiddleware(
        RequestDelegate next,
        DistributedRateLimiter limiter,
        RateLimitConfig config)
    {
        _next = next;
        _limiter = limiter;
        _config = config;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        string clientId = ExtractClientId(context);
        string endpoint = context.Request.Path.Value;

        // Find matching rate limit rule
        var rule = _config.GetRule(endpoint, clientId);
        if (rule == null)
        {
            await _next(context);
            return;
        }

        // Check rate limit
        var result = await _limiter.CheckAsync(rule, clientId);

        // Apply headers to every response
        result.ApplyToHeaders(context.Response);

        if (!result.Allowed)
        {
            context.Response.StatusCode = 429;
            context.Response.Headers["Retry-After"] =
                result.RetryAfterSeconds.ToString();
            await context.Response.WriteAsJsonAsync(new
            {
                error = "Too Many Requests",
                message = $"Rate limit exceeded. " +
                    $"Try again in {result.RetryAfterSeconds} seconds.",
                retryAfter = result.RetryAfterSeconds
            });
            return;
        }

        await _next(context);
    }

    private string ExtractClientId(HttpContext context)
    {
        // Priority: API key > JWT user ID > client IP
        if (context.Request.Headers.TryGetValue(
            "X-API-Key", out var apiKey))
            return $"apikey:{apiKey}";

        string userId = context.User?.FindFirst("sub")?.Value;
        if (!string.IsNullOrEmpty(userId))
            return $"user:{userId}";

        string ip = context.Connection.RemoteIpAddress?.ToString()
            ?? "unknown";
        return $"ip:{ip}";
    }
}

11.4 Configuration Service

C#
public class RateLimitConfig
{
    private readonly ConcurrentDictionary<string, RateLimitRule> _rules = new();
    private readonly IWatcher<ConfigChange> _etcdWatcher;

    public RateLimitConfig(IWatcher<ConfigChange> etcdWatcher)
    {
        _etcdWatcher = etcdWatcher;
        _etcdWatcher.OnChange += ApplyConfigChange;
    }

    public RateLimitRule GetRule(string endpoint, string clientId)
    {
        // Try endpoint-specific rule first
        if (_rules.TryGetValue(endpoint, out var endpointRule))
            return endpointRule;

        // Try tier-based rule
        string tier = ResolveTier(clientId);
        if (_rules.TryGetValue($"tier:{tier}", out var tierRule))
            return tierRule;

        // Default rule
        return _rules.GetOrAdd("default", _ => new RateLimitRule
        {
            Key = "default",
            MaxRequests = 60,
            WindowSeconds = 60,
            Algorithm = RateLimitAlgorithm.SlidingWindowCounter
        });
    }

    private void ApplyConfigChange(ConfigChange change)
    {
        foreach (var (key, rule) in change.Rules)
        {
            _rules[key] = rule;
        }
    }

    private string ResolveTier(string clientId)
    {
        if (clientId.StartsWith("apikey:ent_")) return "enterprise";
        if (clientId.StartsWith("apikey:pro_")) return "pro";
        return "free";
    }
}

12. Data Flow and Request Lifecycle

Understanding the complete request lifecycle from client to backend is essential for debugging, monitoring, and optimization. In this section, we trace the journey of a single HTTP request through every component of the rate limiting system.

12.1 Happy Path (Request Allowed)

The client sends an HTTP GET request to /api/users/123 with an API key in the Authorization header. The API Gateway receives the request and the rate limit middleware extracts the key: apikey:sk_live_abc123. It looks up the rate limit rule for the /api/users endpoint and finds the limit for the "pro" tier: 1000 requests per 60 seconds.

The middleware constructs a gRPC call to the Rate Limiter Service with the full key (rl:apikey:sk_live_abc123:60), the limit (1000), and the window (60). The service checks the local cache for this key. On a cache hit (95% of the time for active keys), the service increments the local count and returns ALLOW immediately without touching Redis. The Gateway adds the rate limit headers to the response context and forwards the request to the user service. The backend processes the request, returns a 200 OK with the user data, and the Gateway adds the rate limit headers to the outgoing response.

12.2 Rate Limit Exceeded

The client has already sent 1000 requests in the current window. The 1001st request arrives. The rate limiter service runs the Lua script, which increments the counter, computes the effective count (1001), and compares it to the limit (1000). Since 1001 > 1000, the script decrements the counter (undoing the increment) and returns DENY with a retry_after value of 45 seconds. The Gateway receives the denial, sets the response status to 429, adds the Retry-After header, and returns a JSON error response. The client reads the Retry-After header and waits 45 seconds before retrying.

12.3 Degraded Mode (Redis Failure)

Redis becomes unreachable due to a network partition. The next rate limit check fails with a connection timeout. The circuit breaker increments the failure count. After 5 consecutive failures, the circuit trips open. All subsequent requests are served from local in-memory counters. Each rate limiter instance tracks its own count independently. If there are 10 instances, a client could theoretically send 10x the intended limit by hitting a different instance for each request. This is the accepted trade-off for availability over accuracy.

13. Hot Key Handling and Local Caching

A hot key is a rate limit key that receives an unusually high number of requests per second. For example, a popular API key used by a large SaaS platform might receive 50,000 requests per second. All of these requests would target the same Redis shard, potentially exhausting its CPU and creating a bottleneck. Hot key handling is one of the most important optimizations in a production rate limiter.

13.1 Local In-Memory Cache

The primary defense against hot keys is a local in-memory cache on each rate limiter instance. When a request arrives for a hot key, the local cache serves the decision without hitting Redis. The cache has a very short TTL (50-100 milliseconds), which means it absorbs the vast majority of requests while maintaining acceptable accuracy. A 100ms staleness window means the rate limiter might allow up to 100ms worth of excess traffic, which is negligible for most use cases.

For a key receiving 50,000 requests per second distributed across 10 rate limiter instances, each instance handles 5,000 requests per second for that key. With a 100ms local cache TTL, each instance makes one Redis call per 100ms (10 calls per second) instead of 5,000 calls per second. This is a 500x reduction in Redis load for that key.

13.2 Cache Eviction Strategy

The local cache should use an LRU (Least Recently Used) eviction policy with a maximum size of 1 million entries. At approximately 200 bytes per entry, this uses 200 MB of memory per rate limiter instance, which is acceptable for a dedicated rate limiting service. Keys that are no longer accessed are evicted automatically, and their rate limiting reverts to the standard Redis path.

13.3 Hot Key Detection

Proactively detect hot keys by monitoring Redis command latency per shard. If a shard's latency exceeds the p99 threshold (1ms), identify the hot key using Redis's MONITOR command or the --hotkeys flag in redis-cli. Once identified, you can add the key to the local cache proactively or redistribute it across shards using Redis Cluster's MIGRATE command.

14. Fault Tolerance and Circuit Breaker Pattern

A rate limiter that fails closed (denies all traffic when Redis is down) can cause more damage than no rate limiter at all. A rate limiter that fails open (allows all traffic when Redis is down) preserves availability but risks backend overload. The right strategy depends on your SLA, but most production systems use fail-open with alerting and manual escalation.

14.1 Circuit Breaker States

The circuit breaker monitors Redis health by tracking consecutive failures. When failures exceed a threshold (5 consecutive failures), the circuit trips to the OPEN state. In the OPEN state, all rate limit decisions are served from local in-memory counters without attempting to reach Redis. After a configurable timeout (30 seconds), the circuit enters HALF-OPEN state and sends a single probe request to Redis. If the probe succeeds, the circuit closes and normal operation resumes. If it fails, the circuit reopens.

14.2 Fail-Open Strategy

When the rate limiter fails open, it allows all traffic through without enforcement. This is the default strategy for public APIs where revenue and user experience take priority over backend protection. However, fail-open must be paired with aggressive monitoring and alerting. If the rate limiter is down for more than 60 seconds, an alert fires and the on-call engineer investigates. During the outage, backend services must be able to handle the full traffic load (or have their own independent rate limiting).

14.3 Fail-Closed Strategy

Fail-closed denies all traffic when the rate limiter is unavailable. This protects backend services from being overwhelmed but causes a complete outage for all users. This strategy is appropriate for systems where the cost of backend failure is catastrophic (e.g., financial trading systems, healthcare platforms). Fail-closed should only be used with redundant rate limiter deployments across multiple regions, so a regional failure does not cause a global outage.

14.4 Bulkhead Pattern

Separate the Redis connection pool used for rate limiting from other Redis usage. If the rate limiter's Redis pool is exhausted, it should not affect other services that use Redis (e.g., session caching, feature flags). Use a dedicated Redis connection pool with a maximum of 50 connections and a timeout of 50ms per operation.

C#
// Circuit breaker implementation for Redis failures
public class CircuitBreaker
{
    private int _failureCount = 0;
    private volatile bool _isOpen = false;
    private DateTime _openedAt;
    private readonly int _failureThreshold;
    private readonly TimeSpan _openDuration;

    public CircuitBreaker(int failureThreshold = 5,
        int openDurationSeconds = 30)
    {
        _failureThreshold = failureThreshold;
        _openDuration = TimeSpan.FromSeconds(openDurationSeconds);
    }

    public bool IsOpen => _isOpen &&
        (DateTime.UtcNow - _openedAt) < _openDuration;

    public bool ShouldProbe => _isOpen &&
        (DateTime.UtcNow - _openedAt) >= _openDuration;

    public void RecordSuccess()
    {
        Interlocked.Exchange(ref _failureCount, 0);
        _isOpen = false;
    }

    public void RecordFailure()
    {
        int count = Interlocked.Increment(ref _failureCount);
        if (count >= _failureThreshold)
        {
            _isOpen = true;
            _openedAt = DateTime.UtcNow;
        }
    }
}

15. High Availability and Multi-Region Design

High availability for a rate limiter means that rate limit decisions are always available, even during partial system failures. This requires redundant instances, automatic failover, and a clear degradation strategy.

15.1 Deployment Architecture

The rate limiter is deployed as a Kubernetes Deployment with a minimum of 10 replicas distributed across three availability zones. A Horizontal Pod Autoscaler scales the deployment based on CPU utilization and gRPC request rate. The minimum replicas ensure that the loss of an entire availability zone does not reduce capacity below peak demand.

15.2 Redis Cluster Topology

Redis Cluster runs 6 master shards with 2 replicas each (18 total instances). Masters are distributed evenly across 3 AZs. Redis Sentinel monitors the masters and automatically promotes replicas when a master fails. The failover typically completes in 10-30 seconds. During failover, the affected shard is temporarily unavailable, and rate limiter instances fall back to local counters for keys on that shard.

15.3 Cross-Region Design

Each region runs its own rate limiter deployment and Redis Cluster. Regional limits (per-minute, per-hour) are enforced locally with no cross-region communication. This keeps latency low (sub-millisecond within a region). Global limits (per-day across all regions) require cross-region synchronization.

For global limits, use Redis CRDT (Conflict-Free Replicated Data Types) with Redis Enterprise Active-Active. Each region increments its local counter independently, and counters converge asynchronously. During a network partition between regions, each region enforces the local count, which may temporarily exceed the global limit. Once the partition heals, the counters converge. The maximum overshoot during a partition equals the number of regions times the per-region limit increment.

16. Observability, Monitoring, and Alerting

A rate limiter without observability is a black box. You need to know whether it is working correctly, whether clients are approaching their limits, and whether the infrastructure is healthy. We implement the three pillars of observability: metrics, logs, and traces.

16.1 Metrics (Prometheus)

Expose the following metrics from the rate limiter service:

• rate_limiter_requests_total{key_type, algorithm, result} — Total rate limit decisions. Labels differentiate between API keys, IPs, and user IDs. The result label is either "allow" or "deny".
• rate_limiter_redis_latency_ms{percentile} — Redis round-trip latency. Track p50, p95, and p99. An increase in Redis latency often precedes circuit breaker trips.
• rate_limiter_local_cache_hits_total — Decisions served from the local cache. A high hit rate (above 90%) indicates the hot key cache is effective.
• rate_limiter_circuit_breaker_state — Current circuit breaker state: 0 for closed, 1 for half-open, 2 for open.
• rate_limiter_denied_requests_total{key, reason} — Denied requests with the reason (limit_exceeded, whitelist_miss, circuit_open).

16.2 Dashboards (Grafana)

Create dashboards for three audiences. The operations dashboard shows Redis latency, circuit breaker state, pod count, and error rate. The product dashboard shows rate limit hit rate per API key, top 20 violating clients, and usage trends over time. The business dashboard shows quota utilization per tier, revenue impact of rate limiting, and client onboarding velocity.

16.3 Tracing (OpenTelemetry)

Instrument the rate limiter with OpenTelemetry. Each rate limit decision creates a span with attributes for the key, algorithm, decision, remaining quota, and Redis latency. These spans are correlated with the parent span from the API Gateway, enabling end-to-end tracing of the request lifecycle. Use Jaeger or Tempo as the trace backend.

16.4 Alerting Rules

• P1: Redis error rate exceeds 1% for 5 minutes. This indicates a Redis health issue that needs immediate attention.
• P1: Circuit breaker is open for more than 2 minutes. This means Redis has been unreachable and all decisions are local.
• P2: Rate limit p99 latency exceeds 5ms. This indicates Redis latency degradation or a hot key issue.
• P2: Rate limit denial rate exceeds 10% for any API key. This might indicate a misconfigured limit or a client that needs quota adjustment.
• P3: Local cache hit rate drops below 80%. This might indicate a cold start after deployment or a change in traffic patterns.

Observability Stack
═══════════════════════════════════════════════════════════
Metrics:     Prometheus + Grafana
Logs:        Structured JSON → Fluentd → Elasticsearch
Traces:      OpenTelemetry → Jaeger/Tempo
Alerts:      Prometheus Alertmanager → PagerDuty
Dashboards:  Operations / Product / Business
═══════════════════════════════════════════════════════════

17. Security Considerations

A rate limiter is a security-sensitive component. If it can be bypassed, the entire backend is exposed to abuse. If it leaks information, it becomes an attack vector. We must secure the rate limiter at every layer.

17.1 Authentication and Authorization

The rate limiter is an internal service not exposed to the public internet. Only API Gateway instances communicate with it. Communication uses mutual TLS (mTLS), ensuring that only authenticated Gateway pods can issue rate limit checks. The rate limiter does not need to authenticate individual clients; it trusts the Gateway to have already authenticated the request.

17.2 Rate Limit Bypass Prevention

Attackers may attempt to bypass rate limiting by cycling through API keys, rotating IP addresses, or manipulating headers. Defenses include: combining multiple rate limit dimensions (per-IP and per-API-key simultaneously), fingerprinting clients by user agent and TLS fingerprint, using a sliding window over the combined key to detect rapid cycling, and blocking keys that exhibit bypass patterns.

17.3 Header Injection Prevention

Rate limit response headers (X-RateLimit-Remaining, Retry-After) must be set by the API Gateway, not by the upstream service. This prevents a compromised backend from lying about its rate limit status. The Gateway should overwrite any existing rate limit headers before sending the response to the client.

17.4 Secrets Management

Redis passwords, API keys, and mTLS certificates are stored in HashiCorp Vault and mounted as Kubernetes secrets. They are rotated automatically every 30 days. No secrets are committed to source code or container images.

17.5 DDoS Protection

The rate limiter itself needs protection from DDoS attacks. If an attacker sends millions of unique rate limit keys, they could exhaust the rate limiter's memory. Defenses include: a global request rate limit at the load balancer level (before traffic reaches the rate limiter), a maximum number of concurrent rate limit checks (queue overflow protection), and a per-connection rate limit at the TCP level. For extreme DDoS scenarios, the CDN layer (Cloudflare, AWS Shield) provides volumetric attack mitigation before traffic reaches the data center.

18. Performance Benchmarks and Optimization

Performance is paramount for a rate limiter. Every microsecond of added latency impacts every request your platform serves. In this section, we present benchmark results and optimization techniques.

18.1 Benchmark Results

18.2 Optimization Techniques

Redis pipelining. When a single request requires checking multiple rate limits (per-IP and per-API-key), send both Lua scripts in a single pipelined Redis call. This reduces latency from 2 * RTT to 1 * RTT. Pipelining improves throughput by 40-60% for multi-key checks.

gRPC with HTTP/2. Use gRPC for Gateway-to-RateLimiter communication. gRPC uses HTTP/2 multiplexing, which allows multiple concurrent requests over a single TCP connection. This eliminates connection setup overhead and reduces latency for subsequent requests.

Connection pre-warming. After deployment, proactively establish connections to all Redis shards and rate limiter pods. This eliminates cold-start latency spikes that can affect the first 100-200 requests after a deployment.

Binary serialization. Use Protocol Buffers for rate limit request/response serialization instead of JSON. Protobuf messages are 3-5x smaller and 10x faster to serialize/deserialize.

Affinity routing. For rate limit keys that are extremely hot, route all requests for that key to a single rate limiter instance. This maximizes local cache hit rate and minimizes Redis load. The Gateway can use consistent hashing on the rate limit key to determine which instance to target.

19. Cost Estimation

Understanding the infrastructure cost of the rate limiter helps justify the investment and optimize resource allocation. Here is a monthly cost estimate for a system handling 1 billion requests per day.

Cost optimization strategies include using spot instances for rate limiter pods (they are stateless and can be interrupted without data loss), using Redis on Graviton instances (30% cheaper than x86), and compressing analytics events with Snappy before sending to Kafka. With spot instances, the pod cost drops to approximately $540 per month, bringing the total to approximately $4,000.

20. Evolution Roadmap — V1 through V5

A rate limiter does not need to be built to its final form on day one. An incremental approach allows you to learn from production usage and invest in complexity only when needed.

V1: Embedded In-Memory (Week 1)

A simple in-memory rate limiter embedded in the API Gateway as middleware. Uses a Fixed Window Counter algorithm. Single instance, no distributed state. Handles 10,000 requests per second. Suitable for a pre-launch product with a single server. No Redis, no configuration service, no monitoring.

V2: Standalone Service with Redis (Month 1)

Extract the rate limiter into a standalone microservice. Use Redis as the backing store with a Sliding Window Counter algorithm. Configuration via YAML files that require a redeploy to change. Handles 100,000 requests per second with 3-5 instances. Add basic Prometheus metrics and Grafana dashboards.

V3: Distributed with Hot Key Cache (Month 3)

Deploy on Kubernetes with Redis Cluster (6 shards). Add local in-memory caching for hot keys. Implement circuit breaker fallback. Switch to gRPC for Gateway communication. Add admin API for runtime configuration changes. Handles 1,000,000+ requests per second. Add OpenTelemetry tracing.

V4: Multi-Region with Global Limits (Month 6)

Deploy in 3 regions with regional Redis Clusters. Add cross-region synchronization via Redis CRDT for global daily limits. Implement webhook notifications when clients approach limits. Add a real-time analytics dashboard. Implement tiered rate limits (free, pro, enterprise) with automatic provisioning.

V5: Self-Tuning and Adaptive (Month 12)

Implement adaptive rate limiting where limits adjust based on real-time backend health metrics (CPU, latency, error rate). When backend latency increases, the rate limiter automatically reduces limits proportionally. Add ML-based anomaly detection for identifying sophisticated bypass attempts. Implement cost-aware rate limiting where expensive operations (complex queries, ML inference) receive lower limits. Add A/B testing for rate limit configurations.

Evolution Roadmap
═══════════════════════════════════════════════════════════
V1  │ Embedded, in-memory, 10K RPS
V2  │ Standalone + Redis, YAML config, 100K RPS
V3  │ Distributed, hot cache, circuit breaker, 1M+ RPS
V4  │ Multi-region, CRDT, webhooks, analytics dashboard
V5  │ Adaptive limits, ML anomaly detection, cost-aware
═══════════════════════════════════════════════════════════

21. Common Bottlenecks and Production Incidents

Real-world production systems encounter failure modes that are difficult to anticipate in design documents. This section covers the most common bottlenecks and incident scenarios that affect rate limiters.

21.1 Redis CPU Exhaustion

Redis is single-threaded. A single Lua script that runs for 1ms limits throughput to 1,000 QPS per shard. At peak, a hot key can drive one shard to 100% CPU. Detection: monitor Redis used_cpu_user metric. Mitigation: shard horizontally (add more master nodes), optimize the Lua script to minimize computation, and increase local cache hit rate to reduce the number of Redis calls.

21.2 Clock Skew Across Nodes

If rate limiter nodes have system clocks that differ by more than a few seconds, the window calculation produces inconsistent results. A request on Node A (clock ahead by 5 seconds) might compute a different window ID than the same request on Node B. Mitigation: configure NTP with tight tolerance (max 100ms drift), use monotonic clocks for elapsed-time calculations, and log clock offset as a metric.

21.3 Memory Exhaustion in Local Cache

If the traffic pattern changes rapidly (e.g., a DDoS attack uses randomized API keys), the local cache fills up with keys that are each seen once. Without LRU eviction, the cache grows unbounded and causes an out-of-memory crash. Mitigation: enforce a maximum cache size (1 million entries), use LRU eviction, and monitor cache size as a metric.

21.4 Network Partition Between Gateway and Rate Limiter

If the network link between the Gateway and the rate limiter service is disrupted, gRPC calls time out after 50ms. The Gateway must decide whether to fail open (allow) or fail closed (deny). Most implementations fail open. Detection: monitor gRPC error rates and latency. Mitigation: deploy rate limiter instances in the same AZ as the Gateway, and use multiple AZs for redundancy.

21.5 Rate Limit Misconfiguration

An administrator accidentally sets a limit to 0 for a critical API endpoint, blocking all traffic. Or sets a limit so high that it provides no protection. Mitigation: implement config validation (reject limits of 0, require minimum window sizes), add a config diff review process, and implement shadow mode where new configurations are logged but not enforced for 24 hours.

21.6 Leap Second Handling

A leap second causes the system clock to jump backward or forward by one second. If the window calculation uses wall-clock time, a backward jump can cause the current window to become the "previous" window, resetting counters unexpectedly. Mitigation: use NTP leap second smoothing, or use monotonic time for window calculations. The impact is typically negligible (a 1-second window drift).

21.7 Thundering Herd After Recovery

When Redis recovers from an outage, all rate limiter instances simultaneously flush their local caches and start hitting Redis again. This creates a sudden spike of Redis traffic that can cause a second failure. Mitigation: implement staggered cache warm-up (randomize TTLs across instances), add Redis connection retry with exponential backoff, and pre-warm caches gradually.

22. Interview Q&A — Senior and Staff Level

Rate limiting is one of the most frequently tested topics in system design interviews. Below are detailed Q&A pairs that cover the depth expected at senior and staff engineer levels.

Q1: Which rate limiting algorithm would you choose and why?

A: The Sliding Window Counter algorithm. It approximates a true sliding window using two fixed-window counters, providing excellent accuracy with minimal memory (two integers per key). It eliminates the boundary spike problem of Fixed Window Counter while being far more memory-efficient than the Sliding Window Log (which stores every request timestamp in a sorted set). It supports bursts through configurable bucket sizes and is the algorithm used by Cloudflare, Envoy, and most production systems. For APIs that require strict burst control, Token Bucket is a strong alternative.

Q2: How do you handle distributed rate limiting across multiple data centers?

A: Each data center runs its own rate limiter and Redis cluster for regional limits. This keeps latency sub-millisecond. For global limits (e.g., total daily API calls per user across all regions), use Redis CRDT (Conflict-Free Replicated Data Types) with Redis Enterprise Active-Active. Each region increments its local counter independently. During normal operation, counters converge within sub-second replication lag. During a network partition, each region continues enforcing limits independently, potentially allowing the global count to temporarily exceed the limit by the number of regions times the per-region increment. Once the partition heals, counters converge. The trade-off is availability over strict consistency, which is the correct choice for rate limiting.

Q3: How do you prevent race conditions in distributed rate limiting?

A: Use a single Redis Lua script that atomically checks the counter, computes the effective count, decides whether to allow or deny, and increments the counter. Redis Lua scripts execute atomically on a single shard — no other client command is processed while the script runs. This means two concurrent requests for the same key on the same shard are serialized. Since Redis Cluster routes all operations for a given key to the same shard (via CRC16 hashing), there are no cross-shard race conditions for a single key.

Q4: What happens when Redis goes down? Walk me through the failure modes.

A: The circuit breaker detects consecutive Redis failures. After 5 failures, it trips open and all decisions fall back to local in-memory counters. Each rate limiter instance tracks counts independently. This means a client can exceed the global limit by up to N times (where N is the number of instances), because each instance enforces its own count. After 30 seconds, the circuit enters half-open and sends a probe. If Redis is healthy, normal operation resumes. The key design decision is fail-open vs fail-closed. We choose fail-open (allow traffic during failure) because the cost of denying all legitimate traffic is higher than the cost of allowing some excess traffic. Backend services should have their own independent rate limiting as a safety net.

Q5: How do you handle a hot key — a single API key receiving millions of requests per second?

A: The primary defense is a local in-memory cache on each rate limiter instance with a 50-100ms TTL. For a key receiving 50,000 RPS distributed across 10 instances, each instance serves 5,000 RPS. With a 100ms cache TTL, each instance makes only 10 Redis calls per second for that key (down from 5,000). The local cache absorbs 99.8% of Redis calls. For even hotter keys, implement affinity routing where the Gateway directs all requests for that key to a single instance, maximizing local cache hit rate. Monitor Redis shard CPU as a leading indicator of hot key issues.

Q6: Design a rate limiter that supports 10 different rate limit rules per API endpoint.

A: Each rule evaluates independently. When a request arrives, extract all applicable rate limit dimensions: client IP, API key, user ID, endpoint, HTTP method, and request size. For each dimension, check the corresponding rate limit rule. If any rule is exceeded, deny the request. Use Redis pipelining to send all rule checks in a single network round trip. Return the most restrictive remaining quota in the response headers. Example: a request to POST /api/search from a pro-tier user with IP 1.2.3.4 might be checked against: per-IP (60/min), per-API-key (1000/min), per-user (500/min), per-endpoint (100/min), per-method-POST (50/min). All five must pass for the request to be allowed.

Q7: How would you test a rate limiter in production safely?

A: Use shadow mode. Deploy the rate limiter making decisions but not enforcing them. Log all decisions (allow/deny) to Kafka for analysis. Compare the rate limiter's decisions against the actual traffic patterns. Gradually shift to enforcement using feature flags, starting with a single low-traffic API key. Monitor error rates, latency, and Redis health. Once confident, roll out enforcement to all keys over 1-2 weeks. Keep the ability to instantly disable enforcement via the feature flag if issues arise.

Q8: Staff-level: How do you design self-tuning rate limits that adapt to backend health?

A: Monitor backend health metrics (CPU utilization, p99 latency, error rate, queue depth). Define a health score that combines these metrics. When the health score degrades (e.g., backend CPU exceeds 80%), automatically reduce rate limits proportionally. Use a feedback control loop: compute the desired reduction factor (e.g., 0.5x for 80-90% CPU, 0.25x for 90-95%), apply it to all active rate limit rules, and monitor whether the backend recovers. When the health score normalizes, gradually restore original limits. This creates a self-healing system where the rate limiter acts as a pressure valve, automatically protecting the backend during traffic spikes without human intervention. The key challenge is avoiding oscillation — limits bouncing up and down. Use a slow ramp-up (10% per minute) and fast ramp-down (50% instant) to stabilize.

Q9: How do you handle rate limiting for WebSocket connections or long-lived connections?

A: WebSocket connections do not make per-request rate limit checks. Instead, rate limit at the connection level: limit the number of concurrent connections per client, limit the message send rate per connection, and limit the total bandwidth per connection. Use a token bucket for message rate limiting (tokens represent send permits). The connection limit is enforced at the WebSocket upgrade handshake. If the client exceeds the connection limit, reject the upgrade with 429.

Q10: How would you handle rate limiting in a microservices architecture where Service A calls Service B calls Service C?

A: Implement rate limiting at two levels. The API Gateway enforces external client rate limits (the "edge" rate limiter). Internal services enforce inter-service rate limits to prevent cascading failures. For example, if Service A calls Service B at 10,000 RPS, Service B should enforce its own rate limit on Service A's calls. Use service-specific API keys or mTLS identities to identify callers. Internal rate limits should be more generous than external limits (since one external call might fan out to multiple internal calls). Use a separate rate limit key namespace for internal traffic.

23. Key Takeaways

Designing a distributed rate limiter is a multi-faceted engineering challenge that spans algorithms, distributed systems, performance engineering, and operational excellence. Here are the most important lessons from this guide.

1. Choose the Sliding Window Counter algorithm for most use cases. It provides the best balance of accuracy, memory efficiency, and implementation simplicity. Use Token Bucket when burst handling is a first-class requirement.
2. Use Redis Lua scripts for atomic check-and-increment operations. Lua scripts eliminate race conditions and execute in under 0.1ms. Load scripts at startup and cache SHA hashes to avoid recompilation overhead.
3. Implement local in-memory caching for hot keys with a 50-100ms TTL. This reduces Redis load by 100-500x for popular keys and brings p99 latency to under 0.5ms.
4. Deploy a circuit breaker with local fallback. When Redis is unreachable, serve decisions from local in-memory counters. Accept the accuracy degradation (clients may exceed the global limit by N times) as a trade-off for availability.
5. Always return rate limit headers (X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, Retry-After). These headers are essential for client-side rate limiting and developer experience.
6. Deploy across multiple availability zones with Redis Cluster for high availability. Use at least 3 AZs and 6 Redis shards with replicas. RTO should be under 30 seconds for any single failure.
7. Monitor everything. Redis latency, circuit breaker state, denial rate, local cache hit rate, and per-key usage patterns. Alert on anomalies before they become incidents.
8. Build incrementally. Start with V1 (in-memory, single node) and evolve to V5 (adaptive, ML-powered) as your system matures. Do not over-engineer on day one.
9. Know your failure mode. Decide explicitly whether your rate limiter fails open or fails closed. Document this decision and ensure all stakeholders understand the implications.
10. Secure the rate limiter. Use mTLS for internal communication, Redis AUTH and TLS for data store access, and never expose Redis directly to the internet.

Rate Limiter Design Checklist
═══════════════════════════════════════════════════════════
[✓] Algorithm: Sliding Window Counter (recommended)
[✓] State store: Redis Cluster with Lua scripts
[✓] Local cache: 100ms TTL, LRU, 1M entries max
[✓] Circuit breaker: 5 failures → open, 30s probe
[✓] Fail mode: Fail-open for most APIs
[✓] Deployment: K8s, 10+ pods, 3 AZs
[✓] Redis: 6 shards, 2 replicas each
[✓] Transport: gRPC with HTTP/2
[✓] Headers: X-RateLimit-*, Retry-After
[✓] Monitoring: Prometheus + Grafana + OpenTelemetry
[✓] Config: etcd with runtime hot-reload
[✓] Security: mTLS, Redis AUTH, no public exposure
═══════════════════════════════════════════════════════════

Rate limiting is not a one-time implementation. It is a living system that must evolve with your platform. As your traffic grows, your rate limiter must scale. As your business model changes, your rate limits must adapt. As new attack vectors emerge, your defenses must evolve. The architecture described in this guide provides a solid foundation that can grow from handling thousands of requests per second to handling millions, from a single data center to a global deployment, from static rules to self-tuning adaptive limits. The key is to start simple, measure everything, and invest in complexity only when the data justifies it.

Frequently Asked Questions

What is the best rate limiting algorithm for distributed systems?

The Sliding Window Counter algorithm is the most practical choice for distributed systems. It approximates a true sliding window using two fixed-window counters, balancing accuracy and memory. It avoids the boundary-spike problem of fixed windows while using only two integers per key, making it far more memory-efficient than the Sliding Window Log approach.

How do you achieve atomic rate limiting with Redis?

Use a Redis Lua script that atomically reads the current counter, increments it, sets the TTL, computes the effective sliding window count, and returns the decision. Because Redis executes Lua scripts atomically on a single shard, there are no race conditions between concurrent requests.

Should a rate limiter fail open or fail closed?

It depends on business priorities. Fail-open allows traffic when the rate limiter is down, protecting revenue and user experience but risking backend overload. Fail-closed blocks all traffic when the rate limiter is down, protecting backend services at the cost of availability. Most public APIs choose fail-open with alerting and manual escalation.

How do you handle hot keys in a distributed rate limiter?

Use a local in-memory cache on each rate limiter instance with a very short TTL (50-100ms). For a key receiving thousands of requests per second, the local cache absorbs the vast majority of decisions without hitting Redis. On cache miss, the request goes to Redis and populates the local cache.

What Redis data structure is used for rate limiting?

Most production rate limiters use Redis strings with Lua scripting for the Sliding Window Counter algorithm. Two string keys per rate limit key store the current and previous window counts. For the Sliding Window Log algorithm, Redis Sorted Sets store individual request timestamps.

How do you test a rate limiter in production safely?

Use shadow mode where the rate limiter makes decisions but does not enforce them. Log all decisions (allow/deny) for analysis. Gradually shift from shadow mode to enforcement for specific API keys. Use feature flags to control rollout per tenant.