How to Design an API Gateway & Rate Limiter

Building a Kong/AWS API Gateway-Scale System — Routing, Throttling, Authentication, Observability & Extensibility

Senior+ System Design Guide C# · Mermaid · Real-World Case Studies

Table of Contents

1. Introduction & Why API Gateways Matter
2. Functional & Non-Functional Requirements
3. Capacity Estimation & Back-of-Envelope Math
4. Data Model & Storage Schema
5. High-Level Architecture Overview
6. Request Routing & Load Balancing
7. Rate Limiting Algorithms (Token Bucket, Sliding Window)
8. Authentication & Authorization (JWT / OAuth 2.0)
9. Request & Response Transformation
10. API Versioning Strategies
11. Circuit Breaker & Retry Logic
12. Caching Layer & Response Cache
13. API Analytics & Monitoring
14. Developer Portal & Documentation
15. Plugin / Extension System
16. WebSocket & gRPC Support
17. Multi-Tenancy & Workspace Isolation
18. Security — DDoS, Injection & Threat Mitigation
19. Reliability & Failure Modes
20. Cost Estimation & Infrastructure Sizing
21. Interview Q&A Deep Dive

1. Introduction & Why API Gateways Matter

In the era of microservices, a single user request often fans out across dozens of backend services. Without a centralized entry point, clients must know the exact location, protocol, and authentication requirements of every service. This creates a tightly coupled system that is fragile, difficult to evolve, and nearly impossible to secure uniformly. The API Gateway pattern solves this problem by acting as the single front door for all client traffic — routing requests, enforcing policies, transforming payloads, and providing a unified observability surface.

Commercial and open-source API Gateways like Kong, NGINX, AWS API Gateway, Apigee, Tyk, and Traefik have become foundational infrastructure for companies of every size. At Netflix, the Zuul gateway handles billions of API calls per day. At Shopify, a custom gateway mediates traffic between merchants, partners, and internal microservices. At Cloudflare, the gateway serves as the edge enforcement point for rate limiting, bot detection, and DDoS mitigation across millions of domains.

The core value proposition of an API Gateway can be distilled into five pillars:

• Abstraction: Clients interact with a stable API surface while backend services evolve independently. The gateway decouples the external contract from internal service topology.
• Security: Authentication, authorization, encryption, and threat mitigation are applied once at the gateway rather than duplicated across every microservice.
• Traffic Management: Rate limiting, load balancing, circuit breaking, retries, and timeouts are centrally managed, providing consistent resilience policies.
• Observability: All traffic flows through the gateway, making it the natural point for logging, metrics collection, distributed tracing, and anomaly detection.
• Extensibility: Plugin systems allow operators to inject custom logic — authentication providers, request transforms, logging adapters — without modifying gateway source code.

Building an API Gateway from scratch is a non-trivial undertaking that touches nearly every area of distributed systems design: high-performance networking, concurrent request processing, distributed state management, hot-reloadable configuration, and graceful degradation. This article walks through the complete system design of an API Gateway and Rate Limiter, covering the same depth you would encounter in a staff-level system design interview or an architecture review at a cloud-scale company.

Target Audience: This guide is written for Senior, Staff, and Principal Engineers who need to design, evaluate, or operate API Gateway infrastructure. We assume familiarity with HTTP, gRPC, distributed systems, and basic networking concepts.

Real-World Gateway Comparison

Gateway	Type	Languages	Key Differentiator
Kong	Open-source / Enterprise	Lua (OpenResty)	Nginx-based, massive plugin ecosystem
AWS API Gateway	Managed	—	Deep AWS integration, serverless
NGINX	Open-source / Plus	C	Extreme performance, reverse proxy
Apigee	Managed (Google)	Java	Enterprise policy management, analytics
Traefik	Open-source	Go	Auto-discovery, Kubernetes native
Tyk	Open-source / Enterprise	Go	GraphQL support, built-in analytics

2. Functional & Non-Functional Requirements

Before designing any system, we must clearly delineate what it must do and how well it must do it. For an API Gateway, requirements span routing correctness, security enforcement, performance targets, and operational qualities.

Functional Requirements

• Request Routing: Map incoming HTTP/gRPC/WebSocket requests to appropriate backend services based on path, headers, method, and query parameters. Support path-based, host-based, and header-based routing rules.
• Load Balancing: Distribute traffic across multiple instances of a backend service using round-robin, weighted, least-connections, or consistent-hashing strategies.
• Rate Limiting: Enforce per-client, per-endpoint, per-tenant, and global rate limits using configurable algorithms (Token Bucket, Sliding Window, Fixed Window, Leaky Bucket).
• Authentication & Authorization: Validate JWT tokens, enforce OAuth 2.0 flows, verify API keys, and support mTLS for service-to-service authentication.
• Request/Response Transformation: Modify headers, bodies, query parameters, and paths. Support JSON-to-XML translation, field masking, and payload enrichment.
• API Versioning: Route traffic to different backend versions based on URL path, header, query parameter, or content-type negotiation.
• Circuit Breaking: Detect failing backend services and temporarily stop sending traffic to prevent cascading failures.
• Caching: Cache responses at the gateway level to reduce backend load and latency for cacheable endpoints.
• Logging & Analytics: Record detailed request/response logs, compute latency percentiles, track error rates, and expose metrics via Prometheus/OpenTelemetry.
• Developer Portal: Provide auto-generated API documentation, interactive API explorer, SDK generation, and developer onboarding workflows.
• Plugin System: Allow custom plugins to be loaded at runtime for authentication, transformation, logging, and other cross-cutting concerns.
• Multi-Tenancy: Isolate workspaces, configurations, and rate limits per tenant or organization.

Non-Functional Requirements

Requirement	Target	Rationale
Throughput	500,000+ requests/second per gateway node	Must handle enterprise-scale traffic without becoming a bottleneck
Latency Overhead	< 5ms p99 added latency	Gateway should be nearly transparent to the request path
Availability	99.99% uptime	Single point of failure — must be more available than any backend service
Scalability	Linear horizontal scaling	Capacity grows proportionally with added gateway instances
Config Propagation	< 5 seconds globally	Route changes, rate limit updates, and plugin deployments must propagate quickly
Graceful Degradation	Continue serving during partial failures	Backend outages or cache failures should not cascade to the gateway
Hot Reload	Zero-downtime configuration updates	New routes, certificates, and plugins must be applied without restarts

Interview Tip: Always clarify whether the gateway is a reverse proxy (handling all HTTP traffic) or an API gateway (focused on API-specific concerns like authentication, rate limiting, and transformation). The scope difference significantly impacts complexity and performance requirements.

3. Capacity Estimation & Back-of-Envelope Math

Let us size the system for a moderately large deployment — an API Gateway serving a mid-to-large SaaS platform with millions of daily active users.

Traffic Assumptions

Metric	Value	Notes
Daily Active Users	10,000,000	Mobile + Web + API consumers
Average requests per user per day	50	Mix of read and write operations
Total daily requests	500,000,000	500M requests/day
Requests per second (avg)	~5,800	500M / 86,400 seconds
Peak requests per second	~29,000	5x average for peak hours
Average request payload	2 KB	Headers + body for typical API call
Average response payload	10 KB	JSON responses with data
Peak sustained throughput	~290 MB/s inbound, ~1.45 GB/s outbound	At peak RPS with avg payloads

Storage Estimation

For request logging and analytics, we need to estimate the storage footprint:

• Each log entry (request ID, timestamp, method, path, status code, latency, client IP, tenant ID): approximately 500 bytes.
• Daily log volume: 500M requests * 500 bytes = 250 GB/day.
• Monthly log volume: 250 GB * 30 = 7.5 TB/month.
• With compression (roughly 4:1 for structured logs): ~1.9 TB/month compressed.
• Retention of 90 days: ~5.7 TB compressed. With hot/warm/cold tiering, hot tier (7 days) needs ~1.9 TB of fast storage.

Rate Limiter State Estimation

• Unique API keys: 100,000 (tenants with multiple keys).
• Sliding window counters per key per endpoint: with 500 popular endpoints, worst case 50M counter entries.
• Each counter entry (key + window timestamp + count): ~64 bytes.
• Total rate limiter state: 50M * 64 bytes = ~3.2 GB — easily fits in a distributed in-memory store like Redis.

Bandwidth & Connection Estimation

• At 29,000 RPS with average 10 connections per backend service, we need ~290,000 concurrent open connections from the gateway pool.
• TCP connection overhead: ~3 KB per connection (kernel buffers + userspace state). Total: ~870 MB just for connection state.
• With HTTP/2 multiplexing, this drops significantly — a single HTTP/2 connection can carry thousands of concurrent streams.

Key Insight: The gateway is CPU and network bound, not memory or storage bound. The critical path is packet processing, TLS termination, header parsing, and rule evaluation — all of which benefit from zero-copy networking, kernel bypass (DPDK/io_uring), and lock-free data structures.

4. Data Model & Storage Schema

The API Gateway maintains a rich configuration and metadata model. While the hot path (request processing) primarily uses in-memory data structures and caches, the configuration store, analytics backend, and developer portal rely on persistent storage.

Core Entities

C#
public class GatewayRoute
{
    public string Id { get; set; }
    public string Name { get; set; }
    public string TenantId { get; set; }
    public List<RouteMatch> Matches { get; set; }
    public List<ServiceTarget> Targets { get; set; }
    public LoadBalancingStrategy LbStrategy { get; set; }
    public List<Plugin> Plugins { get; set; }
    public RateLimitConfig RateLimit { get; set; }
    public RetryConfig Retry { get; set; }
    public CircuitBreakerConfig CircuitBreaker { get; set; }
    public CacheConfig Cache { get; set; }
    public int Priority { get; set; }
    public bool Enabled { get; set; }
    public DateTime CreatedAt { get; set; }
    public DateTime UpdatedAt { get; set; }
}

public class RouteMatch
{
    public string PathPattern { get; set; }
    public string Method { get; set; }
    public Dictionary<string, string> Headers { get; set; }
    public string HostPattern { get; set; }
    public Dictionary<string, string> QueryParams { get; set; }
}

public class ServiceTarget
{
    public string ServiceId { get; set; }
    public string Url { get; set; }
    public int Weight { get; set; }
    public TimeSpan Timeout { get; set; }
    public string Protocol { get; set; }
}

public class RateLimitConfig
{
    public string Algorithm { get; set; }
    public int RequestsPerWindow { get; set; }
    public int WindowSizeSeconds { get; set; }
    public int BurstSize { get; set; }
    public string Scope { get; set; }
    public string KeyExtractor { get; set; }
    public Dictionary<string, int> OverrideLimits { get; set; }
}

Plugin Configuration

C#
public class Plugin
{
    public string Id { get; set; }
    public string Name { get; set; }
    public string Version { get; set; }
    public PluginPhase Phase { get; set; }
    public Dictionary<string, object> Config { get; set; }
    public bool Enabled { get; set; }
    public List<string> Conditions { get; set; }
}

public enum PluginPhase
{
    Access,
    HeaderFilter,
    BodyFilter,
    Log,
    Rewrite,
    Certify
}

Analytics & Metrics Schema

Table / Index	Key Columns	Storage Engine
request_logs	request_id, timestamp, tenant_id, route_id, method, path, status, latency_ms, request_size, response_size, client_ip, user_agent	ClickHouse / TimescaleDB
rate_limit_counters	key, window_start, count, limit, remaining	Redis (in-memory)
api_keys	key_hash, tenant_id, scopes, rate_limit_tier, created_at, expires_at, revoked	PostgreSQL
certificate_store	cert_id, domain, cert_pem, key_pem, expires_at, auto_renew	Vault / encrypted PostgreSQL
plugins_config	plugin_id, route_id, tenant_id, config_json, enabled, version	PostgreSQL + Redis cache

Design Choice: We use PostgreSQL as the source of truth for configuration and Redis as the hot cache. Configuration changes are written to PostgreSQL, then a change-data-capture (CDC) pipeline pushes updates to Redis and notifies gateway nodes to reload. This gives us durability, transactional consistency, and sub-second propagation.

5. High-Level Architecture Overview

The API Gateway sits between clients and backend services. Its architecture must handle the full request lifecycle: connection termination, protocol detection, authentication, rate limiting, routing, load balancing, proxying, response transformation, and logging.

graph TB subgraph Clients A[Mobile App] B[Web Browser] C[Third-Party API] D[Internal Service] end subgraph Edge Layer E[Global Load Balancer] F[WAF / DDoS Protection] end subgraph API Gateway Cluster G[Gateway Node 1] H[Gateway Node 2] I[Gateway Node N] end subgraph Gateway Internals J[Listener / Connection Manager] K[Plugin Pipeline] L[Router / Matcher] M[Rate Limiter] N[Auth Engine] O[Transformer] P[Load Balancer] Q[Health Checker] R[Log Aggregator] end subgraph Control Plane S[Config API Server] T[PostgreSQL Config DB] U[Redis Cache] V[Plugin Registry] W[Certificate Manager] end subgraph Data Plane Analytics X[ClickHouse] Y[Prometheus] Z[Jaeger/Tempo] end subgraph Backend Services AA[User Service] AB[Order Service] AC[Payment Service] AD[Notification Service] end A --> E B --> E C --> E D --> E E --> F F --> G F --> H F --> I G --> J J --> K K --> L K --> M K --> N K --> O L --> P P --> AA P --> AB P --> AC P --> AD G --> R R --> X R --> Y R --> Z S --> T S --> U S --> V S --> W G -.->|config sync| S

Request Lifecycle

1. Connection Accept: The listener accepts the TCP/TLS connection, performs TLS termination, and allocates a connection from the worker pool.
2. Protocol Detection: The gateway detects whether the request is HTTP/1.1, HTTP/2, gRPC, or WebSocket and routes to the appropriate handler.
3. Access Phase (Plugins): Authentication plugins validate tokens/API keys. Rate limiter checks quotas. CORS plugin validates origin. Custom plugins execute in configured order.
4. Route Matching: The router evaluates the request against all configured routes (sorted by priority) and selects the matching route and target service.
5. Rewrite Phase: Request path, headers, and body are transformed according to route and plugin configuration.
6. Load Balancing & Proxy: The load balancer selects a healthy backend instance and proxies the request, managing timeouts, retries, and circuit breaking.
7. Header Filter Phase: Response headers are modified (add/remove/cors).
8. Body Filter Phase: Response body is optionally transformed (compression, field filtering).
9. Log Phase: Request/response data is asynchronously logged to the analytics pipeline.

C#
public class GatewayRequestPipeline
{
    private readonly IReadOnlyList<IPlugin> _accessPlugins;
    private readonly IReadOnlyList<IRewritePlugin> _rewritePlugins;
    private readonly IReadOnlyList<ILogPlugin> _logPlugins;
    private readonly IRouter _router;
    private readonly ILoadBalancer _loadBalancer;
    private readonly IHealthChecker _healthChecker;

    public async Task<GatewayResponse> ProcessAsync(GatewayContext context)
    {
        var sw = Stopwatch.StartNew();

        foreach (var plugin in _accessPlugins)
        {
            if (!plugin.Matches(context)) continue;
            var result = await plugin.ExecuteAsync(context);
            if (result.Rejected)
            {
                sw.Stop();
                await LogAsync(context, sw.ElapsedMilliseconds, result.StatusCode);
                return GatewayResponse.Rejected(result.StatusCode, result.Message);
            }
        }

        var route = _router.Match(context.Request);
        if (route == null)
        {
            sw.Stop();
            await LogAsync(context, sw.ElapsedMilliseconds, 404);
            return GatewayResponse.NotFound("No matching route");
        }

        context.Route = route;

        foreach (var plugin in _rewritePlugins)
        {
            if (plugin.Matches(context))
                await plugin.RewriteAsync(context);
        }

        var target = _loadBalancer.SelectTarget(route.Targets, context.Request);
        if (target == null || !_healthChecker.IsHealthy(target.ServiceId))
        {
            sw.Stop();
            await LogAsync(context, sw.ElapsedMilliseconds, 503);
            return GatewayResponse.ServiceUnavailable("No healthy upstream");
        }

        var proxyResponse = await ProxyAsync(context.Request, target, route.Retry);
        sw.Stop();

        _ = LogAsync(context, sw.ElapsedMilliseconds, proxyResponse.StatusCode);

        return GatewayResponse.FromUpstream(proxyResponse);
    }
}

Architecture Decision: We choose a sidecar-free, shared-nothing architecture for the gateway data plane. Each gateway node is stateless — it loads configuration from the control plane and holds only ephemeral state (connection pools, rate limit counters via Redis). This makes horizontal scaling trivial: add a node behind the load balancer and it immediately starts serving traffic.

6. Request Routing & Load Balancing

Routing is the most performance-critical path in the gateway. Every incoming request must be matched against potentially hundreds or thousands of route rules with sub-millisecond latency. The routing engine must support complex matching criteria while remaining fast enough to handle hundreds of thousands of lookups per second per gateway node.

Route Matching Strategies

Naive route matching — iterating through all routes and evaluating regex patterns — is too slow for production use. Instead, we use a radix trie (prefix tree) for path matching, which reduces path lookup from O(n) to O(k) where k is the path depth.

C#
public class RadixTrieRouter : IRouter
{
    private readonly TrieNode _root = new();
    private readonly ReaderWriterLockSlim _lock = new();

    public void AddRoute(GatewayRoute route)
    {
        _lock.EnterWriteLock();
        try
        {
            foreach (var match in route.Matches)
            {
                var segments = match.PathPattern.Split('/',
                    StringSplitOptions.RemoveEmptyEntries);
                var current = _root;

                foreach (var segment in segments)
                {
                    if (segment.StartsWith('{') && segment.EndsWith('}'))
                    {
                        var paramName = segment[1..^1];
                        current = current.GetOrCreateParamChild(paramName);
                    }
                    else
                    {
                        current = current.GetOrCreateLiteralChild(segment);
                    }
                }

                current.AddRoute(route, match);
            }
        }
        finally
        {
            _lock.ExitWriteLock();
        }
    }

    public RouteMatchResult Match(GatewayRequest request)
    {
        _lock.EnterReadLock();
        try
        {
            var segments = request.Path.Split('/', StringSplitOptions.RemoveEmptyEntries);
            var candidates = new List<(GatewayRoute route, RouteMatch match,
                Dictionary<string, string> params)>();

            MatchRecursive(_root, segments, 0, new Dictionary<string, string>(),
                candidates, request);

            if (candidates.Count == 0) return RouteMatchResult.NoMatch;

            return candidates
                .OrderByDescending(c => c.route.Priority)
                .ThenByDescending(c => c.match.Specificity)
                .Select(c => RouteMatchResult.Matched(c.route, c.match, c.params))
                .First();
        }
        finally
        {
            _lock.ExitReadLock();
        }
    }

    private void MatchRecursive(TrieNode node, string[] segments, int depth,
        Dictionary<string, string>routeParams,
        List<(GatewayRoute, RouteMatch, Dictionary<string, string>)> candidates,
        GatewayRequest request)
    {
        if (depth == segments.Length)
        {
            foreach (var (route, match) in node.TerminalRoutes)
            {
                if (MatchMethod(match, request.Method) &&
                    MatchHeaders(match, request.Headers) &&
                    MatchHost(match, request.Host))
                {
                    candidates.Add((route, match,
                        new Dictionary<string, string>(routeParams)));
                }
            }
            return;
        }

        var segment = segments[depth];

        if (node.Children.TryGetValue(segment, out var literalChild))
        {
            MatchRecursive(literalChild, segments, depth + 1, routeParams,
                candidates, request);
        }

        foreach (var (paramName, paramChild) in node.ParamChildren)
        {
            routeParams[paramName] = Uri.UnescapeDataString(segment);
            MatchRecursive(paramChild, segments, depth + 1, routeParams,
                candidates, request);
            routeParams.Remove(paramName);
        }
    }
}

Load Balancing Algorithms

Algorithm	Description	Best For	Complexity
Round Robin	Cycle through targets sequentially	Uniform backends, simple setup	O(1)
Weighted Round Robin	Honor per-target weights	Heterogeneous backend capacity	O(1)
Least Connections	Route to target with fewest active connections	Varying request durations	O(log n)
Power of Two Choices	Randomly pick two, choose the one with fewer connections	Good balance of simplicity and performance	O(1) amortized
Consistent Hash	Hash request attribute to pin to a target	Cache-friendly routing, session affinity	O(log n)
Least Response Time	Route to target with lowest avg response time	Latency-sensitive workloads	O(log n)

C#
public class PowerOfTwoChoicesBalancer : ILoadBalancer
{
    private readonly Random _random = new();
    private readonly ConcurrentDictionary<string, ConnectionPool> _pools = new();
    private readonly IHealthChecker _healthChecker;

    public ServiceTarget SelectTarget(IReadOnlyList<ServiceTarget> targets,
        GatewayRequest request)
    {
        var healthy = targets
            .Where(t => _healthChecker.IsHealthy(t.ServiceId)).ToList();
        if (healthy.Count == 0) return null;
        if (healthy.Count == 1) return healthy[0];

        var totalWeight = healthy.Sum(t => t.Weight);
        var pick1 = WeightedPick(healthy, totalWeight);
        var pick2 = WeightedPick(healthy, totalWeight);

        var pool1 = _pools.GetOrAdd(pick1.ServiceId,
            id => new ConnectionPool(id));
        var pool2 = _pools.GetOrAdd(pick2.ServiceId,
            id => new ConnectionPool(id));

        return pool1.ActiveConnections <= pool2.ActiveConnections ? pick1 : pick2;
    }

    private ServiceTarget WeightedPick(List<ServiceTarget> targets, int totalWeight)
    {
        var roll = _random.Next(totalWeight);
        var cumulative = 0;
        foreach (var target in targets)
        {
            cumulative += target.Weight;
            if (roll < cumulative) return target;
        }
        return targets[^1];
    }
}

Production Tip: Power of Two Choices is the default load balancing algorithm in many production gateways (Envoy, Kong). It achieves near-optimal load distribution with O(1) cost per request, avoiding the O(log n) overhead of tracking true least-connections while avoiding the pathological imbalance of pure random selection.

7. Rate Limiting Algorithms

Rate limiting is one of the most critical functions of an API Gateway. It protects backend services from traffic spikes, ensures fair resource allocation across tenants, prevents abuse, and provides predictable capacity planning. The choice of algorithm has profound implications for accuracy, memory usage, and user experience.

Token Bucket Algorithm

The Token Bucket algorithm models rate limiting as a bucket that holds tokens. Each request consumes one token. Tokens are added to the bucket at a fixed rate (the allowed request rate). The bucket has a maximum capacity (the burst size). If the bucket is empty, the request is rejected.

graph LR A[Token Generator] -->|Add tokens| B[Token Bucket] B -->|Check and consume| C{Has tokens?} C -->|Yes| D[Allow Request] C -->|No| E[Reject 429]

C#
public class TokenBucketRateLimiter : IRateLimiter
{
    private readonly ConcurrentDictionary<string, TokenBucket> _buckets = new();
    private readonly IRedisClient _redis;
    private readonly TokenBucketConfig _defaultConfig;

    public TokenBucketRateLimiter(IRedisClient redis, TokenBucketConfig config)
    {
        _redis = redis;
        _defaultConfig = config;
    }

    public async Task<RateLimitResult> CheckAsync(string key, RateLimitContext ctx)
    {
        var config = _defaultConfig;

        var bucket = _buckets.AddOrUpdate(key,
            _ => new TokenBucket(config.BurstSize, config.BurstSize,
                config.RefillRate, DateTime.UtcNow),
            (_, existing) =>
            {
                var elapsed = (DateTime.UtcNow - existing.LastRefill).TotalSeconds;
                var tokensToAdd = elapsed * config.RefillRate;
                var currentTokens = Math.Min(config.BurstSize,
                    existing.Tokens + tokensToAdd);
                return existing with
                {
                    Tokens = currentTokens,
                    LastRefill = DateTime.UtcNow
                };
            });

        if (bucket.Tokens >= 1)
        {
            _buckets[key] = bucket with { Tokens = bucket.Tokens - 1 };
            return RateLimitResult.Allowed(
                remaining: (int)Math.Floor(bucket.Tokens - 1),
                limit: config.BurstSize,
                resetAt: CalculateResetTime(bucket, config));
        }

        return RateLimitResult.Rejected(
            retryAfter: CalculateRetryAfter(bucket, config),
            limit: config.BurstSize,
            remaining: 0);
    }

    private double CalculateRetryAfter(TokenBucket bucket, TokenBucketConfig config)
    {
        var tokensNeeded = 1 - bucket.Tokens;
        return Math.Ceiling(tokensNeeded / config.RefillRate);
    }
}

public record TokenBucket(
    double Tokens,
    DateTime LastRefill,
    double RefillRate,
    double BurstSize);

Sliding Window Log Algorithm

The Sliding Window Log maintains a sorted set of timestamps for each request. When a new request arrives, it removes all entries outside the current window, counts remaining entries, and decides based on the limit. This provides the most accurate rate limiting but at higher memory cost.

C#
public class SlidingWindowLogRateLimiter : IRateLimiter
{
    private readonly IRedisClient _redis;
    private readonly SlidingWindowConfig _config;

    public async Task<RateLimitResult> CheckAsync(string key, RateLimitContext ctx)
    {
        var now = DateTimeOffset.UtcNow;
        var windowStart = now.AddSeconds(-_config.WindowSizeSeconds);
        var windowKey = $"ratelimit:{key}";

        var script = @"
            local key = KEYS[1]
            local window_start = tonumber(ARGV[1])
            local now = tonumber(ARGV[2])
            local limit = tonumber(ARGV[3])
            local window_size = tonumber(ARGV[4])
            local member = ARGV[5]
            redis.call('ZREMRANGEBYSCORE', key, '-inf', window_start)
            local current_count = redis.call('ZCARD', key)
            if current_count < limit then
                redis.call('ZADD', key, now, member)
                redis.call('EXPIRE', key, window_size + 1)
                return {1, limit - current_count - 1, 0}
            else
                local oldest = redis.call('ZRANGE', key, 0, 0, 'WITHSCORES')
                local retry_after = 0
                if #oldest > 0 then
                    retry_after = math.ceil(
                        tonumber(oldest[2]) + window_size - now)
                end
                return {0, 0, retry_after}
            end
        ";

        var result = await _redis.EvalAsync(script,
            keys: new[] { windowKey },
            args: new object[]
            {
                windowStart.ToUnixTimeMilliseconds(),
                now.ToUnixTimeMilliseconds(),
                _config.Limit,
                _config.WindowSizeSeconds,
                $"{now.ToUnixTimeMilliseconds()}-{Guid.NewGuid():N}"
            });

        var allowed = (long)result[0] == 1;
        var remaining = (long)result[1];
        var retryAfter = (long)result[2];

        return allowed
            ? RateLimitResult.Allowed(remaining, _config.Limit,
                now.AddSeconds(_config.WindowSizeSeconds))
            : RateLimitResult.Rejected(retryAfter, _config.Limit, 0);
    }
}

Sliding Window Counter (Hybrid)

The Sliding Window Counter is a hybrid approach that combines Fixed Window accuracy with Sliding Window smoothness. It approximates the count in the current sliding window by using a weighted sum of the previous window count and the current window count.

C#
public class SlidingWindowCounterRateLimiter : IRateLimiter
{
    private readonly IRedisClient _redis;
    private readonly int _windowSizeSeconds;
    private readonly int _limit;

    public async Task<RateLimitResult> CheckAsync(string key, RateLimitContext ctx)
    {
        var now = DateTimeOffset.UtcNow;
        var currentWindow = now.ToUnixTimeSeconds() / _windowSizeSeconds;
        var previousWindow = currentWindow - 1;
        var windowElapsed = now.ToUnixTimeSeconds() % _windowSizeSeconds;

        var prevKey = $"rl:{key}:{previousWindow}";
        var currKey = $"rl:{key}:{currentWindow}";

        var prevCount = ParseCount(await _redis.StringGetAsync(prevKey));
        var currCount = ParseCount(await _redis.StringGetAsync(currKey));

        var weight = 1.0 - ((double)windowElapsed / _windowSizeSeconds);
        var estimatedCount = (prevCount * weight) + currCount;

        if (estimatedCount < _limit)
        {
            await _redis.StringIncrementAsync(currKey);
            await _redis.KeyExpireAsync(currKey,
                TimeSpan.FromSeconds(_windowSizeSeconds * 2));

            var remaining = (int)Math.Max(0, _limit - estimatedCount - 1);
            return RateLimitResult.Allowed(remaining, _limit,
                now.AddSeconds(_windowSizeSeconds - windowElapsed));
        }

        var retryAfter = _windowSizeSeconds - windowElapsed;
        return RateLimitResult.Rejected(retryAfter, _limit, 0);
    }

    private static long ParseCount(string val)
    {
        return long.TryParse(val, out var count) ? count : 0;
    }
}

Algorithm Comparison

Algorithm	Accuracy	Memory	Burst Handling	Distributed	Complexity
Token Bucket	High	O(1) per key	Configurable burst	Requires sync	Medium
Sliding Window Log	Perfect	O(n) per key	Smooth	Redis sorted set	High
Sliding Window Counter	~99% accurate	O(1) per key	Smooth	Two counters	Low
Fixed Window	Low (edge bursts)	O(1) per key	2x burst at edges	Single counter	Very Low
Leaky Bucket	High	O(1) per key	No burst	Requires sync	Medium

Edge Case Warning: Fixed Window rate limiting has a well-known vulnerability at window boundaries. If a client sends 100 requests at 11:59:59.999 and 100 requests at 12:00:00.001, they effectively double their rate. The Sliding Window Counter eliminates this problem with minimal additional complexity and is the recommended default for production systems.

Distributed Rate Limiting Architecture

For multi-node gateway deployments, rate limit state must be shared. We use Redis as the centralized counter store. Each gateway node executes Lua scripts atomically against Redis to ensure correctness under concurrent access.

C#
public class DistributedRateLimiter : IRateLimiter
{
    private readonly RedisCluster _redisCluster;
    private readonly RateLimitConfig _config;
    private readonly LocalFallbackCache _localCache;

    public async Task<RateLimitResult> CheckAsync(string key, RateLimitContext ctx)
    {
        try
        {
            var result = await ExecuteRedisScript(key, ctx);
            _localCache.Set(key, result, TimeSpan.FromSeconds(5));
            return result;
        }
        catch (RedisConnectionException)
        {
            Logger.Warn("Redis unavailable, failing open for key: {Key}", key);
            var cached = _localCache.Get(key);
            return cached ?? RateLimitResult.Allowed(
                int.MaxValue, int.MaxValue, DateTime.MaxValue);
        }
    }
}

8. Authentication & Authorization (JWT / OAuth 2.0)

Authentication verifies the identity of the caller; authorization determines whether the caller has permission to perform the requested action. The API Gateway is the ideal enforcement point because it can validate credentials once and propagate identity context to all downstream services.

JWT Validation Pipeline

C#
public class JwtAuthPlugin : IAccessPlugin
{
    private readonly JwtSecurityTokenHandler _tokenHandler = new();
    private readonly TokenValidationParameters _validationParams;
    private readonly IKeyStore _keyStore;

    public JwtAuthPlugin(IKeyStore keyStore, JwtGatewayConfig config)
    {
        _keyStore = keyStore;
        _validationParams = new TokenValidationParameters
        {
            ValidateIssuer = true,
            ValidIssuer = config.Issuer,
            ValidateAudience = true,
            ValidAudience = config.Audience,
            ValidateLifetime = true,
            ValidateIssuerSigningKey = true,
            ClockSkew = TimeSpan.FromSeconds(30),
            RequireExpirationTime = true
        };
    }

    public async Task<PluginResult> ExecuteAsync(GatewayContext context)
    {
        if (!context.Request.Headers.TryGetValue("Authorization",
            out var authHeader) ||
            !authHeader.StartsWith("Bearer ",
                StringComparison.OrdinalIgnoreCase))
        {
            return PluginResult.Rejected(401,
                "Missing or invalid Authorization header");
        }

        var tokenString = authHeader["Bearer ".Length..].Trim();

        try
        {
            var principal = _tokenHandler.ValidateToken(
                tokenString, _validationParams, out var validatedToken);

            if (validatedToken is not JwtSecurityToken jwt)
                return PluginResult.Rejected(401, "Invalid token format");

            var jti = jwt.Claims
                .FirstOrDefault(c => c.Type == JwtRegisteredClaimNames.Jti)?
                .Value;
            if (jti != null && await _keyStore.IsTokenRevokedAsync(jti))
                return PluginResult.Rejected(401, "Token has been revoked");

            context.Identity = new GatewayIdentity
            {
                Subject = principal.FindFirst(ClaimTypes.NameIdentifier)?.Value,
                Email = principal.FindFirst(ClaimTypes.Email)?.Value,
                Roles = principal.FindAll(ClaimTypes.Role)
                    .Select(c => c.Value).ToList(),
                Scopes = jwt.Claims
                    .Where(c => c.Type == "scope")
                    .SelectMany(c => c.Value.Split(' '))
                    .ToList(),
                TenantId = jwt.Claims
                    .FirstOrDefault(c => c.Type == "tenant_id")?.Value,
                TokenExpiry = jwt.ValidTo,
                CustomClaims = jwt.Claims
                    .ToDictionary(c => c.Type, c => c.Value)
            };

            var authorized = await AuthorizeAsync(context);
            if (!authorized)
                return PluginResult.Rejected(403, "Insufficient permissions");

            return PluginResult.Allowed();
        }
        catch (SecurityTokenExpiredException)
        {
            return PluginResult.Rejected(401, "Token has expired");
        }
        catch (SecurityTokenException ex)
        {
            return PluginResult.Rejected(401,
                $"Token validation failed: {ex.Message}");
        }
    }

    private async Task<bool> AuthorizeAsync(GatewayContext context)
    {
        var route = context.Route;
        if (route.RequiredRoles == null || route.RequiredRoles.Count == 0)
            return true;

        return route.RequiredRoles.Any(role =>
            context.Identity.Roles.Contains(role,
                StringComparer.OrdinalIgnoreCase));
    }
}

OAuth 2.0 Integration

The gateway supports multiple OAuth 2.0 grant types depending on the client type:

Grant Type	Client Type	Flow	Security Notes
Authorization Code + PKCE	SPA, Mobile	Redirect-based with code exchange	Most secure for public clients
Client Credentials	Server-to-server	Direct token request	No user context; service-to-service
Device Authorization	Smart TV, IoT	Device code polling	User authorizes on separate device
JWT Bearer	Service-to-service	Self-signed JWT exchange	Eliminates shared secrets

Defense in Depth: Even though the gateway validates JWT tokens, backend services should also validate tokens (or verify a signed identity header). This defense-in-depth approach protects against gateway misconfigurations and ensures identity propagation is trustworthy.

9. Request & Response Transformation

API Gateways frequently need to modify requests and responses as they pass through. Common transformations include adding/removing headers, rewriting URL paths, extracting fields from JWT tokens into headers, masking sensitive response fields, and converting between data formats.

Transformer Pipeline

C#
public class RequestTransformer : IRewritePlugin
{
    public PluginPhase Phase => PluginPhase.Rewrite;

    public async Task<GatewayResponse> RewriteAsync(GatewayContext context)
    {
        var config = context.Route.TransformationConfig;

        if (config.AddHeaders != null)
        {
            foreach (var (name, value) in config.AddHeaders)
            {
                var resolvedValue = TemplateEngine.Resolve(value, context);
                context.Request.Headers[name] = resolvedValue;
            }
        }

        if (config.RemoveHeaders != null)
        {
            foreach (var header in config.RemoveHeaders)
                context.Request.Headers.Remove(header);
        }

        if (config.PathRewrite != null)
            context.Request.Path = TemplateEngine.Resolve(
                config.PathRewrite, context);

        if (config.IdentityHeaders && context.Identity != null)
        {
            context.Request.Headers["X-User-ID"] = context.Identity.Subject;
            context.Request.Headers["X-Tenant-ID"] =
                context.Identity.TenantId;
            context.Request.Headers["X-User-Roles"] =
                string.Join(",", context.Identity.Roles);
        }

        if (config.BodyTransform != null && context.Request.HasBody)
        {
            var body = await context.Request.ReadBodyAsJsonAsync();
            if (config.BodyTransform.RemoveFields != null)
            {
                foreach (var field in config.BodyTransform.RemoveFields)
                    body.Remove(field);
            }
            if (config.BodyTransform.AddFields != null)
            {
                foreach (var (field, value) in config.BodyTransform.AddFields)
                    body[field] = TemplateEngine.Resolve(value, context);
            }
            context.Request.SetBody(body);
        }

        return GatewayResponse.Next();
    }
}

public class ResponseTransformer : IHeaderFilterPlugin
{
    public PluginPhase Phase => PluginPhase.HeaderFilter;

    public async Task<GatewayResponse> TransformHeadersAsync(
        GatewayContext context, UpstreamResponse response)
    {
        response.Headers.Remove("X-Internal-Trace-ID");
        response.Headers.Remove("X-Backend-Server");

        response.Headers["X-Gateway-Version"] = "2.0";
        response.Headers["Strict-Transport-Security"] =
            "max-age=31536000; includeSubDomains";

        if (context.Route.CorsConfig != null)
        {
            var cors = context.Route.CorsConfig;
            var origin = context.Request.Headers.GetValueOrDefault("Origin");
            if (cors.AllowedOrigins.Contains("*") ||
                cors.AllowedOrigins.Contains(origin))
            {
                response.Headers["Access-Control-Allow-Origin"] =
                    origin ?? "*";
                response.Headers["Access-Control-Allow-Methods"] =
                    string.Join(", ", cors.AllowedMethods);
                response.Headers["Access-Control-Allow-Headers"] =
                    string.Join(", ", cors.AllowedHeaders);
                response.Headers["Access-Control-Max-Age"] =
                    cors.MaxAge.ToString();
            }
        }

        return GatewayResponse.Next();
    }
}

Performance Consideration: Response body transformation is expensive because it requires buffering the entire response. For large responses (files, streams), transformations should be applied selectively or the response should be streamed through without buffering. Always measure the latency impact of body transformations under realistic payload sizes.

10. API Versioning Strategies

API versioning allows backends to evolve without breaking existing clients. The gateway is the natural place to manage version routing because it can inspect requests and route them to the appropriate backend version.

Versioning Approaches

Strategy	Example	Pros	Cons
URL Path	/api/v1/users	Explicit, cacheable, easy to understand	URL proliferation, breaks RESTful constraints
Query Parameter	/api/users?version=2	No URL changes needed	Easy to forget, not cacheable by default
Request Header	Accept: application/vnd.api.v2+json	Clean URLs, content negotiation	Hidden from URL, harder to test in browser
Content-Type	application/vnd.myapi.v2+json	Leverages HTTP content negotiation	Complex, limited tooling support
Host-based	v2.api.example.com	Complete isolation	DNS management overhead, cert complexity

C#
public class VersionRouter : IRouter
{
    private readonly Dictionary<string, List<GatewayRoute>> _versionedRoutes
        = new();

    public RouteMatchResult Match(GatewayRequest request)
    {
        var version = ExtractVersion(request);
        var routes = _versionedRoutes.GetValueOrDefault(version)
            ?? _versionedRoutes["latest"];

        foreach (var route in routes.OrderByDescending(r => r.Priority))
        {
            if (route.Matches.Any(m => m.Matches(request)))
            {
                return RouteMatchResult.Matched(route,
                    route.Matches.First(m => m.Matches(request)));
            }
        }

        return RouteMatchResult.NoMatch;
    }

    private string ExtractVersion(GatewayRequest request)
    {
        var pathMatch = Regex.Match(request.Path, @"/api/v(\d+)/");
        if (pathMatch.Success) return $"v{pathMatch.Groups[1].Value}";

        if (request.Headers.TryGetValue("X-API-Version", out var hv))
            return $"v{hv}";

        if (request.QueryParams.TryGetValue("version", out var qv))
            return $"v{qv}";

        if (request.Headers.TryGetValue("Accept", out var accept))
        {
            var mediaMatch = Regex.Match(accept, @"vnd\.api\.(v\d+)");
            if (mediaMatch.Success) return mediaMatch.Groups[1].Value;
        }

        return "latest";
    }
}

Best Practice: Support at least two previous API versions simultaneously. Use a deprecation policy that gives clients 6-12 months notice before sunsetting a version. The gateway should return Sunset and Deprecation headers on deprecated versions to enable programmatic migration.

11. Circuit Breaker & Retry Logic

When a backend service is struggling — responding slowly or returning errors — continuing to send traffic to it wastes resources and can trigger cascading failures across the entire system. The Circuit Breaker pattern detects these conditions and temporarily stops sending traffic, allowing the failing service time to recover.

Circuit Breaker State Machine

stateDiagram-v2 [*] --> Closed Closed --> Open: Failure threshold exceeded Open --> HalfOpen: Timeout expires HalfOpen --> Closed: Probe succeeds HalfOpen --> Open: Probe fails

C#
public class CircuitBreaker : ICircuitBreaker
{
    private readonly ConcurrentDictionary<string, CircuitState> _circuits = new();
    private readonly CircuitBreakerConfig _config;

    public CircuitBreaker(CircuitBreakerConfig config)
    {
        _config = config;
    }

    public async Task<CircuitBreakerResult> ExecuteAsync<T>(
        string serviceId, Func<Task<T>> action)
    {
        var state = _circuits.GetOrAdd(serviceId,
            _ => new CircuitState(CircuitStatus.Closed));

        switch (state.Status)
        {
            case CircuitStatus.Open:
                if (DateTime.UtcNow >
                    state.OpenedAt.Add(_config.OpenDuration))
                {
                    state.Status = CircuitStatus.HalfOpen;
                    state.ProbeCount = 0;
                    Logger.Info(
                        "Circuit {ServiceId} -> HalfOpen", serviceId);
                }
                else
                {
                    return CircuitBreakerResult.Rejected(
                        "Circuit is open",
                        state.OpenedAt.Add(_config.OpenDuration));
                }
                break;

            case CircuitStatus.HalfOpen:
                if (state.ProbeCount >= _config.ProbeCount)
                {
                    return CircuitBreakerResult.Rejected(
                        "Half-open, max probes reached",
                        state.OpenedAt.Add(_config.OpenDuration));
                }
                state.ProbeCount++;
                break;
        }

        try
        {
            var result = await action();
            OnSuccess(state, serviceId);
            return CircuitBreakerResult.Success(result);
        }
        catch (Exception ex)
        {
            OnFailure(state, serviceId, ex);
            throw;
        }
    }

    private void OnSuccess(CircuitState state, string serviceId)
    {
        lock (state.Lock)
        {
            state.FailureCount = 0;
            state.SuccessCount++;

            if (state.Status == CircuitStatus.HalfOpen &&
                state.SuccessCount >= _config.SuccessThreshold)
            {
                state.Status = CircuitStatus.Closed;
                Logger.Info(
                    "Circuit {ServiceId} closed", serviceId);
            }
        }
    }

    private void OnFailure(CircuitState state, string serviceId, Exception ex)
    {
        lock (state.Lock)
        {
            state.FailureCount++;
            state.LastFailure = DateTime.UtcNow;

            if (state.Status == CircuitStatus.Closed &&
                state.FailureCount >= _config.FailureThreshold)
            {
                state.Status = CircuitStatus.Open;
                state.OpenedAt = DateTime.UtcNow;
                Logger.Warn(
                    "Circuit {ServiceId} opened after {Count} failures",
                    serviceId, state.FailureCount);
            }
            else if (state.Status == CircuitStatus.HalfOpen)
            {
                state.Status = CircuitStatus.Open;
                state.OpenedAt = DateTime.UtcNow;
            }
        }
    }
}

Retry with Exponential Backoff and Jitter

C#
public class RetryPolicy
{
    private static readonly Random _random = new();

    public static async Task<T> ExecuteAsync<T>(
        Func<Task<T>> action,
        RetryConfig config,
        CancellationToken ct = default)
    {
        var attempt = 0;
        Exception lastException = null;

        while (attempt <= config.MaxRetries)
        {
            try
            {
                return await action();
            }
            catch (Exception ex) when (
                IsRetryable(ex) && attempt < config.MaxRetries)
            {
                lastException = ex;
                attempt++;

                var baseDelay = Math.Min(
                    config.InitialDelay.TotalMilliseconds *
                        Math.Pow(2, attempt - 1),
                    config.MaxDelay.TotalMilliseconds);

                var jitter = _random.NextDouble() * baseDelay;
                var delay = TimeSpan.FromMilliseconds(jitter);

                Logger.Debug(
                    "Retry {Attempt}/{Max} after {Delay}ms: {Error}",
                    attempt, config.MaxRetries,
                    delay.TotalMilliseconds, ex.Message);

                await Task.Delay(delay, ct);
            }
        }

        throw new RetryExhaustedException(
            $"All {config.MaxRetries} retries exhausted",
            lastException);
    }

    private static bool IsRetryable(Exception ex)
    {
        return ex is HttpRequestException ||
               ex is TimeoutException ||
               ex is TaskCanceledException ||
               ex is IOException;
    }
}

Critical Rule: Never retry non-idempotent operations (POST, PUT with side effects) without explicit client confirmation. Retrying a payment request or order creation can cause duplicate charges. The gateway must inspect the HTTP method and only retry on 502/503/504 for GET, HEAD, OPTIONS, and DELETE methods. For POST/PUT, the client should be responsible for idempotency keys.

12. Caching Layer & Response Cache

Caching at the gateway level provides significant latency reduction and backend load mitigation. By caching responses for frequently requested, relatively static resources, the gateway avoids round-trips to backend services entirely.

Cache Architecture

graph LR A[Client Request] --> B{Cache Hit?} B -->|Yes| C[Return Cached Response] B -->|No| D[Route to Backend] D --> E[Receive Response] E --> F{Cacheable?} F -->|Yes| G[Store in Cache] F -->|No| H[Return Response] G --> H

C#
public class GatewayCacheLayer : IPlugin
{
    private readonly IDistributedCache _cache;
    private readonly CacheConfig _defaultConfig;

    public async Task<PluginResult> ExecuteAsync(GatewayContext context)
    {
        var config = context.Route.CacheConfig ?? _defaultConfig;
        if (!config.Enabled) return PluginResult.Allowed();

        if (!HttpMethods.IsGet(context.Request.Method) &&
            !HttpMethods.IsHead(context.Request.Method))
            return PluginResult.Allowed();

        if (!config.CacheAuthenticatedRequests &&
            context.Request.Headers.ContainsKey("Authorization"))
            return PluginResult.Allowed();

        var cacheKey = BuildCacheKey(context.Request, config);

        var cached = await _cache.GetAsync<CachedResponse>(cacheKey);
        if (cached != null && !cached.IsExpired)
        {
            context.Response = cached.ToGatewayResponse();
            context.Response.Headers["X-Cache"] = "HIT";
            context.Response.Headers["X-Cache-Age"] =
                ((int)(DateTime.UtcNow - cached.CachedAt)
                    .TotalSeconds).ToString();
            context.SkipProxy = true;
            return PluginResult.Allowed();
        }

        context.Response.Headers["X-Cache"] = "MISS";
        context.CacheKey = cacheKey;
        context.CacheConfig = config;

        return PluginResult.Allowed();
    }

    public async Task OnResponseAsync(GatewayContext context,
        UpstreamResponse response)
    {
        if (context.CacheKey == null) return;

        var config = context.CacheConfig;
        if (!IsCacheable(response, config)) return;

        var cached = new CachedResponse
        {
            StatusCode = response.StatusCode,
            Headers = response.Headers
                .ToDictionary(h => h.Key, h => h.Value),
            Body = await response.ReadBodyAsBytesAsync(),
            CachedAt = DateTime.UtcNow,
            TtlSeconds = DetermineTtl(response, config)
        };

        await _cache.SetAsync(context.CacheKey, cached,
            TimeSpan.FromSeconds(cached.TtlSeconds));
    }

    private string BuildCacheKey(GatewayRequest request, CacheConfig config)
    {
        var parts = new List<string> { "gw_cache", request.Path };

        if (config.IncludeQueryString)
            parts.Add(request.QueryString);

        if (config.VaryByHeaders != null)
        {
            foreach (var header in config.VaryByHeaders)
            {
                if (request.Headers.TryGetValue(header, out var value))
                    parts.Add($"{header}:{value}");
            }
        }

        if (config.PerTenant)
            parts.Add(request.TenantId ?? "global");

        return string.Join(":", parts);
    }

    private bool IsCacheable(UpstreamResponse response, CacheConfig config)
    {
        if (response.StatusCode < 200 || response.StatusCode >= 300)
            return false;

        var cacheControl = response.Headers
            .GetValueOrDefault("Cache-Control");
        if (cacheControl?.Contains("no-store") == true) return false;
        if (response.Headers.ContainsKey("Set-Cookie")) return false;

        return true;
    }
}

Cache Layer	Latency	Capacity	Use Case
In-process Memory (L1)	< 1 us	~500 MB per node	Hot routes, configuration data
Redis Cluster (L2)	~0.5 ms	Tens of GB	Shared cache across gateway nodes
CDN Edge Cache	~5-20 ms	Terrabytes	Static responses, public APIs

13. API Analytics & Monitoring

Comprehensive observability is non-negotiable for an API Gateway. The gateway sees all traffic, making it the richest source of data about system behavior, client patterns, error trends, and performance characteristics. Without strong analytics, you cannot detect anomalies, debug issues, or make informed capacity decisions.

Metrics Collection

C#
public class MetricsCollector
{
    private readonly Meter _meter = new("api-gateway", "2.0");
    private readonly Counter<long> _requestsTotal;
    private readonly Histogram<double> _requestDuration;
    private readonly Counter<long> _requestsByStatus;
    private readonly Counter<long> _rateLimitRejections;
    private readonly UpDownCounter<long> _activeConnections;
    private readonly Histogram<double> _requestSize;
    private readonly Histogram<double> _responseSize;
    private readonly Counter<long> _circuitBreakerTrips;
    private readonly Counter<long> _cacheHits;
    private readonly Counter<long> _cacheMisses;

    public MetricsCollector()
    {
        _requestsTotal = _meter.CreateCounter<long>(
            "gateway.requests.total",
            description: "Total number of requests");
        _requestDuration = _meter.CreateHistogram<double>(
            "gateway.request.duration",
            unit: "ms",
            description: "Request duration in milliseconds");
        _requestsByStatus = _meter.CreateCounter<long>(
            "gateway.requests.status",
            description: "Requests by status code");
        _rateLimitRejections = _meter.CreateCounter<long>(
            "gateway.ratelimit.rejected",
            description: "Rate limit rejections");
        _activeConnections = _meter.CreateUpDownCounter<long>(
            "gateway.connections.active",
            description: "Currently active connections");
        _requestSize = _meter.CreateHistogram<double>(
            "gateway.request.size", unit: "bytes");
        _responseSize = _meter.CreateHistogram<double>(
            "gateway.response.size", unit: "bytes");
        _circuitBreakerTrips = _meter.CreateCounter<long>(
            "gateway.circuit_breaker.trips");
        _cacheHits = _meter.CreateCounter<long>(
            "gateway.cache.hits");
        _cacheMisses = _meter.CreateCounter<long>(
            "gateway.cache.misses");
    }

    public void RecordRequest(GatewayContext context,
        UpstreamResponse response, double durationMs)
    {
        var tags = new TagList
        {
            { "method", context.Request.Method },
            { "route", context.Route?.Name ?? "unknown" },
            { "status", response.StatusCode.ToString() },
            { "tenant", context.Identity?.TenantId ?? "anon" },
            { "upstream", context.Target?.ServiceId ?? "unknown" }
        };

        _requestsTotal.Add(1, tags);
        _requestDuration.Record(durationMs, tags);
        _requestsByStatus.Add(1, new TagList
        {
            { "status", response.StatusCode.ToString() }
        });
        _requestSize.Record(context.Request.ContentLength, tags);
        _responseSize.Record(response.ContentLength, tags);

        if (context.Response.Headers.TryGetValue("X-Cache", out var c)
            && c == "HIT")
            _cacheHits.Add(1);
        else
            _cacheMisses.Add(1);
    }

    public void RecordRateLimitRejection(string key, string route)
    {
        _rateLimitRejections.Add(1, new TagList
        {
            { "key", key }, { "route", route }
        });
    }
}

Key Dashboards and Alerts

• Traffic Dashboard: Request rate by route, method, tenant, status code. Geographic distribution. Peak vs. average ratio.
• Latency Dashboard: P50, P95, P99, P99.9 latency by route and upstream service. Latency breakdown (gateway processing vs. upstream processing).
• Error Dashboard: 4xx and 5xx rates. Error rate by route and upstream. Circuit breaker state transitions.
• Rate Limiting Dashboard: Rejection rate by tenant and route. Top throttled clients. Limit utilization (% of limit consumed).
• Capacity Dashboard: Active connections, connection pool utilization, CPU/memory usage per gateway node, request queue depth.

Distributed Tracing

C#
public class TracingMiddleware
{
    private readonly ActivitySource _activitySource =
        new("api-gateway");

    public async Task<GatewayResponse> ProcessAsync(
        GatewayContext context,
        Func<Task<GatewayResponse>> next)
    {
        using var activity = _activitySource.StartActivity(
            $"{context.Request.Method} {context.Request.Path}",
            ActivityKind.Server);

        if (ActivityContext.TryParse(
            context.Request.Headers
                .GetValueOrDefault("traceparent"),
            null, out var parentContext))
        {
            activity = _activitySource.StartActivity(
                $"{context.Request.Method} {context.Request.Path}",
                ActivityKind.Server, parentContext);
        }

        activity?.SetTag("http.method", context.Request.Method);
        activity?.SetTag("http.url", context.Request.Uri);
        activity?.SetTag("http.host", context.Request.Host);
        activity?.SetTag("tenant.id", context.Identity?.TenantId);

        context.Activity = activity;
        context.TraceId = activity?.TraceId.ToString();

        var response = await next();
        if (activity != null)
            response.Headers["X-Trace-ID"] =
                activity.TraceId.ToString();

        activity?.SetTag("http.status_code", response.StatusCode);
        activity?.SetStatus(response.StatusCode >= 400
            ? ActivityStatusCode.Error
            : ActivityStatusCode.Ok);

        return response;
    }
}

14. Developer Portal & Documentation

A great API Gateway is only as useful as its adoptability. The developer portal is the interface through which internal and external developers discover, understand, test, and integrate with APIs. It should provide auto-generated documentation, interactive testing, SDK generation, and onboarding workflows.

OpenAPI-First Documentation

The gateway can auto-generate OpenAPI specifications from route configurations and request/response schemas observed in production traffic. This ensures documentation stays synchronized with actual API behavior.

C#
public class OpenApiGenerator
{
    public OpenApiDocument GenerateFromRoutes(
        IReadOnlyList<GatewayRoute> routes)
    {
        var doc = new OpenApiDocument
        {
            Info = new OpenApiInfo
            {
                Title = "API Gateway",
                Version = "1.0.0",
                Description = "Auto-generated from gateway config"
            },
            Servers = new List<OpenApiServer>
            {
                new() { Url = "https://api.example.com" }
            }
        };

        doc.Components ??= new OpenApiComponents();
        doc.Components.SecuritySchemes["BearerAuth"] =
            new OpenApiSecurityScheme
            {
                Type = SecuritySchemeType.Http,
                Scheme = "bearer",
                BearerFormat = "JWT",
                Description = "JWT token from auth server"
            };

        foreach (var route in routes.Where(r => r.Enabled))
        {
            foreach (var match in route.Matches)
            {
                var pathItem = doc.Paths
                    .GetValueOrDefault(match.PathPattern)
                    ?? new OpenApiPathItem();

                var operation = new OpenApiOperation
                {
                    OperationId =
                        $"{route.Name}_{match.Method.ToLower()}",
                    Summary = route.Description,
                    Tags = new List<string>
                        { route.Group ?? "default" },
                    Security = new List<
                        IDictionary<string, IEnumerable<string>>>
                    {
                        new Dictionary<string, IEnumerable<string>>
                        {
                            ["BearerAuth"] = Array.Empty<string>()
                        }
                    }
                };

                pathItem.Operations[match.Method.ToUpper()] =
                    operation;
                doc.Paths[match.PathPattern] = pathItem;
            }
        }

        return doc;
    }
}

Portal Features Matrix

Feature	Description	Priority
API Explorer	Interactive Swagger UI / Redoc for trying APIs directly	P0 — Must Have
Authentication Guide	Step-by-step OAuth/JWT setup with code samples	P0 — Must Have
API Key Management	Self-service key creation, rotation, revocation	P0 — Must Have
Usage Dashboard	Real-time request counts, error rates, latency per API key	P1 — Should Have
Rate Limit Visibility	Show current usage vs. limits per endpoint	P1 — Should Have
SDK Generation	Auto-generate client SDKs from OpenAPI spec	P1 — Should Have
Webhook Management	Configure and test webhook endpoints	P2 — Nice to Have
Changelog	API version changelog and migration guides	P1 — Should Have
Issue Reporting	Report API issues with trace ID context	P2 — Nice to Have

15. Plugin / Extension System

The plugin system is what transforms an API Gateway from a simple reverse proxy into a programmable platform. By allowing custom logic to be injected at various points in the request lifecycle, the gateway becomes extensible without requiring source code modifications.

Plugin Interface and Lifecycle

C#
public interface IPlugin
{
    string Name { get; }
    string Version { get; }
    PluginPhase Phase { get; }
    bool Matches(GatewayContext context);
    Task<PluginResult> ExecuteAsync(GatewayContext context);
}

public interface IRewritePlugin : IPlugin
{
    Task<GatewayResponse> RewriteAsync(GatewayContext context);
}

public interface IHeaderFilterPlugin : IPlugin
{
    Task<GatewayResponse> TransformHeadersAsync(
        GatewayContext context, UpstreamResponse response);
}

public interface IBodyFilterPlugin : IPlugin
{
    Task<Stream> TransformBodyAsync(
        GatewayContext context, Stream upstreamBody);
}

public interface ILogPlugin : IPlugin
{
    Task OnResponseAsync(
        GatewayContext context, UpstreamResponse response);
}

public class PluginPipeline
{
    private readonly Dictionary<PluginPhase, List<IPlugin>> _plugins;
    private readonly PluginLoader _loader;

    public PluginPipeline(PluginLoader loader)
    {
        _loader = loader;
        _plugins = Enum.GetValues<PluginPhase>()
            .ToDictionary(p => p, _ => new List<IPlugin>());
    }

    public async Task LoadPluginsAsync(
        IEnumerable<PluginConfig> configs)
    {
        foreach (var config in configs.Where(c => c.Enabled))
        {
            try
            {
                var plugin = await _loader.LoadAsync(
                    config.Name, config.Version);
                plugin.Configure(config.Config);
                _plugins[plugin.Phase].Add(plugin);

                Logger.Info(
                    "Loaded plugin {Name} v{Version} in {Phase}",
                    plugin.Name, plugin.Version, plugin.Phase);
            }
            catch (Exception ex)
            {
                Logger.Error(ex,
                    "Failed to load plugin {Name}", config.Name);
            }
        }
    }

    public IReadOnlyList<IPlugin> GetPlugins(PluginPhase phase)
    {
        return _plugins[phase].AsReadOnly();
    }
}

public class PluginLoader
{
    private readonly string _pluginDirectory;

    public async Task<IPlugin> LoadAsync(
        string pluginName, string version)
    {
        var pluginPath = Path.Combine(
            _pluginDirectory, pluginName, version,
            $"{pluginName}.dll");

        if (File.Exists(pluginPath))
        {
            var assembly = Assembly.LoadFrom(pluginPath);
            var pluginType = assembly.GetTypes()
                .FirstOrDefault(t =>
                    typeof(IPlugin).IsAssignableFrom(t) &&
                    !t.IsAbstract);

            if (pluginType != null)
                return (IPlugin)Activator.CreateInstance(pluginType);
        }

        var pluginBytes =
            await FetchPluginFromRegistry(pluginName, version);
        var loaded = Assembly.Load(pluginBytes);
        var type = loaded.GetTypes()
            .First(t =>
                typeof(IPlugin).IsAssignableFrom(t) &&
                !t.IsAbstract);

        return (IPlugin)Activator.CreateInstance(type);
    }
}

Built-in Plugin Catalog

Plugin	Phase	Description
jwt-auth	Access	Validate JWT tokens and extract identity
key-auth	Access	Validate API keys from header or query param
rate-limiter	Access	Enforce rate limits using configurable algorithms
cors	Access / Header Filter	Handle CORS preflight and response headers
request-transformer	Rewrite	Add/remove/rename headers, path rewrite
response-transformer	Header Filter	Modify response headers and body
ip-restriction	Access	Allow/deny requests by IP CIDR ranges
bot-detection	Access	Detect and block automated traffic
request-logger	Log	Log request/response to file, Kafka, or HTTP
prometheus-exporter	Log	Expose metrics in Prometheus format
oauth-introspect	Access	Validate tokens via OAuth 2.0 introspection
request-size-limiter	Access	Reject requests exceeding size limits
file-cache	Access / Log	Cache responses in Redis or local memory
grpc-web	Access	Convert gRPC-Web to gRPC for browser clients

Hot Reloading: Plugins must support hot reloading. When a plugin configuration is updated via the admin API, the gateway should seamlessly transition to the new configuration without dropping in-flight requests. This is achieved using a copy-on-write pattern where new configuration is prepared atomically and atomically swapped.

16. WebSocket & gRPC Support

Modern applications increasingly use protocols beyond traditional HTTP/1.1 request-response. WebSockets enable real-time bidirectional communication, while gRPC provides high-performance binary RPC with strong typing. A production API Gateway must support both protocols natively.

WebSocket Proxying

C#
public class WebSocketProxy : IProtocolHandler
{
    public bool CanHandle(HttpRequest request)
    {
        return request.Headers.TryGetValue("Upgrade",
            out var upgrade) &&
            upgrade.Equals("websocket",
                StringComparison.OrdinalIgnoreCase);
    }

    public async Task HandleAsync(HttpContext context,
        ServiceTarget target)
    {
        var upstreamUri = new Uri(
            $"ws://{target.Host}:{target.Port}{context.Request.Path}");

        if (!ValidateOrigin(context, target))
        {
            context.Response.StatusCode = 403;
            return;
        }

        var upstreamSocket = new ClientWebSocket();
        foreach (var header in GetFilteredHeaders(
            context.Request.Headers))
        {
            upstreamSocket.Options.SetRequestHeader(
                header.Key, header.Value);
        }

        try
        {
            await upstreamSocket.ConnectAsync(upstreamUri,
                context.RequestAborted);

            var downstreamSocket =
                await context.WebSockets.AcceptWebSocketAsync();

            var upTask = ProxyStreamAsync(
                downstreamSocket, upstreamSocket,
                context.RequestAborted);
            var downTask = ProxyStreamAsync(
                upstreamSocket, downstreamSocket,
                context.RequestAborted);

            await Task.WhenAny(upTask, downTask);

            if (downstreamSocket.State == WebSocketState.Open)
                await downstreamSocket.CloseAsync(
                    WebSocketCloseStatus.NormalClosure,
                    "", CancellationToken.None);
            if (upstreamSocket.State == WebSocketState.Open)
                await upstreamSocket.CloseAsync(
                    WebSocketCloseStatus.NormalClosure,
                    "", CancellationToken.None);
        }
        finally
        {
            upstreamSocket.Dispose();
        }
    }

    private async Task ProxyStreamAsync(WebSocket source,
        WebSocket target, CancellationToken ct)
    {
        var buffer = new byte[8192];
        while (!ct.IsCancellationRequested)
        {
            var result = await source.ReceiveAsync(buffer, ct);
            if (result.MessageType ==
                WebSocketMessageType.Close) break;

            await target.SendAsync(
                buffer.AsMemory(0, result.Count),
                result.MessageType,
                result.EndOfMessage,
                ct);
        }
    }
}

gRPC Proxying

gRPC proxying is more complex than HTTP because it uses HTTP/2 with binary framing. The gateway must handle stream multiplexing, flow control, and trailer propagation.

C#
public class GrpcProxyHandler : IProtocolHandler
{
    private readonly HttpMessageInvoker _invoker;

    public GrpcProxyHandler()
    {
        _invoker = new HttpMessageInvoker(new SocketsHttpHandler
        {
            ConnectTimeout = TimeSpan.FromSeconds(10),
            KeepAlivePingDelay = TimeSpan.FromSeconds(30),
            KeepAlivePingTimeout = TimeSpan.FromSeconds(60),
            EnableMultipleHttp2Connections = true
        });
    }

    public bool CanHandle(HttpRequest request)
    {
        return request.ContentType == "application/grpc" ||
            request.ContentType?.StartsWith(
                "application/grpc+") == true;
    }

    public async Task HandleAsync(HttpContext context,
        ServiceTarget target)
    {
        var upstreamUri =
            $"{target.Url}{context.Request.Path}";

        var upstreamRequest = new HttpRequestMessage
        {
            Method = HttpMethod.Post,
            RequestUri = new Uri(upstreamUri),
            Version = new Version(2, 0),
            Content = new StreamContent(context.Request.Body)
        };

        CopyGrpcHeaders(context.Request.Headers,
            upstreamRequest.Headers);

        var upstreamResponse = await _invoker.SendAsync(
            upstreamRequest, context.RequestAborted);

        context.Response.StatusCode =
            (int)upstreamResponse.StatusCode;

        foreach (var header in upstreamResponse.Headers)
        {
            context.Response.Headers[header.Key] =
                header.Value.ToArray();
        }

        var upstreamStream =
            await upstreamResponse.Content.ReadAsStreamAsync();
        await upstreamStream.CopyToAsync(
            context.Response.Body, context.RequestAborted);

        if (upstreamResponse.TrailingHeaders != null)
        {
            foreach (var trailer in upstreamResponse.TrailingHeaders)
            {
                context.Response.Headers.Append(
                    trailer.Key, trailer.Value.ToArray());
            }
        }
    }
}

gRPC Streaming: gRPC supports four communication patterns: Unary, Server Streaming, Client Streaming, and Bidirectional Streaming. The gateway must correctly proxy all four, ensuring that flow control signals (HTTP/2 WINDOW_UPDATE frames) are properly propagated and that half-closed connections are handled correctly.

17. Multi-Tenancy & Workspace Isolation

Enterprise API Gateways serve multiple teams, products, and sometimes external customers from a shared infrastructure. Multi-tenancy ensures that each tenant's configuration, traffic, rate limits, and analytics are completely isolated.

Tenancy Architecture

graph TB subgraph Control Plane A[Admin API] B[Config Store PostgreSQL] end subgraph Tenant Workspaces C[Tenant A] D[Tenant B] E[Tenant C] end subgraph Data Isolation F[Route Configs per Tenant] G[Rate Limits per Tenant] H[API Keys per Tenant] I[Analytics per Tenant] end A --> B B --> C B --> D B --> E

C#
public class TenantResolver
{
    private readonly TenantConfigStore _configStore;
    private readonly ConcurrentDictionary<string, TenantContext>
        _tenantCache = new();

    public async Task<TenantContext> ResolveAsync(
        GatewayRequest request)
    {
        var tenantId = ExtractTenantId(request);
        if (tenantId == null)
            return TenantContext.Anonymous;

        return await _tenantCache.GetOrAddAsync(tenantId,
            async _ => await LoadTenantContext(tenantId));
    }

    private string ExtractTenantId(GatewayRequest request)
    {
        if (request.Headers.TryGetValue("X-Tenant-ID",
            out var headerTenant))
            return headerTenant;

        if (request.Headers.TryGetValue("X-API-Key",
            out var apiKey) ||
            request.QueryParams.TryGetValue("api_key", out apiKey))
        {
            return ResolveTenantFromApiKey(apiKey);
        }

        if (request.Headers.TryGetValue("Authorization",
            out var auth) && auth.StartsWith("Bearer "))
        {
            return ExtractTenantFromJwt(
                auth["Bearer ".Length..]);
        }

        var host = request.Host;
        var match = Regex.Match(host,
            @"^(?<tenant>[a-z0-9-]+)\.api\.example\.com$");
        if (match.Success) return match.Groups["tenant"].Value;

        return null;
    }

    private async Task<TenantContext> LoadTenantContext(
        string tenantId)
    {
        var config =
            await _configStore.GetTenantConfigAsync(tenantId);

        return new TenantContext
        {
            TenantId = tenantId,
            Name = config.Name,
            Plan = config.Plan,
            RouteFilters = config.RouteFilters,
            RateLimits = config.RateLimits,
            Quotas = config.Quotas,
            CustomDomains = config.CustomDomains,
            SSLCertificates = config.SslCertificates,
            Features = config.EnabledFeatures,
            CreatedAt = config.CreatedAt,
            Metadata = config.Metadata
        };
    }
}

Resource Isolation by Plan

Resource	Free	Starter	Pro	Enterprise
Routes	5	25	100	Unlimited
Rate Limit (req/min)	60	600	6,000	Custom
API Keys	2	10	50	Unlimited
Plugins	3	8	All	All + Custom
Analytics Retention	7 days	30 days	90 days	365 days
Support	Community	Email	Priority	Dedicated
SLA	None	99.5%	99.9%	99.99%

18. Security — DDoS, Injection & Threat Mitigation

As the outermost layer of the application stack, the API Gateway is the primary defense against a wide range of attacks. Its security responsibilities include DDoS mitigation, bot detection, injection prevention, data exfiltration controls, and compliance enforcement.

DDoS Protection Layers

graph TB subgraph Layer 1 - Network A[BGP Anycast Traffic Distribution] B[Volume Attack Absorption] end subgraph Layer 2 - Edge C[WAF Rules] D[Bot Detection] E[Geo-blocking] end subgraph Layer 3 - Gateway F[Connection Limits per IP] G[Request Size Limits] H[Slow Loris Protection] I[Header Validation] end subgraph Layer 4 - Application J[Authentication JWT/API Key] K[Rate Limiting Per Tenant] L[Input Validation] end A --> B --> C --> D --> E --> F --> G --> H --> I --> J --> K --> L

C#
public class SecurityPlugin : IAccessPlugin
{
    private readonly SecurityConfig _config;
    private readonly ConcurrentDictionary<string, IpBlockEntry>
        _blockedIps = new();

    private readonly Regex _sqlInjectionPattern = new(
        @"(\b(SELECT|INSERT|UPDATE|DELETE|DROP|UNION|ALTER|CREATE|EXEC)\b)|" +
        @"(--)|(\b(OR|AND)\b\s+\w+\s*=\s*\w+)|" +
        @"('|"")\s*(OR|AND)\s*('|"")",
        RegexOptions.IgnoreCase | RegexOptions.Compiled);

    private readonly Regex _xssPattern = new(
        @"<script[^>]*>|javascript:|on\w+\s*=|<iframe|<object|<embed",
        RegexOptions.IgnoreCase | RegexOptions.Compiled);

    public async Task<PluginResult> ExecuteAsync(
        GatewayContext context)
    {
        var request = context.Request;

        if (IsBlocked(request.ClientIp))
            return PluginResult.Rejected(403, "Access denied");

        if (!CheckConnectionRate(request.ClientIp))
        {
            await AutoBlockAsync(request.ClientIp,
                TimeSpan.FromMinutes(5));
            return PluginResult.Rejected(429,
                "Too many connections");
        }

        if (request.ContentLength >
            _config.MaxRequestSizeBytes)
        {
            return PluginResult.Rejected(413,
                "Request entity too large");
        }

        if (request.Headers.Count > _config.MaxHeaders)
            return PluginResult.Rejected(431, "Too many headers");

        foreach (var (name, value) in request.Headers)
        {
            if (name.Length > _config.MaxHeaderNameLength ||
                value.Length > _config.MaxHeaderValueLength)
            {
                return PluginResult.Rejected(431,
                    "Header too large");
            }

            if (value.Contains('\r') || value.Contains('\n'))
            {
                return PluginResult.Rejected(400,
                    "Invalid header value");
            }
        }

        if (request.Path.Contains("..") ||
            request.Path.Contains("%2e%2e"))
        {
            return PluginResult.Rejected(400, "Invalid path");
        }

        var queryString = request.QueryString;
        if (_sqlInjectionPattern.IsMatch(queryString))
        {
            Logger.Warn(
                "SQL injection from {IP}: {Path}{Query}",
                request.ClientIp, request.Path, queryString);
            return PluginResult.Rejected(400, "Invalid request");
        }

        if (_xssPattern.IsMatch(queryString))
        {
            Logger.Warn(
                "XSS attempt from {IP}: {Path}{Query}",
                request.ClientIp, request.Path, queryString);
            return PluginResult.Rejected(400, "Invalid request");
        }

        if ((request.Method == "POST" ||
             request.Method == "PUT") &&
            request.ContentType != null &&
            !_config.AllowedContentTypes.Contains(
                request.ContentType))
        {
            return PluginResult.Rejected(415,
                "Unsupported media type");
        }

        return PluginResult.Allowed();
    }

    private bool IsBlocked(string ip)
    {
        if (_blockedIps.TryGetValue(ip, out var entry))
        {
            if (entry.ExpiresAt > DateTime.UtcNow) return true;
            _blockedIps.TryRemove(ip, out _);
        }
        return false;
    }

    private async Task AutoBlockAsync(string ip, TimeSpan duration)
    {
        _blockedIps[ip] = new IpBlockEntry
        {
            Ip = ip,
            BlockedAt = DateTime.UtcNow,
            ExpiresAt = DateTime.UtcNow.Add(duration),
            Reason = "Auto-blocked: rate exceeded"
        };
    }
}

Security Checklist

• TLS Everywhere: Enforce TLS 1.2+ for all client connections. Use TLS 1.3 when possible. Support SNI for multi-tenant SSL.
• Request Validation: Validate Content-Type, content length, header count, and header sizes to prevent resource exhaustion attacks.
• Input Sanitization: Detect and block SQL injection, XSS, path traversal, and command injection in query parameters, headers, and bodies.
• Response Headers: Strip server version headers, add security headers (HSTS, X-Content-Type-Options, X-Frame-Options, CSP).
• Secrets Management: Store API keys, JWT signing keys, and TLS certificates in a secrets vault. Never log secrets.
• Audit Logging: Record all authentication failures, rate limit violations, and security policy triggers with full request context.

Never Trust the Client: All security validations at the gateway are defense-in-depth. Backend services must independently validate inputs and enforce authorization. A compromised or misconfigured gateway should never grant unauthorized access to backend services.

19. Reliability & Failure Modes

As a critical piece of infrastructure sitting in front of all traffic, the API Gateway must be designed for maximum reliability. We must analyze every possible failure mode and design the system to degrade gracefully rather than fail catastrophically.

Failure Mode Analysis

Failure	Impact	Detection	Mitigation
Gateway node crash	Temporary capacity reduction	Health check failures	Auto-replacement, LB removes from pool
Redis failure	Rate limiting disabled	Redis health checks	Fail open, local fallback cache
Config DB failure	Cannot update routes	DB connection timeouts	Continue with last-known-good config
Backend service down	503 for affected routes	Circuit breaker, health checks	Circuit breaker opens, cached responses
Network partition	Partial connectivity	Connection failures	Retry with backoff, alternate paths
Certificate expiry	TLS handshake failures	Certificate monitoring	Auto-renewal via Let's Encrypt
DNS resolution failure	Cannot reach backends	DNS query failures	Local DNS cache, multiple DNS servers
CPU saturation	Request queuing, timeouts	CPU metrics, latency spike	Auto-scaling, load shedding
Memory exhaustion	OOM kill, process crash	Memory metrics	Connection limits, memory pools
Upstream timeout storms	Connection pool exhaustion	Timeout metrics	Circuit breaker, adaptive timeouts

Health Check System

C#
public class HealthCheckService : BackgroundService
{
    private readonly ServiceRegistry _registry;
    private readonly HealthCheckConfig _config;
    private readonly ConcurrentDictionary<string, ServiceHealth>
        _healthStates = new();

    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            var services = _registry.GetAllServices();
            var tasks = services.Select(CheckHealthAsync);
            await Task.WhenAll(tasks);
            await Task.Delay(_config.CheckInterval, stoppingToken);
        }
    }

    private async Task CheckHealthAsync(ServiceInfo service)
    {
        var sw = Stopwatch.StartNew();
        try
        {
            using var client = new HttpClient
            {
                Timeout = _config.Timeout
            };

            var response = await client.GetAsync(
                $"{service.BaseUrl}{_config.HealthPath}");

            sw.Stop();

            var healthy = response.IsSuccessStatusCode &&
                sw.ElapsedMilliseconds <
                    _config.LatencyThresholdMs;

            UpdateHealth(service.Id, new ServiceHealth
            {
                ServiceId = service.Id,
                Healthy = healthy,
                StatusCode = (int)response.StatusCode,
                LatencyMs = sw.ElapsedMilliseconds,
                CheckedAt = DateTime.UtcNow,
                ConsecutiveFailures = healthy ? 0 :
                    (_healthStates
                        .GetValueOrDefault(service.Id)
                        ?.ConsecutiveFailures ?? 0) + 1,
                Message = healthy ? "OK" :
                    $"Status: {response.StatusCode}"
            });
        }
        catch (Exception ex)
        {
            sw.Stop();
            UpdateHealth(service.Id, new ServiceHealth
            {
                ServiceId = service.Id,
                Healthy = false,
                LatencyMs = sw.ElapsedMilliseconds,
                CheckedAt = DateTime.UtcNow,
                ConsecutiveFailures =
                    (_healthStates
                        .GetValueOrDefault(service.Id)
                        ?.ConsecutiveFailures ?? 0) + 1,
                Message = ex.Message
            });
        }
    }

    private void UpdateHealth(string serviceId, ServiceHealth health)
    {
        _healthStates[serviceId] = health;

        if (!health.Healthy &&
            health.ConsecutiveFailures ==
                _config.FailureThreshold)
        {
            AlertService.SendAlertAsync(new Alert
            {
                Severity = AlertSeverity.Warning,
                Title = $"Service {serviceId} unhealthy",
                Details = $"Failed {health.ConsecutiveFailures}" +
                    $" health checks. Error: {health.Message}",
                Timestamp = DateTime.UtcNow
            });
        }
    }

    public bool IsHealthy(string serviceId)
    {
        return _healthStates.GetValueOrDefault(serviceId)
            ?.Healthy ?? false;
    }
}

Graceful Degradation Strategies

• Load Shedding: When CPU exceeds 80%, start rejecting lowest-priority traffic. Use the AdaptiveConcurrency algorithm to dynamically adjust the number of concurrent requests sent to each backend based on observed latency.
• Stale Cache Serving: When a backend is unavailable, serve stale cached responses with a Warning: 199 - Stale Response header. This is better than a 503 for many use cases.
• Default Responses: For non-critical features (analytics, recommendations), return default/stub responses when the backend is down rather than failing the entire request.
• Priority Queuing: Classify requests by priority (critical = authentication, high = data reads, normal = standard writes, low = background tasks). During overload, shed low-priority traffic first.

Netflix Pattern — Adaptive Concurrency: Netflix's gateway (Zuul) uses an adaptive concurrency limiter that dynamically adjusts the maximum number of in-flight requests per route based on observed latency. When a backend starts responding slowly, the limiter automatically reduces the concurrency, preventing the gateway from overwhelming the struggling service while maintaining optimal throughput for healthy services.

20. Cost Estimation & Infrastructure Sizing

Understanding the infrastructure cost of an API Gateway deployment is essential for capacity planning and budget justification. Let us estimate costs for a production deployment serving 500 million requests per day.

Infrastructure Sizing

Component	Spec	Count	Monthly Cost (Est.)
Gateway Nodes (Data Plane)	8 vCPU, 16 GB RAM, 10 Gbps NIC	6	$2,400 (6 x $400)
Redis Cluster (Rate Limiting/Cache)	3-node cluster, 8 GB RAM each	3	$900 (3 x $300)
PostgreSQL (Config Store)	4 vCPU, 16 GB RAM, 500 GB SSD	2 (primary + replica)	$600
ClickHouse (Analytics)	8 vCPU, 32 GB RAM, 2 TB NVMe	3	$3,600 (3 x $1,200)
Prometheus + Grafana	4 vCPU, 8 GB RAM, 500 GB SSD	1	$200
Control Plane (API Server)	4 vCPU, 8 GB RAM	2	$400 (2 x $200)
Load Balancer (Cloud)	Application LB with TLS	2	$300
Bandwidth	~2 TB egress/month	—	$170 ($0.085/GB)
Total			~$8,570/month

Cost Optimization Strategies

• Right-Sizing: Monitor actual CPU and memory utilization. Most gateway nodes operate at 20-40% capacity, so starting with smaller instances and scaling up based on actual usage can save 40-60%.
• Spot/Preemptible Instances: Gateway data plane nodes are stateless and can be replaced quickly. Using spot instances for 60-70% of the fleet can reduce compute costs by 50-70%.
• Compression: Enable gzip/brotli compression for API responses. For JSON-heavy APIs, this typically achieves 70-80% reduction in response size, directly reducing bandwidth costs.
• Cache Hit Ratio: A 60% cache hit ratio means 60% fewer backend requests. Improving cache hit ratio from 40% to 70% can save the equivalent of 30% of backend infrastructure.
• Log Retention Tiering: Keep 7 days of logs in hot storage (SSD), 30 days in warm storage (HDD/S3 Standard), and 90 days in cold storage (S3 Glacier). This can reduce analytics storage costs by 60-80%.

Cost Per Request

Unit Economics: At 500M requests/day and ~$8,570/month in infrastructure, the cost per API request is approximately $0.0000057 (0.00057 cents). This makes the API Gateway one of the most cost-effective components in the stack. However, this excludes bandwidth costs for large payloads and the cost of backend services the gateway routes to.

21. Interview Q&A Deep Dive

This section covers the most frequently asked questions about API Gateway design in staff-level system design interviews, along with detailed answers that demonstrate depth of understanding.

Q: How would you handle a sudden 10x traffic spike?

Answer: The response has three layers: Immediate (seconds): Enable load shedding at the gateway — reject lowest-priority traffic, enforce rate limits aggressively, and serve cached responses for cacheable endpoints. Short-term (minutes): Auto-scale the gateway fleet horizontally (the data plane is stateless, so new nodes can join in under 60 seconds). Scale Redis cluster if rate limiting becomes the bottleneck. Long-term (hours): Backend services auto-scale behind the gateway. Enable circuit breakers for non-essential services to shed load. Communicate with clients via status page.

Q: How do you ensure the gateway itself is not a single point of failure?

Answer: Deploy gateway nodes across multiple availability zones behind a global load balancer (DNS-based or Anycast). Each node is stateless — it holds no persistent state, only ephemeral connection pools and caches that are rebuilt from Redis/config store. The control plane (config API, database) also runs multi-AZ with automatic failover. Health checks continuously monitor gateway nodes, and unhealthy nodes are automatically removed from the load balancer pool. The gateway fleet should be sized for N-1 redundancy — able to handle full traffic even if one AZ goes down.

Q: How would you design rate limiting that works across multiple gateway nodes?

Answer: Use Redis as the centralized counter store with atomic Lua scripts for check-and-increment operations. The Sliding Window Counter algorithm provides the best accuracy-to-performance ratio. Each gateway node executes a Redis EVAL command that atomically reads the previous window count, reads the current window count, calculates the weighted estimate, and increments the counter — all in a single round trip. For ultra-low-latency requirements, use a two-tier approach: local in-memory counters for coarse limiting (e.g., per-second) and Redis for fine-grained per-minute limits. If Redis is unavailable, fail open to avoid blocking legitimate traffic.

Q: How do you handle configuration changes without downtime?

Answer: Use a publish-subscribe model with a change-data-capture pipeline. Configuration changes are written to PostgreSQL (source of truth). A CDC connector (Debezium) detects changes and publishes events to a Kafka topic. Each gateway node subscribes to this topic and atomically swaps its in-memory configuration when changes arrive. The swap uses a copy-on-write pattern: prepare the new configuration, validate it, then atomically replace the reference. In-flight requests continue using the old configuration until they complete. This gives sub-second propagation with zero downtime.

Q: How would you handle the N+1 query problem in the plugin pipeline?

Answer: The N+1 problem in a gateway context manifests when multiple plugins each make independent external calls (e.g., JWT validation hitting an introspection endpoint, rate limiter hitting Redis, analytics hitting a logging service). We solve this with: (1) Parallel execution — plugins in the same phase that don't depend on each other run concurrently using Task.WhenAll. (2) Batching — multiple Redis operations from different plugins are batched into a single pipeline/transaction. (3) Connection pooling — long-lived connections to Redis, databases, and upstream services are reused across requests. (4) Caching — frequently accessed data (JWKS keys, route configs) is cached locally with appropriate TTLs.

Q: Compare API Gateway vs. Service Mesh. When would you use each?

Answer: An API Gateway handles north-south traffic (client-to-service) with centralized policy enforcement: authentication, rate limiting, request transformation, and analytics. A Service Mesh (Istio, Linkerd) handles east-west traffic (service-to-service) with per-proxy sidecars providing mutual TLS, load balancing, retries, and observability. Use a gateway when: you need to enforce policies on external traffic, transform requests for external consumers, or provide a unified API surface. Use a service mesh when: you need to secure and observe internal microservice communication. Use both when: you have a microservices architecture with both external and internal traffic that needs policy enforcement. The gateway handles the edge; the mesh handles the interior.

Q: How do you test a rate limiter for correctness under high concurrency?

Answer: Correctness testing for distributed rate limiters requires careful coordination: (1) Unit tests — verify each algorithm's behavior in isolation with deterministic time. (2) Concurrency tests — use a concurrent test harness with 100+ parallel tasks all hitting the same rate limit key simultaneously. Verify that the total allowed count never exceeds the configured limit. (3) Integration tests — deploy the rate limiter against a real Redis cluster and verify behavior under realistic network conditions. (4) Chaos tests — kill Redis nodes during rate limit checks and verify the fail-open behavior. (5) Boundary tests — test at exact window boundaries (second 59 to second 0) to verify sliding window counters don't allow burst double-counting.

Q: Design a rate limiter that supports different limits per geographic region.

Answer: Extend the rate limiter key to include a geographic component. The key format becomes: rl:{tenant_id}:{endpoint}:{region}:{window}. The region is determined by the client's IP address using a GeoIP lookup (MaxMind GeoLite2 database, cached locally). Each region can have different limit configurations stored in the tenant's rate limit config. The gateway node performs the GeoIP lookup once per request (cached for repeat clients) and constructs the appropriate rate limit key. Redis stores separate counters per region, allowing fine-grained control. For global rate limits, use a separate key without the region component and sum across all regional counters.

Q: How would you implement canary routing through the gateway?

Answer: Canary routing sends a small percentage of traffic to a new version of a service for testing. The gateway implements this through weighted routing: configure two targets for the same route — the stable version (95% weight) and the canary version (5% weight). The Power of Two Choices load balancer naturally distributes traffic based on these weights. For more sophisticated canary logic (e.g., routing based on specific headers, user segments, or geographic regions), use a canary plugin that inspects request attributes and overrides the default target selection. The plugin can read canary configuration from the control plane and update routing rules in real-time, allowing operators to gradually increase canary traffic from 1% to 100%.

Q: How do you prevent the gateway from becoming a bottleneck during response body transformation?

Answer: Response body transformation forces the gateway to buffer the entire response before forwarding, which destroys streaming capabilities and increases memory pressure. Strategies to mitigate: (1) Header-only transforms — apply the majority of transformations at the header level, avoiding body buffering entirely. (2) Streaming transforms — for body transforms that can be decomposed per-chunk (e.g., JSON streaming field filtering), use streaming pipelines that transform data as it flows through. (3) Selective transformation — only apply body transforms for specific routes/content-types where it is truly needed. (4) Offload to backend — for complex transformations, let the backend service handle it and let the gateway proxy transparently. (5) Zero-copy proxying — when no transformation is needed, use kernel-level zero-copy mechanisms (sendfile, splice) to forward data without user-space copies.

API Gateway System Design — Senior+ Guide