1. Introduction: Redis at Scale

Redis (Remote Dictionary Server) is an open-source, in-memory data structure store that has become one of the most critical building blocks in modern distributed systems. Originally created by Salvatore Sanfilippo in 2009, Redis has evolved from a simple key-value cache into a versatile, high-performance database capable of supporting a wide array of data structures including strings, lists, sets, sorted sets, hashes, streams, HyperLogLogs, geospatial indexes, and bitmaps. It routinely delivers over 100,000 read and write operations per second on commodity hardware, making it the go-to solution for latency-sensitive workloads that demand sub-millisecond response times.

The fundamental design philosophy behind Redis is simplicity married to extreme performance. By keeping all data in memory, Redis eliminates the disk I/O bottleneck that plagues traditional databases. The single-threaded execution model, while seemingly limiting, actually removes the overhead of locks, context switches, and thread synchronization, allowing Redis to process commands sequentially at blistering speeds. This architectural decision, combined with I/O multiplexing using epoll, kqueue, or IOCP, means that a single Redis instance can handle tens of thousands of concurrent connections without breaking a sweat.

Redis is deployed across virtually every major technology company on the planet. Twitter uses it for timelines and rate limiting. GitHub employs it for caching and job queues. Stack Overflow leverages it for real-time analytics. Snapchat uses it for ephemeral data storage. Netflix relies on it for session management and recommendation caching. Pinterest uses it for feed ranking. In each of these cases, Redis serves as the high-speed data layer that sits between the application and slower persistent storage, absorbing traffic spikes and reducing tail latency to levels that would be impossible with disk-based systems alone.

The versatility of Redis extends far beyond simple caching. It is used as a primary database for small datasets that require lightning-fast access, as a message broker through its Pub/Sub and Streams capabilities, as a distributed lock manager using Redlock, as a session store for web applications, as a real-time analytics engine using sorted sets and HyperLogLogs, as a geospatial database for location-based services, and as a rate limiter using sliding window algorithms. This extraordinary range of use cases, combined with a rich ecosystem of client libraries, modules, and tools, makes Redis an indispensable tool in the senior engineer's arsenal.

Why Redis Over Alternatives?

When compared to other in-memory solutions, Redis stands out for several reasons. Unlike Memcached, which only supports simple key-value strings, Redis provides rich data structures that eliminate the need for application-level serialization and deserialization of complex objects. Unlike KeyDB, which is a multithreaded fork of Redis, the original Redis benefits from a larger ecosystem and more battle-tested stability. Unlike Hazelcast or Apache Ignite, which require JVM deployment, Redis is written in C with minimal memory overhead and predictable garbage collection behavior. The combination of performance, data structure richness, persistence options, and operational maturity makes Redis the default choice for most in-memory data store requirements.

Redis in System Design Interviews

Redis is one of the most frequently discussed components in system design interviews. Interviewers expect candidates to understand when to use Redis, how to integrate it into a larger architecture, what trade-offs it introduces, and how to operate it reliably at scale. Being able to articulate the difference between using Redis as a cache versus a persistent store, understanding the CAP theorem implications of Redis Cluster, knowing how to prevent cache stampedes and thundering herds, and explaining the consistency guarantees of Redis replication are all topics that separate senior candidates from their peers. This guide equips you with the knowledge to handle all of these scenarios with confidence and precision.

The sections that follow provide deep technical coverage of every major Redis subsystem. We begin with the core architecture, then progress through data structures, memory management, persistence, replication, clustering, messaging, scripting, caching patterns, high availability, transactions, security, performance tuning, module ecosystem, and a comparative analysis with alternatives. Each section includes practical code examples, architectural diagrams, and operational best practices drawn from real-world production deployments. By the end of this guide, you will have a comprehensive understanding of Redis internals and the expertise to design, deploy, and operate Redis at any scale.

2. Core Architecture: Event Loop, Single Thread, and I/O Multiplexing

The architectural elegance of Redis lies in its simplicity. At its core, Redis operates on a single-threaded event loop that processes commands sequentially. This design eliminates the need for complex locking mechanisms, mutexes, and thread synchronization primitives that plague multithreaded systems. By processing one command at a time, Redis guarantees that there are no race conditions within the core command execution path, dramatically simplifying both the codebase and the mental model for reasoning about correctness.

The event loop is the heartbeat of Redis. It works by wrapping file descriptors (representing client connections, disk I/O operations, and inter-node communication) in a platform-specific I/O multiplexing mechanism. On Linux, Redis uses epoll; on macOS and BSD, it uses kqueue; and on Windows, it can use IOCP. The event loop iterates over all ready file descriptors, processes the associated events, executes the corresponding commands, and writes responses back to clients. This cycle repeats indefinitely, with the event loop spending most of its time sleeping in the I/O multiplexing call, waiting for new events to arrive.

When a client sends a command to Redis, the data first lands in an input buffer. The event loop detects that the client socket is readable, reads the command from the input buffer, parses it, executes it, and then queues the response in the output buffer. If the output buffer is full (for example, when sending a large result set), the event loop defers the write until the socket becomes writable again. This non-blocking I/O approach ensures that a single slow client cannot block the processing of commands for all other connected clients.

Threaded I/O for Network Operations

Starting with Redis 6.0, the project introduced a significant architectural evolution: threaded I/O for network operations. While the core command execution remains single-threaded, the parsing of client requests and the writing of responses can now be offloaded to background threads. This improvement is particularly impactful for workloads involving large payloads, such as fetching or storing values in the megabyte range. The threaded I/O feature is disabled by default and can be enabled by setting the io-threads configuration directive. In practice, setting io-threads to 4 or 8 (matching the number of CPU cores) can improve throughput by 1-2x for workloads dominated by large values, while having negligible impact on small-value workloads where the single-threaded model is already optimal.

The Redis Server Cron

Running alongside the event loop is a periodic task handler called the server cron. This function executes once per server tick (typically 10 times per second, controlled by the hz configuration directive) and is responsible for a wide range of housekeeping duties. These include scanning for expired keys using a probabilistic algorithm, updating statistics and metrics, triggering background persistence operations, performing replication heartbeats, rehashing dictionaries as needed, and managing cluster state. The server cron is designed to execute quickly and yield control back to the event loop frequently, ensuring that it does not introduce latency spikes for client command processing. Redis 7.0 introduced dynamic hertz adjustment, allowing the server to increase the cron frequency when idle and reduce it when under load, balancing responsiveness with CPU efficiency.

Internal Data Structures and Dictionaries

Redis maintains its own internal hash table implementation called a dictionary, which is used to store the mapping from keys to values. This dictionary is implemented as two hash tables (ht[0] and ht[1]) to facilitate incremental rehashing. When the hash table needs to grow (because the load factor exceeds a threshold), Redis allocates a larger hash table and gradually migrates entries from the old table to the new one over subsequent operations. This incremental rehashing approach avoids the latency spike that would result from rehashing millions of keys in a single operation. The hash function used is based on SipHash, which provides good distribution and resistance to hash flooding attacks. Each bucket in the hash table points to a linked list (or a more compact encoding for small hash tables) to handle hash collisions.

Command Dispatch Pipeline

When Redis receives a complete command (delimited by CRLF), it looks up the command in a command table that maps command names to their handler functions. Each handler function receives the client state, parses its arguments, executes the operation on the relevant data structure, and populates the reply. The command table is defined statically in server.c and includes metadata about each command such as its arity, flags (read-only, write, admin, pub-sub, etc.), and the positions of key arguments. This metadata enables Redis to make intelligent decisions about command routing in cluster mode, where it can determine which slot a command targets based on the key positions specified in the command definition. Understanding this dispatch pipeline is essential for writing custom Redis modules, as modules must register their commands with the same metadata structure.

Why Single-Threaded Is Fast Enough

A common question from engineers accustomed to multithreaded systems is why Redis uses a single thread at all. The answer lies in the nature of Redis's workload. Redis operates primarily on data that fits in memory, meaning there is no disk I/O during command execution. Memory access is fast (approximately 100 nanoseconds for L3 cache), and the single-threaded model eliminates all synchronization overhead including locks, condition variables, memory barriers, and context switches. A multithreaded system processing the same simple key-value operations would spend more time on synchronization than on actual computation. The single-threaded model also provides deterministic latency — every command takes roughly the same time to execute regardless of system load, which is a critical property for real-time systems. For workloads that truly benefit from parallelism (such as large value I/O or Lua script execution), Redis provides thread pools and the module API for offloading heavy computations.

3. Data Structures: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLog

One of Redis's greatest strengths is its rich set of native data types. Unlike Memcached, which provides only simple string key-value pairs, Redis supports eight distinct data structure types, each optimized for specific access patterns and use cases. Understanding these data structures deeply — including their internal encodings, memory overhead, and performance characteristics — is essential for designing efficient systems on top of Redis.

Strings

Strings are the most fundamental Redis data type. A Redis string can hold any binary data up to 512 MB in size. Internally, strings are implemented using a simple dynamic string (SDS) structure that stores the length of the string alongside the data, enabling O(1) length queries and preventing buffer overflow attacks. Strings are used not only for storing text and binary data but also as the building block for implementing counters (INCR/DECR), bitmaps (SETBIT/GETBIT), and simple caches. Strings can encode small integers as integer objects to save memory, and short strings (under 44 bytes) use an embedded encoding called embstr that avoids a separate heap allocation.

Lists

Redis lists are doubly-linked lists that support O(1) push and pop operations from both ends. They are commonly used to implement queues (LPUSH + RPOP) and stacks (LPUSH + LPOP). Internally, Redis optimizes list storage using two encodings: ziplist (for small lists with few elements and short values) and quicklist (a doubly-linked list of ziplists, which combines the memory efficiency of ziplists with the O(1) insertion performance of linked lists). Lists also support blocking operations (BLPOP, BRPOP) that allow clients to wait for elements to appear, making them useful for implementing simple message queues.

Sets

Redis sets are unordered collections of unique strings. They support O(1) membership testing, addition, removal, and intersection/union/difference operations. Internally, sets use either intset (for sets containing only integers) or hashtable encoding. Intset is a compact, sorted array of integers that uses binary search for O(log N) membership testing with minimal memory overhead. Set operations like SINTER, SUNION, and SDIFF are used extensively in social networking features such as finding mutual friends and building recommendation sets.

Sorted Sets (ZSets)

Sorted sets are one of Redis's most powerful data structures. Each element is associated with a floating-point score, and elements are maintained in ascending order by score. Sorted sets support O(log N) insertion, removal, and score updates, as well as O(log N + M) range queries. Internally, sorted sets use ziplist encoding for small sets and skiplist + hashtable encoding for larger sets. Sorted sets are used for leaderboards, priority queues, time-series indexing, rate limiting with sliding windows, and delayed task scheduling.

Hashes

Redis hashes are collections of field-value pairs, similar to dictionaries or maps in other programming languages. They are ideal for representing objects with multiple attributes, such as user profiles or product details. Internally, hashes use ziplist encoding for small hashes and hashtable encoding for larger ones. Using hashes instead of multiple string keys reduces memory overhead significantly because Redis only needs to allocate a single key for the entire object rather than one key per field.

Streams

Introduced in Redis 5.0, streams are an append-only log data structure that supports consumer groups for reliable message processing. Streams combine the durability of a persistent log with the performance of in-memory processing. Each stream entry has a unique ID composed of a timestamp and sequence number. Consumer groups allow multiple consumers to divide the work of processing stream entries, with automatic acknowledgment and pending entry tracking for fault tolerance.

HyperLogLog

HyperLogLog is a probabilistic data structure used for cardinality estimation — counting the number of unique elements in a set without storing the actual elements. A HyperLogLog in Redis uses only 12 KB of memory regardless of how many elements have been added, yet it provides an accuracy of approximately 0.81% standard error. This makes it ideal for counting unique visitors, unique search queries, or any scenario where approximate cardinality is acceptable.

Choosing the Right Data Structure

Choosing the appropriate Redis data structure for a given problem is a critical skill. For example, if you need to track the most recent 100 actions per user, a sorted set with timestamps as scores is far more efficient than maintaining a list and manually pruning it. If you need to count unique daily visitors, a HyperLogLog is orders of magnitude more memory-efficient than a set. If you need to implement a rate limiter, a sorted set with sliding window scoring provides precise control. Understanding the internal encodings also matters: storing a set of integers as an intset can save 10x or more memory compared to hashtable encoding, so being aware of the thresholds that trigger encoding changes helps you design memory-efficient data models.

C#
// C# example: Using StackExchange.Redis to interact with multiple data structures
using StackExchange.Redis;

var conn = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
var db = conn.GetDatabase();

// String operations
await db.StringSetAsync("user:1001:name", "Alice", TimeSpan.FromHours(24));
string name = await db.StringGetAsync("user:1001:name");

// Counter operations
await db.StringIncrementAsync("page:/home/views");
long views = await db.StringIncrementAsync("page:/home/views");

// Hash operations for user profile
var profile = new HashEntry[] {
    new HashEntry("name", "Alice"),
    new HashEntry("email", "alice@example.com"),
    new HashEntry("age", "30")
};
await db.HashSetAsync("user:1001", profile);
HashEntry[] fields = await db.HashGetAllAsync("user:1001");

// Sorted Set for leaderboard
await db.SortedSetAddAsync("leaderboard", new SortedSetEntry[] {
    new SortedSetEntry("Alice", 1500),
    new SortedSetEntry("Bob", 2200),
    new SortedSetEntry("Charlie", 1800)
});
RedisValue[] topPlayers = await db.SortedSetRangeByRankAsync("leaderboard", 0, 9, Order.Descending);

// List for task queue
await db.ListRightPushAsync("task:queue", "process_order_12345");
string task = await db.ListLeftPopAsync("task:queue");

// Set operations
await db.SetAddAsync("tags:post:1", new RedisValue[] { "redis", "caching", "performance" });
await db.SetAddAsync("tags:post:2", new RedisValue[] { "redis", "distributed-systems" });
RedisValue[] common = await db.SetIntersectionAsync("tags:post:1", "tags:post:2");

4. Memory Management: Jemalloc, Eviction Policies, and Optimization

Memory management is arguably the most critical operational concern when running Redis in production. Because Redis stores all data in RAM, running out of memory is a catastrophic event that causes all write operations to fail. Understanding how Redis manages memory, what allocator it uses, how eviction policies work, and how to optimize memory usage is essential for every senior engineer.

Jemalloc: Redis's Memory Allocator

Redis uses jemalloc as its default memory allocator, chosen for its excellent performance in multi-threaded scenarios and its efficient handling of small allocations. Jemalloc organizes memory into size classes and arenas to minimize fragmentation and maximize cache locality. Redis's data structures are designed to work cooperatively with jemalloc — small objects are rounded up to predefined size classes, which jemalloc can efficiently manage. The memory reported by Redis's INFO command (used_memory) may differ from what the operating system reports (used_memory_rss), because jemalloc may hold onto freed memory pages for potential reuse rather than returning them to the OS immediately. Understanding this discrepancy is important for capacity planning and monitoring.

Memory Overhead Per Key

Every key in Redis consumes approximately 50-60 bytes of overhead on a 64-bit system in addition to the key name and value. This means that millions of small keys can consume significantly more memory than expected. The INFO MEMORY command provides detailed breakdowns of memory usage, including overhead per key, internal fragmentation, and active versus idle memory. For workloads with millions of small keys, using hash-based encoding (HSET with small field-value pairs stored in a ziplist) can reduce per-key overhead dramatically.

Eviction Policies Deep Dive

When Redis reaches its configured memory limit (maxmemory), it must evict existing keys to make room for new writes (unless the policy is set to noeviction). The eviction policy determines which keys are selected for removal. LRU eviction tracks the last access time of keys using a sampled approach — Redis periodically samples a small number of keys and evicts the one with the oldest access time. This sampling approach trades accuracy for speed, and the sample size can be tuned with maxmemory-samples. LFU, introduced in Redis 4.0, uses a probabilistic counting algorithm inspired by the COUNT-MIN sketch to track access frequency with minimal memory overhead. LFU is generally superior for workloads with power-law access distributions, while LRU works better for more uniform temporal locality.

Memory Optimization Techniques

Several practical techniques can dramatically reduce Redis memory consumption. First, use short, meaningful key names — every byte of a key name is multiplied across millions of keys. Second, prefer hashes over multiple string keys for structured objects, as the ziplist encoding for small hashes eliminates per-key dictionary overhead. Third, use the smallest data type that meets your requirements — store integers with INCR rather than separate hash fields, use bitmaps (SETBIT) for boolean flags, and use HyperLogLog for cardinality estimation. Fourth, set appropriate TTLs on all cached data to ensure automatic cleanup. Fifth, configure maxmemory-policy to an eviction strategy that matches your access pattern. Sixth, monitor memory fragmentation with INFO MEMORY and restart instances if fragmentation becomes excessive (ratio above 1.5). Seventh, consider using Redis's object encoding optimization where small data structures are automatically encoded in compact representations.

Active Memory Defragmentation

Redis 4.0 introduced active memory defragmentation, enabled with activedefrag yes. When enabled, Redis periodically scans its internal data structures and relocates objects in memory to consolidate fragmented memory pages, which can then be returned to the operating system. The defragmentation process runs in the main thread and is controlled by two thresholds: active-defrag-threshold-lower and active-defrag-threshold-upper. While active defragmentation can significantly reduce RSS without requiring a restart, it introduces CPU overhead and should be enabled judiciously. In practice, monitoring the RSS-to-used ratio and scheduling restarts during maintenance windows is often preferable to running active defragmentation in production.

C#
// C# example: Monitoring and managing Redis memory with StackExchange.Redis
using StackExchange.Redis;
using System.Text.Json;

var conn = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
var db = conn.GetDatabase();
var server = conn.GetServer(conn.GetEndPoints().First());

// Monitor memory usage
var info = await server.InfoRawAsync("memory");
string usedMemory = await db.ExecuteAsync("INFO", "memory") as string;

// Configure maxmemory via CONFIG SET
await db.ExecuteAsync("CONFIG", "SET", "maxmemory", "2gb");
await db.ExecuteAsync("CONFIG", "SET", "maxmemory-policy", "allkeys-lru");

// Store data with TTL for automatic cleanup
var products = new[] { "laptop", "phone", "tablet", "monitor", "keyboard" };
foreach (var product in products)
{
    var data = JsonSerializer.Serialize(new { Name = product, Stock = Random.Shared.Next(1, 100) });
    await db.StringSetAsync($"product:{product}", data, TimeSpan.FromMinutes(30));
}

// Monitor eviction statistics
var stats = await db.ExecuteAsync("INFO", "stats");
string evictedKeys = await db.ExecuteAsync("INFO", "stats") as string;

// Use OBJECT ENCODING to check internal encoding of keys
RedisValue encoding = await db.ExecuteAsync("OBJECT", "ENCODING", "user:1001");
Console.WriteLine($"Key encoding: {encoding}");

// Use MEMORY USAGE to check per-key memory consumption
long memUsage = (long)await db.ExecuteAsync("MEMORY", "USAGE", "user:1001");
Console.WriteLine($"Key memory usage: {memUsage} bytes");

5. Persistence: RDB Snapshots, AOF, and Hybrid Persistence

By default, Redis stores all data in memory. If the Redis process crashes or the server restarts, all data is lost unless persistence is configured. Redis provides two independent persistence mechanisms: RDB (Redis Database Backup) snapshots and AOF (Append-Only File) logging. Understanding the trade-offs between these approaches is critical for designing reliable systems.

RDB Snapshots

RDB persistence works by forking the Redis process and creating a compressed binary snapshot of the entire dataset at a point in time. The fork operation uses the operating system's copy-on-write (CoW) mechanism, which means the child process shares memory pages with the parent until a write occurs, at which point the child gets its own copy of the modified page. This approach allows Redis to create snapshots without blocking the main process for more than the initial fork time (typically milliseconds to low seconds). The snapshot is written to a temporary file and then atomically renamed to the configured dump location. RDB snapshots are configured using the save directive — for example, save 900 1 means "create a snapshot if at least 1 write operation occurs within 900 seconds."

Append-Only File (AOF)

AOF persistence records every write command received by Redis in an append-only log file. On restart, Redis replays the AOF to reconstruct the dataset. AOF provides stronger durability guarantees than RDB because it can be configured to fsync after every write command, every second, or let the operating system handle flushing. The three fsync policies are controlled by the appendfsync directive: always (fsync after every write — strongest durability, lowest throughput), everysec (fsync once per second — balanced, can lose up to 1 second of data), and no (let the OS decide — highest throughput, weakest durability). The AOF file grows over time, so Redis periodically rewrites it to remove redundant commands. AOF rewrite uses a fork-based approach with copy-on-write to avoid blocking the main process.

Hybrid Persistence (Redis 7+)

Redis 7.0 introduced the Multi-Part AOF, which combined with aof-use-rdb-preamble yes allows AOF files to start with an RDB preamble followed by incremental AOF commands. This hybrid approach provides the best of both worlds: fast restart times from the RDB preamble and strong durability from the incremental AOF. The multi-part AOF also introduces a manifest file that tracks multiple AOF segments, enabling more efficient rewrite operations and better crash recovery.

Persistence Strategy Recommendations

For pure caching workloads where data loss on restart is acceptable, persistence can be disabled entirely. For session stores where moderate data loss is acceptable but fast restart is important, RDB-only with frequent snapshots provides a good balance. For messaging or queue workloads where data loss is unacceptable, AOF with appendfsync everysec provides strong durability. For workloads requiring both strong durability and fast restart, hybrid persistence with Redis 7+ Multi-Part AOF is the optimal choice. In all cases, test your persistence configuration by simulating crashes and verifying data integrity after restart.

C#
// C# example: Configuring and monitoring Redis persistence
using StackExchange.Redis;

var conn = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
var db = conn.GetDatabase();
var server = conn.GetServer(conn.GetEndPoints().First());

// Check current persistence configuration
var rdbConfig = await db.ExecuteAsync("CONFIG", "GET", "save");
Console.WriteLine($"RDB save config: {rdbConfig}");

var aofConfig = await db.ExecuteAsync("CONFIG", "GET", "appendonly");
Console.WriteLine($"AOF enabled: {aofConfig}");

var fsyncConfig = await db.ExecuteAsync("CONFIG", "GET", "appendfsync");
Console.WriteLine($"Fsync policy: {fsyncConfig}");

// Trigger manual RDB snapshot
await db.ExecuteAsync("BGSAVE");
Console.WriteLine("Background RDB save initiated");

// Trigger AOF rewrite
await db.ExecuteAsync("BGREWRITEAOF");
Console.WriteLine("Background AOF rewrite initiated");

// Monitor persistence status
var persistenceInfo = await db.ExecuteAsync("INFO", "persistence");
Console.WriteLine($"Persistence info: {persistenceInfo}");

// Check last save time
var lastSave = await db.ExecuteAsync("LASTSAVE");
Console.WriteLine($"Last save timestamp: {lastSave}");

// Verify data survives simulated restart cycle
await db.StringSetAsync("persistence-test", "survives-restart", TimeSpan.FromHours(1));
Console.WriteLine("Data written for persistence test");

6. Replication: Primary/Replica, Partial Resync, and PSYNC

Replication is the foundation of Redis's high availability story. By maintaining one or more replicas that continuously replicate data from a primary instance, Redis provides read scaling, data redundancy, and the foundation for automatic failover.

Primary/Replica Architecture

In Redis's replication model, one instance acts as the primary and one or more instances act as replicas. The primary handles all write operations, while replicas maintain a copy of the primary's dataset. Replicas can serve read operations, allowing horizontal scaling of read-heavy workloads. Replication is asynchronous by default, meaning the primary does not wait for replicas to acknowledge receiving a write before considering it complete. This provides the best write performance but means that a small window of data loss is possible if the primary fails before replicating recent writes. Synchronous replication can be configured using the WAIT command, which blocks until a specified number of replicas have acknowledged the write, but at the cost of increased write latency.

Full Resynchronization (FULLRESYNC)

When a replica connects to a primary for the first time, or when a partial resynchronization is not possible, the primary performs a full resynchronization. The process begins with the replica sending a PSYNC command with the special offset -1. The primary responds with a FULLRESYNC reply containing its run ID (a unique identifier generated at startup) and the current replication offset. The primary then creates an RDB snapshot using fork and sends it to the replica over the replication connection. While the RDB is being transferred, the primary buffers all incoming write commands in a replication output buffer. Once the RDB transfer completes, the primary sends the buffered commands to the replica, which applies them in order. After this, the replica is fully synchronized and enters incremental replication mode.

Partial Resynchronization (PSYNC)

When a replica temporarily disconnects and reconnects (for example, due to a network blip), Redis attempts to perform a partial resynchronization instead of a full one. The replica sends a PSYNC command with its last known run ID and replication offset. The primary checks whether the requested offset is still available in its replication backlog (a circular buffer of recent write commands). If the offset is valid, the primary responds with +CONTINUE and sends only the commands that the replica missed. If the offset is too old (the requested data has been overwritten in the backlog), the primary falls back to a full resynchronization. The replication backlog size is controlled by repl-backlog-size (default 1 MB, recommended to increase to 256 MB or more for large write volumes) and repl-backlog-ttl (the time the backlog is retained after all replicas disconnect).

Cascading Replication

Redis supports cascading (chained) replication where a replica can itself have replicas. This creates a replication chain: Primary → Replica A → Replica B. This topology reduces the replication load on the primary but introduces additional replication lag for downstream replicas. Cascading replication is useful in geographically distributed deployments where a regional replica serves as the primary for local replicas, reducing cross-region replication bandwidth. However, it must be used carefully because a failure in the middle of the chain disconnects all downstream replicas. For most production deployments, a flat topology (all replicas connected directly to the primary) is preferred because it provides more predictable latency and simpler failover semantics.

Replication Offset and Consistency

Every write operation on the primary increments the replication offset by the number of bytes written to the output buffers. Replicas track their own offset to know how far behind they are from the primary. The INFO replication command displays the primary's offset, each replica's offset, and the lag (difference between primary and replica offsets divided by the write rate). Understanding replication lag is crucial for building consistent read patterns — if you need strongly consistent reads, you should read from the primary or use the WAIT command to ensure replicas have caught up. In practice, replication lag in well-configured Redis deployments is typically under 1 millisecond under normal load, but can spike during high write bursts, network congestion, or replica restarts.

C#
// C# example: Managing Redis replication with StackExchange.Redis
using StackExchange.Redis;

var conn = await ConnectionMultiplexer.ConnectAsync("primary-host:6379");
var db = conn.GetDatabase();

// Check replication status on primary
var server = conn.GetServer(conn.GetEndPoints().First());
string replInfo = await db.ExecuteAsync("INFO", "replication");
Console.WriteLine($"Replication info:\n{replInfo}");

// Configure replica replication (run on replica)
// CONFIG SET replicaof primary-host 6379

// Use WAIT to ensure writes are replicated before returning
await db.StringSetAsync("critical-data", "important-value");
long acknowledgedReplicas = (long)await db.ExecuteAsync("WAIT", "1", "5000");
Console.WriteLine($"Replicas acknowledged: {acknowledgedReplicas}");

// Monitor replication lag
var primaryInfo = await db.ExecuteAsync("INFO", "replication");
Console.WriteLine($"Primary replication info:\n{primaryInfo}");

// Check replication backlog configuration
var backlogSize = await db.ExecuteAsync("CONFIG", "GET", "repl-backlog-size");
Console.WriteLine($"Backlog size: {backlogSize}");

// Set a large backlog for production workloads
await db.ExecuteAsync("CONFIG", "SET", "repl-backlog-size", "256mb");
Console.WriteLine("Repl backlog set to 256MB");

// Monitor replica offset for consistency verification
string info = await db.ExecuteAsync("INFO", "replication") as string;
Console.WriteLine($"Replication offset info: {info}");

7. Redis Cluster: Hash Slots, Resharding, and Failover

Redis Cluster is the official sharding solution for Redis, enabling horizontal scaling of both reads and writes across multiple Redis nodes. It provides automatic partitioning of the key space into 16,384 hash slots, automatic failover when primary nodes fail, and a gossip-based protocol for cluster state management. Understanding how Redis Cluster works is essential for operating Redis at scale beyond what a single instance can handle.

Hash Slots

Redis Cluster divides the entire key space into 16,384 hash slots. Each key is assigned to a slot using the CRC16 hash function: CRC16(key) mod 16384. Each primary node in the cluster is responsible for a subset of these slots. When a client sends a command to any node in the cluster, the node checks whether the target key (or keys, for multi-key commands) falls within its assigned slots. If it does, the command is executed locally. If not, the node returns a MOVED redirect telling the client which node handles the target slot. Clients cache these redirects to avoid unnecessary round-trips. During resharding (moving slots between nodes), the node may return a temporary ASK redirect that the client follows for that specific command only.

Resharding

Resharding is the process of moving hash slots from one node to another, allowing you to rebalance the cluster as data grows or shrinks. Redis Cluster supports online resharding — slots can be moved while the cluster continues to serve traffic. During resharding, a slot enters one of three states: STABLE (owned by the source node, normal operation), MIGRATING (being moved out of the source node), and IMPORTING (being moved into the target node). When a slot is in the MIGRATING state, the source node accepts commands for keys that still exist locally but returns ASK redirects for keys that have already been migrated. When a slot is in the IMPORTING state, the target node only accepts commands preceded by an ASKING command (sent automatically by the client after receiving an ASK redirect). The redis-cli --cluster reshard command automates this process, including the sequential key migration using DUMP/RESTORE commands with atomic key transfer.

Cluster Bus and Gossip Protocol

Redis nodes communicate through a dedicated cluster bus that runs on a separate port (typically the main port + 10000). The cluster bus uses a gossip protocol to disseminate cluster state information efficiently. Each node periodically sends gossip messages to randomly selected peers, containing its own view of the cluster state (which nodes are alive, which slots each node owns, and the current configuration epoch). When nodes receive gossip messages, they merge the information with their own state, eventually reaching consensus across the cluster. This gossip-based approach eliminates the need for a central coordinator and ensures that the cluster can operate correctly even when network partitions occur. The gossip protocol also handles node discovery — new nodes join the cluster by connecting to any existing node and exchanging gossip messages until they have a complete view of the cluster.

Failover

When a primary node becomes unresponsive (due to crash, network partition, or excessive load), the cluster initiates an automatic failover process. Each primary has one or more replicas that monitor the primary's health through the cluster bus. If a replica detects that its primary is failing (no PONG response within the configured timeout, controlled by cluster-node-timeout), it initiates a failover election. The replica with the highest replication offset among the replicas of the failing primary is elected as the new primary. This replica promotes itself to primary, takes ownership of the failing primary's hash slots, and begins accepting write commands. Other nodes in the cluster update their routing tables to reflect the new slot ownership. The entire failover process typically completes within a few seconds, depending on the cluster-node-timeout configuration (default 15 seconds, often reduced to 5 seconds in production).

Hash Tags for Multi-Key Operations

A key limitation of Redis Cluster is that multi-key operations (such as SUNION, MGET, or transactional commands involving multiple keys) require all target keys to reside in the same hash slot. To work around this, Redis supports hash tags — portions of the key name that are used for slot computation. If the key name contains a {...} pattern, only the content inside the curly braces is used for the CRC16 hash. For example, user:{1001}:profile and user:{1001}:sessions both hash to the same slot because the hash tag is "1001". This allows you to co-locate related data for the same user or entity in the same slot, enabling multi-key operations. However, hash tags must be used carefully — if too many keys map to the same slot, that node becomes a hot spot, undermining the benefit of sharding.

8. Pub/Sub and Streams: Message Brokering and Consumer Groups

Redis provides two distinct messaging mechanisms: the classic Pub/Sub (Publish/Subscribe) system and the more modern Streams data structure. While both enable message-driven architectures, they differ significantly in their delivery guarantees, persistence characteristics, and operational complexity. Understanding when to use each is essential for building reliable event-driven systems.

Redis Pub/Sub

Redis Pub/Sub is a lightweight, fire-and-forget messaging system. Publishers send messages to named channels, and all connected subscribers to those channels receive the message instantly. Pub/Sub is implemented entirely in memory with no persistence — if a subscriber is disconnected when a message is published, the message is lost. This makes Pub/Sub suitable for real-time notifications, live feeds, and ephemeral broadcasts where occasional message loss is acceptable. Pub/Sub messages are not buffered, and there is no acknowledgment mechanism. The simplicity of Pub/Sub is both its strength and its weakness: it provides the lowest possible latency for message delivery (typically microseconds) but offers no durability guarantees whatsoever.

Redis Streams

Redis Streams, introduced in Redis 5.0, provide a durable, append-only log data structure with consumer groups for reliable message processing. Unlike Pub/Sub, Stream messages are persisted to the AOF/RDB and survive restarts. Stream entries have monotonically increasing IDs and can be read multiple times by different consumer groups. Consumer groups provide automatic load balancing (each message is delivered to only one consumer in the group), acknowledgment tracking (XACK), and pending entry lists (PEL) for fault tolerance. If a consumer crashes before acknowledging a message, the message can be reclaimed by another consumer using XPENDING and XCLAIM. Streams also support blocking reads (XREAD with BLOCK), enabling event-driven architectures with strong delivery guarantees.

Consumer Groups in Detail

Consumer groups are the most powerful feature of Redis Streams. A consumer group maintains three critical pieces of state: the last-delivered ID (the high-water mark of messages sent to the group), the pending entries list (PEL — messages that have been delivered but not yet acknowledged), and per-consumer PELs. When a consumer reads from a stream using XREADGROUP with a group name, Redis delivers new messages (messages with IDs greater than the last-delivered ID) to the consumer and adds them to the PEL. The consumer must explicitly acknowledge each message with XACK. If a consumer fails to acknowledge within a configurable time, the message can be re-delivered using XPENDING (to list pending messages) and XCLAIM (to take ownership of a pending message). This mechanism provides at-least-once delivery semantics, which is sufficient for most message processing workloads. For exactly-once semantics, the application must implement idempotent message handling.

When to Use Pub/Sub vs. Streams

Use Pub/Sub when you need real-time broadcast to currently connected subscribers, messages are ephemeral and can be lost, you want minimal infrastructure complexity, and the system can tolerate missed messages during disconnections. Use Streams when you need durable message storage with replay capability, at-least-once delivery with consumer group semantics, the ability to process historical messages, backpressure handling through consumer group load balancing, and audit trails of all messages. In practice, most production systems prefer Streams over Pub/Sub because the durability guarantees and consumer group features significantly simplify error handling and recovery.

C#
// C# example: Redis Pub/Sub and Streams with StackExchange.Redis
using StackExchange.Redis;
using System.Text.Json;

var conn = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
var db = conn.GetDatabase();
var sub = conn.GetSubscriber();

// === Pub/Sub Pattern ===
// Subscribe to a channel
await sub.SubscribeAsync("notifications:alerts", (channel, message) =>
{
    Console.WriteLine($"Alert received: {message}");
});

// Publish a message
await sub.PublishAsync("notifications:alerts", "High CPU usage on server-1");

// === Streams Pattern ===
// Create a consumer group
try
{
    await db.ExecuteAsync("XGROUP", "CREATE", "orders:stream", "order-processors", "$", "MKSTREAM");
}
catch (RedisServerException) { /* Group already exists */ }

// Add messages to the stream
var orderId = Guid.NewGuid().ToString();
var order = JsonSerializer.Serialize(new
{
    OrderId = orderId,
    Amount = 99.99m,
    Timestamp = DateTime.UtcNow
});
await db.ExecuteAsync("XADD", "orders:stream", "*",
    "order-data", order, "priority", "high");

// Read from consumer group (blocking)
var result = await db.ExecuteAsync("XREADGROUP", "GROUP", "order-processors", "worker-1",
    "COUNT", "10", "BLOCK", "5000", "STREAMS", "orders:stream", ">");

// Acknowledge processed messages
await db.ExecuteAsync("XACK", "orders:stream", "order-processors", messageId);

// Check pending entries (for failed consumers)
var pending = await db.ExecuteAsync("XPENDING", "orders:stream", "order-processors",
    "COUNT", "100");
Console.WriteLine($"Pending entries: {pending}");

// Claim abandoned messages from crashed consumers
var claimed = await db.ExecuteAsync("XCLAIM", "orders:stream", "order-processors",
    "worker-2", "60000", "0-0", "COUNT", "10");

9. Lua Scripting: Atomic Operations and Custom Commands

Redis includes a built-in Lua interpreter (LuaJIT) that allows you to execute Lua scripts directly on the server. Lua scripts are executed atomically — the Redis server blocks all other commands while a script is running, guaranteeing that the script's operations see a consistent snapshot of the data and that no other commands interfere with the script's execution. This atomicity guarantee makes Lua scripts the primary mechanism for implementing complex operations that require multiple commands without the overhead of distributed locking.

Why Lua Scripting Matters

In many real-world scenarios, you need to perform multiple Redis operations as an atomic unit. For example, implementing a distributed rate limiter might require checking a counter, incrementing it, and setting an expiry — three separate commands that must execute without interruption. Without Lua scripting, you would need to use WATCH/MULTI/EXEC transactions, which are limited to optimistic locking and cannot branch based on intermediate results. Lua scripts can contain arbitrary control flow (if/else, loops, function calls) and can make decisions based on the results of individual commands within the script. The EVAL command sends a Lua script to the Redis server along with the number of keys the script accesses and the key names. Inside the script, you use the redis.call() function to execute Redis commands, and the results are returned to the client as if they were the response to a single command.

C#
// C# example: Redis Lua scripting for atomic rate limiting and conditional updates
using StackExchange.Redis;

var conn = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
var db = conn.GetDatabase();

// Atomic sliding window rate limiter using Lua
string rateLimitScript = @"
    local key = KEYS[1]
    local limit = tonumber(ARGV[1])
    local window = tonumber(ARGV[2])
    local now = tonumber(ARGV[3])
    
    -- Remove expired entries
    redis.call('ZREMRANGEBYSCORE', key, 0, now - window)
    
    -- Count current entries in window
    local count = redis.call('ZCARD', key)
    
    if count < limit then
        -- Add the new request
        redis.call('ZADD', key, now, now .. '-' .. math.random(1000000))
        redis.call('PEXPIRE', key, window)
        return {1, limit - count - 1}
    else
        return {0, 0}
    end
";

var rateLimitResult = await db.ScriptEvaluateAsync(rateLimitScript,
    new RedisKey[] { "ratelimit:user:1001" },
    new RedisValue[] { 100, 60000, DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() });

var allowed = (int)((RedisResult[])rateLimitResult)[0];
var remaining = (int)((RedisResult[])rateLimitResult)[1];
Console.WriteLine($"Rate limit: Allowed={allowed}, Remaining={remaining}");

// Atomic check-and-set operation
string casScript = @"
    local current = redis.call('GET', KEYS[1])
    if current == ARGV[1] then
        redis.call('SET', KEYS[1], ARGV[2])
        return 1
    end
    return 0
";

var casResult = await db.ScriptEvaluateAsync(casScript,
    new RedisKey[] { "config:feature:dark-mode" },
    new RedisValue[] { "enabled", "disabled" });

// Atomic inventory decrement (prevents overselling)
string decrementScript = @"
    local stock = tonumber(redis.call('GET', KEYS[1]) or '0')
    local requested = tonumber(ARGV[1])
    if stock >= requested then
        redis.call('DECRBY', KEYS[1], requested)
        return stock - requested
    end
    return -1
";

var stockResult = await db.ScriptEvaluateAsync(decrementScript,
    new RedisKey[] { "inventory:product:laptop" },
    new RedisValue[] { 3 });

Console.WriteLine($"Stock after order: {stockResult}");

Script Caching and SHA1

Redis caches loaded Lua scripts on the server side, identified by their SHA1 hash. Instead of sending the entire script body with every EVAL call, you can use EVALSHA to execute a previously loaded script by its SHA1 digest. This reduces network bandwidth for frequently executed scripts. The script cache is per-instance and is cleared when the instance restarts. To ensure scripts are loaded before EVALSHA is used, the application should first send the script via EVAL (or SCRIPT LOAD) and then use EVALSHA for subsequent executions. The SCRIPT FLUSH command clears the script cache, and SCRIPT EXISTS checks whether a script is cached.

Script Debugging and Best Practices

Redis provides a built-in Lua debugger accessed via EVAL DEBUG that supports breakpoints, stepping, variable inspection, and breakpoints on Redis commands. For production use, follow these best practices: keep scripts short and focused (long scripts block the server), use KEYS[] for all key names (required for cluster mode), avoid blocking operations within scripts, handle edge cases gracefully, and always test scripts with realistic data volumes. Script timeouts are controlled by lua-time-limit (default 5 seconds) — if a script runs longer than this, other clients receive aBUSY error and can issue SCRIPT KILL to terminate the script (only if the script has not yet performed any write operations).

10. Caching Patterns: Cache-Aside, Write-Through, Write-Behind, and Cache Invalidation

Choosing the right caching pattern is one of the most impactful architectural decisions in a system that uses Redis. Each pattern offers different trade-offs between consistency, performance, complexity, and data freshness. A deep understanding of these patterns — and their failure modes — is essential for building systems that perform well under load while maintaining correct behavior.

Cache-Aside (Lazy Loading)

Cache-Aside is the most common caching pattern. The application first checks the cache for the requested data. If found (cache hit), the data is returned directly from Redis. If not found (cache miss), the application fetches the data from the primary database, stores it in Redis with a TTL, and returns it to the caller. The key property of Cache-Aside is that the cache is populated on demand — only data that has been requested is cached. This makes it memory-efficient but means the first request for any piece of data always incurs the latency of a database query. Cache-Aside works well for read-heavy workloads where the access distribution follows a power law — a small percentage of keys receive the vast majority of traffic.

Write-Through

In the Write-Through pattern, every write operation updates both the cache and the database synchronously. When the application writes data, it writes to the cache first, and the cache (or a cache layer) writes to the database. This ensures that the cache is always consistent with the database, at the cost of write latency (every write hits both the cache and the database). Write-Through is useful when data freshness is critical and the write volume is moderate. The main disadvantage is that cold data in the cache is only populated during writes, so the cache may not contain data that is frequently read but infrequently written.

Write-Behind (Write-Back)

Write-Behind is similar to Write-Through, but the database write is asynchronous. The application writes to the cache, and the cache periodically flushes changes to the database in the background. This provides the lowest possible write latency but introduces the risk of data loss if the cache fails before flushing. Write-Behind is ideal for write-heavy workloads where write latency is critical and some data loss is acceptable (for example, analytics counters, view counts, or rate limiter state). The implementation typically uses a Redis Stream or List to buffer writes, with a background worker that batches and flushes them to the database.

Cache Invalidation Strategies

Cache invalidation — the process of removing or updating stale data in the cache — is notoriously difficult. Phil Karlton's famous quote, "There are only two hard things in Computer Science: cache invalidation and naming things," captures the essence of the challenge. The three primary invalidation strategies are TTL-based expiry (data expires after a configured duration), event-driven invalidation (the database publishes change events that the cache subscriber uses to invalidate entries), and version-based invalidation (cache keys include a version number that is incremented on updates, causing old entries to become orphaned and eventually evicted). Each strategy has trade-offs: TTL-based expiry is simple but may serve stale data until expiry; event-driven invalidation provides near-real-time consistency but adds complexity and requires reliable event delivery; version-based invalidation avoids explicit invalidation logic but uses more memory for orphaned keys.

Cache Stampede and Thundering Herd

Cache stampede (also called thundering herd) occurs when a popular cache key expires and a large number of concurrent requests all miss the cache simultaneously, each hitting the database to rebuild the same data. This can overwhelm the database and cause cascading failures. Solutions include: mutex locking (use a Redis distributed lock so only one process rebuilds the cache while others wait), probabilistic early expiration (randomly refresh the cache before it actually expires), and request coalescing (deduplicate concurrent requests for the same key at the application layer). Implementing these patterns correctly is critical for production reliability.

C#
// C# example: Cache-Aside pattern with stampede prevention
using StackExchange.Redis;
using System.Text.Json;

var conn = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
var db = conn.GetDatabase();

// Cache-Aside with distributed lock for stampede prevention
async Task<Product?> GetProductAsync(string productId)
{
    // Step 1: Check cache
    var cached = await db.StringGetAsync($"product:{productId}");
    if (cached.HasValue)
    {
        return JsonSerializer.Deserialize<Product>(cached);
    }

    // Step 2: Acquire lock to prevent stampede
    var lockKey = $"lock:product:{productId}";
    var lockValue = Guid.NewGuid().ToString();
    bool acquired = await db.LockTakeAsync(lockKey, lockValue, TimeSpan.FromSeconds(10));

    if (!acquired)
    {
        // Another process is rebuilding the cache; wait and retry
        await Task.Delay(100);
        var retryCache = await db.StringGetAsync($"product:{productId}");
        return retryCache.HasValue ? JsonSerializer.Deserialize<Product>(retryCache) : null;
    }

    try
    {
        // Step 3: Double-check cache (another process may have populated it)
        var doubleCheck = await db.StringGetAsync($"product:{productId}");
        if (doubleCheck.HasValue)
        {
            return JsonSerializer.Deserialize<Product>(doubleCheck);
        }

        // Step 4: Fetch from database
        var product = await FetchProductFromDatabase(productId);
        if (product != null)
        {
            var serialized = JsonSerializer.Serialize(product);
            await db.StringSetAsync($"product:{productId}", serialized, TimeSpan.FromMinutes(30));
        }
        return product;
    }
    finally
    {
        // Step 5: Release lock
        await db.LockReleaseAsync(lockKey, lockValue);
    }
}

// Write-Behind pattern for analytics counters
async Task TrackEventAsync(string eventType, string entityId)
{
    var entry = JsonSerializer.Serialize(new
    {
        Event = eventType,
        EntityId = entityId,
        Timestamp = DateTime.UtcNow
    });

    // Write to Redis stream for async processing
    await db.ExecuteAsync("XADD", $"events:{eventType}:buffer", "*",
        "data", entry);

    // Increment in-memory counter for immediate reads
    await db.StringIncrementAsync($"analytics:{eventType}:count");
}

record Product(string Id, string Name, decimal Price, int Stock);

11. Redis Sentinel: High Availability and Automatic Failover

Redis Sentinel is the official high availability solution for Redis deployments that do not use Redis Cluster. It provides automatic failover, configuration provider, and monitoring for Redis primary-replica topologies. Sentinel ensures that when a primary node fails, a replica is automatically promoted to primary and all other replicas are reconfigured to replicate from the new primary, minimizing downtime and manual intervention.

Sentinel Architecture

A Sentinel deployment consists of three or more Sentinel processes running on separate machines (to avoid single points of failure). Sentinels communicate with each other using the same gossip-based protocol as Redis Cluster, constantly exchanging information about the state of monitored Redis instances. Each Sentinel monitors the primary and all replicas, performing health checks at regular intervals (controlled by down-after-milliseconds). When a majority of Sentinels agree that the primary is unreachable (quorum-based detection), they initiate a failover election. The Sentinel with the lowest configuration epoch is elected as the leader, and it coordinates the failover process.

Failover Process

The Sentinel failover process follows a carefully orchestrated sequence of steps. First, Sentinels detect the primary is down using the subjective-down (SDOWN) and objective-down (ODOWN) mechanisms. SDOWN occurs when a single Sentinel's health check fails; ODOWN occurs when a quorum of Sentinels agree the primary is down. Next, a Sentinel leader is elected through a Raft-like consensus protocol. The leader selects the best replica for promotion based on a scoring algorithm that considers: replication offset (preferring the most up-to-date replica), priority (configurable per replica via replica-priority), and ID (tiebreaker). The selected replica is promoted using the SLAVEOF NO ONE command, and all other replicas are reconfigured to replicate from the new primary. Finally, the old primary (if it recovers) is automatically reconfigured as a replica of the new primary. The entire failover process typically completes within 10-30 seconds, depending on the down-after-milliseconds configuration.

Client Connection Handling

Applications connect to Redis through Sentinel by using a Sentinel-aware client library (such as StackExchange.Redis for C#, Jedis or Lettuce for Java). The client queries the Sentinels to discover the current primary address and subscribes to Sentinel events to receive failover notifications in real-time. When a failover occurs, the client automatically reconnects to the new primary without application restart. This transparent failover is critical for zero-downtime deployments. The client maintains a connection pool to both the primary and replicas, routing read commands to replicas and write commands to the primary (or to the primary after failover). It is important to configure the client with the full list of Sentinel endpoints, not just one, to ensure connectivity even if individual Sentinels are unavailable.

Sentinel Configuration and Operations

A minimal Sentinel configuration requires specifying the primary name, IP, port, and quorum: sentinel monitor mymaster 127.0.0.1 6379 2. This tells Sentinel to monitor a primary named "mymaster" at 127.0.0.1:6379 with a quorum of 2 (meaning at least 2 Sentinels must agree before a failover is initiated). Other important configurations include sentinel down-after-milliseconds (how long a primary can be unresponsive before being declared down), sentinel failover-timeout (maximum time for a failover to complete), and sentinel parallel-syncs (how many replicas can sync simultaneously after failover — setting this too high can cause a performance hit). In production, always run at least 3 Sentinels on separate machines, use a quorum of 2 or more, and monitor Sentinel logs for failover events and configuration changes.

C#
// C# example: Connecting to Redis through Sentinel
using StackExchange.Redis;

// Configure Sentinel connection with multiple Sentinel endpoints
var sentinelEndpoints = new EndPoint[]
{
    new DnsEndPoint("sentinel-1.example.com", 26379),
    new DnsEndPoint("sentinel-2.example.com", 26379),
    new DnsEndPoint("sentinel-3.example.com", 26379)
};

var sentinelOptions = new ConfigurationOptions
{
    ServiceName = "mymaster",
    AbortOnConnectFail = false,
    Password = "your-password"
};

// Sentinel-aware connection - auto-discovers primary
foreach (var ep in sentinelEndpoints)
    sentinelOptions.EndPoints.Add(ep);

var conn = await ConnectionMultiplexer.ConnectAsync(sentinelOptions);
var db = conn.GetDatabase();

// Subscribe to failover events
conn.ConnectionFailed += (sender, args) =>
{
    Console.WriteLine($"Connection failed: {args.EndPoint} - {args.FailureType}");
};

conn.ConnectionRestored += (sender, args) =>
{
    Console.WriteLine($"Connection restored: {args.EndPoint}");
};

// Normal operations - client handles failover transparently
await db.StringSetAsync("session:user123", "active", TimeSpan.FromMinutes(30));
var value = await db.StringGetAsync("session:user123");
Console.WriteLine($"Session status: {value}");

// Get information about the current topology
var server = conn.GetServer(conn.GetEndPoints().First());
Console.WriteLine($"Connected to: {server.EndPoint} (IsMaster: {server.IsMaster})");

12. Transactions and Lua: MULTI/EXEC, Optimistic Locking

Redis provides transaction support through the MULTI/EXEC command pair, which allows multiple commands to be executed as a single atomic unit. However, Redis transactions differ fundamentally from traditional database transactions — they do not support rollback, and their isolation level is weaker than what most databases provide. Understanding these limitations and knowing when to use transactions versus Lua scripts is essential for building correct systems.

MULTI/EXEC Transactions

A Redis transaction begins with the MULTI command and ends with EXEC. All commands issued between MULTI and EXEC are queued and then executed sequentially as a single unit. No other client commands are interleaved during execution, providing atomicity in the sense that the transaction either runs completely or not at all. However, unlike database transactions, Redis transactions do not provide rollback — if a command within the transaction fails (for example, trying to increment a non-numeric string), subsequent commands in the transaction still execute. This means that partial updates may occur within a transaction, which is a critical difference from ACID transactions in relational databases. The DISCARD command can be used to abort a transaction before EXEC without executing any of the queued commands.

Optimistic Locking with WATCH

WATCH provides optimistic locking for Redis transactions. When you WATCH a key, Redis monitors it for changes. If any other client modifies the watched key before your EXEC, the transaction is aborted with a nil response, indicating that the watched key was modified. This allows you to implement compare-and-swap patterns: WATCH a key, read its value, compute the new value, MULTI/EXEC the update, and if EXEC returns nil, retry the entire operation. WATCH is useful for scenarios like inventory management (decrement stock only if it has not changed since you read it) or configuration updates (change a setting only if it has not been modified by another process). However, WATCH only detects changes at the EXEC point — if the value changes and then changes back between WATCH and EXEC, the transaction still succeeds, which may not be the desired behavior.

When to Use Transactions vs. Lua Scripts

MULTI/EXEC transactions are appropriate for simple atomic operations where all the commands are known in advance and no conditional logic is needed. For example, atomically transferring a value between two keys (DECRBY key1 100; INCRBY key2 100). Lua scripts are necessary when you need conditional logic within the atomic operation — for example, "decrement inventory only if sufficient stock exists, and if so, create an order record." Lua scripts can read intermediate results and make decisions, whereas transactions cannot. Lua scripts also provide stronger isolation guarantees within the script execution because they execute as a single unit with no command interleaving. In practice, most complex atomic operations are better implemented with Lua scripts, while simple sequences of unconditional commands can use MULTI/EXEC.

C#
// C# example: Redis transactions and optimistic locking
using StackExchange.Redis;

var conn = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
var db = conn.GetDatabase();

// Simple MULTI/EXEC transaction: Transfer between accounts
var tx = db.CreateTransaction();
tx.StringDecrementAsync("account:alice:balance", 500);
tx.StringIncrementAsync("account:bob:balance", 500);
await tx.ExecuteAsync();
Console.WriteLine("Transfer completed atomically");

// WATCH + MULTI/EXEC: Optimistic locking for inventory
async Task<bool> TryPlaceOrderAsync(string productId, int quantity)
{
    var key = $"inventory:{productId}";
    var lockToken = new RedisValue(Guid.NewGuid().ToString());

    for (int attempt = 0; attempt < 3; attempt++)
    {
        // Watch the inventory key
        await db.KeyExpireAsync(key, TimeSpan.FromSeconds(30)); // ensure it exists
        var stock = (long?)await db.StringGetAsync(key);
        if (stock == null || stock < quantity)
        {
            Console.WriteLine("Insufficient stock");
            return false;
        }

        // Create transaction
        var transaction = db.CreateTransaction();
        _ = transaction.StringDecrementAsync(key, quantity);
        _ = transaction.ListLeftPushAsync($"orders:{productId}", $"order-{Guid.NewGuid()}");

        // Execute transaction (returns false if watched key was modified)
        bool success = await transaction.ExecuteAsync();
        if (success)
        {
            Console.WriteLine("Order placed successfully");
            return true;
        }

        Console.WriteLine($"Attempt {attempt + 1} failed, retrying...");
        await Task.Delay(50); // Brief backoff before retry
    }
    return false;
}

bool orderResult = await TryPlaceOrderAsync("laptop", 1);
Console.WriteLine($"Order result: {orderResult}");

// Using Lua script for the same operation (often preferred)
string luaScript = @"
    local stock = tonumber(redis.call('GET', KEYS[1]) or '0')
    if stock >= tonumber(ARGV[1]) then
        redis.call('DECRBY', KEYS[1], ARGV[1])
        redis.call('LPUSH', KEYS[2], ARGV[2])
        return 1
    end
    return 0
";

var luaResult = await db.ScriptEvaluateAsync(luaScript,
    new RedisKey[] { "inventory:laptop", "orders:laptop" },
    new RedisValue[] { 1, $"order-{Guid.NewGuid()}" });
Console.WriteLine($"Lua-based order: {(long)luaResult == 1 ? "Success" : "Failed"}");

13. Security: ACLs, TLS, and Command Renaming

Redis security has historically been a concern, as early versions ran without any authentication by default. Modern Redis (6.0+) provides robust security features including Access Control Lists (ACLs), TLS encryption, command renaming/disabling, and network-level protections. A comprehensive security strategy for Redis involves multiple layers of defense, each addressing different threat vectors.

Access Control Lists (ACLs)

Redis 6.0 introduced Access Control Lists (ACLs), which allow administrators to create multiple user accounts with fine-grained permissions. Each user can be granted access to specific commands and specific key patterns. For example, you can create a user that can only execute GET and SET on keys matching cache:*, or a user that can only execute READ operations on all keys. ACLs are configured via the ACL command or an external ACL file (aclfile). The default user retains full access for backward compatibility. ACLs are essential for multi-tenant environments, microservice architectures where different services need different levels of access, and any deployment where the Redis instance is accessible to multiple teams or applications.

TLS Encryption

Redis 6.0 added native TLS support, enabling encrypted communication between clients and the server, as well as between nodes in a cluster or between primary and replica. TLS encryption protects against eavesdropping, man-in-the-middle attacks, and credential theft. Configuring TLS requires providing certificate and key files via the tls-port, tls-cert-file, tls-key-file, and optionally tls-ca-cert-file configuration directives. When TLS is enabled, the non-encrypted port can be disabled by setting port 0. In production environments, TLS should always be enabled for deployments where the network is not fully trusted (cloud environments, shared infrastructure, cross-datacenter replication). TLS adds approximately 5-15% overhead to Redis operations due to the encryption/decryption cost, which is generally acceptable for the security benefits.

Command Renaming and Disabling

Redis provides the rename-command directive to rename or disable dangerous commands. Commands like FLUSHALL, FLUSHDB, DEBUG, and CONFIG can cause data loss or information disclosure if misused. By renaming these commands to obfuscated names (or disabling them entirely by renaming to an empty string), you reduce the attack surface. For example, rename-command FLUSHALL "" disables the FLUSHALL command entirely, while rename-command CONFIG "SECURE_CONFIG" renames the CONFIG command so that only those who know the new name can use it. Note that renaming commands can cause issues with Redis Cluster (which uses certain administrative commands internally) and should be tested thoroughly before deployment.

Security Best Practices

In production Redis deployments, follow these security practices: always require authentication using either the traditional requirepass directive (for single-password access) or ACLs (for multi-user environments with fine-grained permissions). Enable TLS for all communication channels, including client-to-server, replication, and cluster bus communication. Bind Redis to specific network interfaces rather than all interfaces (0.0.0.0). Enable protected mode, which prevents connections from non-localhost addresses when no password is set. Disable or rename dangerous commands. Run Redis as a non-root user with minimal file system permissions. Keep Redis updated to the latest stable version to receive security patches. Monitor Redis logs for unauthorized access attempts and suspicious command patterns. Use network firewalls and security groups to restrict access to Redis ports (default 6379 for client, 6380 for TLS, port+10000 for cluster bus). Regularly audit ACL configurations and rotate passwords.

C#
// C# example: Redis security configuration and ACL management
using StackExchange.Redis;

// Connect with authentication and TLS
var options = new ConfigurationOptions
{
    EndPoints = { "redis-server.example.com:6380" },
    Password = "strong-password-here",
    Ssl = true,
    SslProtocols = System.Security.Authentication.SslProtocols.Tls12,
    AbortOnConnectFail = false,
    ConnectTimeout = 5000,
    SyncTimeout = 5000
};

var conn = await ConnectionMultiplexer.ConnectAsync(options);
var db = conn.GetDatabase();

// ACL management (requires admin user)
await db.ExecuteAsync("ACL", "SETUSER", "app-service",
    "on",
    "+@read", "+@write",
    "~cache:*",
    "&cache:*",
    ">service-password-123");

await db.ExecuteAsync("ACL", "SETUSER", "readonly-service",
    "on",
    "+@read",
    "~cache:*",
    ">readonly-password-456");

// List current ACL configuration
var aclList = await db.ExecuteAsync("ACL", "LIST");
Console.WriteLine($"ACL users: {aclList}");

// Check effective permissions for current user
var aclWhoami = await db.ExecuteAsync("ACL", "WHOAMI");
Console.WriteLine($"Current user: {aclWhoami}");

// Test command authorization
try
{
    await db.ExecuteAsync("FLUSHALL");
}
catch (RedisServerException ex) when (ex.Message.Contains("NOPERM"))
{
    Console.WriteLine("Command not authorized - FLUSHALL is disabled");
}

// Secure key access patterns - use specific key prefixes
await db.StringSetAsync("cache:products:123", "laptop-data");
var data = await db.StringGetAsync("cache:products:123");
Console.WriteLine($"Secure data access: {data}");

14. Performance Tuning: Pipelining, Connection Pooling, and Large Keys

While Redis is inherently fast, suboptimal usage patterns can dramatically degrade performance. Understanding how to use pipelining effectively, manage connection pools, avoid large keys, and tune Redis configuration for your specific workload is essential for achieving maximum throughput and minimum latency in production environments.

Redis Pipelining

Redis pipelining is a technique that allows clients to send multiple commands in a single network round-trip without waiting for each response. In normal (non-pipelined) operation, each command requires a round-trip: send command, wait for response, send next command. With pipelining, the client sends N commands in sequence, then reads all N responses. This can improve throughput by 5-10x for small commands on high-latency networks because it eliminates the round-trip latency overhead. However, pipelining is not a silver bullet — it increases memory usage on both client and server (because all responses must be buffered), and it reduces the ability to interleave operations for other clients. Pipelining is most effective for bulk operations like inserting or updating many keys, warming a cache, or migrating data. It is less useful for single-command operations or workloads where each command's result is needed before the next command can be issued.

Connection Pooling

Connection pooling is essential for applications that make frequent Redis calls. Creating and tearing down TCP connections is expensive (involves DNS resolution, TCP handshake, and possibly authentication), so maintaining a pool of persistent connections dramatically reduces connection overhead. StackExchange.Redis (the primary .NET Redis client) automatically manages a connection pool internally — a single ConnectionMultiplexer instance maintains a pool of connections to each Redis endpoint. Best practices include: sharing a single ConnectionMultiplexer across the entire application (it is thread-safe and designed for concurrent use), setting appropriate syncTimeout and connectTimeout values, monitoring pool saturation via the ConnectionCounters property, and avoiding creating separate connections for different parts of the application.

Large Key Problems

Large keys (strings larger than 100 KB, collections with more than 10,000 elements) cause several operational problems: they increase memory fragmentation, slow down replication (transferring large keys over the network), cause latency spikes during RDB snapshots (fork + copy-on-write for large keys), and can cause command timeouts if operations like DEL on a large collection block the server for too long. Redis provides the UNLINK command as a non-blocking alternative to DEL — it marks the key for lazy deletion in a background thread rather than synchronously freeing memory. Large hash, set, sorted set, and list keys should be split into smaller sub-keys using hash tags or numeric suffixes. Monitoring for large keys can be done using the MEMORY USAGE command, the --bigkeys flag in redis-cli, or external monitoring tools that periodically scan the key space.

Slow Log and Latency Monitoring

Redis provides the SLOWLOG command to track commands that exceed a configurable execution time threshold. The slow log records the command, execution time, timestamp, and client address for each slow command, making it easy to identify performance bottlenecks. Configure slowlog-log-slower-than (in microseconds, default 10000 = 10ms) and slowlog-max-len (number of entries to retain, default 128). The LATENCY MONITOR command provides real-time latency tracking with percentile statistics, and the LATENCY HISTORY command shows latency samples over time. For continuous monitoring, export Redis metrics (latency, throughput, memory, connections) to a monitoring system like Prometheus with Grafana dashboards, and set up alerts for latency spikes, memory pressure, and connection pool exhaustion.

C#
// C# example: Performance optimization with pipelining and connection pooling
using StackExchange.Redis;

// Share a single ConnectionMultiplexer across the application (thread-safe)
var conn = await ConnectionMultiplexer.ConnectAsync("localhost:6379");
var db = conn.GetDatabase();

// === Pipelining for Bulk Operations ===
var tasks = new List<Task>();
for (int i = 0; i < 10000; i++)
{
    tasks.Add(db.StringSetAsync($"bulk:key:{i}", $"value-{i}", TimeSpan.FromHours(1)));
}
await Task.WhenAll(tasks);
Console.WriteLine("Bulk insert with pipelining completed");

// More explicit pipelining using batch
var batch = db.CreateBatch();
var batchTasks = new List<Task>();
for (int i = 0; i < 10000; i++)
{
    batchTasks.Add(db.StringSetAsync($"batch:key:{i}", $"value-{i}"));
}
batch.Execute(); // Sends all commands in one network round-trip
await Task.WhenAll(batchTasks);
Console.WriteLine("Batch insert completed");

// === Large Key Detection ===
var server = conn.GetServer(conn.GetEndPoints().First());
long bigKeyCount = 0;
foreach (var key in server.Keys(pattern: "*", pageSize: 100))
{
    long memUsage = (long)await db.ExecuteAsync("MEMORY", "USAGE", key);
    if (memUsage > 102400) // 100 KB threshold
    {
        Console.WriteLine($"Large key: {key} ({memUsage} bytes)");
        bigKeyCount++;
    }
}
Console.WriteLine($"Found {bigKeyCount} large keys");

// Use UNLINK instead of DEL for large keys
await db.ExecuteAsync("UNLINK", "large:collection:key");

// === Slow Log Analysis ===
var slowLog = await db.ExecuteAsync("SLOWLOG", "GET", "10");
Console.WriteLine($"Recent slow commands:\n{slowLog}");

// Check slow log length
var slowLogLen = await db.ExecuteAsync("SLOWLOG", "LEN");
Console.WriteLine($"Slow log entries: {slowLogLen}");

// Reset slow log
await db.ExecuteAsync("SLOWLOG", "RESET");

// === Connection Pool Monitoring ===
var connCounters = conn.GetCounters();
Console.WriteLine($"Total connections: {connCounters.TotalConnections}");
Console.WriteLine($"Active connections: {connCounters.InteractiveConnectionCount}");

15. Redis Modules: RedisJSON, RediSearch, and RedisTimeSeries

The Redis Modules system, introduced in Redis 4.0, allows extending Redis with new data structures, commands, and capabilities without modifying the core codebase. Modules are shared libraries (.so on Linux, .dll on Windows) that are loaded at runtime using the MODULE LOAD command or the loadmodule configuration directive. The module ecosystem has grown to include dozens of high-quality modules that transform Redis from a key-value store into a multi-purpose data platform. Understanding the module ecosystem and knowing which modules are available for common use cases is essential for designing comprehensive solutions on top of Redis.

RedisJSON

RedisJSON provides native JSON document storage and manipulation within Redis. It stores JSON documents in a binary format that allows efficient partial updates without deserializing the entire document. RedisJSON supports JSONPath expressions for querying and modifying nested structures, atomic operations on individual fields, and indexing for fast queries. With RediSearch integration, you can create secondary indexes on JSON document fields and perform complex queries, full-text search, and aggregations directly on JSON documents stored in Redis. RedisJSON is particularly useful for storing user profiles, product catalogs, configuration objects, and any structured data that benefits from JSON-native storage and querying.

RediSearch

RediSearch is a full-text search and secondary indexing engine for Redis. It provides capabilities that rival dedicated search engines like Elasticsearch for moderate-scale use cases. RediSearch supports full-text search with stemming, phonetic matching, and auto-complete, as well as numeric range queries, geo-spatial queries, and tag-based filtering. It can index both standalone Redis keys and fields within RedisJSON documents. RediSearch supports aggregations (GROUP BY, REDUCE), sort-by, paging, and highlighting. For applications that need search functionality on data already stored in Redis, RediSearch eliminates the need to maintain a separate search infrastructure. However, for very large datasets (billions of documents) or complex search requirements, dedicated search engines like Elasticsearch may still be more appropriate.

RedisTimeSeries

RedisTimeSeries provides a native time-series data type for Redis, optimized for high-throughput ingestion and range queries on time-stamped data. It supports automatic compaction (downsampling) to reduce storage for older data, label-based indexing for efficient filtering, aggregation functions (sum, avg, min, max, count, range, etc.) over time ranges, and retention policies that automatically expire old data. RedisTimeSeries is ideal for IoT sensor data, application metrics, financial tick data, and any workload that involves high-volume time-stamped data with range query requirements. It can be combined with RediSearch for secondary indexing on time-series labels.

Module Selection Guidelines

When evaluating Redis modules, consider the following factors. First, assess whether the module is actively maintained and compatible with your Redis version. Second, evaluate the memory overhead of the module's data structures — some modules use significantly more memory than equivalent native Redis data structures. Third, consider the operational complexity of running modules in production, including monitoring, backup, and upgrade procedures. Fourth, verify that the module supports your deployment topology (standalone, Sentinel, or Cluster). Fifth, benchmark the module's performance with your specific workload before committing to production use. In general, prefer native Redis data structures for simple use cases and reach for modules only when the built-in capabilities are insufficient for your requirements.

16. Comparison: Memcached, KeyDB, Dragonfly, and Alternatives

While Redis dominates the in-memory data store landscape, several alternatives offer different trade-offs in terms of performance, features, and operational characteristics. Understanding how Redis compares to these alternatives helps architects make informed technology choices and understand the boundaries of what Redis does well versus where alternatives might be preferable.

Redis vs. Memcached

Memcached is the original high-performance distributed caching system, designed for simplicity and speed. It supports only string key-value pairs with a maximum value size of 1 MB. Memcached uses a multithreaded architecture, which gives it an advantage for pure caching workloads on multi-core servers because it can utilize all CPU cores without the overhead of running multiple Redis instances. However, Memcached lacks persistence, replication, rich data structures, pub/sub, scripting, and most of the advanced features that make Redis so versatile. For simple caching workloads where multithreaded performance on a single node is paramount, Memcached can be a reasonable choice. For virtually all other use cases, Redis provides significantly more capability with comparable or better performance.

Redis vs. KeyDB

KeyDB is a multithreaded fork of Redis that claims 2x-5x performance improvement over single-threaded Redis by utilizing multiple CPU cores for command processing. KeyDB maintains full compatibility with the Redis API and protocol, making it a potential drop-in replacement. KeyDB also supports Active Replication (multi-master), FLASH storage (tiered storage to SSD), and subkey expiry. However, KeyDB's smaller community, fewer production deployments, and the maintenance overhead of tracking upstream Redis changes are significant considerations. For most organizations, running multiple Redis instances on the same server (sharding manually) provides comparable multithreaded performance with the stability and ecosystem of the original Redis project.

Redis vs. Dragonfly

Dragonfly is a newer in-memory datastore that uses a shared-nothing, multithreaded architecture designed to fully utilize modern multi-core servers. Dragonfly claims to be compatible with Redis and Memcached protocols while achieving significantly higher throughput on large instances (claiming up to 16x throughput improvement on 16-core machines). Dragonfly uses a novel memory management approach based on dash tables and inline allocation that reduces fragmentation. However, Dragonfly is relatively new (first released in 2022) and has a smaller ecosystem, fewer production deployments, and unproven long-term stability compared to Redis. For greenfield projects that need extreme single-instance performance and can tolerate the risk of a newer technology, Dragonfly is worth evaluating.

Choosing the Right Solution

The choice between Redis and its alternatives depends on specific requirements and constraints. Redis is the safest default choice for most use cases due to its maturity, ecosystem breadth, operational tooling, and community support. Memcached is preferred when you need pure multithreaded caching for simple string values and want to avoid Redis's single-threaded overhead on a single server. KeyDB is worth considering when you need Redis compatibility but want native multithreading and active replication without managing multiple Redis instances. Dragonfly is an emerging option for workloads that need extreme single-instance performance and can tolerate the risk of a newer technology stack. In all cases, benchmark with your specific workload, evaluate operational maturity for your team, and consider the long-term maintenance implications of each choice.

17. Interview Q&A: System Design with Redis

Redis is one of the most frequently tested topics in system design interviews at all levels, from mid-level through principal engineer. The following questions cover the most common Redis-related interview scenarios, providing concise, high-quality answers that demonstrate senior-level understanding.

Q1: How would you design a rate limiter using Redis?

A rate limiter using Redis can be implemented using several approaches, each with different precision and performance trade-offs. The simplest approach is the Fixed Window Counter using INCR with a TTL: increment a counter for each request, and set an expiry on the key equal to the window duration. When the counter exceeds the limit, reject the request. This is easy to implement but suffers from boundary issues (a burst at the edge of one window can effectively double the rate). The Sliding Window Log approach uses a sorted set where scores are timestamps — for each request, remove expired entries and count remaining entries. This provides precise rate limiting but uses more memory. The Sliding Window Counter combines both approaches by interpolating between the previous and current window counts, providing a good balance of precision and memory usage. For distributed rate limiting across multiple Redis instances, use a Lua script to ensure atomicity of the check-and-increment operation.

Q2: How do you prevent cache stampede in a high-traffic system?

Cache stampede occurs when a popular cache key expires and concurrent requests all miss the cache simultaneously, overwhelming the backend database. Several strategies prevent this. Mutex locking uses a Redis distributed lock (SETNX with expiry) so that only one process rebuilds the cache while others wait or serve stale data. Probabilistic early expiration randomly refreshes the cache before it expires, spreading the refresh load over time. The Refresh-ahead pattern proactively refreshes popular keys before they expire using a background worker. Request coalescing at the application layer deduplicates concurrent requests for the same key. Using stale-while-revalidate (serve the stale data while asynchronously refreshing) provides the best user experience. In practice, a combination of these approaches — a mutex lock plus a background refresh worker plus a short grace period for stale data — provides the most robust protection.

Q3: Explain Redis Cluster's consistency guarantees and the CAP theorem implications.

Redis Cluster prioritizes availability and partition tolerance over consistency, making it an AP system in the CAP theorem. During a network partition, the cluster may continue to serve reads from the minority partition (using stale replicas) while the majority partition handles writes. This means that during partitions, different clients may see different data depending on which partition they are connected to. Writes to the primary are acknowledged immediately without waiting for replica confirmation (asynchronous replication), so a primary failure immediately after a write can lose that write even in the absence of partitions. The WAIT command can be used to require replica acknowledgment, trading availability for stronger consistency. For most use cases, Redis's eventual consistency is acceptable because the data is either cacheable (and will eventually become consistent) or can tolerate brief inconsistencies. For use cases requiring strong consistency, use WATCH/MULTI/EXEC for single-node operations or Lua scripts with careful consideration of cluster slot constraints.

Q4: Design a distributed session store using Redis for a web application with 100 million users.

A distributed session store for 100 million users requires careful consideration of memory, throughput, and availability. Store sessions using Redis hashes (HSET) for structured session data, which provides memory-efficient encoding for small hashes. Set TTLs on all session keys to ensure automatic cleanup of expired sessions (e.g., 24 hours for active sessions). Use Redis Cluster with enough shards to distribute the load — for 100 million sessions at an average of 500 bytes each, you need approximately 50 GB, which fits on 3-4 nodes with comfortable headroom. Configure maxmemory-policy to allkeys-lru so that the most inactive sessions are evicted when memory is full. Use Sentinel or Cluster for high availability. Implement session affinity (sticky sessions) at the load balancer level to reduce cross-shard session access. For session data, use async replication with WAIT to balance durability and performance. Monitor eviction rates, memory usage, and session hit rates to ensure SLA compliance.

Q5: How would you implement a distributed lock using Redis? What are the pitfalls?

A distributed lock using Redis is implemented with the SET command: SET lock:resource_name unique_value NX PX 30000. The NX flag ensures the key is only set if it does not exist (mutual exclusion), PX sets a millisecond expiry (prevents deadlock if the lock holder crashes), and unique_value identifies the lock holder. To release the lock, use a Lua script that checks the unique_value before deleting: if redis.call("GET", KEYS[1]) == ARGV[1] then redis.call("DEL", KEYS[1]) end. The main pitfalls include: lock expiry before the critical section completes (leading to two processes holding the lock simultaneously), clock drift in distributed environments, and the lack of fencing tokens. Redlock (proposed by Redis's creator) attempts to solve the clock drift problem by acquiring the lock on multiple independent Redis instances, but it has been criticized by Martin Kleppmann for not providing sufficient safety guarantees under certain failure scenarios. For most applications, a single Redis instance lock with careful expiry management is sufficient, and Redlock should only be considered when the cost of incorrect locking is high.

Q6: Design a real-time leaderboards system for a gaming platform with 50 million players.

A real-time leaderboard using Redis sorted sets provides O(log N) updates and O(log N + M) range queries. Use ZADD to update player scores and ZREVRANK to get a player's position. For the full leaderboard, use ZREVRANGE. With 50 million players, a single sorted set can handle the data (approximately 10-20 GB depending on key sizes), but may not handle the throughput alone. Partition the leaderboard by game or region using separate sorted sets, and aggregate results at the application layer when a global leaderboard is needed. Use Redis Cluster to distribute sorted sets across nodes. For extremely high update rates, batch score updates using pipelining. Implement tier-based leaderboards (Bronze, Silver, Gold) by maintaining separate sorted sets for each tier. Use ZRANGEBYSCORE to find players in specific score ranges for matchmaking. Cache frequently accessed leaderboards (top 100, player's rank) with short TTLs to reduce sorted set query load.

Q7: How do you handle data migration between Redis instances without downtime?

Data migration between Redis instances without downtime uses the MIGRATE command, which atomically transfers a key from one Redis instance to another. For bulk migration, use DUMP to serialize a key and MIGRATE to transfer it atomically. During migration, the source instance serves reads and writes normally while keys are being transferred. For cluster resharding, Redis Cluster handles slot migration online — the source node marks a slot as MIGRATING (still serves existing keys, redirects new keys) and the target node marks the slot as IMPORTING (only accepts keys via ASKING). The redis-cli --cluster reshard tool automates this process. For application-level migration with zero downtime, use a dual-write approach: write to both old and new instances, read from old until migration is complete, then switch reads to new and stop writes to old. Monitor migration progress using SCAN on the source instance and verify data consistency with periodic checksums.

Q8: Explain the memory fragmentation issue in Redis and how to diagnose and fix it.

Memory fragmentation in Redis occurs when jemalloc allocates memory from the OS in pages but cannot reuse freed pages efficiently due to fragmentation of live objects. The fragmentation ratio is calculated as used_memory_rss / used_memory. A ratio of 1.0 means no fragmentation; ratios above 1.5 indicate significant fragmentation that wastes memory. Common causes include: high churn of differently-sized objects, a mix of long-lived and short-lived keys, and aggressive memory reclamation settings. To diagnose fragmentation, use INFO MEMORY to check the mem_fragmentation_ratio, mem_allocator, and active_defrag_running metrics. To fix fragmentation: enable active defragmentation (activedefrag yes) for online defragmentation; restart the instance to completely reset the memory layout (scheduled during maintenance windows); use MEMORY PURGE to request jemalloc to release unused pages; or switch to a different allocator (tcmalloc or jemalloc with different configuration) if the default allocator is not performing well for your workload.

Q9: Design a pub/sub notification system that can handle 1 million concurrent subscribers.

A pub/sub system for 1 million concurrent subscribers requires careful capacity planning. Redis Pub/Sub broadcasts each message to all subscribers on a channel, so the throughput requirement is: messages_per_second × subscribers_per_channel = total messages delivered. For 100 messages/second on a popular channel with 1 million subscribers, this means 100 million message deliveries per second — far beyond what a single Redis instance can handle. The solution is to shard channels across multiple Redis instances (using consistent hashing or channel name hashing) and have each subscriber connect to the shard responsible for its channels. Each Redis instance handles a fraction of the subscriber base. Alternatively, use Redis Streams with consumer groups for a more durable approach where each subscriber reads independently. For geographic distribution, use Redis pub/sub relay across regions. Monitor memory usage carefully because each subscriber connection consumes memory for its output buffer, and slow subscribers can cause output buffer growth and memory pressure.

Q10: When would you NOT use Redis?

Redis is not always the right choice. Do not use Redis as a primary database when: the dataset exceeds available memory (Redis is fundamentally memory-bound), you need complex joins or relational queries (use a traditional RDBMS), you need strong ACID transactions with rollback across multiple operations (Redis transactions are limited), your workload is write-heavy and the data must survive all failures (use a disk-based system like PostgreSQL), you need full-text search on large document collections (use Elasticsearch or Solr), or you need graph traversal queries on large graphs (use Neo4j or a graph database). Also avoid Redis when: operational simplicity is paramount and you want a managed database service, when latency requirements are not sub-millisecond (a traditional database with proper indexing may suffice), and when data durability is the primary concern (Redis persistence has known limitations). Redis excels as a cache, session store, rate limiter, real-time analytics engine, message broker for small-scale workloads, and as a fast data layer between applications and slower persistent storage — but it should not be the default for all data storage needs.