1. Introduction — Why Distributed Locks Matter

In modern cloud-native systems, applications are deployed across dozens or even hundreds of servers. When multiple instances of a service need to coordinate access to a shared resource — a database row, a file, a message queue partition, an API endpoint — we need a mechanism that guarantees only one process can act at a time. That mechanism is a distributed lock.

A distributed lock is the distributed-systems analog of a mutex or semaphore in single-process programming. But whereas a mutex relies on shared memory and OS-level primitives like pthread_mutex_lock, a distributed lock must operate across process boundaries, across machine boundaries, and sometimes across data-center boundaries — with no shared memory at all.

Single-Node vs. Distributed

Consider a single Node.js process processing background jobs. A simple in-memory boolean flag or a Redis SETNX from one connection is enough. But once you scale to 50 instances of that service behind a load balancer, that in-memory flag is invisible to every other instance. You need an external coordination service that all instances agree on.

Use Cases for Distributed Locks

• Leader Election: In a cluster of worker nodes, only one should be the active leader for tasks like cron scheduling or partition assignment (e.g., Kafka consumer group coordination).
• Rate Limiting: Ensuring a global rate limit of, say, 1000 requests per second across all service instances requires atomic counter manipulation protected by a lock.
• Inventory Reservation: An e-commerce flash sale must prevent two customers from purchasing the last item simultaneously. A distributed lock on the SKU guarantees atomic decrement.
• Unique Resource Creation: Creating a unique username or booking a flight seat requires check-then-act logic protected by a lock to prevent TOCTOU (time-of-check-to-time-of-use) races.
• Distributed Cron: A scheduled job should run exactly once across a fleet of nodes, even if nodes fail and restart.
• Session Management: Preventing duplicate session processing in OAuth flows or payment processing pipelines.

Without distributed locks, systems resort to fragile polling, database deadlocks, or silent data corruption — all of which become exponentially harder to debug at scale.

2. Functional & Non-Functional Requirements

Functional Requirements

Non-Functional Requirements

3. Capacity Estimation

Before designing the system, we need to size it properly. Let's estimate the load for a mid-scale distributed lock service.

Assumptions

Storage Estimation

Network Estimation

Each lock operation involves a request-response round trip. At 5,000 peak QPS with an average payload of 512 bytes:

• Inbound bandwidth: 5,000 × 512 bytes = 2.56 MB/s ≈ 20.5 Mbps
• Outbound bandwidth (response): 5,000 × 128 bytes = 0.64 MB/s ≈ 5.1 Mbps
• Total: ~26 Mbps — well within a single 1 Gbps NIC

4. Data Model

The data model for a distributed lock service is deceptively simple on the surface but has important nuances for correctness.

Core Entities

// Lock entity
Lock {
    Key:          string    // Unique resource identifier (e.g., "inventory:sku-12345")
    OwnerID:      string    // Unique process/instance identifier
    LockID:       uuid      // Unique ID for this specific lock acquisition
    FencingToken: long      // Monotonically increasing token per key
    AcquiredAt:   timestamp // When the lock was acquired
    ExpiresAt:    timestamp // When the lock auto-expires (TTL)
    Version:      long      // Optimistic concurrency version
    Metadata:     map       // Arbitrary key-value pairs (trace ID, service name)
}

// Lock owner registration
Owner {
    OwnerID:      string    // Instance identifier
    ServiceName:  string    // Owning service
    InstanceID:   string    // Pod/container ID
    HeartbeatAt:  timestamp // Last heartbeat timestamp
    RegisteredAt: timestamp
}

// Audit log entry
AuditLog {
    ID:           uuid
    Key:          string
    OwnerID:      string
    Action:       enum(ACQUIRE, RELEASE, EXTEND, EXPIRE, STEAL)
    FencingToken: long
    Timestamp:    timestamp
    TraceID:      string
    Duration:     duration  // How long the lock was held
}

State Machine