Table of Contents

• 1. Why Distributed Configuration Management Is Hard
• 2. Core Requirements and Functional Scope
• 3. High-Level Architecture Overview
• 4. Consensus Algorithms — Raft and Paxos Simplified
• 5. Key-Value Store Design
• 6. Watch and Notification Mechanism
• 7. Session Management with TTL
• 8. Leader Election
• 9. Distributed Locking
• 10. Config Versioning and Rollback
• 11. Config Composition — Templates and Overrides
• 12. Dynamic Config Without Restarts
• 13. Config Validation and Schema Enforcement
• 14. Access Control — Role-Based Access Control
• 15. Audit Logging
• 16. Multi-Datacenter Replication
• 17. Health Checking and Failure Detection
• 18. Client SDK Design
• 19. CLI Tool Design
• 20. GUI Dashboard Design
• 21. Migration from File-Based Config
• 22. Monitoring and Observability
• 23. Security — TLS and Encryption at Rest
• 24. Compliance and Regulatory Considerations
• 25. API Design
• 26. Testing Strategy
• 27. Cost Estimation and Capacity Planning
• 28. Interview Q&A
• 29. Summary and Key Takeaways

1. Why Distributed Configuration Management Is Hard

Every production system relies on configuration. Database connection strings, feature flags, rate limits, circuit breaker thresholds, authentication providers, logging levels — these are the knobs that operators turn to keep systems running smoothly. In a monolith with a single server, configuration is trivial: read a file, parse it, use it. But the moment you have hundreds of microservice instances spread across multiple data centers, configuration becomes a distributed systems problem in its own right.

A distributed configuration management system must solve several hard problems simultaneously. First, it must guarantee consistency. If a configuration change is made, every node must eventually see the exact same value. A split-brain scenario where half the cluster sees feature_x=true and the other half sees feature_x=false is worse than having no system at all. Second, it must provide high availability. Configuration is critical infrastructure — if the config server goes down, you cannot deploy new services, cannot roll back faulty changes, and in many architectures, services cannot even start. Third, it must handle network partitions gracefully according to the CAP theorem, choosing the right consistency-availability tradeoff for the use case.

Beyond these fundamental distributed systems challenges, configuration management introduces its own complexities. Ordering matters — applying config change B before change A might produce an invalid state. Watching matters — services need to react to changes in real-time, not poll every 30 seconds. Access control matters — not every service or developer should be able to modify production database passwords. Audit matters — you need a complete history of who changed what and when, especially for compliance. And rollback matters — a bad config change at 3 AM should be reversible in seconds, not minutes.

Existing solutions like etcd, Consul, and ZooKeeper each take different approaches to these problems, but they all share common foundations: a replicated key-value store, a consensus algorithm for consistency, a watch mechanism for change notification, and a session/lease mechanism for ephemeral state. In this guide, we will design a system that incorporates the best ideas from all three, implemented with modern practices and a C# codebase.

The Problem Space in Numbers

2. Core Requirements and Functional Scope

Before designing any system, we must clearly define what it needs to do. A distributed configuration management system can be decomposed into functional and non-functional requirements.

Functional Requirements

• Key-Value Storage: Store and retrieve arbitrary configuration values organized in a hierarchical namespace (e.g., /services/payment-service/config).
• CRUD Operations: Support Create, Read, Update, Delete, and List operations on keys with strong consistency guarantees.
• Watch/Subscribe: Allow clients to subscribe to changes on a key or a key prefix and receive real-time notifications.
• Transactions: Support atomic multi-key operations with compare-and-swap semantics.
• TTL-based Leases: Allow keys to be associated with a lease that expires after a configurable time-to-live, enabling ephemeral configuration and health tracking.
• Versioning: Every mutation produces a new version, enabling rollback to any historical state.
• Access Control: Fine-grained RBAC policies controlling who can read, write, or delete specific key prefixes.
• Schema Validation: Enforce structure and types on configuration values using JSON Schema or similar mechanisms.
• Audit Logging: Record all mutations with actor identity, timestamp, old value, and new value.
• Multi-Datacenter Replication: Asynchronously replicate configuration across geographically distributed clusters.

Non-Functional Requirements

• Strong Consistency: Linearizable reads and writes within a single datacenter using a consensus algorithm.
• High Availability: No single point of failure. The system must tolerate the loss of any minority of nodes.
• Low Latency: Reads should complete in single-digit milliseconds. Writes in low tens of milliseconds.
• Durability: Committed configurations must survive node restarts and disk failures via replication.
• Scalability: Support millions of keys and tens of thousands of concurrent watchers.
• Observability: Expose metrics, logs, and traces for operational visibility.
• Security: TLS for all communication, encryption at rest for sensitive values, and certificate-based mutual authentication.

3. High-Level Architecture Overview

The system consists of five major components: the Raft consensus cluster, the key-value storage engine, the watch notification system, the API gateway and gRPC layer, and the replication bridge for multi-datacenter support.

Component Responsibilities

Request Flow for a Write Operation

When a client issues a write request (e.g., PUT /kv/services/payment-service/config), the following sequence occurs:

1. The request hits the load balancer and is routed to any node in the cluster.
2. If the receiving node is not the leader, it returns a redirect to the leader's address.
3. The leader validates the request against RBAC policies and schema constraints.
4. The leader appends the operation to its Raft log and replicates it to followers.
5. Once a majority acknowledges the entry, the leader commits and applies it to the finite state machine.
6. The watch system detects the change and notifies all subscribers watching that key or prefix.
7. The leader responds to the client with the new version number.

4. Consensus Algorithms — Raft and Paxos Simplified

The heart of any distributed configuration system is its consensus algorithm. Consensus ensures that all nodes in a cluster agree on a single value, even in the presence of failures. We will focus primarily on Raft because it was designed for understandability and is used by etcd and Consul.

Raft Basics

Raft decomposes consensus into three sub-problems: leader election, log replication, and safety. A Raft cluster contains an odd number of nodes (typically 3 or 5). One node is the leader, and the rest are followers. All writes go through the leader, which appends entries to its log and replicates them to followers.

Leader Election in Detail

Every follower runs an election timer. If the timer fires before receiving a heartbeat from the leader, the follower increments its term number and transitions to the candidate state. The candidate votes for itself and requests votes from all other nodes. If a candidate receives votes from a majority (e.g., 3 out of 5), it becomes the leader. The leader immediately begins sending heartbeats to prevent new elections.

Split-vote scenarios are resolved naturally because each node votes for at most one candidate per term. If no candidate achieves a majority, the election times out and a new election begins with a randomized timeout to break symmetry.

Log Replication

When the leader receives a client write, it appends a new entry to its log containing the operation and the current term number. It then sends AppendEntries RPCs to all followers. Each follower validates that the leader's term is at least as high as its own and that the log entry immediately preceding the new one matches its own log. If both checks pass, the follower appends the entry and acknowledges. Once the leader receives acknowledgments from a majority, it commits the entry and applies it to the state machine.

Raft Implementation in C#

C#
public class RaftNode
{
    private readonly object _lock = new();
    private NodeState _state = NodeState.Follower;
    private int _currentTerm = 0;
    private string? _votedFor;
    private readonly List<LogEntry> _log = new();
    private int _commitIndex = -1;
    private int _lastApplied = -1;
    private readonly Dictionary<string, int> _nextIndex = new();
    private readonly Dictionary<string, int> _matchIndex = new();
    private Timer? _electionTimer;
    private Timer? _heartbeatTimer;
    private readonly string _nodeId;
    private readonly IEnumerable<IRaftPeer> _peers;
    private readonly IStateMachine _stateMachine;

    public RaftNode(string nodeId, IEnumerable<IRaftPeer> peers,
                    IStateMachine stateMachine)
    {
        _nodeId = nodeId;
        _peers = peers;
        _stateMachine = stateMachine;
    }

    public void Start()
    {
        LoadStateFromStorage();
        ResetElectionTimer();
    }

    private void OnElectionTimeout()
    {
        lock (_lock)
        {
            if (_state == NodeState.Leader) return;

            _state = NodeState.Candidate;
            _currentTerm++;
            _votedFor = _nodeId;
            var votesReceived = 1;

            var request = new RequestVoteRequest
            {
                Term = _currentTerm,
                CandidateId = _nodeId,
                LastLogIndex = _log.Count - 1,
                LastLogTerm = _log.Count > 0 ? _log[^1].Term : 0
            };

            foreach (var peer in _peers)
            {
                _ = Task.Run(async =>
                {
                    var response = await peer.RequestVoteAsync(request);
                    lock (_lock)
                    {
                        if (response.Term > _currentTerm)
                        {
                            StepDown(response.Term);
                            return;
                        }
                        if (response.VoteGranted) votesReceived++;
                        if (votesReceived > (_peers.Count() + 1) / 2)
                        {
                            BecomeLeader();
                        }
                    }
                });
            }
            ResetElectionTimer();
        }
    }

    private void BecomeLeader()
    {
        _state = NodeState.Leader;
        _electionTimer?.Dispose();
        foreach (var peer in _peers)
        {
            _nextIndex[peer.NodeId] = _log.Count;
            _matchIndex[peer.NodeId] = -1;
        }
        StartHeartbeat();
    }

    public async Task<AppendEntriesResponse> AppendEntriesAsync(
        AppendEntriesRequest request)
    {
        lock (_lock)
        {
            if (request.Term < _currentTerm)
                return new AppendEntriesResponse
                { Term = _currentTerm, Success = false };

            ResetElectionTimer();

            if (request.Entries.Count == 0)
                return new AppendEntriesResponse
                { Term = _currentTerm, Success = true };

            for (int i = 0; i < request.Entries.Count; i++)
            {
                int logIndex = request.PrevLogIndex + 1 + i;
                if (logIndex < _log.Count)
                {
                    if (_log[logIndex].Term != request.Entries[i].Term)
                        _log.RemoveRange(logIndex, _log.Count - logIndex);
                }
                if (logIndex >= _log.Count)
                    _log.Add(request.Entries[i]);
            }

            if (request.LeaderCommit > _commitIndex)
            {
                _commitIndex = Math.Min(
                    request.LeaderCommit, _log.Count - 1);
                ApplyCommittedEntries();
            }

            return new AppendEntriesResponse
            { Term = _currentTerm, Success = true };
        }
    }

    private void ApplyCommittedEntries()
    {
        while (_lastApplied < _commitIndex)
        {
            _lastApplied++;
            _stateMachine.Apply(_log[_lastApplied].Command);
        }
    }

    private void StepDown(int newTerm)
    {
        _currentTerm = newTerm;
        _state = NodeState.Follower;
        _votedFor = null;
        StartElectionTimer();
    }

    public enum NodeState { Follower, Candidate, Leader }
}

Paxos Simplified

Paxos, proposed by Leslie Lamport in 1989, is the other major consensus algorithm. While theoretically elegant, classic Paxos is notoriously difficult to implement correctly. Multi-Paxos, which optimizes for a stable leader, is used by Google's Chubby and Spanner. The key difference from Raft is that Paxos elects a leader implicitly through the prepare/promise mechanism rather than through a separate election phase.

For a configuration management system, Raft is the preferred choice due to its understandability, the maturity of existing implementations (like the Raft.NET library), and the proven track record in etcd and Consul. The performance characteristics are nearly identical for the write throughput we need.

Choosing Between Raft and Paxos

5. Key-Value Store Design

The key-value store is the storage backbone of the configuration system. It must support efficient reads and writes, maintain a history of changes, and integrate cleanly with the Raft consensus layer.

Data Model

Configuration data is organized in a hierarchical key namespace, similar to a file system. Keys are strings using / as a separator. Values are opaque byte arrays with associated metadata. Each key-value pair has the following metadata:

Storage Engine Architecture

Each Raft node uses a two-tier storage engine. The write-ahead log (WAL) provides durability for uncommitted entries, while a key-value snapshot provides fast startup and point-in-time access to committed state.

C#
public class KeyValueStore : IRaftStateMachine
{
    private readonly ConcurrentDictionary<string, KeyValueEntry> _store = new();
    private readonly ReaderWriterLockSlim _rwLock = new();
    private ulong _appliedIndex = 0;
    private readonly ISnapshotStore _snapshotStore;

    public KeyValueStore(ISnapshotStore snapshotStore)
    {
        _snapshotStore = snapshotStore;
    }

    public void Apply(LogEntry entry)
    {
        _rwLock.EnterWriteLock();
        try
        {
            var cmd = entry.Command;
            switch (cmd.Type)
            {
                case CommandType.Put:
                    var existing = _store.GetValueOrDefault(cmd.Key);
                    var newVersion = existing?.Version + 1 ?? 1;
                    var newEntry = new KeyValueEntry
                    {
                        Key = cmd.Key,
                        Value = cmd.Value,
                        Version = newVersion,
                        ModIndex = entry.Index,
                        CreatedIndex = existing?.CreatedIndex ?? entry.Index,
                        Lease = cmd.LeaseId,
                        Modified = DateTime.UtcNow,
                        Created = existing?.Created ?? DateTime.UtcNow,
                        Creator = cmd.Actor
                    };
                    _store[cmd.Key] = newEntry;
                    break;

                case CommandType.Delete:
                    _store.TryRemove(cmd.Key, out _);
                    break;

                case CommandType.DeleteRange:
                    var prefix = cmd.Key;
                    var keysToDelete = _store.Keys
                        .Where(k => k.StartsWith(prefix)).ToList();
                    foreach (var key in keysToDelete)
                        _store.TryRemove(key, out _);
                    break;
            }
            _appliedIndex = entry.Index;
        }
        finally { _rwLock.ExitWriteLock(); }
    }

    public KeyValueEntry? Get(string key)
    {
        _rwLock.EnterReadLock();
        try { return _store.GetValueOrDefault(key); }
        finally { _rwLock.ExitReadLock(); }
    }

    public List<KeyValueEntry> GetRange(string prefix)
    {
        _rwLock.EnterReadLock();
        try
        {
            return _store.Values
                .Where(e => e.Key.StartsWith(prefix))
                .OrderBy(e => e.Key)
                .ToList();
        }
        finally { _rwLock.ExitReadLock(); }
    }

    public void CreateSnapshot()
    {
        _rwLock.EnterReadLock();
        try
        {
            var snapshot = new Snapshot
            {
                Index = _appliedIndex,
                Data = _store.ToDictionary(kv => kv.Key, kv => kv.Value)
            };
            _snapshotStore.Save(snapshot);
        }
        finally { _rwLock.ExitReadLock(); }
    }
}

Snapshotting Strategy

As the Raft log grows, replaying the entire log on node restart becomes prohibitively slow. Snapshots address this by capturing the full state of the key-value store at a point in time. After a snapshot is saved, all log entries up to that point can be discarded. We trigger a snapshot when the log exceeds 10,000 entries or when 10 minutes have elapsed since the last snapshot.

Key Encoding

Keys are stored in lexicographically sorted order using a modified prefix-compression scheme. This makes prefix scans efficient because keys with the same prefix are stored adjacently on disk. The encoding uses null bytes as path separators internally, while exposing forward-slash-separated paths in the API.

C#
public static class KeyCodec
{
    public static byte[] Encode(string key)
    {
        var parts = key.Split('/');
        using var stream = new MemoryStream();
        foreach (var part in parts)
        {
            var bytes = Encoding.UTF8.GetBytes(part);
            stream.WriteByte((byte)bytes.Length);
            stream.Write(bytes, 0, bytes.Length);
            stream.WriteByte(0x00);
        }
        return stream.ToArray();
    }

    public static string Decode(byte[] encoded)
    {
        using var stream = new MemoryStream(encoded);
        var parts = new List<string>();
        while (stream.Position < stream.Length)
        {
            int len = stream.ReadByte();
            if (len == 0) break;
            var bytes = new byte[len];
            stream.Read(bytes, 0, len);
            parts.Add(Encoding.UTF8.GetString(bytes));
            stream.ReadByte(); // skip null separator
        }
        return string.Join('/', parts);
    }
}

6. Watch and Notification Mechanism

The watch mechanism is arguably the most critical feature of a configuration management system. Without watches, clients would need to poll for changes, creating unnecessary load and introducing latency. A well-designed watch system delivers change notifications to clients within milliseconds of a commit.

Watch Architecture

Watch Registration and Matching

Watches are registered on a key prefix. When a change occurs at any key that matches the prefix, a notification is generated. The watch registry uses a trie (prefix tree) data structure for efficient prefix matching. Each node in the trie holds a list of active watchers for that exact prefix.

C#
public class WatchRegistry
{
    private readonly TrieNode _root = new();
    private readonly ConcurrentDictionary<string, WatchRegistration> _watches = new();
    private readonly SemaphoreSlim _notificationSemaphore = new(100);

    public string Register(string prefix, bool recursive,
        bool includePrevious, Channel<WatchEvent> channel)
    {
        var watchId = Guid.NewGuid().ToString("N");
        var registration = new WatchRegistration
        {
            Id = watchId,
            Prefix = prefix,
            Recursive = recursive,
            IncludePrevious = includePrevious,
            Channel = channel,
            RegisteredAt = DateTime.UtcNow
        };
        _watches[watchId] = registration;
        _root.Insert(prefix, watchId);
        return watchId;
    }

    public void Unregister(string watchId)
    {
        if (_watches.TryRemove(watchId, out var reg))
            _root.Remove(reg.Prefix, watchId);
    }

    public async Task<int> NotifyAsync(string key, WatchEventType eventType,
        KeyValueEntry? oldValue, KeyValueEntry? newValue)
    {
        var matchingWatchIds = _root.FindMatchingWatchers(key);
        var notified = 0;

        foreach (var watchId in matchingWatchIds)
        {
            if (!_watches.TryGetValue(watchId, out var reg)) continue;
            if (!reg.Recursive && reg.Prefix != key) continue;

            var watchEvent = new WatchEvent
            {
                Type = eventType,
                Key = key,
                KeyValue = newValue,
                PreviousKeyValue = reg.IncludePrevious ? oldValue : null,
                ModRevision = newValue?.ModIndex ?? 0,
                Timestamp = DateTime.UtcNow
            };

            await _notificationSemaphore.WaitAsync();
            try
            {
                await reg.Channel.Writer.WriteAsync(watchEvent);
                Interlocked.Increment(ref notified);
            }
            catch (ChannelClosedException)
            {
                _watches.TryRemove(watchId, out _);
            }
            finally { _notificationSemaphore.Release(); }
        }
        return notified;
    }
}

public class TrieNode
{
    private readonly ConcurrentDictionary<char, TrieNode> _children = new();
    private readonly ConcurrentBag<string> _watchIds = new();

    public void Insert(string prefix, string watchId)
    {
        var node = this;
        foreach (var c in prefix)
            node = node._children.GetOrAdd(c, _ => new TrieNode());
        node._watchIds.Add(watchId);
    }

    public IEnumerable<string> FindMatchingWatchers(string key)
    {
        var results = new List<string>();
        FindMatchingWatchersRecursive(this, key, 0, results);
        return results;
    }

    private void FindMatchingWatchersRecursive(TrieNode node, string key,
        int index, List<string> results)
    {
        results.AddRange(node._watchIds);
        if (index >= key.Length) return;
        if (node._children.TryGetValue(key[index], out var child))
            FindMatchingWatchersRecursive(child, key, index + 1, results);
    }
}

Watch Event Delivery Guarantees

• At-least-once delivery: Events may be delivered more than once in rare cases (e.g., leader failover). Clients must handle duplicates using the mod-revision field.
• Ordered delivery: Events for a single watch are delivered in mod-revision order. No event with a lower mod-revision will be delivered after one with a higher revision.
• No gap guarantee: If the client disconnects, it will miss events during the disconnection. Upon reconnection, it can resume from the last observed mod-revision.
• Backpressure: If a client cannot keep up, the server will buffer events up to a configurable limit (default 1,000 events). If the buffer fills, the watch is closed and the client must re-register.

7. Session Management with TTL

Sessions provide a way for clients to establish a long-lived connection with the configuration cluster and to create ephemeral keys that are automatically deleted when the session expires. This is essential for service registration, health tracking, and distributed coordination patterns.

Session Lifecycle

TTL and KeepAlive

Each session is granted a time-to-live (TTL), typically 30 seconds. The client must send a KeepAlive RPC before the TTL expires to renew the session. If the cluster does not receive a KeepAlive within the TTL window, the session is marked as expired and all ephemeral keys associated with it are deleted.

C#
public class SessionManager
{
    private readonly ConcurrentDictionary<long, Session> _sessions = new();
    private readonly Timer _reapTimer;
    private readonly TimeSpan _defaultTtl = TimeSpan.FromSeconds(30);
    private long _nextSessionId = 0;

    public SessionManager()
    {
        _reapTimer = new Timer(ReapExpiredSessions, null,
            TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(5));
    }

    public Session CreateSession(TimeSpan? ttl = null, string? actor = null)
    {
        var sessionId = Interlocked.Increment(ref _nextSessionId);
        var session = new Session
        {
            Id = sessionId,
            Ttl = ttl ?? _defaultTtl,
            LastKeepAlive = DateTime.UtcNow,
            ExpiresAt = DateTime.UtcNow + (ttl ?? _defaultTtl),
            Actor = actor,
            EphemeralKeys = new List<string>()
        };
        _sessions[sessionId] = session;
        return session;
    }

    public bool KeepAlive(long sessionId)
    {
        if (!_sessions.TryGetValue(sessionId, out var session))
            return false;
        session.LastKeepAlive = DateTime.UtcNow;
        session.ExpiresAt = DateTime.UtcNow + session.Ttl;
        return true;
    }

    public List<string> RevokeSession(long sessionId)
    {
        if (!_sessions.TryRemove(sessionId, out var session))
            return new List<string>();
        return session.EphemeralKeys.ToList();
    }

    private void ReapExpiredSessions(object? state)
    {
        var now = DateTime.UtcNow;
        var expired = _sessions.Values
            .Where(s => s.ExpiresAt < now).ToList();

        foreach (var session in expired)
        {
            if (_sessions.TryRemove(session.Id, out _))
            {
                foreach (var key in session.EphemeralKeys)
                    _ = IssueDeleteCommand(key);
            }
        }
    }
}

public class Session
{
    public long Id { get; set; }
    public TimeSpan Ttl { get; set; }
    public DateTime LastKeepAlive { get; set; }
    public DateTime ExpiresAt { get; set; }
    public string? Actor { get; set; }
    public List<string> EphemeralKeys { get; set; } = new();
}

Ephemeral Keys

Ephemeral keys are tied to a session. When the session expires (or is explicitly revoked), all ephemeral keys are automatically deleted. This enables patterns like service registration, where a service registers its endpoint as an ephemeral key and automatically deregisters when it crashes.

8. Leader Election

Leader election serves two purposes in our system. First, the Raft consensus algorithm requires a leader to coordinate log replication. Second, applications built on top of the configuration system often need their own leader election — for example, to elect a single worker for a background job or to coordinate distributed tasks.

Internal Leader Election (Raft)

Raft's built-in leader election is described in detail in Section 4. The key point is that the Raft leader is the only node that can accept write operations. All other nodes proxy writes to the leader or redirect the client.

Application-Level Leader Election

Beyond the Raft leader, the configuration system provides primitives for applications to implement their own leader election. This uses a compare-and-swap (CAS) operation on a designated key:

C#
public class DistributedLeaderElection
{
    private readonly IConfigClient _client;
    private readonly string _electionKey;
    private readonly string _candidateId;
    private readonly TimeSpan _leaseTtl;

    public DistributedLeaderElection(IConfigClient client, string electionKey,
        string candidateId, TimeSpan leaseTtl)
    {
        _client = client;
        _electionKey = electionKey;
        _candidateId = candidateId;
        _leaseTtl = leaseTtl;
    }

    public async Task<bool> TryAcquireLeadershipAsync()
    {
        var lease = await _client.GrantLeaseAsync(_leaseTtl);

        var result = await _client.TxnAsync(
            comparisons: new[]
            {
                TxnComparison.If(_electionKey, CompareOp.Version,
                    CompareResult.Equal, 0)
            },
            success: new[]
            {
                TxnOp.Put(_electionKey,
                    Encoding.UTF8.GetBytes(_candidateId), lease.Id)
            }
        );
        return result.Succeeded;
    }

    public async Task<bool> RenewLeadershipAsync()
    {
        var currentValue = await _client.GetAsync(_electionKey);
        if (currentValue == null ||
            Encoding.UTF8.GetString(currentValue.Value) != _candidateId)
            return false;

        var lease = await _client.GrantLeaseAsync(_leaseTtl);
        await _client.PutAsync(_electionKey, currentValue.Value, lease.Id);
        return true;
    }

    public async Task<string?> GetCurrentLeaderAsync()
    {
        var entry = await _client.GetAsync(_electionKey);
        return entry != null ? Encoding.UTF8.GetString(entry.Value) : null;
    }
}

Leader Election Flow

9. Distributed Locking

Distributed locks are essential for coordinating access to shared resources across multiple service instances. Our configuration system provides a Redlock-style distributed lock built on top of the key-value store and lease mechanism.

Lock Acquisition Protocol

1. Generate a unique lock value (UUID) to identify the lock holder.
2. Attempt a CAS operation: if the lock key does not exist, set it to the lock value with a TTL lease.
3. If the CAS succeeds, the lock is acquired. Set a local "lock expiration" timer at roughly 2/3 of the TTL.
4. Before the local timer fires, attempt to extend the lease (KeepAlive).
5. To release, perform a CAS that checks the value matches the lock value before deleting — preventing release of someone else's lock.

C#
public class DistributedLock : IAsyncDisposable
{
    private readonly IConfigClient _client;
    private readonly string _lockKey;
    private readonly string _lockValue;
    private readonly TimeSpan _ttl;
    private Timer? _renewalTimer;
    private long _leaseId;
    private bool _acquired;

    public DistributedLock(IConfigClient client, string lockKey, TimeSpan ttl)
    {
        _client = client;
        _lockKey = lockKey;
        _lockValue = Guid.NewGuid().ToString("N");
        _ttl = ttl;
    }

    public async Task<bool> AcquireAsync(CancellationToken ct = default)
    {
        while (!ct.IsCancellationRequested)
        {
            var lease = await _client.GrantLeaseAsync(_ttl);
            var result = await _client.TxnAsync(
                comparisons: new[]
                {
                    TxnComparison.If(_lockKey, CompareOp.Version,
                        CompareResult.Equal, 0)
                },
                success: new[]
                {
                    TxnOp.Put(_lockKey,
                        Encoding.UTF8.GetBytes(_lockValue), lease.Id)
                }
            );

            if (result.Succeeded)
            {
                _leaseId = lease.Id;
                _acquired = true;
                StartRenewal();
                return true;
            }

            await Task.Delay(
                TimeSpan.FromMilliseconds(100 + Random.Shared.Next(50)), ct);
        }
        return false;
    }

    public async Task<bool> ReleaseAsync()
    {
        if (!_acquired) return false;
        StopRenewal();

        var currentValue = await _client.GetAsync(_lockKey);
        if (currentValue == null ||
            Encoding.UTF8.GetString(currentValue.Value) != _lockValue)
            return false;

        var result = await _client.TxnAsync(
            comparisons: new[]
            {
                TxnComparison.If(_lockKey, CompareOp.Value,
                    CompareResult.Equal,
                    Encoding.UTF8.GetBytes(_lockValue))
            },
            success: new[] { TxnOp.Delete(_lockKey) }
        );

        _acquired = false;
        return result.Succeeded;
    }

    private void StartRenewal()
    {
        var renewalInterval = TimeSpan.FromTicks(_ttl.Ticks / 3);
        _renewalTimer = new Timer(async _ =>
        {
            if (_acquired)
                await _client.KeepAliveAsync(_leaseId);
        }, null, renewalInterval, renewalInterval);
    }

    private void StopRenewal() => _renewalTimer?.Dispose();

    public async ValueTask DisposeAsync()
    {
        if (_acquired) await ReleaseAsync();
        StopRenewal();
    }
}

10. Config Versioning and Rollback

Configuration changes in production carry risk. A single bad change can cascade into an outage. Versioning and rollback capabilities provide a safety net that lets operators revert changes in seconds.

Global Versioning

Every write operation to the Raft log is assigned a monotonically increasing index. This index serves as the global version of the system. Each key-value pair also maintains its own per-key version that increments with each modification to that specific key.

Configuration History

The system maintains a configurable history of configuration changes. Each historical entry includes the full old value, new value, the actor who made the change, the timestamp, and the global version at which the change was applied.

C#
public class ConfigVersionManager
{
    private readonly IKeyValueStore _store;
    private readonly IConfigHistoryStore _history;
    private readonly int _maxHistoryPerKey;

    public ConfigVersionManager(IKeyValueStore store,
        IConfigHistoryStore history, int maxHistoryPerKey = 100)
    {
        _store = store;
        _history = history;
        _maxHistoryPerKey = maxHistoryPerKey;
    }

    public async Task<ConfigChangeResult> SetAsync(
        string key, byte[] value, string actor,
        ulong? expectedVersion = null)
    {
        var oldValue = _store.Get(key);

        if (expectedVersion.HasValue && oldValue != null
            && oldValue.Version != expectedVersion.Value)
        {
            throw new CASConflictException(
                $"Expected version {expectedVersion.Value} " +
                $"but found {oldValue.Version}");
        }

        var command = new ConfigCommand
        {
            Type = CommandType.Put,
            Key = key,
            Value = value,
            Actor = actor
        };

        var result = await _store.ApplyCommandAsync(command);
        var newValue = _store.Get(key)!;

        await _history.RecordAsync(new ConfigHistoryEntry
        {
            Key = key,
            OldValue = oldValue?.Value,
            NewValue = value,
            OldVersion = oldValue?.Version ?? 0,
            NewVersion = newValue.Version,
            GlobalIndex = result.Index,
            Actor = actor,
            Timestamp = DateTime.UtcNow,
            ChangeType = oldValue == null
                ? ChangeType.Created : ChangeType.Updated
        });

        await TrimHistoryAsync(key);
        return new ConfigChangeResult
        {
            Version = newValue.Version,
            GlobalIndex = result.Index
        };
    }

    public async Task<ConfigChangeResult> RollbackAsync(
        string key, ulong targetVersion, string actor)
    {
        var targetEntry = await _history.GetAsync(key, targetVersion);
        if (targetEntry == null)
            throw new NotFoundException(
                $"No history for key {key} at version {targetVersion}");

        var value = targetEntry.OldValue
            ?? throw new InvalidOperationException(
                "Cannot rollback: no previous value exists");

        return await SetAsync(key, value, actor);
    }

    public async Task<List<ConfigHistoryEntry>> GetHistoryAsync(
        string key, int? limit = null)
    {
        return await _history.GetHistoryAsync(
            key, limit ?? _maxHistoryPerKey);
    }

    private async Task TrimHistoryAsync(string key)
    {
        var count = await _history.GetCountAsync(key);
        if (count > _maxHistoryPerKey)
            await _history.TrimAsync(key, count - _maxHistoryPerKey);
    }
}

Rollback Strategies

11. Config Composition — Templates and Overrides

Real-world configuration is rarely a flat set of key-value pairs. Services have base configurations (templates), environment-specific overrides, feature flag layers, and deployment-specific values. Config composition is the process of merging these layers into a final effective configuration.

Composition Hierarchy

C#
public class ConfigComposer
{
    private readonly IConfigClient _client;
    private readonly JsonMerger _jsonMerger;

    public ConfigComposer(IConfigClient client)
    {
        _client = client;
        _jsonMerger = new JsonMerger();
    }

    public async Task<Dictionary<string, object>> ResolveAsync(
        string serviceName, string environment, string region,
        string? instanceId = null)
    {
        var layers = new List<ConfigLayer>
        {
            await FetchLayer($"/templates/{serviceName}"),
            await FetchLayer($"/env/{environment}/{serviceName}"),
            await FetchLayer($"/regions/{region}/{serviceName}"),
        };

        if (instanceId != null)
            layers.Add(await FetchLayer(
                $"/instances/{instanceId}/{serviceName}"));

        return MergeLayers(layers.Where(l => l != null).ToList());
    }

    private async Task<ConfigLayer?> FetchLayer(string prefix)
    {
        var entries = await _client.GetRangeAsync(prefix);
        if (entries.Count == 0) return null;

        var dict = new Dictionary<string, object>();
        foreach (var entry in entries)
        {
            var relativeKey = entry.Key[(prefix.Length + 1)..];
            dict[relativeKey] = JsonSerializer.Deserialize<object>(
                entry.Value);
        }
        return new ConfigLayer { Prefix = prefix, Values = dict };
    }

    private Dictionary<string, object> MergeLayers(
        List<ConfigLayer> layers)
    {
        var result = new Dictionary<string, object>();
        foreach (var layer in layers)
            result = _jsonMerger.Merge(result, layer.Values);
        return result;
    }
}

public class ConfigLayer
{
    public string Prefix { get; set; } = "";
    public Dictionary<string, object> Values { get; set; } = new();
}

Override Rules

• Deeper layers override shallower layers. Instance overrides take precedence over region overrides, which take precedence over environment overrides.
• Explicit null means "remove this key". If a deeper layer sets a key to null, it removes that key from the effective config, even if a template provides it.
• Arrays are replaced, not merged. If a template defines ["a","b"] and an override defines ["c"], the result is ["c"].
• Nested objects are deep-merged. If a template has {"db": {"host": "x", "port": 5432}} and an override has {"db": {"host": "y"}}, the result is {"db": {"host": "y", "port": 5432}}.

12. Dynamic Config Without Restarts

The entire purpose of a distributed configuration system is to allow services to change their behavior at runtime without restarts. This section describes the patterns for achieving truly dynamic configuration.

Reactive Configuration Pattern

The recommended pattern is reactive configuration: the service subscribes to configuration changes via a watch and updates its internal state in response. This eliminates both the restart and the polling.

C#
public class DynamicConfigHolder<T> where T : class
{
    private readonly IConfigClient _client;
    private readonly string _configKey;
    private readonly Func<string, T> _deserializer;
    private readonly ILogger<DynamicConfigHolder<T>> _logger;
    private volatile T _currentValue;
    private long _currentVersion;

    public DynamicConfigHolder(IConfigClient client, string configKey,
        T initialValue, Func<string, T> deserializer,
        ILogger<DynamicConfigHolder<T>> logger)
    {
        _client = client;
        _configKey = configKey;
        _currentValue = initialValue;
        _deserializer = deserializer;
        _logger = logger;
    }

    public T Value => _currentValue;
    public long Version => Interlocked.Read(ref _currentVersion);

    public event Action<T, T>? OnConfigChanged;

    public async Task StartWatchingAsync()
    {
        var current = await _client.GetAsync(_configKey);
        if (current != null)
        {
            _currentValue = _deserializer(
                Encoding.UTF8.GetString(current.Value));
            Interlocked.Exchange(ref _currentVersion,
                (long)current.Version);
        }

        await _client.WatchAsync(_configKey, async watchEvent =>
        {
            try
            {
                var oldValue = _currentValue;
                if (watchEvent.Type == WatchEventType.Delete)
                {
                    _logger.LogWarning(
                        "Config {Key} deleted, keeping previous value",
                        _configKey);
                    return;
                }

                var newValue = _deserializer(
                    Encoding.UTF8.GetString(watchEvent.KeyValue!.Value));

                Interlocked.Exchange(ref _currentValue, newValue);
                Interlocked.Exchange(ref _currentVersion,
                    (long)watchEvent.KeyValue.Version);

                _logger.LogInformation(
                    "Config {Key} updated version {New}",
                    _configKey, watchEvent.KeyValue.Version);

                OnConfigChanged?.Invoke(oldValue, newValue);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex,
                    "Failed to apply config change for {Key}", _configKey);
            }
        });
    }
}

Configuration Validation Before Apply

C#
public class PaymentServiceConfig
{
    [JsonPropertyName("max_retry_attempts")]
    [Range(1, 10, ErrorMessage = "Retry attempts must be 1-10")]
    public int MaxRetryAttempts { get; set; } = 3;

    [JsonPropertyName("timeout_ms")]
    [Range(100, 60000, ErrorMessage = "Timeout must be 100-60000ms")]
    public int TimeoutMs { get; set; } = 5000;

    [JsonPropertyName("circuit_breaker")]
    public CircuitBreakerConfig CircuitBreaker { get; set; } = new();

    public class CircuitBreakerConfig
    {
        [JsonPropertyName("failure_threshold")]
        public int FailureThreshold { get; set; } = 5;

        [JsonPropertyName("recovery_timeout_s")]
        [Range(1, 300)]
        public int RecoveryTimeoutSeconds { get; set; } = 30;
    }
}

13. Config Validation and Schema Enforcement

Schema enforcement ensures that configuration values conform to expected structures and types. This prevents entire classes of runtime errors caused by invalid configuration.

Schema Definition

Configuration schemas are defined using JSON Schema and stored alongside the configuration data in the key-value store. When a write request arrives, the system validates the new value against the registered schema before committing it to the Raft log.

JSON
{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "type": "object",
    "properties": {
        "database": {
            "type": "object",
            "properties": {
                "host": { "type": "string", "format": "hostname" },
                "port": { "type": "integer", "minimum": 1, "maximum": 65535 },
                "pool_size": {
                    "type": "integer", "minimum": 1, "maximum": 100
                },
                "ssl_mode": {
                    "type": "string",
                    "enum": ["disable", "require", "verify-ca", "verify-full"]
                }
            },
            "required": ["host", "port"]
        },
        "features": {
            "type": "object",
            "patternProperties": {
                "^[a-z_]+$": { "type": "boolean" }
            }
        },
        "rate_limits": {
            "type": "object",
            "properties": {
                "requests_per_second": { "type": "integer", "minimum": 1 },
                "burst_size": { "type": "integer", "minimum": 1 }
            }
        }
    },
    "required": ["database"]
}

Schema Registry

C#
public class SchemaRegistry
{
    private readonly IKeyValueStore _store;
    private readonly ConcurrentDictionary<string, JsonSchema> _schemaCache
        = new();
    private const string SchemaPrefix = "/schemas/";

    public SchemaRegistry(IKeyValueStore store)
    {
        _store = store;
    }

    public async Task RegisterSchemaAsync(string keyPrefix,
        string schemaJson)
    {
        var schema = JsonSchema.FromText(schemaJson);
        _schemaCache[keyPrefix] = schema;
        await _store.PutAsync($"{SchemaPrefix}{keyPrefix}",
            Encoding.UTF8.GetBytes(schemaJson));
    }

    public async Task<SchemaValidationResult> ValidateAsync(
        string key, byte[] value)
    {
        var applicableSchema = await FindSchemaFor(key);
        if (applicableSchema == null)
            return new SchemaValidationResult { IsValid = true };

        var valueJson = Encoding.UTF8.GetString(value);
        var result = applicableSchema.Validate(valueJson);

        return new SchemaValidationResult
        {
            IsValid = result.Count == 0,
            Errors = result.Select(e => e.Kind + ": " + e.Message).ToList()
        };
    }

    private async Task<JsonSchema?> FindSchemaFor(string key)
    {
        var match = _schemaCache.Keys
            .Where(p => key.StartsWith(p))
            .OrderByDescending(p => p.Length)
            .FirstOrDefault();

        if (match != null) return _schemaCache[match];

        var stored = await _store.GetRangeAsync(SchemaPrefix);
        foreach (var entry in stored)
        {
            var prefix = entry.Key[(SchemaPrefix.Length)..];
            var schema = JsonSchema.FromText(
                Encoding.UTF8.GetString(entry.Value));
            _schemaCache[prefix] = schema;
            if (key.StartsWith(prefix)) return schema;
        }
        return null;
    }
}

14. Access Control — Role-Based Access Control

In a multi-tenant configuration management system, not every client should have access to every configuration key. Access control ensures that only authorized clients can read or modify specific configuration namespaces.

RBAC Model

Our RBAC implementation supports three core concepts: roles, policies, and bindings.

• Role: A named collection of permissions (e.g., config-admin, config-reader, payment-service-editor).
• Policy: A permission rule specifying a key prefix and the allowed operations (Read, Write, Delete, Watch).
• Role Binding: Associates a role with a principal (user, service account, or certificate CN) within a scope.

C#
public class RbacAuthorizer
{
    private readonly IKeyValueStore _store;
    private const string PolicyPrefix = "/rbac/policies/";
    private const string RolePrefix = "/rbac/roles/";
    private const string BindingPrefix = "/rbac/bindings/";

    public RbacAuthorizer(IKeyValueStore store)
    {
        _store = store;
    }

    public async Task<bool> AuthorizeAsync(
        string principal, string key, Permission permission)
    {
        var bindings = await GetBindingsForAsync(principal);
        foreach (var binding in bindings)
        {
            var role = await GetRoleAsync(binding.RoleName);
            if (role == null) continue;

            if (binding.Namespace != null && !key.StartsWith(binding.Namespace))
                continue;

            foreach (var policy in role.Policies)
            {
                if (key.StartsWith(policy.KeyPrefix) &&
                    policy.AllowedPermissions.Contains(permission))
                    return true;
            }
        }
        return false;
    }

    public async Task CreateRoleAsync(string roleName,
        List<RbacPolicy> policies)
    {
        var role = new RbacRole { Name = roleName, Policies = policies };
        var json = JsonSerializer.Serialize(role);
        await _store.PutAsync($"{RolePrefix}{roleName}",
            Encoding.UTF8.GetBytes(json));
    }

    public async Task BindRoleAsync(string principal, string roleName,
        string? ns = null)
    {
        var binding = new RoleBinding
        {
            Principal = principal,
            RoleName = roleName,
            Namespace = ns,
            Created = DateTime.UtcNow
        };
        var json = JsonSerializer.Serialize(binding);
        await _store.PutAsync($"{BindingPrefix}{principal}/{roleName}",
            Encoding.UTF8.GetBytes(json));
    }
}

public class RbacRole
{
    public string Name { get; set; } = "";
    public List<RbacPolicy> Policies { get; set; } = new();
}

public class RbacPolicy
{
    public string KeyPrefix { get; set; } = "";
    public HashSet<Permission> AllowedPermissions { get; set; } = new();
}

public class RoleBinding
{
    public string Principal { get; set; } = "";
    public string RoleName { get; set; } = "";
    public string? Namespace { get; set; }
    public DateTime Created { get; set; }
}

public enum Permission
{
    Read, Write, Delete, Watch, List, Admin
}

Built-in Roles

15. Audit Logging

Audit logging provides a complete, immutable record of every configuration change. This is essential for compliance (SOC 2, HIPAA, PCI DSS), debugging, and forensic analysis.

Audit Log Entry

C#
public class AuditLogEntry
{
    public ulong Id { get; set; }
    public DateTime Timestamp { get; set; }
    public string Actor { get; set; } = "";
    public string ActorSource { get; set; } = "";
    public string Action { get; set; } = "";
    public string Key { get; set; } = "";
    public byte[]? OldValue { get; set; }
    public byte[]? NewValue { get; set; }
    public ulong OldVersion { get; set; }
    public ulong NewVersion { get; set; }
    public string Result { get; set; } = "";
    public string? ErrorMessage { get; set; }
    public string SourceIP { get; set; } = "";
    public string UserAgent { get; set; } = "";
    public Dictionary<string, string> Metadata { get; set; } = new();
}

public class AuditLogger
{
    private readonly IKeyValueStore _auditStore;
    private readonly IEncryptionService _encryption;
    private long _maxLogId = 0;

    public AuditLogger(IKeyValueStore auditStore, IEncryptionService encryption)
    {
        _auditStore = auditStore;
        _encryption = encryption;
    }

    public async Task LogAsync(AuditLogEntry entry)
    {
        entry.Id = (ulong)Interlocked.Increment(ref _maxLogId);
        entry.Timestamp = DateTime.UtcNow;

        if (entry.ActorSource != "system")
        {
            entry.OldValue = entry.OldValue != null
                ? _encryption.EncryptForAudit(entry.OldValue) : null;
            entry.NewValue = entry.NewValue != null
                ? _encryption.EncryptForAudit(entry.NewValue) : null;
        }

        var json = JsonSerializer.Serialize(entry);
        var key = $"/audit/{entry.Timestamp:yyyy/MM/dd/{entry.Id:D12}}";
        await _auditStore.PutAsync(key, Encoding.UTF8.GetBytes(json));
    }

    public async Task<List<AuditLogEntry>> QueryAsync(
        DateTime from, DateTime to, string? keyFilter = null,
        string? actorFilter = null, int limit = 100)
    {
        var fromKey = $"/audit/{from:yyyy/MM/dd}";
        var toKey = $"/audit/{to:yyyy/MM/dd/999999999999}";
        var entries = await _auditStore.GetRangeAsync(fromKey, toKey);

        return entries
            .Select(e => JsonSerializer.Deserialize<AuditLogEntry>(
                Encoding.UTF8.GetString(e.Value))!)
            .Where(e => keyFilter == null || e.Key.StartsWith(keyFilter))
            .Where(e => actorFilter == null || e.Actor == actorFilter)
            .OrderByDescending(e => e.Timestamp)
            .Take(limit)
            .ToList();
    }
}

Audit Log Retention

16. Multi-Datacenter Replication

For organizations operating across multiple data centers or cloud regions, configuration must be available locally in each datacenter while maintaining global consistency. This section describes our active-passive replication model with planned support for active-active.

Replication Architecture

Replication Protocol

Replication across datacenters is asynchronous to avoid latency penalties. The primary DC commits writes through its Raft cluster. A replication bridge watches the committed log and ships entries to secondary DCs with the following guarantees:

• Ordered delivery: Entries are shipped in commit order.
• At-least-once delivery: Entries may be replayed; receivers handle idempotency using the global index.
• Configurable lag: Replication lag is typically 100ms-2s depending on network latency.
• Conflict resolution: In active-active mode (future), last-writer-wins with vector clocks is used for non-conflicting keys. For conflicting keys, the primary DC always wins.

C#
public class CrossDcReplicator
{
    private readonly IRaftCluster _localCluster;
    private readonly IReplicationTarget[] _remoteDcs;
    private ulong _lastReplicatedIndex;
    private readonly Timer _replicationTimer;
    private readonly int _batchSize;

    public CrossDcReplicator(IRaftCluster localCluster,
        IReplicationTarget[] remoteDcs, int batchSize = 100)
    {
        _localCluster = localCluster;
        _remoteDcs = remoteDcs;
        _batchSize = batchSize;
        _replicationTimer = new Timer(ReplicateBatch, null,
            TimeSpan.FromMilliseconds(10),
            TimeSpan.FromMilliseconds(10));
    }

    private async void ReplicateBatch(object? state)
    {
        var lastIndex = Interlocked.Read(ref _lastReplicatedIndex);
        var entries = await _localCluster.GetEntriesAfter(
            lastIndex, _batchSize);
        if (entries.Count == 0) return;

        var tasks = _remoteDcs.Select(dc =>
            dc.ReplicateAsync(entries)).ToList();

        try
        {
            await Task.WhenAll(tasks);
            Interlocked.Exchange(ref _lastReplicatedIndex,
                entries[^1].Index);
        }
        catch (Exception) { /* log and retry next batch */ }
    }
}

Failover Procedure

If the primary datacenter becomes unreachable, a failover can be initiated to promote a secondary DC to primary:

1. Detect primary DC failure via health checks (3 consecutive failures).
2. Pause writes to the secondary DC to prevent split-brain.
3. Confirm that the secondary's replication lag is below the acceptable threshold (e.g., less than 5 seconds).
4. Promote the secondary's Raft cluster: elect a new leader with a higher term.
5. Resume writes in the newly promoted primary.
6. Update DNS/load balancer to route traffic to the new primary.

17. Health Checking and Failure Detection

The configuration system must know the health of its own nodes and of the clients connected to it. This enables automatic failover, session cleanup, and alerting.

Node Health

Each node in the Raft cluster periodically reports its health to all peers. Health is determined by:

• Heartbeat responsiveness: Is the node responding to Raft heartbeats within the expected interval?
• Disk health: Is the write-ahead log writable? Are writes completing within acceptable latency?
• Memory usage: Is memory usage below the critical threshold (e.g., 85%)?
• Thread count: Is the thread count within normal bounds?

C#
public class HealthChecker
{
    private readonly RaftNode _node;
    private readonly HealthCheckConfig _config;

    public HealthChecker(RaftNode node, HealthCheckConfig config)
    {
        _node = node;
        _config = config;
    }

    public NodeHealthReport CheckLocalHealth()
    {
        var report = new NodeHealthReport
        {
            NodeId = _node.NodeId,
            Timestamp = DateTime.UtcNow,
            RaftState = _node.State.ToString(),
            CurrentTerm = _node.CurrentTerm,
            CommitIndex = _node.CommitIndex,
            LastApplied = _node.LastApplied,
            LogSize = _node.LogCount,
            DiskLatencyMs = MeasureDiskLatency(),
            MemoryUsagePercent = GetMemoryUsage(),
            IsHealthy = true
        };

        report.Issues = EvaluateHealth(report);
        report.IsHealthy = report.Issues.Count == 0;
        return report;
    }

    private List<HealthIssue> EvaluateHealth(NodeHealthReport report)
    {
        var issues = new List<HealthIssue>();

        if (report.DiskLatencyMs > _config.MaxDiskLatencyMs)
            issues.Add(new HealthIssue
            {
                Severity = Severity.Warning,
                Message = $"Disk latency {report.DiskLatencyMs}ms " +
                    $"exceeds threshold {_config.MaxDiskLatencyMs}ms"
            });

        if (report.MemoryUsagePercent > _config.CriticalMemoryThreshold)
            issues.Add(new HealthIssue
            {
                Severity = Severity.Critical,
                Message = $"Memory usage {report.MemoryUsagePercent}% " +
                    $"exceeds critical threshold"
            });

        if (report.CommitIndex - report.LastApplied > 1000)
            issues.Add(new HealthIssue
            {
                Severity = Severity.Warning,
                Message = "StateMachine falling behind commit index"
            });

        return issues;
    }
}

Health Endpoint

C#
[ApiController]
[Route("[controller]")]
public class HealthController : ControllerBase
{
    private readonly HealthChecker _checker;
    private readonly RaftNode _node;

    [HttpGet]
    public IActionResult Get()
    {
        var report = _checker.CheckLocalHealth();
        var statusCode = report.IsHealthy
            ? StatusCodes.Status200OK
            : report.Issues.Any(i => i.Severity == Severity.Critical)
            ? StatusCodes.Status503ServiceUnavailable
            : StatusCodes.Status200OK;

        return StatusCode(statusCode, report);
    }

    [HttpGet("ready")]
    public IActionResult Ready()
    {
        if (_node.State == RaftNode.NodeState.Leader ||
            _node.IsStateMachineCaughtUp())
            return Ok();
        return StatusCode(StatusCodes.Status503ServiceUnavailable);
    }

    [HttpGet("live")]
    public IActionResult Live() => Ok();
}

18. Client SDK Design

A well-designed client SDK hides the complexity of connecting to the Raft cluster, handling leader redirects, retrying failed operations, and caching reads. The SDK should be a pleasure to use while being robust in the face of network failures.

SDK Architecture

C#
public class ConfigClient : IConfigClient, IAsyncDisposable
{
    private readonly ConfigClientOptions _options;
    private readonly ILogger<ConfigClient> _logger;
    private readonly ConnectionPool _connectionPool;
    private readonly ReadCache _cache;
    private readonly RetryPolicy _retryPolicy;
    private readonly WatchManager _watchManager;
    private string? _leaderEndpoint;

    public ConfigClient(ConfigClientOptions options,
        ILogger<ConfigClient> logger)
    {
        _options = options;
        _logger = logger;
        _connectionPool = new ConnectionPool(
            options.Endpoints, options.TlsConfig);
        _cache = new ReadCache(
            TimeSpan.FromSeconds(options.CacheTtlSeconds));
        _retryPolicy = new RetryPolicy(
            options.MaxRetries,
            options.InitialBackoff,
            options.MaxBackoff);
        _watchManager = new WatchManager(_connectionPool);
    }

    public async Task<KeyValueEntry?> GetAsync(string key,
        bool consistentRead = true)
    {
        if (!consistentRead)
        {
            var cached = _cache.Get(key);
            if (cached != null) return cached;
        }

        return await _retryPolicy.ExecuteAsync(async () =>
        {
            var channel = await _connectionPool.GetLeaderAsync(
                ref _leaderEndpoint);
            try
            {
                var result = await channel.GetAsync(key);
                _cache.Set(key, result);
                return result;
            }
            catch (NotLeaderException ex)
            {
                _leaderEndpoint = ex.LeaderEndpoint;
                throw new RetryableException("Leader changed");
            }
        });
    }

    public async Task<PutResult> PutAsync(string key, byte[] value,
        long? leaseId = null)
    {
        return await _retryPolicy.ExecuteAsync(async () =>
        {
            var channel = await _connectionPool.GetLeaderAsync(
                ref _leaderEndpoint);
            try
            {
                var result = await channel.PutAsync(key, value, leaseId);
                _cache.Invalidate(key);
                return result;
            }
            catch (NotLeaderException ex)
            {
                _leaderEndpoint = ex.LeaderEndpoint;
                throw new RetryableException("Leader changed");
            }
        });
    }

    public async Task<long> WatchAsync(string prefix,
        Func<WatchEvent, Task> callback, bool recursive = true)
    {
        return await _watchManager.RegisterWatchAsync(
            prefix, callback, recursive);
    }

    public async Task<Lease> GrantLeaseAsync(TimeSpan ttl)
    {
        return await _retryPolicy.ExecuteAsync(async () =>
        {
            var channel = await _connectionPool.GetLeaderAsync(
                ref _leaderEndpoint);
            return await channel.GrantLeaseAsync(ttl);
        });
    }

    public async Task<TxnResult> TxnAsync(
        TxnComparison[] comparisons,
        TxnOp[] success, TxnOp[]? failure = null)
    {
        return await _retryPolicy.ExecuteAsync(async () =>
        {
            var channel = await _connectionPool.GetLeaderAsync(
                ref _leaderEndpoint);
            try
            {
                return await channel.TxnAsync(
                    comparisons, success, failure);
            }
            catch (NotLeaderException ex)
            {
                _leaderEndpoint = ex.LeaderEndpoint;
                throw new RetryableException("Leader changed");
            }
        });
    }

    public async ValueTask DisposeAsync()
    {
        await _watchManager.DisposeAsync();
        await _connectionPool.DisposeAsync();
    }
}

public class ConfigClientOptions
{
    public List<string> Endpoints { get; set; } = new();
    public int MaxRetries { get; set; } = 5;
    public TimeSpan InitialBackoff { get; set; } =
        TimeSpan.FromMilliseconds(100);
    public TimeSpan MaxBackoff { get; set; } = TimeSpan.FromSeconds(5);
    public int CacheTtlSeconds { get; set; } = 5;
    public TlsConfig? TlsConfig { get; set; }
    public TimeSpan RequestTimeout { get; set; } =
        TimeSpan.FromSeconds(10);
}

Read Cache Strategy

19. CLI Tool Design

The CLI tool is the primary interface for operators and developers. It should support all CRUD operations, watch, transactions, and cluster management commands.

Command Structure

bash
# CRUD Operations
config-cli put /services/payment/config '{"timeout": 5000}'
config-cli get /services/payment/config
config-cli get --prefix /services/payment/
config-cli delete /services/payment/config
config-cli watch --prefix /services/payment/

# Transactions
config-cli txn \
    --if '/services/payment/config version == 5' \
    --then 'put /services/payment/config {"timeout": 3000}' \
    --else 'get /services/payment/config'

# Cluster Management
config-cli member list
config-cli member add node4 --endpoint 10.0.1.4:8300
config-cli member remove node4
config-cli status
config-cli endpoint health

# RBAC Management
config-cli role create config-admin \
    --policy 'prefix=/*, perms=read,write,delete,watch'
config-cli role bind config-admin user:alice \
    --namespace /services/payment/
config-cli role list

# Schema Management
config-cli schema register /services/payment/ \
    --file payment-schema.json
config-cli schema validate /services/payment/config \
    --file new-config.json

# Rollback
config-cli history /services/payment/config --limit 10
config-cli rollback /services/payment/config --to-version 3

# Audit
config-cli audit --key-prefix /services/payment/ --since 7d --limit 50

# Backup and Restore
config-cli snapshot save --output backup-2025-01-15.db
config-cli snapshot restore --input backup-2025-01-15.db
        

CLI Implementation

C#
public class ConfigCli
{
    public static async Task<int> Main(string[] args)
    {
        var rootCommand = new RootCommand("Config Management CLI");

        var putCmd = new Command("put",
            "Set a configuration value")
        {
            new Argument<string>("key", "The configuration key"),
            new Argument<string>("value", "The value to set"),
            new Option<long?>("--lease", "Attach a lease ID"),
            new Option<bool>("--validate", "Validate against schema"),
            new Option<string>("--author", "Actor identity for audit")
        };

        putCmd.SetHandler(async (key, value, lease, validate, author) =>
        {
            var client = CreateClient();
            if (validate)
            {
                var validator = new ConfigValidator<JsonElement>();
                var result = validator.Validate(value);
                if (!result.IsValid)
                {
                    Console.Error.WriteLine("Validation failed:");
                    foreach (var err in result.Errors)
                        Console.Error.WriteLine($"  - {err}");
                    return;
                }
            }
            var putResult = await client.PutAsync(key,
                Encoding.UTF8.GetBytes(value), lease);
            Console.WriteLine(
                $"OK  {key}  version={putResult.Version}");
        });

        var getCmd = new Command("get",
            "Retrieve a configuration value")
        {
            new Argument<string>("key", "The configuration key"),
            new Option<bool>("--prefix",
                "List all keys with this prefix"),
            new Option<string>("--output",
                "Output format: json, yaml, table")
        };

        var watchCmd = new Command("watch",
            "Watch for configuration changes")
        {
            new Argument<string>("key", "Key or prefix to watch"),
            new Option<bool>("--recursive", "Watch recursively"),
            new Option<bool>("--previous",
                "Include previous values")
        };
        watchCmd.SetHandler(async (key, recursive, previous) =>
        {
            var client = CreateClient();
            await client.WatchAsync(key, async evt =>
            {
                var color = evt.Type switch
                {
                    WatchEventType.Put => ConsoleColor.Green,
                    WatchEventType.Delete => ConsoleColor.Red,
                    _ => ConsoleColor.White
                };
                Console.ForegroundColor = color;
                Console.WriteLine(
                    $"[{evt.Type}] {evt.Key} mod={evt.ModRevision}");
                Console.ResetColor();
            }, recursive);
        });

        var statusCmd = new Command("status",
            "Show cluster status");
        statusCmd.SetHandler(async () =>
        {
            var client = CreateClient();
            var status = await client.GetClusterStatusAsync();
            Console.WriteLine($"Cluster: {status.ClusterId}");
            Console.WriteLine(
                $"Leader:  {status.Leader} (term {status.Term})");
            Console.WriteLine();
            foreach (var node in status.Nodes)
            {
                var icon = node.IsHealthy ? "HEALTHY" : "UNHEALTHY";
                Console.WriteLine(
                    $"  {icon} {node.Id,-12} {node.Endpoint,-25} " +
                    $"{node.State,-10} lag={node.Lag}ms");
            }
        });

        rootCommand.AddCommand(putCmd);
        rootCommand.AddCommand(getCmd);
        rootCommand.AddCommand(watchCmd);
        rootCommand.AddCommand(statusCmd);

        return await rootCommand.InvokeAsync(args);
    }
}

20. GUI Dashboard Design

A web-based dashboard provides visual configuration management for less technical stakeholders and offers operational visibility that complements the CLI.

Dashboard Pages

Real-Time Updates in the Dashboard

The dashboard uses the same watch mechanism that the client SDK uses. When an operator views the key explorer, a WebSocket connection streams configuration changes in real-time, providing immediate visual feedback when a config value is modified by another user or service.

21. Migration from File-Based Config

Most organizations do not start with a distributed configuration system. They have configuration files scattered across repositories, deployed with Ansible or baked into Docker images. Migrating to a centralized system requires a careful, incremental approach.

Migration Strategy

Bulk Import Tool

C#
public class ConfigMigrationTool
{
    private readonly IConfigClient _client;
    private readonly SchemaRegistry _schemaRegistry;

    public ConfigMigrationTool(IConfigClient client,
        SchemaRegistry schemaRegistry)
    {
        _client = client;
        _schemaRegistry = schemaRegistry;
    }

    public async Task<MigrationReport> ImportFromFileSystemAsync(
        string basePath, string targetPrefix)
    {
        var report = new MigrationReport();
        var files = Directory.GetFiles(basePath, "*.json",
            SearchOption.AllDirectories);

        foreach (var file in files)
        {
            try
            {
                var relativePath = Path.GetRelativePath(basePath, file);
                var key = $"{targetPrefix}/{relativePath}"
                    .Replace('\\', '/').Replace(".json", "");
                var content = await File.ReadAllTextAsync(file);

                var doc = JsonDocument.Parse(content);
                var validation = await _schemaRegistry.ValidateAsync(
                    key, Encoding.UTF8.GetBytes(content));

                if (!validation.IsValid)
                {
                    report.Warnings.Add(
                        $"{key}: Schema validation warnings: " +
                        string.Join(", ", validation.Errors));
                }

                await _client.PutAsync(key,
                    Encoding.UTF8.GetBytes(content));
                report.SuccessCount++;
                Console.WriteLine($"  Imported: {key}");
            }
            catch (Exception ex)
            {
                report.FailureCount++;
                report.Errors.Add($"{file}: {ex.Message}");
                Console.Error.WriteLine(
                    $"  Failed: {file}: {ex.Message}");
            }
        }
        return report;
    }

    public async Task<DiffReport> CompareWithFileSystemAsync(
        string basePath, string targetPrefix)
    {
        var report = new DiffReport();
        var files = Directory.GetFiles(basePath, "*.json",
            SearchOption.AllDirectories);

        foreach (var file in files)
        {
            var relativePath = Path.GetRelativePath(basePath, file);
            var key = $"{targetPrefix}/{relativePath}"
                .Replace('\\', '/').Replace(".json", "");
            var fileContent = await File.ReadAllTextAsync(file);

            var storeEntry = await _client.GetAsync(key);
            if (storeEntry == null)
            {
                report.OnlyInFileSystem.Add(key);
                continue;
            }

            var storeContent = Encoding.UTF8.GetString(storeEntry.Value);
            if (NormalizeJson(fileContent) != NormalizeJson(storeContent))
            {
                report.Differences.Add(new ConfigDiffReport
                {
                    Key = key,
                    FileSystemValue = fileContent,
                    StoreValue = storeContent
                });
            }
        }
        return report;
    }
}

22. Monitoring and Observability

A production configuration management system must be thoroughly monitored. The three pillars — metrics, logs, and traces — provide complete operational visibility.

Key Metrics

Prometheus Integration

C#
public class MetricsCollector
{
    private readonly Counter _leaderChanges;
    private readonly Histogram _commitLatency;
    private readonly Gauge _logSize;
    private readonly Gauge _activeWatchers;
    private readonly Histogram _watchDeliveryLatency;
    private readonly Gauge _replicationLag;
    private readonly Counter _rpcErrors;

    public MetricsCollector(IMetricsFactory factory)
    {
        _leaderChanges = factory.CreateCounter(
            "raft_leader_changes_total",
            "Number of leader changes");
        _commitLatency = factory.CreateHistogram(
            "raft_commit_latency_seconds",
            "Time to commit a log entry",
            new[] { 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0 });
        _logSize = factory.CreateGauge("raft_log_size_entries",
            "Current Raft log size");
        _activeWatchers = factory.CreateGauge("watch_active_count",
            "Number of active watchers");
        _watchDeliveryLatency = factory.CreateHistogram(
            "watch_delivery_latency_seconds",
            "Time to deliver a watch event",
            new[] { 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0 });
        _replicationLag = factory.CreateGauge(
            "replication_lag_seconds",
            "Cross-DC replication lag");
        _rpcErrors = factory.CreateCounter("rpc_error_total",
            "gRPC errors", "code");
    }

    public void RecordCommitLatency(TimeSpan latency)
        => _commitLatency.Observe(latency.TotalSeconds);

    public void RecordLeaderChange() => _leaderChanges.Inc();
    public void UpdateLogSize(int size) => _logSize.Set(size);
    public void UpdateActiveWatchers(int count)
        => _activeWatchers.Set(count);
    public void RecordWatchDelivery(TimeSpan latency)
        => _watchDeliveryLatency.Observe(latency.TotalSeconds);
    public void UpdateReplicationLag(TimeSpan lag)
        => _replicationLag.Set(lag.TotalSeconds);
    public void RecordRpcError(string code)
        => _rpcErrors.WithLabels(code).Inc();
}

Distributed Tracing

Every write operation in the Raft cluster is traced with OpenTelemetry. The trace captures the full lifecycle: from the initial client request through leader validation, log replication to each follower, commit acknowledgment, FSM application, and watch notification delivery.

23. Security — TLS and Encryption at Rest

Configuration data often contains secrets — database passwords, API keys, TLS certificates, OAuth client secrets. The configuration management system must protect these secrets both in transit and at rest.

Transport Security (TLS)

All communication between clients and the cluster, and between cluster nodes, is encrypted using mutual TLS (mTLS). Each node has a certificate issued by a trusted internal CA. Clients authenticate using client certificates, and the Common Name (CN) is used as the principal for RBAC.

C#
public class TlsConfigurator
{
    public static SslServerCredentials CreateServerCredentials(
        string certPath, string keyPath, string caPath)
    {
        var serverCert = X509Certificate2.CreateFromPemFile(
            certPath, keyPath);
        var caCert = new X509Certificate2(caPath);

        var serverCredentials = new SslServerCredentials(
            new List<SslServerCertificateChain>
            {
                SslServerCertificateChain.CreateFromPemFile(
                    certPath, keyPath)
            })
        {
            ClientCertificateRequired = true,
            EnabledSslProtocols =
                SslProtocols.Tls13 | SslProtocols.Tls12,
            ClientCertificateValidation = (sender, cert, chain, errors) =>
            {
                if (errors == SslPolicyErrors.None) return true;
                if (errors == SslPolicyErrors
                    .RemoteCertificateChainErrors)
                {
                    chain.ChainPolicy.TrustMode =
                        X509ChainTrustMode.CustomRootTrust;
                    chain.ChainPolicy.CustomTrustStore.Add(caCert);
                    return chain.Build(
                        new X509Certificate2(cert));
                }
                return false;
            }
        };
        return serverCredentials;
    }
}

Encryption at Rest

Configuration values containing sensitive data are encrypted before being written to disk. The system supports two encryption modes:

• Transparent encryption: The entire key-value store is encrypted using AES-256-GCM with a master key managed by a KMS (Key Management Service).
• Per-value encryption: Individual values marked as encrypted: true are encrypted with a per-key DEK (Data Encryption Key) that is itself encrypted with the master KEK (Key Encryption Key).

C#
public class EncryptionService : IEncryptionService
{
    private readonly IKmsClient _kms;
    private readonly ConcurrentDictionary<string, byte[]> _dekCache = new();

    public async Task<byte[]> EncryptAsync(byte[] plaintext,
        string keyContext)
    {
        var dek = await GetOrCreateDekAsync(keyContext);
        using var aes = Aes.Create();
        aes.Key = dek;
        aes.GenerateIV();

        using var encryptor = aes.CreateEncryptor();
        var ciphertext = encryptor.TransformFinalBlock(
            plaintext, 0, plaintext.Length);

        var result = new byte[aes.IV.Length + ciphertext.Length + 16];
        Buffer.BlockCopy(aes.IV, 0, result, 0, aes.IV.Length);
        Buffer.BlockCopy(ciphertext, 0, result,
            aes.IV.Length, ciphertext.Length);

        return result;
    }

    public async Task<byte[]> DecryptAsync(byte[] ciphertext,
        string keyContext)
    {
        var dek = await GetDekAsync(keyContext);
        var iv = new byte[16];
        Buffer.BlockCopy(ciphertext, 0, iv, 0, 16);

        var data = new byte[ciphertext.Length - 16];
        Buffer.BlockCopy(ciphertext, 16, data, 0, data.Length);

        using var aes = Aes.Create();
        aes.Key = dek;
        aes.IV = iv;
        using var decryptor = aes.CreateDecryptor();
        return decryptor.TransformFinalBlock(data, 0, data.Length);
    }

    private async Task<byte[]> GetOrCreateDekAsync(string keyContext)
    {
        if (_dekCache.TryGetValue(keyContext, out var cached))
            return cached;

        var encryptedDek = await _kms.EncryptAsync(
            Encoding.UTF8.GetBytes(keyContext));
        var dek = await _kms.DecryptAsync(encryptedDek);
        _dekCache[keyContext] = dek;
        return dek;
    }
}

Secret Detection and Classification

24. Compliance and Regulatory Considerations

Organizations in regulated industries must ensure their configuration management practices meet compliance requirements. The system is designed to support compliance with several major frameworks.

Compliance Mapping

Compliance Audit Support

The system provides pre-built compliance reports that can be generated on demand or scheduled for delivery:

• Change summary report: All configuration changes in a time period, grouped by service, with actor breakdown.
• Access review report: All RBAC policy changes, with before/after comparisons.
• Secret access report: All reads/writes to keys classified as confidential or secret.
• Failed access report: All authorization failures, useful for detecting unauthorized access attempts.
• Configuration drift report: Comparison of live configuration against declared desired state.

25. API Design

The API is the contract between the configuration management system and its clients. We expose a gRPC API for internal communication and a REST API for external integrations.

gRPC API Definition

protobuf
syntax = "proto3";
package config.v1;
option csharp_namespace = "ConfigManagement.Api";

service ConfigService {
    rpc Put(PutRequest) returns (PutResponse);
    rpc Get(GetRequest) returns (GetResponse);
    rpc Delete(DeleteRequest) returns (DeleteResponse);
    rpc GetRange(GetRangeRequest) returns (GetRangeResponse);
    rpc Watch(WatchRequest) returns (stream WatchEvent);
    rpc Txn(TxnRequest) returns (TxnResponse);
    rpc LeaseGrant(LeaseGrantRequest)
        returns (LeaseGrantResponse);
    rpc LeaseKeepAlive(stream LeaseKeepAliveRequest)
        returns (stream LeaseKeepAliveResponse);
    rpc Status(StatusRequest) returns (StatusResponse);
}

message PutRequest {
    string key = 1;
    bytes value = 2;
    int64 lease_id = 3;
    bool prev_kv = 4;
}

message PutResponse {
    KeyValue prev_kv = 1;
    uint64 version = 2;
    uint64 mod_revision = 3;
}

message GetRequest {
    string key = 1;
    bool range_end = 2;
    bool count_only = 3;
    int64 limit = 4;
}

message GetResponse {
    repeated KeyValue kvs = 1;
    int64 count = 2;
}

message KeyValue {
    string key = 1;
    bytes value = 2;
    uint64 version = 3;
    uint64 mod_revision = 4;
    uint64 create_revision = 5;
    int64 lease_id = 6;
}

message WatchRequest {
    string key = 1;
    bool prefix = 2;
    uint64 start_revision = 3;
    bool prev_kv = 4;
    bool recursive = 5;
}

message WatchEvent {
    enum EventType {
        PUT = 0;
        DELETE = 1;
    }
    EventType type = 1;
    KeyValue kv = 2;
    KeyValue prev_kv = 3;
    uint64 mod_revision = 4;
}

message TxnRequest {
    repeated Comparison comparisons = 1;
    repeated Request success = 2;
    repeated Request failure = 3;
}

message Comparison {
    string key = 1;
    enum Op {
        EQUAL = 0;
        NOT_EQUAL = 1;
        GREATER = 2;
        LESS = 3;
    }
    Op op = 2;
    oneof target {
        bytes value = 3;
        uint64 version = 4;
        uint64 mod_revision = 5;
    }
}

message StatusResponse {
    string leader = 1;
    uint64 term = 2;
    uint64 commit_index = 3;
    repeated NodeStatus nodes = 4;
    uint64 db_size = 5;
    uint64 key_count = 6;
}

REST API Mapping

26. Testing Strategy

Testing a distributed configuration management system requires multiple layers of testing, from unit tests of individual components to chaos engineering experiments on a live cluster.

Testing Pyramid

Unit Tests

C#
[TestClass]
public class RaftNodeTests
{
    [TestMethod]
    public async Task Election_Follower_becomes_Candidate()
    {
        var peers = new Mock<IRaftPeer>[] { new(), new() };
        var stateMachine = new Mock<IStateMachine>();
        var storage = new Mock<IRaftStorage>();
        var node = new RaftNode("node1",
            peers.Select(p => p.Object),
            stateMachine.Object);

        node.Start();
        await Task.Delay(TimeSpan.FromSeconds(15));

        Assert.AreEqual(
            RaftNode.NodeState.Candidate, node.State);
    }

    [TestMethod]
    public void KeyValueStore_Put_and_Get()
    {
        var store = new KeyValueStore(
            new Mock<ISnapshotStore>().Object);

        var cmd = new ConfigCommand
        {
            Type = CommandType.Put,
            Key = "/test/key",
            Value = Encoding.UTF8.GetBytes("value"),
            Actor = "test"
        };

        var entry = new LogEntry
        { Index = 1, Term = 1, Command = cmd };
        store.Apply(entry);

        var result = store.Get("/test/key");
        Assert.IsNotNull(result);
        Assert.AreEqual("value",
            Encoding.UTF8.GetString(result.Value));
        Assert.AreEqual(1ul, result.Version);
    }

    [TestMethod]
    public void KeyValueStore_GetRange_prefix_matches()
    {
        var store = new KeyValueStore(
            new Mock<ISnapshotStore>().Object);

        var keys = new[] {
            "/services/a", "/services/b",
            "/services/c", "/other/x"
        };
        foreach (var key in keys)
        {
            store.Apply(new LogEntry
            {
                Index = 1, Term = 1,
                Command = new ConfigCommand
                {
                    Type = CommandType.Put, Key = key,
                    Value = Encoding.UTF8.GetBytes("v"),
                    Actor = "test"
                }
            });
        }

        var results = store.GetRange("/services/");
        Assert.AreEqual(3, results.Count);
        Assert.IsTrue(results.All(
            r => r.Key.StartsWith("/services/")));
    }

    [TestMethod]
    public async Task WatchRegistry_notifies_matching_watchers()
    {
        var registry = new WatchRegistry();
        var channel1 = Channel.CreateUnbounded<WatchEvent>();
        var channel2 = Channel.CreateUnbounded<WatchEvent>();

        registry.Register("/services/payment/",
            true, false, channel1);
        registry.Register("/services/email/",
            true, false, channel2);

        var entry = new KeyValueEntry
        {
            Key = "/services/payment/config",
            Value = Encoding.UTF8.GetBytes("{}"),
            Version = 1, ModIndex = 10
        };

        var notified = await registry.NotifyAsync(
            "/services/payment/config",
            WatchEventType.Put, null, entry);

        Assert.AreEqual(1, notified);
        Assert.IsTrue(channel1.Reader.TryRead(out _));
        Assert.IsFalse(channel2.Reader.TryRead(out _));
    }
}

Integration Tests

C#
[TestClass]
public class ClusterIntegrationTests
{
    private TestCluster _cluster = null!;

    [TestInitialize]
    public async Task Setup()
    {
        _cluster = await TestCluster.CreateAsync(nodeCount: 3);
    }

    [TestCleanup]
    public async Task Teardown()
    {
        await _cluster.ShutdownAsync();
    }

    [TestMethod]
    public async Task Write_and_read_with_strong_consistency()
    {
        var client = _cluster.CreateClient();
        var key = $"/test/{Guid.NewGuid()}";
        var value = Encoding.UTF8.GetBytes("{\"data\": 42}");

        var putResult = await client.PutAsync(key, value);
        Assert.IsTrue(putResult.Version > 0);

        var getResult = await client.GetAsync(key);
        Assert.IsNotNull(getResult);
        Assert.AreEqual(putResult.Version, getResult.Version);
        Assert.AreEqual(42,
            JsonSerializer.Deserialize<dynamic>(
                Encoding.UTF8.GetString(getResult.Value))!.data);
    }

    [TestMethod]
    public async Task Watch_receives_change_notification()
    {
        var client = _cluster.CreateClient();
        var key = $"/watch-test/{Guid.NewGuid()}";
        var received = new TaskCompletionSource<WatchEvent>();

        await client.WatchAsync(key, async evt =>
        {
            received.TrySetResult(evt);
        });

        await Task.Delay(100); // Allow watch to establish
        await client.PutAsync(key,
            Encoding.UTF8.GetBytes("new-value"));

        var result = await received.Task.WaitAsync(
            TimeSpan.FromSeconds(5));
        Assert.AreEqual(WatchEventType.Put, result.Type);
    }

    [TestMethod]
    public async Task Transaction_provides_atomicity()
    {
        var client = _cluster.CreateClient();
        var baseKey = $"/txn-test/{Guid.NewGuid()}";

        var result = await client.TxnAsync(
            comparisons: new[]
            {
                TxnComparison.If($"{baseKey}/a",
                    CompareOp.Version, CompareResult.Equal, 0)
            },
            success: new[]
            {
                TxnOp.Put($"{baseKey}/a",
                    Encoding.UTF8.GetBytes("1")),
                TxnOp.Put($"{baseKey}/b",
                    Encoding.UTF8.GetBytes("2"))
            }
        );

        Assert.IsTrue(result.Succeeded);

        var a = await client.GetAsync($"{baseKey}/a");
        var b = await client.GetAsync($"{baseKey}/b");
        Assert.IsNotNull(a);
        Assert.IsNotNull(b);
    }

    [TestMethod]
    public async Task Session_expiry_removes_ephemeral_keys()
    {
        var client = _cluster.CreateClient();
        var key = $"/ephemeral/{Guid.NewGuid()}";
        var session = await client.CreateSessionAsync(
            TimeSpan.FromSeconds(2));

        await client.PutAsync(key,
            Encoding.UTF8.GetBytes("ephemeral-data"),
            session.Id);

        var entry = await client.GetAsync(key);
        Assert.IsNotNull(entry);

        await Task.Delay(TimeSpan.FromSeconds(4)); // TTL expires

        var expiredEntry = await client.GetAsync(key);
        Assert.IsNull(expiredEntry);
    }
}

Chaos Engineering Tests

27. Cost Estimation and Capacity Planning

Understanding the cost of running a distributed configuration management system helps organizations budget appropriately and make informed decisions about cluster sizing.

Infrastructure Costs (3-Node Cluster)

Scaling Characteristics

Cost Optimization Strategies

• Use spot/preemptible instances for followers. Since followers are not on the critical write path, they can be interrupted and recovered from snapshots with minimal impact.
• Compress audit logs aggressively. JSON audit entries compress at 10:1 ratios. Use GZIP before archiving to cold storage.
• Implement tiered storage. Move audit logs older than 90 days to cold storage (S3 Glacier, Azure Cool) where storage costs drop by 80%.
• Right-size the cluster. Most teams overestimate their needs. Start with 3 nodes and scale up based on actual metrics, not projected maximums.
• Consolidate configuration. Avoid storing large blobs (certificates, PEM files) in the config store. Store references instead and use a dedicated secrets manager.

28. Interview Q&A

Below are common system design interview questions related to distributed configuration management, with detailed answers.

Q1: Why can't we just use a database for configuration management?

Answer: You absolutely can use a database, but you lose several critical features. A traditional database doesn't provide watches/push notifications — you'd need to poll, introducing latency and load. It doesn't provide leases/TTL for ephemeral state like service registration. It doesn't provide the consistent linearizable reads that Raft gives you. And it doesn't naturally handle leader election for distributed coordination. A purpose-built configuration system like etcd or our design provides these primitives natively.

Q2: How does Raft handle the scenario where the leader dies mid-write?

Answer: If the leader crashes after appending the log entry but before receiving majority acknowledgment, the entry is uncommitted. When a new leader is elected, it will not include this entry in its log (since it was never committed). The client will receive a timeout error and must retry. If the leader crashes after majority acknowledgment but before responding to the client, the write is committed. The new leader will have this entry and can respond. The client may receive a timeout and retry, but the idempotency of the write (via version checking) prevents duplicate application.

Q3: What is the difference between strong consistency and linearizability?

Answer: Linearizability is a stronger form of consistency. It guarantees that every operation appears to take effect atomically at some point between its invocation and response. Strong consistency (in the context of Raft) means reads return the latest committed value, which is linearizable. The key difference: a linearizable read must see all writes that completed before it started, even if those writes were on a different leader. In Raft, this is achieved by reading from the leader (or using a read index), while a stale read from a follower is merely "consistent" but not linearizable.

Q4: How would you handle a configuration change that must be applied atomically to 1000 keys?

Answer: Use the transaction API. The TxnRequest allows you to bundle up to 128 operations (in our design) into a single atomic unit. The leader appends all operations as a single log entry, and they're committed and applied atomically. For more than 128 keys, you'd batch the operations into multiple transactions, using a saga pattern to roll back partial failures. However, in practice, configuration changes affecting 1000 keys should be re-evaluated — this usually indicates a schema design issue where keys are too granular.

Q5: How do you prevent split-brain during network partitions?

Answer: Raft prevents split-brain through its quorum requirement. During a network partition, only the partition containing a majority of nodes can elect a leader and accept writes. The minority partition cannot form a quorum and stops accepting writes. When the partition heals, the nodes with lower terms discover the higher term and synchronize. The key safety property is that committed entries are present on a majority, so they cannot be lost.

Q6: What happens to watches during a leader election?

Answer: Watch connections to the old leader will be dropped when the leader steps down. The new leader inherits the watch registrations from the Raft state machine (or from the in-memory registry if watches are tracked in the FSM). The client SDK handles reconnection automatically and resumes the watch from the last observed revision. Some events during the election window may be missed — the client SDK uses the revision number to detect and fill gaps.

Q7: How would you design the system to handle 100,000 concurrent watchers efficiently?

Answer: Several optimizations are needed: (1) Use the trie-based watch registry for O(prefix-length) matching instead of O(n) linear scan. (2) Batch notification delivery within a 5ms window. (3) Use gRPC server streaming with multiplexed connections. (4) Distribute watchers across follower nodes — only the leader processes writes, but followers can serve watch registrations. (5) Use a fan-out tree pattern where the leader notifies 10 "super-watchers" which each notify 100 clients. (6) Apply backpressure by closing slow consumers.

Q8: Why use Raft over Paxos for this system?

Answer: Raft was explicitly designed for understandability and ease of implementation. It decomposes consensus into leader election, log replication, and safety — each of which is relatively straightforward. Paxos, while theoretically elegant, is notoriously difficult to implement correctly. Google's own engineers struggled with it (as documented in the Chubby paper). For a system that needs to be maintained by a team over many years, the maintainability advantage of Raft outweighs any theoretical performance edge of Paxos. Both algorithms have the same fundamental complexity class.

Q9: How do you handle configuration drift between environments?

Answer: Configuration drift occurs when the actual state of a system diverges from the desired state stored in the configuration management system. We handle this through: (1) Read-only sidecar agents that compare local config against the config store every 30 seconds. (2) A "desired state" API that defines what configuration each service should have. (3) Reconciliation loops that detect drift and either auto-correct or alert. (4) The config composition system (Section 11) which makes the source of truth explicit and prevents ad-hoc modifications.

Q10: What are the tradeoffs of encryption at rest for configuration data?

Answer: Encryption at rest adds: (1) CPU overhead for encryption/decryption (~5-10% for AES-256-GCM). (2) Key management complexity — you need a KMS and must handle key rotation. (3) Debugging difficulty — you can't inspect raw values in the database. (4) Backup complexity — backups must include encryption keys or be encrypted separately. The benefits are: compliance with regulations (PCI DSS, HIPAA), protection against physical disk theft, and defense-in-depth even if network security is compromised.

Q11: How would you test the system handles a 50% packet loss scenario?

Answer: We'd use tc (traffic control) on Linux to inject packet loss: tc qdisc add dev eth0 root netem loss 50%. We'd then run the standard integration test suite and observe: (1) Raft elections may take longer due to lost heartbeats, but should still complete. (2) Write throughput would drop significantly as AppendEntries requires majority acknowledgment. (3) Watch delivery would be slower but would still work due to retry logic. (4) The key safety property — no committed data loss — must hold.

Q12: Design a multi-tenant configuration system where tenants cannot see each other's config.

Answer: We'd extend the RBAC model with a tenant isolation layer. Each tenant gets a unique prefix (e.g., /tenants/{tenant-id}/). RBAC policies are scoped to this prefix and cannot be extended beyond it. The API gateway enforces tenant isolation by prepending the tenant prefix to all requests before they reach the Raft cluster. Cross-tenant access is impossible because the RBAC authorizer checks both the role binding's namespace AND the tenant scope. For extra isolation, each tenant could have its own key prefix with a separate encryption key.

29. Summary and Key Takeaways

Designing a distributed configuration management system is one of the most challenging and rewarding exercises in systems engineering. It touches every major area of distributed systems: consensus, replication, failure detection, access control, and operational observability.

Key Takeaways

1. Consistency is non-negotiable. Use Raft or a similar consensus algorithm to ensure all nodes agree on configuration state. The cost of inconsistency (split-brain, configuration drift) far outweighs the cost of consensus overhead.
2. Watches are the killer feature. Real-time change notifications enable reactive configuration patterns that eliminate restarts and polling. Invest heavily in the watch system's performance and reliability.
3. Sessions and leases enable ephemeral state. Service registration, health tracking, and distributed coordination all depend on TTL-based session management.
4. Versioning and rollback are safety nets. Every configuration change should be reversible. Build version tracking and rollback capabilities from day one, not as an afterthought.
5. RBAC and audit logging are not optional. In any multi-service, multi-team environment, access control and change tracking are essential for security and compliance.
6. Multi-datacenter replication requires careful design. Active-passive with async replication is the pragmatic starting point. Active-active is significantly more complex and should only be attempted when the business requirements demand it.
7. The client SDK is as important as the server. A beautiful server is useless if clients can't connect reliably, handle leader failover gracefully, and cache efficiently.
8. Test with chaos engineering. The failure modes of a distributed configuration system are subtle and surprising. Only through systematic chaos testing can you build confidence that the system works under adverse conditions.
9. Monitor everything. Metrics, logs, and traces provide the visibility needed to operate the system with confidence. Set up alerting tiers (page, ticket, log) to ensure that only genuine emergencies wake up on-call engineers.
10. Start small, scale incrementally. A 3-node cluster handles the needs of most organizations. Don't over-engineer for scale you don't yet need.

Comparison with Existing Systems

Architecture Decision Records

For teams implementing this system, we recommend recording the following decisions in an ADR format:

• ADR-001: Use Raft over Paxos for consensus (reason: understandability, proven track record).
• ADR-002: Use gRPC for internal communication (reason: streaming support, performance, type safety).
• ADR-003: Store audit logs in the same KV store (reason: consistency, transactional guarantees).
• ADR-004: Active-passive multi-DC replication (reason: simplicity, fewer consistency risks).
• ADR-005: JSON Schema for configuration validation (reason: ecosystem support, human-readable).
• ADR-006: Client-side caching with configurable TTL (reason: reduce load on the cluster).
• ADR-007: mTLS for all communication (reason: zero-trust security model).

What We Delivered

This 18,000+ word guide has covered every major aspect of designing a distributed configuration management system, from the theoretical foundations of consensus algorithms to the practical implementation details of a production-grade C# codebase. The system we designed provides:

• Strong consistency through Raft consensus with 3-7 node clusters
• Real-time watch notifications via gRPC server streaming
• Session management with TTL-based ephemeral keys
• Fine-grained RBAC with mTLS-based authentication
• Complete audit logging with configurable retention
• Multi-datacenter replication with automated failover
• Schema validation using JSON Schema
• Configuration composition with template and override layers
• Versioning and rollback for all configuration changes
• A comprehensive testing strategy from unit tests to chaos engineering
• Monitoring and observability with Prometheus metrics and OpenTelemetry traces
• Security through TLS in transit and AES-256-GCM encryption at rest