1. Introduction & Why Distributed File Systems Matter

A distributed file system (DFS) is the foundational storage layer that enables organizations to store, access, and manage files across a cluster of commodity machines as though they were accessing a single, unified filesystem. Every modern cloud service, big data platform, and AI training pipeline depends on some form of distributed storage. When you watch a video on YouTube, the underlying blob storage is a distributed file system spanning thousands of machines across multiple data centers. When a data scientist trains a large language model on petabytes of text data, that data is served by a distributed filesystem like HDFS or Lustre. When you use Dropbox or Google Drive, you are interacting with a DFS that transparently syncs your files across devices and data centers.

The core problem a distributed file system solves is deceptively simple: store files on many machines and let clients read and write them transparently. But the devil is in the details. Consider a single 10 GB video file stored across 50 machines. If one machine crashes, the file must still be readable from the remaining replicas. If a client starts reading the file while another client is appending data to it, the reader must see a consistent view. If two clients try to rename the same file simultaneously, exactly one operation must succeed. If the cluster grows from 50 to 5,000 machines, the metadata service must scale without becoming a bottleneck. If a disk in the cluster develops silent data corruption (bit rot), the system must detect and repair it automatically.

These challenges arise from three fundamental tensions in distributed systems: the tension between consistency and availability (you cannot have both in the face of network partitions, per the CAP theorem), the tension between performance and durability (fsync on every write kills throughput, but skipping it risks data loss), and the tension between simplicity and features (a simple key-value store is easy to reason about, but applications need directories, atomic renames, and POSIX semantics).

Understanding the lineage of distributed file systems helps ground our design. Google's Google File System (GFS), published in 2003, was the seminal paper that established the modern DFS architecture: a single master for metadata, chunked file storage on many chunk servers, and a simplified API that relaxes POSIX semantics for performance. Hadoop Distributed File System (HDFS) adopted the GFS design for the open-source world and became the storage backbone of big data. Ceph unified block, file, and object storage under a single architecture using the CRUSH algorithm for data placement. Microsoft's Azure File Service provides SMB-compatible distributed file shares. Facebook's Haystack optimized for billions of small photos by reducing metadata overhead. Each system made different tradeoffs based on its workload.

Real-World Case Studies

Before diving into our design, let us examine how major companies have built distributed file systems, each optimizing for different workload characteristics:

GFS established the template that most modern DFS implementations follow: separate metadata management from data storage, chunk files into fixed-size blocks, replicate chunks across machines, and use a single authoritative master for namespace operations. However, GFS's single-master design created a scalability bottleneck that Google later addressed with Colossus (GFS2), which introduced a distributed metadata layer. Our design follows this evolutionary path: we start with a clear single-master architecture and then discuss how to distribute the metadata service for extreme scale.

The practical motivation for building a DFS has only grown stronger. Modern AI workloads demand storage systems that can deliver hundreds of gigabytes per second of aggregate throughput for model training. Edge computing demands storage that replicates data closer to users across geographic regions. Compliance requirements demand immutable storage with fine-grained access controls and audit trails. And cost pressure demands intelligent data lifecycle management that automatically tiers, compresses, and deduplicates data to minimize storage costs. A well-designed distributed file system must address all of these requirements.

2. Functional & Non-Functional Requirements

Functional Requirements

1. File Operations: Create, read, write (overwrite and append), delete, rename, and stat files. Files are identified by a hierarchical path (e.g., /data/2026/reports/quarterly.pdf).
2. Directory Operations: Create, delete, rename directories. Directories form a tree structure. List contents of a directory. Support empty and non-empty directory deletion (recursive).
3. Large File Support: Files up to 16 TB in size. Files are transparently chunked into fixed-size blocks (e.g., 64 MB each) and distributed across chunk servers.
4. High Throughput: Optimized for large sequential reads and writes. Support streaming reads and append-only writes. Aggregate throughput scales linearly with cluster size.
5. Concurrent Access: Multiple clients can read the same file simultaneously. Concurrent writes to different regions of a file are supported. Write-write conflicts are resolved by last-writer-wins at the chunk level.
6. Data Replication: Configurable replication factor per file or directory (default 3x). Replicas are placed across different racks for fault tolerance. Automatic re-replication when replicas are lost.
7. Snapshot and Copy-on-Write: Point-in-time snapshots of files and directories for backup and recovery. Snapshots use copy-on-write to minimize storage overhead.
8. Access Control: POSIX-like permissions (owner/group/other, read/write/execute) augmented with ACLs for fine-grained access control. Support for authentication via Kerberos or OAuth tokens.

Non-Functional Requirements

Key Design Tradeoffs

3. High-Level Architecture Overview

The architecture of a distributed file system follows a classic three-tier design: clients interact with a metadata service for namespace operations and directly with chunk servers for data read/write operations. This separation is the single most important architectural decision: by removing the metadata service from the data path, we ensure that data throughput scales with the number of chunk servers rather than being bottlenecked by a single metadata server.

Component Responsibilities

The data flow for a file write is: (1) Client asks the metadata master for the chunk locations for the file. (2) The master returns the primary replica and secondary replicas for each chunk. (3) The client pushes data to all replicas in a pipeline (first to the primary, then the primary forwards to secondaries). (4) Once all replicas acknowledge, the write is complete. This pipeline approach maximizes network utilization by overlapping the transmission of data between replicas.

The data flow for a file read is simpler: (1) Client asks the master for chunk locations. (2) The master returns a list of chunk servers holding replicas of the requested chunk, ordered by network proximity to the client. (3) The client reads from the nearest replica. If that replica is unavailable, the client falls back to the next closest replica. This replica selection strategy minimizes read latency while providing fault tolerance.

Data Plane vs Control Plane Separation

A critical architectural principle is the strict separation of the data plane (chunk read/write) from the control plane (metadata operations, rebalancing, garbage collection). The data plane must be optimized for throughput and latency — it handles gigabytes of data per second and must not be blocked by metadata operations. The control plane handles lower-frequency operations like file creation, deletion, rebalancing, and recovery — these are important but can tolerate higher latency. By separating these planes, a slow rebalancing operation cannot impact foreground read/write performance.

4. API Design

The API surface of a distributed file system should be clean, minimal, and familiar to developers accustomed to POSIX or cloud storage APIs. We design both a low-level chunk API (used internally and by advanced clients) and a high-level file API (used by most applications).

High-Level File API

C#
public interface IDistributedFileSystem
{
    // File operations
    Task<FileStream> OpenAsync(string path, FileMode mode, FileAccess access);
    Task<FileMetadata> StatAsync(string path);
    Task<bool> DeleteAsync(string path);
    Task<bool> RenameAsync(string sourcePath, string destinationPath);
    Task<IEnumerable<DirectoryEntry>> ListDirectoryAsync(string directoryPath);

    // Directory operations
    Task<bool> MakeDirectoryAsync(string path, bool recursive = false);
    Task<bool> RemoveDirectoryAsync(string path, bool recursive = false);

    // Write operations
    Task<long> WriteAsync(string path, byte[] data, long offset = 0);
    Task<long> AppendAsync(string path, byte[] data);

    // Read operations
    Task<byte[]> ReadAsync(string path, long offset = 0, int length = -1);

    // Advanced operations
    Task<SnapshotId> CreateSnapshotAsync(string path, string? snapshotName = null);
    Task<bool> SetReplicationFactorAsync(string path, short replicationFactor);
    Task<StorageInfo> GetStorageInfoAsync(string path);
    Task<bool> SetAclAsync(string path, AclEntry[] entries);
}

public record FileMetadata(
    string Path,
    long Size,
    DateTime CreatedAt,
    DateTime ModifiedAt,
    DateTime AccessedAt,
    short ReplicationFactor,
    string Owner,
    string Group,
    FilePermissions Permissions,
    IDictionary<string, string> ExtendedAttributes,
    IEnumerable<ChunkInfo> Chunks
);

public record ChunkInfo(
    long ChunkIndex,
    Guid ChunkId,
    long Offset,
    int Length,
    IEnumerable<ChunkServerLocation> Locations,
    ChecksumData Checksum
);

public record ChunkServerLocation(
    Guid ServerId,
    string Hostname,
    int Port,
    int RackId,
    ReplicaState State  // Primary, Secondary, Learning
);
        

Low-Level Chunk API (Internal)

C#
public interface IChunkServer
{
    // Data operations
    Task<byte[]> ReadChunkAsync(Guid chunkId, int offset, int length);
    Task<WriteResult> WriteChunkAsync(Guid chunkId, int offset, byte[] data, long version);
    Task<WriteResult> AppendChunkAsync(Guid chunkId, byte[] data, long version);
    Task<bool> DeleteChunkAsync(Guid chunkId, long version);

    // Replica management
    Task<bool> CreateChunkAsync(Guid chunkId, ChunkMetadata metadata);
    Task<ChunkSnapshot> SnapshotChunkAsync(Guid chunkId, Guid newChunkId);
    Task<bool> ReplicateChunkAsync(Guid chunkId, ChunkServerLocation target);

    // Health reporting
    Task<ChunkServerStatus> GetStatusAsync();
    Task<IEnumerable<Guid>> GetStoredChunksAsync();
}

public interface IMetadataMaster
{
    // Namespace operations
    Task<FileHandle> OpenAsync(string path, OpenMode mode);
    Task<FileHandle> CreateAsync(string path, CreateOptions options);
    Task<bool> DeleteAsync(string path);
    Task<bool> RenameAsync(string source, string destination);
    Task<IEnumerable<DirectoryEntry>> ListAsync(string path, int limit, string? cursor);

    // Chunk operations
    Task<ChunkLocations> GetChunkLocationsAsync(Guid fileId, long startChunk, int count);
    Task<ChunkAssignment> AllocateChunkAsync(Guid fileId, int chunkIndex);
    Task<bool> ReportChunkLossAsync(Guid chunkId, Guid serverId, string reason);

    // Cluster management
    Task<ClusterStatus> GetClusterStatusAsync();
    Task<bool> DecommissionServerAsync(Guid serverId);
    Task<RebalanceStatus> TriggerRebalanceAsync();
}
        

REST API Layer (HTTP Gateway)

HTTP
# Create a file
PUT /v1/files/data/reports/quarterly.pdf
Content-Type: application/octet-stream
Authorization: Bearer {token}
X-Replication-Factor: 3
Body: <file data>

# Read a file
GET /v1/files/data/reports/quarterly.pdf
Range: bytes=0-65535
Authorization: Bearer {token}

# List directory
GET /v1/directories/data/reports?limit=100&cursor=abc123
Authorization: Bearer {token}

# Create snapshot
POST /v1/snapshots
{
    "path": "/data/reports",
    "name": "pre-migration-2026-07-12"
}

# Get file metadata
HEAD /v1/files/data/reports/quarterly.pdf
Authorization: Bearer {token}
        

5. File Metadata Service

The metadata service is the brain of the distributed file system. It maintains the entire namespace (the directory tree mapping paths to files), maps files to their chunk IDs, and tracks which chunk servers hold replicas of each chunk. Every file operation starts with a metadata lookup, making the metadata service the single most critical component for both performance and correctness.

Metadata Storage Schema

SQL
-- Inode table: one row per file or directory
CREATE TABLE inodes (
    inode_id        BIGSERIAL PRIMARY KEY,
    parent_inode_id BIGINT REFERENCES inodes(inode_id),
    name            VARCHAR(255) NOT NULL,
    file_type       SMALLINT NOT NULL,  -- 1=file, 2=directory, 3=symlink
    size            BIGINT DEFAULT 0,
    replication     SMALLINT DEFAULT 3,
    owner           VARCHAR(128) NOT NULL,
    grp             VARCHAR(128) NOT NULL,
    permissions     SMALLINT DEFAULT 644,
    created_at      TIMESTAMPTZ DEFAULT NOW(),
    modified_at     TIMESTAMPTZ DEFAULT NOW(),
    accessed_at     TIMESTAMPTZ DEFAULT NOW(),
    flags           INTEGER DEFAULT 0,
    acl_enabled     BOOLEAN DEFAULT FALSE,
    UNIQUE(parent_inode_id, name)
);

-- Extended attributes
CREATE TABLE xattrs (
    inode_id    BIGINT REFERENCES inodes(inode_id),
    attr_name   VARCHAR(255) NOT NULL,
    attr_value  BYTEA NOT NULL,
    PRIMARY KEY(inode_id, attr_name)
);

-- Chunk mapping: one row per chunk of a file
CREATE TABLE chunks (
    chunk_id        UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    inode_id        BIGINT NOT NULL REFERENCES inodes(inode_id),
    chunk_index     INTEGER NOT NULL,
    offset          BIGINT NOT NULL,
    length          INTEGER NOT NULL,
    checksum        BYTEA NOT NULL,  -- CRC32 or xxHash
    version         BIGINT NOT NULL DEFAULT 1,
    created_at      TIMESTAMPTZ DEFAULT NOW(),
    UNIQUE(inode_id, chunk_index)
);

-- Replica locations: where each chunk is stored
CREATE TABLE chunk_replicas (
    chunk_id        UUID REFERENCES chunks(chunk_id),
    server_id       UUID NOT NULL,
    state           VARCHAR(20) NOT NULL DEFAULT 'secondary',
        -- primary, secondary, pending, corrupt
    last_verified   TIMESTAMPTZ DEFAULT NOW(),
    bytes_written   BIGINT DEFAULT 0,
    PRIMARY KEY(chunk_id, server_id)
);

-- Server registry
CREATE TABLE servers (
    server_id       UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    hostname        VARCHAR(255) NOT NULL UNIQUE,
    ip_address      INET NOT NULL,
    port            INTEGER NOT NULL,
    rack_id         VARCHAR(64) NOT NULL,
    datacenter      VARCHAR(64) NOT NULL,
    total_disk_bytes BIGINT NOT NULL,
    used_disk_bytes  BIGINT NOT NULL DEFAULT 0,
    status          VARCHAR(20) DEFAULT 'active',
    last_heartbeat  TIMESTAMPTZ DEFAULT NOW(),
    joined_at       TIMESTAMPTZ DEFAULT NOW()
);

-- Write-ahead log for crash recovery
CREATE TABLE metadata_wal (
    lsn             BIGSERIAL PRIMARY KEY,
    operation       VARCHAR(32) NOT NULL,
    target_inode    BIGINT,
    target_chunk    UUID,
    payload         JSONB NOT NULL,
    created_at      TIMESTAMPTZ DEFAULT NOW(),
    applied         BOOLEAN DEFAULT FALSE
);

CREATE INDEX idx_chunks_inode ON chunks(inode_id, chunk_index);
CREATE INDEX idx_chunk_replicas_server ON chunk_replicas(server_id);
CREATE INDEX idx_wal_applied ON metadata_wal(applied, lsn);
        

Metadata Master Implementation

C#
public class MetadataMaster : IMetadataMaster
{
    private readonly IMetadataStore _store;
    private readonly IWalWriter _wal;
    private readonly IChunkAllocator _chunkAllocator;
    private readonly IServerRegistry _servers;
    private readonly ConcurrentDictionary<string, FileHandle> _openFileCache;
    private readonly ILogger<MetadataMaster> _logger;

    public async Task<FileHandle> OpenAsync(string path, OpenMode mode)
    {
        // 1. Resolve path to inode
        var inode = await _store.ResolvePathAsync(path)
            ?? throw new FileNotFoundException($"Path not found: {path}");

        // 2. Check permissions
        await CheckAccessAsync(inode, mode);

        // 3. Allocate file handle with chunk locations
        var chunks = await _store.GetChunksAsync(inode.InodeId);
        var chunkLocations = new List<ChunkLocationInfo>();

        foreach (var chunk in chunks)
        {
            var replicas = await _store.GetReplicasAsync(chunk.ChunkId);
            chunkLocations.Add(new ChunkLocationInfo(
                chunk.ChunkIndex,
                chunk.ChunkId,
                chunk.Offset,
                chunk.Length,
                replicas.Where(r => r.State == ReplicaState.Primary)
                        .Select(r => r.ServerId)
                        .FirstOrDefault(),
                replicas.Where(r => r.State != ReplicaState.Corrupt)
                        .Select(r => r.ServerId)
                        .ToList()
            ));
        }

        // 4. For write mode, allocate pending chunks if needed
        if (mode == OpenMode.Write || mode == OpenMode.Append)
        {
            await _store.TouchAccessTimeAsync(inode.InodeId);
        }

        var handle = new FileHandle(
            HandleId: Guid.NewGuid(),
            InodeId: inode.InodeId,
            Path: path,
            Size: inode.Size,
            Mode: mode,
            Chunks: chunkLocations,
            OpenedAt: DateTime.UtcNow
        );

        _openFileCache[handle.HandleId.ToString()] = handle;
        return handle;
    }

    public async Task<ChunkAssignment> AllocateChunkAsync(
        Guid fileId, int chunkIndex)
    {
        // 1. Create chunk record
        var chunk = await _store.CreateChunkAsync(fileId, chunkIndex);

        // 2. Select servers for replicas using rack-aware placement
        var servers = await _chunkAllocator.AllocateReplicasAsync(
            count: 3,
            excludeServers: Enumerable.Empty<Guid>(),
            strategy: PlacementStrategy.RackAware
        );

        // 3. Write WAL entry for crash recovery
        await _wal.AppendAsync(new WalEntry(
            Operation: WalOperation.CreateChunk,
            ChunkId: chunk.ChunkId,
            Payload: JsonSerializer.Serialize(new
            {
                ChunkId = chunk.ChunkId,
                FileId = fileId,
                ChunkIndex = chunkIndex,
                ReplicaServers = servers.Select(s => s.ServerId).ToList()
            })
        ));

        // 4. Record replica assignments
        foreach (var server in servers)
        {
            await _store.AddReplicaAsync(chunk.ChunkId, server.ServerId);
        }

        // 5. Designate primary (the server with latest data or oldest lease)
        var primary = servers.OrderBy(s => s.LastHeartbeat).First();
        await _store.SetPrimaryAsync(chunk.ChunkId, primary.ServerId);

        return new ChunkAssignment(
            ChunkId: chunk.ChunkId,
            Primary: primary,
            Secondaries: servers.Where(s => s.ServerId != primary.ServerId).ToList()
        );
    }

    private async Task CheckAccessAsync(Inode inode, OpenMode mode)
    {
        var effectivePerms = CalculateEffectivePermissions(
            inode.Permissions, inode.Owner, inode.Group);

        if (mode == OpenMode.Read && !effectivePerms.HasFlag(FilePermissions.Read))
            throw new UnauthorizedAccessException("Read permission denied");

        if (mode == OpenMode.Write && !effectivePerms.HasFlag(FilePermissions.Write))
            throw new UnauthorizedAccessException("Write permission denied");
    }
}
        

6. Chunking Strategy & Chunk Storage

Chunking is the process of splitting a file into fixed-size blocks that can be independently stored and replicated across chunk servers. The choice of chunk size is one of the most impactful design decisions in a DFS because it affects metadata overhead, read/write efficiency, small-file performance, and replication speed.

Chunk Size Analysis

We choose 64 MB as the default chunk size, matching GFS and HDFS. For the small-file problem, we implement a separate "bundle" mechanism that aggregates many small files (under 1 MB) into a single chunk with an internal index. This gives us the metadata efficiency of large chunks while handling small files efficiently.

Chunk Storage Format

C#
public class ChunkStorageManager
{
    private readonly string _dataRoot;
    private readonly IChecksumAlgorithm _checksum;

    public ChunkStorageManager(string dataRoot, IChecksumAlgorithm checksum)
    {
        _dataRoot = dataRoot;
        _checksum = checksum;
    }

    public async Task<ChunkReadResult> ReadChunkAsync(
        Guid chunkId, int offset, int length)
    {
        var chunkPath = GetChunkPath(chunkId);
        var header = await ReadChunkHeaderAsync(chunkPath);

        if (offset + length > header.DataLength)
            throw new ArgumentOutOfRangeException(
                $"Read range [{offset}, {offset+length}) exceeds chunk size {header.DataLength}");

        using var stream = File.OpenRead(chunkPath);
        stream.Seek(ChunkHeader.Size + offset, SeekOrigin.Begin);

        var buffer = new byte[length];
        int bytesRead = await stream.ReadAsync(buffer, 0, length);

        var expectedChecksum = _checksum.Compute(
            buffer, 0, bytesRead, header.ChecksumSeed);

        return new ChunkReadResult(
            Data: buffer,
            Length: bytesRead,
            ChecksumValid: true,
            Version: header.Version
        );
    }

    public async Task<ChunkWriteResult> WriteChunkAsync(
        Guid chunkId, int offset, byte[] data, long expectedVersion)
    {
        var chunkPath = GetChunkPath(chunkId);
        var header = await ReadChunkHeaderAsync(chunkPath);

        if (header.Version != expectedVersion)
            throw new VersionMismatchException(
                $"Expected version {expectedVersion}, found {header.Version}");

        var tempPath = chunkPath + ".tmp";
        using (var tempStream = File.Create(tempPath))
        {
            using var sourceStream = File.OpenRead(chunkPath);
            sourceStream.Seek(ChunkHeader.Size, SeekOrigin.Begin);
            await sourceStream.CopyToAsync(tempStream);

            tempStream.Seek(offset, SeekOrigin.Begin);
            await tempStream.WriteAsync(data, 0, data.Length);
        }

        var newHeader = header with
        {
            Version = header.Version + 1,
            DataLength = Math.Max(header.DataLength, offset + data.Length),
            ChecksumSeed = _checksum.ComputeSeed(),
            LastModified = DateTime.UtcNow
        };

        WriteChunkHeader(tempPath, newHeader);
        File.Move(tempPath, chunkPath, overwrite: true);

        return new ChunkWriteResult(
            Success: true,
            NewVersion: newHeader.Version,
            BytesWritten: data.Length
        );
    }

    private string GetChunkPath(Guid chunkId)
    {
        var shard1 = chunkId.ToString("N").Substring(0, 2);
        var shard2 = chunkId.ToString("N").Substring(2, 2);
        return Path.Combine(_dataRoot, shard1, shard2, chunkId.ToString("N"));
    }
}

public record ChunkHeader(
    Guid ChunkId,
    long Version,
    int DataLength,
    int ChecksumSeed,
    DateTime CreatedAt,
    DateTime LastModified,
    int ReplicaCount
)
{
    public static readonly int Size = 128;
}
        

Small File Optimization: Bundling

Small files (under 1 MB) are a well-known problem in distributed file systems. If we store each 10 KB file as a separate 64 MB chunk, we waste 99.98% of each chunk's capacity. The solution is to bundle multiple small files into a single "bundle chunk" with an internal index that maps file offsets to data regions within the chunk.

C#
public class BundleChunkManager
{
    private const int BundleChunkSize = 64 * 1024 * 1024;
    private const int MaxFilesPerBundle = 10000;

    public async Task<BundleWriteResult> WriteSmallFileAsync(
        string path, byte[] data, Guid bundleChunkId)
    {
        var bundleIndex = await ReadBundleIndexAsync(bundleChunkId);
        var entry = bundleIndex.GetEntry(path);

        if (entry != null)
        {
            if (data.Length <= entry.AllocatedSize)
            {
                await WriteToBundleAsync(bundleChunkId, entry.DataOffset, data);
                entry.DataLength = data.Length;
                entry.Checksum = ComputeChecksum(data);
                await SaveBundleIndexAsync(bundleChunkId, bundleIndex);
                return BundleWriteResult.Overwritten;
            }
        }

        if (bundleIndex.TotalSize + data.Length > BundleChunkSize
            || bundleIndex.FileCount >= MaxFilesPerBundle)
        {
            return BundleWriteResult.BundleFull;
        }

        var newOffset = bundleIndex.TotalSize;
        await WriteToBundleAsync(bundleChunkId, newOffset, data);

        bundleIndex.AddEntry(path, new BundleFileEntry(
            DataOffset: newOffset,
            DataLength: data.Length,
            AllocatedSize: AlignTo512(data.Length),
            Checksum: ComputeChecksum(data),
            CreatedAt: DateTime.UtcNow
        ));

        await SaveBundleIndexAsync(bundleChunkId, bundleIndex);
        return BundleWriteResult.Written;
    }
}
        

7. File Upload Pipeline

The file upload pipeline transforms a client's file into replicated chunks distributed across the cluster. This is a multi-stage process that must be crash-safe, efficient, and provide clear progress feedback to the client.

Pipelined Data Flow

The key to achieving high write throughput is pipelined data flow. Rather than waiting for each replica to acknowledge before sending data to the next, the client streams data to the primary, which immediately forwards it to the first secondary while still receiving data from the client. This creates a pipeline where data is simultaneously in transit across all three replicas, maximizing network utilization.

C#
public class FileUploadPipeline
{
    private readonly IMetadataMaster _master;
    private readonly IChunkServerClientFactory _clientFactory;
    private readonly UploadConfig _config;

    public async Task UploadFileAsync(
        string path, Stream dataStream, UploadProgressCallback progress)
    {
        long totalSize = dataStream.Length;
        int chunkSize = _config.ChunkSizeBytes;
        int totalChunks = (int)Math.Ceiling((double)totalSize / chunkSize);

        var fileHandle = await _master.CreateAsync(path, new CreateOptions
        {
            Size = totalSize,
            ReplicationFactor = _config.DefaultReplication
        });

        var semaphore = new SemaphoreSlim(_config.MaxConcurrentChunkUploads);
        var tasks = new List<Task>();

        for (int i = 0; i < totalChunks; i++)
        {
            int chunkIndex = i;
            long chunkOffset = (long)chunkIndex * chunkSize;
            int chunkLength = (int)Math.Min(chunkSize, totalSize - chunkOffset);

            tasks.Add(Task.Run(async () =>
            {
                await semaphore.WaitAsync();
                try
                {
                    var assignment = await _master.AllocateChunkAsync(
                        fileHandle.InodeId, chunkIndex);

                    var buffer = new byte[chunkLength];
                    lock (dataStream)
                    {
                        dataStream.Seek(chunkOffset, SeekOrigin.Begin);
                        dataStream.Read(buffer, 0, chunkLength);
                    }

                    await PipelineWriteAsync(
                        assignment.Primary,
                        assignment.Secondaries,
                        assignment.ChunkId,
                        buffer,
                        assignment.ExpectedVersion);

                    await _master.ReportChunkWrittenAsync(
                        fileHandle.InodeId, chunkIndex,
                        assignment.ChunkId, ComputeChecksum(buffer));

                    progress?.Report(new UploadProgress
                    {
                        ChunkIndex = chunkIndex,
                        TotalChunks = totalChunks,
                        BytesUploaded = Interlocked.Add(
                            ref _bytesUploaded, chunkLength),
                        TotalBytes = totalSize
                    });
                }
                finally
                {
                    semaphore.Release();
                }
            }));
        }

        await Task.WhenAll(tasks);
        await _master.CloseFileAsync(fileHandle.HandleId);
    }
}
        

8. File Download Pipeline

Reading a file requires the client to first obtain chunk locations from the metadata master, then read directly from chunk servers. The download pipeline must handle replica selection, read-ahead prefetching, parallel chunk reads, and transparent failover to alternate replicas.

C#
public class FileDownloadPipeline
{
    private readonly IMetadataMaster _master;
    private readonly IChunkServerClientFactory _clientFactory;
    private readonly IReadCache _cache;
    private readonly DownloadConfig _config;

    public async Task<byte[]> ReadChunkAsync(
        FileHandle handle, int chunkIndex)
    {
        var cacheKey = $"chunk:{handle.InodeId}:{chunkIndex}";
        var cached = await _cache.GetAsync<byte[]>(cacheKey);
        if (cached != null) return cached;

        var locations = handle.Chunks[chunkIndex];
        var sortedReplicas = await SortByProximityAsync(locations.Replicas);

        byte[] data = null;
        Exception lastError = null;

        foreach (var serverId in sortedReplicas)
        {
            try
            {
                var client = _clientFactory.Create(serverId);
                data = await client.ReadChunkAsync(
                    locations.ChunkId, 0, locations.Length);

                var actualChecksum = ComputeChecksum(data);
                if (!actualChecksum.Equals(locations.Checksum))
                {
                    await _master.ReportCorruptChunkAsync(
                        locations.ChunkId, serverId);
                    data = null;
                    continue;
                }
                break;
            }
            catch (Exception ex)
            {
                lastError = ex;
            }
        }

        if (data == null)
        {
            throw new ChunkReadException(
                $"Failed to read chunk {locations.ChunkId} from all replicas",
                lastError);
        }

        await _cache.SetAsync(cacheKey, data, TimeSpan.FromMinutes(5));
        return data;
    }
}
        

Read-Ahead Prefetching

Sequential reads are the dominant access pattern in a DFS. By prefetching upcoming chunks before the client requests them, we can hide read latency and maximize throughput. The prefetcher monitors the access pattern and issues prefetch requests for the next N chunks (typically N=4) in the background.

9. Data Replication & Erasure Coding

Data replication is the primary mechanism for achieving durability and availability in a distributed file system. By storing multiple copies of each chunk on different machines, we ensure that data remains accessible even when machines, disks, or entire racks fail. We implement two replication strategies: 3-way replication for hot data and erasure coding for cold data.

3-Way Replication

In 3-way replication, each chunk is stored on exactly 3 different chunk servers, preferably in different racks. This provides:

• Durability: Survives any single disk failure and any single rack failure simultaneously
• Availability: Readable even if one replica is down (2 of 3 must be available)
• Recovery speed: Fast re-replication by copying from any of the 2 surviving replicas
• Read throughput: Reads can be served from the nearest replica

Erasure Coding for Cold Data

3-way replication is expensive: it triples storage costs. For data that is rarely accessed (logs, archives, backups), we use Reed-Solomon erasure coding. A typical configuration is RS(10,4): split data into 10 data chunks and compute 4 parity chunks. This provides the same durability as 3x replication but uses only 1.4x storage instead of 3x.

C#
public class ErasureCodingManager
{
    private readonly IGFLEncoder _gfEncoder;
    private readonly IChunkServerClientFactory _clientFactory;

    public async Task<EncodedChunks> EncodeAsync(byte[] data)
    {
        int dataChunks = 10;
        int parityChunks = 4;
        int totalChunks = dataChunks + parityChunks;
        int chunkSize = data.Length / dataChunks;

        var chunks = new byte[totalChunks][];
        for (int i = 0; i < dataChunks; i++)
        {
            int offset = i * chunkSize;
            int length = Math.Min(chunkSize, data.Length - offset);
            chunks[i] = new byte[length];
            Array.Copy(data, offset, chunks[i], 0, length);
        }

        for (int p = 0; p < parityChunks; p++)
        {
            chunks[dataChunks + p] = new byte[chunkSize];
            for (int d = 0; d < dataChunks; d++)
            {
                var coeff = _gfEncoder.GetCoefficient(d, p);
                for (int j = 0; j < chunkSize; j++)
                {
                    chunks[dataChunks + p][j] = _gfEncoder.GFMultiply(
                        chunks[d][j], coeff);
                }
            }
        }

        return new EncodedChunks(DataChunks: chunks);
    }

    public async Task<byte[]> DecodeAsync(EncodedChunks chunks, int originalSize)
    {
        var recoveredChunks = await _gfEncoder.DecodeAsync(
            chunks.Chunks,
            chunks.PresentIndices,
            chunks.TotalChunks);

        var result = new byte[originalSize];
        int offset = 0;
        for (int i = 0; i < chunks.DataChunkCount; i++)
        {
            int length = Math.Min(chunks.ChunkSize, originalSize - offset);
            Array.Copy(recoveredChunks[i], 0, result, offset, length);
            offset += length;
        }

        return result;
    }
}
        

10. Consistency Model

Consistency in a distributed file system defines what a client observes when reading data that is being written concurrently by another client. The consistency model is one of the most important design decisions because it directly impacts both performance and application complexity.

Consistency Levels

We adopt a hybrid consistency model inspired by GFS and modern distributed databases:

• Namespace (metadata) operations use strong consistency through the single master. All namespace mutations (create, delete, rename) go through the master sequentially, and the master serializes them using the WAL (Write-Ahead Log). This means two concurrent renames of the same file are serialized correctly.
• Data (chunk) writes use chunk-level consistency. Within a single chunk, data is written to the primary first, then forwarded to secondaries. Once the primary acknowledges the write, the chunk version is incremented and all subsequent reads see the new data. Concurrent writes to the same region of a chunk result in last-writer-wins at the chunk level.
• Cross-chunk consistency is not guaranteed for concurrent writes to different chunks of the same file. If a client writes chunks 1 and 2 concurrently, a reader may see chunk 1's new data but chunk 2's old data. For applications requiring cross-chunk consistency, the API provides an explicit Sync() operation that flushes all pending writes.

C#
public class ConsistencyManager
{
    private readonly MetadataMaster _master;

    public async Task CloseWriterAsync(FileHandle handle)
    {
        await FlushPendingWritesAsync(handle);
        await WaitForReplicaSyncAsync(handle);
        await _master.CommitFileAsync(handle.InodeId, handle.CurrentSize);
        await _master.IncrementFileVersionAsync(handle.InodeId);
    }

    public async Task OpenReaderAsync(FileHandle handle)
    {
        var committedVersion = await _master.GetCommittedVersionAsync(
            handle.InodeId);

        if (handle.OpenedVersion < committedVersion)
        {
            handle = await _master.RefreshHandleAsync(handle);
        }
    }
}
        

11. Directory Structure & Namespace

The directory structure (namespace) of a distributed file system is a tree of directories and files, similar to a local filesystem. The namespace is managed entirely by the metadata master and must support efficient path resolution, directory listing, and atomic directory operations.

C#
public class NamespaceManager
{
    private readonly IMetadataStore _store;
    private readonly ICache<long, Inode> _inodeCache;

    public async Task<Inode> ResolvePathAsync(string path)
    {
        if (path == "/")
            return await _store.GetRootInodeAsync();

        var parts = path.Trim('/').Split('/');
        long currentInodeId = 1;

        foreach (var part in parts)
        {
            var cacheKey = $"{currentInodeId}:{part}";
            var cached = await _inodeCache.GetAsync<Inode>(cacheKey);
            if (cached != null)
            {
                currentInodeId = cached.InodeId;
                continue;
            }

            var child = await _store.LookupChildAsync(currentInodeId, part)
                ?? throw new FileNotFoundException(
                    $"Component '{part}' not found in path");

            await _inodeCache.SetAsync(cacheKey, child,
                TimeSpan.FromMinutes(1));
            currentInodeId = child.InodeId;
        }

        return await _store.GetInodeAsync(currentInodeId);
    }

    public async Task RenameAsync(string source, string destination)
    {
        var sourceParent = await ResolveParentAsync(source);
        var destParent = await ResolveParentAsync(destination);
        var sourceName = Path.GetFileName(source);
        var destName = Path.GetFileName(destination);

        var lockOrder = new[] { sourceParent.InodeId, destParent.InodeId }
            .OrderBy(x => x).ToList();

        using var lock1 = await _store.AcquireLockAsync(lockOrder[0]);
        using var lock2 = await _store.AcquireLockAsync(lockOrder[1]);

        var existing = await _store.LookupChildAsync(
            destParent.InodeId, destName);
        if (existing != null)
        {
            if (existing.FileType == FileType.Directory)
                throw new IOException("Cannot rename over non-empty directory");
            await DeleteRecursiveAsync(existing.InodeId);
        }

        await _store.RenameEntryAsync(
            sourceParent.InodeId, sourceName,
            destParent.InodeId, destName);

        await InvalidateDentryCacheAsync(sourceParent.InodeId, sourceName);
        await InvalidateDentryCacheAsync(destParent.InodeId, destName);
    }

    public async Task<(IEnumerable<DirectoryEntry> entries, string? cursor)>
        ListDirectoryAsync(string path, int limit = 100, string? cursor = null)
    {
        var inode = await ResolvePathAsync(path);
        if (inode.FileType != FileType.Directory)
            throw new IOException("Path is not a directory");

        return await _store.ListChildrenAsync(inode.InodeId, limit, cursor);
    }
}
        

Namespace Partitioning at Scale

At extreme scale (billions of files), a single master cannot store the entire namespace in memory. We partition the namespace by hashing the top-level directory name. For example, with 16 namespace partitions, paths starting with /data/ might go to partition 3, while paths starting with /logs/ go to partition 11. Each partition is managed by a separate master instance, and path resolution for cross-partition operations is handled by a routing layer.

12. File Locking

File locking provides mutual exclusion for concurrent access to the same file. A distributed file system must support both advisory locks (cooperative, applications agree to respect locks) and mandatory locks (enforced by the system). We implement advisory locks using a distributed lock service (ZooKeeper or etcd).

C#
public class DistributedFileLockManager
{
    private readonly IZooKeeperClient _zk;
    private readonly ILogger<DistributedFileLockManager> _logger;

    public async Task<IAsyncDisposable> AcquireWriteLockAsync(
        string path, Guid clientId)
    {
        var lockPath = $"/dfs/locks{path}";
        var myNode = await _zk.CreateAsync(
            $"{lockPath}/lock-",
            data: Encoding.UTF8.GetBytes(clientId.ToString()),
            flags: CreateFlags.Ephemeral | CreateFlags.Sequential);

        var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);

        while (DateTime.UtcNow < deadline)
        {
            var siblings = await _zk.GetChildrenAsync(lockPath);
            var sortedNodes = siblings.OrderBy(n => n).ToList();
            var myIndex = sortedNodes.IndexOf(Path.GetFileName(myNode));

            if (myIndex == 0)
            {
                return new LockReleaseHandle(_zk, myNode);
            }

            var watchNode = sortedNodes[myIndex - 1];
            await _zk.ExistsAsync($"{lockPath}/{watchNode}", watch: true);
            await Task.WhenAny(
                _zk.GetWatchEventAsync(),
                Task.Delay(TimeSpan.FromSeconds(10)));
        }

        await _zk.DeleteAsync(myNode);
        throw new LockTimeoutException(
            $"Failed to acquire write lock on {path}");
    }
}
        

Lock Types and Semantics

13. POSIX Compatibility

Many existing applications expect POSIX filesystem semantics: byte-accurate reads and writes, atomic append (O_APPEND), file locking (fcntl), memory-mapped I/O (mmap), and file metadata operations (stat, chmod). A distributed file system can provide POSIX compatibility through a FUSE (Filesystem in Userspace) adapter or a kernel module.

POSIX Semantics Mapping

C#
public class FuseAdapter : FuseOperations
{
    private readonly IDistributedFileSystem _dfs;
    private readonly IClientCache _openFiles;

    public override int Open(string path, OpenFlags flags, FileOpenInfo info)
    {
        var mode = MapOpenFlags(flags);
        var handle = _dfs.OpenAsync(path, mode).GetAwaiter().GetResult();
        _openFiles.Store(info.FileHandle, handle);
        return 0;
    }

    public override int Read(string path, byte[] buffer, long offset,
        FileReadInfo info)
    {
        var handle = _openFiles.Get(info.FileHandle);
        var result = _dfs.ReadAsync(path, offset, buffer.Length)
            .GetAwaiter().GetResult();
        Array.Copy(result, 0, buffer, 0, result.Length);
        return result.Length;
    }

    public override int Write(string path, byte[] data, long offset,
        FileWriteInfo info)
    {
        var handle = _openFiles.Get(info.FileHandle);
        _dfs.WriteAsync(path, data, offset).GetAwaiter().GetResult();
        return data.Length;
    }

    public override int Flush(string path, FileFlushInfo info)
    {
        var handle = _openFiles.Get(info.FileHandle);
        _dfs.SyncAsync(handle.Path).GetAwaiter().GetResult();
        return 0;
    }

    public override int Fsync(string path, bool datasync, FileFsyncInfo info)
    {
        var handle = _openFiles.Get(info.FileHandle);
        _dfs.FsyncAsync(handle.Path).GetAwaiter().GetResult();
        return 0;
    }

    public override int Release(string path, FileReleaseInfo info)
    {
        _openFiles.Remove(info.FileHandle);
        return 0;
    }
}
        

14. Client SDK Design

The client SDK is the primary interface between applications and the distributed file system. It must handle connection management, chunk location caching, retry logic, automatic failover, and transparent reconnection. A well-designed SDK makes the distributed nature of the filesystem invisible to the application.

C#
public class DfsClient : IDistributedFileSystem, IAsyncDisposable
{
    private readonly IMetadataMasterClient _masterClient;
    private readonly IChunkServerClientFactory _chunkClientFactory;
    private readonly IChunkLocationCache _locationCache;
    private readonly IRetryPolicy _retryPolicy;
    private readonly ClientOptions _options;
    private readonly SemaphoreSlim _connectionSemaphore;

    public DfsClient(ClientOptions options)
    {
        _options = options;
        _masterClient = new MetadataMasterClient(options.MasterEndpoints);
        _chunkClientFactory = new ChunkServerClientFactory();
        _locationCache = new ChunkLocationCache(
            maxSize: options.CacheSizeMb * 1024 * 1024);
        _retryPolicy = new ExponentialBackoffRetry(
            maxRetries: 3, initialDelay: TimeSpan.FromMilliseconds(100));
        _connectionSemaphore = new SemaphoreSlim(
            options.MaxConcurrentConnections);
    }

    public async Task<FileStream> OpenAsync(
        string path, FileMode mode, FileAccess access)
    {
        return await _retryPolicy.ExecuteAsync(async () =>
        {
            await _connectionSemaphore.WaitAsync();
            try
            {
                var fileHandle = await _masterClient.OpenAsync(path, mode);

                foreach (var chunk in fileHandle.Chunks)
                {
                    await _locationCache.PutAsync(
                        $"{fileHandle.InodeId}:{chunk.ChunkIndex}",
                        chunk);
                }

                return new DfsFileStream(this, fileHandle, access);
            }
            finally
            {
                _connectionSemaphore.Release();
            }
        });
    }

    internal async Task<byte[]> ReadChunkWithFailoverAsync(
        Guid fileId, int chunkIndex)
    {
        var locations = await _locationCache.GetAsync(
            $"{fileId}:{chunkIndex}");

        if (locations == null)
        {
            locations = await _masterClient.GetChunkLocationsAsync(
                fileId, chunkIndex, 1);
            await _locationCache.PutAsync(
                $"{fileId}:{chunkIndex}", locations.First());
        }

        foreach (var replica in locations.Replicas)
        {
            try
            {
                var client = _chunkClientFactory.Create(replica);
                return await client.ReadChunkAsync(
                    locations.ChunkId, 0, locations.Length);
            }
            catch (Exception ex) when (IsTransientError(ex))
            {
                _options.Logger?.LogWarning(
                    "Replica {Server} failed for chunk {ChunkId}, trying next",
                    replica.ServerId, locations.ChunkId);
            }
        }

        throw new DfsException(
            "All replicas unavailable for chunk " + locations.ChunkId);
    }

    private bool IsTransientError(Exception ex) =>
        ex is TimeoutException
        or SocketException
        or IOException { InnerException: SocketException };

    public async ValueTask DisposeAsync()
    {
        await _masterClient.DisposeAsync();
        _connectionSemaphore?.Dispose();
    }
}
        

Client SDK Features

15. Caching Layer

A multi-level caching strategy is essential for achieving high read performance. We implement caching at three levels: client-side (per-process memory cache), chunk server (disk read cache), and metadata (in-memory namespace cache).

Cache Hierarchy

C#
public class MultiLevelCache
{
    private readonly IPageCache _l1Cache;
    private readonly ISsdCache _l2Cache;
    private readonly IInodeCache _metadataCache;

    public async Task<byte[]> GetChunkDataAsync(
        Guid chunkId, int offset, int length)
    {
        var cacheKey = $"{chunkId}:{offset}:{length}";

        var l1Hit = await _l1Cache.GetAsync<byte[]>(cacheKey);
        if (l1Hit != null) return l1Hit;

        var l2Hit = await _l2Cache.GetAsync<byte[]>(cacheKey);
        if (l2Hit != null)
        {
            await _l1Cache.SetAsync(cacheKey, l2Hit, CachePriority.High);
            return l2Hit;
        }

        var data = await ReadFromDiskAsync(chunkId, offset, length);
        await _l2Cache.SetAsync(cacheKey, data, CachePriority.Normal);
        await _l1Cache.SetAsync(cacheKey, data, CachePriority.Normal);
        return data;
    }

    public async Task<Inode> ResolvePathCachedAsync(string path)
    {
        var cached = await _metadataCache.GetAsync<Inode>($"path:{path}");
        if (cached != null) return cached;

        var inode = await ResolvePathFromDbAsync(path);
        var parts = path.Trim('/').Split('/');
        var currentPath = "";
        long parentId = 1;

        foreach (var part in parts)
        {
            currentPath += "/" + part;
            var child = await _metadataCache.GetAsync<Inode>(
                $"dentry:{parentId}:{part}");
            if (child != null)
            {
                parentId = child.InodeId;
                continue;
            }
            child = await LookupChildAsync(parentId, part);
            await _metadataCache.SetAsync(
                $"dentry:{parentId}:{part}", child, CachePriority.High);
            parentId = child.InodeId;
        }

        await _metadataCache.SetAsync($"path:{path}", inode, CachePriority.High);
        return inode;
    }
}
        

Cache Invalidation Strategy

Cache invalidation is critical for correctness. When a file is modified, all cached chunk locations and data must be invalidated across all clients. We use a lease-based invalidation mechanism:

16. Deduplication

Data deduplication eliminates redundant storage of identical data blocks. In a DFS serving multiple users or tenants, it is common for identical files (e.g., shared libraries, common datasets, system images) to be stored multiple times. Deduplication identifies these duplicates and stores only one copy, with references from all logical locations.

C#
public class ContentDefinedChunkingDeduplicator
{
    private readonly IHashFunction _hasher;
    private readonly IContentAddressStore _cas;
    private readonly int _minChunkSize = 4 * 1024;
    private readonly int _maxChunkSize = 64 * 1024 * 1024;

    public async Task<List<ContentChunk>> ChunkFileAsync(Stream data)
    {
        var chunks = new List<ContentChunk>();
        var buffer = new byte[_maxChunkSize];
        int bufferPos = 0;
        long fileOffset = 0;

        while (true)
        {
            int bytesRead = await data.ReadAsync(
                buffer, bufferPos, buffer.Length - bufferPos);

            if (bytesRead == 0)
            {
                if (bufferPos > 0)
                    chunks.Add(await CreateChunkAsync(
                        buffer.AsMemory(0, bufferPos), fileOffset - bufferPos));
                break;
            }

            bufferPos += bytesRead;
            int boundary = FindChunkBoundary(buffer, bufferPos);

            if (boundary >= _minChunkSize || bufferPos >= _maxChunkSize)
            {
                int chunkLen = Math.Min(boundary, bufferPos);
                var chunkData = new byte[chunkLen];
                Array.Copy(buffer, chunkData, chunkLen);

                var chunk = await CreateChunkAsync(chunkData, fileOffset);
                chunks.Add(chunk);

                int remaining = bufferPos - chunkLen;
                if (remaining > 0)
                    Array.Copy(buffer, chunkLen, buffer, 0, remaining);
                bufferPos = remaining;
                fileOffset += chunkLen;
            }
        }

        return chunks;
    }

    private async Task<ContentChunk> CreateChunkAsync(
        ReadOnlyMemory<byte> data, long fileOffset)
    {
        var hash = _hasher.ComputeHash(data.Span);
        var existingRef = await _cas.FindAsync(hash);
        if (existingRef != null)
        {
            return new ContentChunk(
                Hash: hash, FileOffset: fileOffset, Size: data.Length,
                IsDuplicate: true, ExistingChunkId: existingRef.ChunkId);
        }

        var chunkId = await _cas.StoreAsync(hash, data.ToArray());
        return new ContentChunk(
            Hash: hash, FileOffset: fileOffset, Size: data.Length,
            IsDuplicate: false, ExistingChunkId: chunkId);
    }
}
        

Deduplication Strategies

17. Compression

Compression reduces storage costs and network transfer time at the expense of CPU overhead. A well-designed DFS applies compression transparently, choosing the right algorithm for each workload type.

C#
public class AdaptiveCompressor
{
    private readonly Dictionary<CompressAlgorithm, ICompressor> _compressors;

    public async Task<CompressedChunk> CompressAsync(
        byte[] data, string mimeType = null)
    {
        var algorithm = SelectAlgorithm(mimeType, data.Length);

        if (algorithm == CompressAlgorithm.None)
        {
            return new CompressedChunk(
                Algorithm: CompressAlgorithm.None,
                CompressedData: data,
                OriginalSize: data.Length,
                CompressedSize: data.Length);
        }

        var compressor = _compressors[algorithm];
        var compressed = await compressor.CompressAsync(data);

        if (compressed.Length >= data.Length * 0.9)
        {
            return new CompressedChunk(
                Algorithm: CompressAlgorithm.None,
                CompressedData: data,
                OriginalSize: data.Length,
                CompressedSize: data.Length);
        }

        return new CompressedChunk(
            Algorithm: algorithm,
            CompressedData: compressed,
            OriginalSize: data.Length,
            CompressedSize: compressed.Length);
    }

    private CompressAlgorithm SelectAlgorithm(string mimeType, int size)
    {
        if (mimeType != null && IsAlreadyCompressed(mimeType))
            return CompressAlgorithm.None;
        if (size < 1024)
            return CompressAlgorithm.None;
        if (IsTextContent(mimeType))
            return CompressAlgorithm.Zstd;
        return CompressAlgorithm.Zstd;
    }

    private bool IsAlreadyCompressed(string mimeType) => mimeType switch
    {
        "image/jpeg" or "image/png" or "image/gif" or "image/webp" => true,
        "video/mp4" or "video/webm" or "video/avi" => true,
        "audio/mpeg" or "audio/ogg" or "audio/wav" => true,
        "application/zip" or "application/gzip" or "application/xz" => true,
        "application/pdf" => true,
        _ => false
    };
}
        

Compression Algorithm Comparison

18. Garbage Collection

Garbage collection (GC) reclaims storage occupied by deleted files, orphaned chunks, and superseded replicas. In a distributed file system, GC must be careful: it must never delete data that is still referenced, and it must handle the race between deletion and in-progress reads.

C#
public class GarbageCollector
{
    private readonly IMetadataStore _metadata;
    private readonly IChunkServerClientFactory _chunkClients;
    private readonly ILogger<GarbageCollector> _logger;

    public async Task RunGcCycleAsync(CancellationToken ct)
    {
        _logger.LogInformation("Starting GC cycle");

        // Phase 1: MARK
        var liveInodes = new HashSet<long>();
        await MarkLiveInodesAsync(liveInodes, ct);

        // Phase 2: IDENTIFY
        var allChunks = await _metadata.GetAllChunksAsync();
        var orphanedChunks = allChunks
            .Where(c => !liveInodes.Contains(c.InodeId))
            .ToList();

        _logger.LogInformation(
            "GC found {Count} orphaned chunks out of {Total}",
            orphanedChunks.Count, allChunks.Count);

        // Phase 3: SWEEP
        int deletedCount = 0;
        foreach (var chunk in orphanedChunks)
        {
            if (ct.IsCancellationRequested) break;

            if (await IsChunkStillReferencedAsync(chunk.ChunkId))
                continue;

            var replicas = await _metadata.GetReplicasAsync(chunk.ChunkId);
            foreach (var replica in replicas)
            {
                try
                {
                    var client = _chunkClients.Create(replica.ServerId);
                    await client.DeleteChunkAsync(chunk.ChunkId, chunk.Version);
                }
                catch (Exception ex)
                {
                    _logger.LogWarning(
                        "Failed to delete chunk {ChunkId}: {Error}",
                        chunk.ChunkId, ex.Message);
                }
            }

            await _metadata.DeleteChunkRecordAsync(chunk.ChunkId);
            deletedCount++;
        }

        // Phase 4: CLEAN UP old WAL entries
        await _metadata.CompactWalAsync();

        _logger.LogInformation(
            "GC cycle complete: deleted {Deleted} chunks", deletedCount);
    }

    private async Task MarkLiveInodesAsync(
        HashSet<long> liveInodes, CancellationToken ct)
    {
        var stack = new Stack<long>();
        stack.Push(1);

        while (stack.Count > 0 && !ct.IsCancellationRequested)
        {
            var inodeId = stack.Pop();
            if (!liveInodes.Add(inodeId)) continue;

            var inode = await _metadata.GetInodeAsync(inodeId);
            if (inode.FileType == FileType.Directory)
            {
                var children = await _metadata.ListChildrenAsync(inodeId);
                foreach (var child in children)
                    stack.Push(child.InodeId);
            }
        }
    }

    private async Task<bool> IsChunkStillReferencedAsync(Guid chunkId)
    {
        var chunk = await _metadata.GetChunkAsync(chunkId);
        if (chunk == null) return false;
        var inode = await _metadata.GetInodeAsync(chunk.InodeId);
        return inode != null &&
               !inode.Flags.HasFlag(InodeFlags.Deleted) &&
               !inode.Flags.HasFlag(InodeFlags.PendingDelete);
    }
}
        

Garbage Collection Phases

19. Rebalancing & Rack Awareness

Rebalancing redistributes chunks across the cluster when new servers are added, old servers are decommissioned, or chunk distribution becomes uneven. Rack awareness ensures that replicas are spread across different racks to survive rack-level failures.

C#
public class Rebalancer
{
    private readonly IMetadataMaster _master;
    private readonly IChunkServerClientFactory _clientFactory;
    private readonly ILogger<Rebalancer> _logger;
    private readonly RebalanceConfig _config;

    public async Task<List<ServerAssignment>> SelectReplicaServersAsync(
        Guid chunkId, int replicationFactor)
    {
        var allServers = await _master.GetActiveServersAsync();
        var rackMap = allServers.GroupBy(s => s.RackId).ToList();

        var selected = new List<ServerAssignment>();
        var usedRacks = new HashSet<string>();

        foreach (var rack in rackMap.OrderBy(_ => Random.Shared.Next()))
        {
            if (selected.Count >= replicationFactor) break;

            var leastLoaded = rack.OrderBy(s => s.UsedDiskBytes).First();
            selected.Add(new ServerAssignment(
                ServerId: leastLoaded.ServerId,
                RackId: rack.Key,
                Reason: "rack-diversity"
            ));
            usedRacks.Add(rack.Key);
        }

        while (selected.Count < replicationFactor)
        {
            var candidate = allServers
                .Where(s => !selected.Any(sel => sel.ServerId == s.ServerId))
                .OrderBy(s => s.UsedDiskBytes)
                .First();

            selected.Add(new ServerAssignment(
                ServerId: candidate.ServerId,
                RackId: candidate.RackId,
                Reason: "load-balancing"
            ));
        }

        return selected;
    }

    public async Task RunRebalanceAsync(CancellationToken ct)
    {
        var servers = await _master.GetActiveServersAsync();
        var avgChunks = servers.Average(s => s.ChunkCount);
        var threshold = avgChunks * _config.ImbalanceThreshold;

        var overloaded = servers.Where(s => s.ChunkCount > threshold).ToList();
        var underloaded = servers.Where(s => s.ChunkCount < avgChunks * 0.8).ToList();

        if (!overloaded.Any() || !underloaded.Any())
        {
            _logger.LogInformation("Cluster is balanced");
            return;
        }

        foreach (var source in overloaded)
        {
            if (ct.IsCancellationRequested) break;

            var excessChunks = source.ChunkCount - (int)avgChunks;
            var chunksToMove = await _master.GetChunksOnServerAsync(
                source.ServerId, limit: excessChunks);

            foreach (var chunk in chunksToMove)
            {
                if (ct.IsCancellationRequested) break;

                var dest = SelectRebalanceDestination(chunk, source, underloaded);
                if (dest == null) continue;

                try
                {
                    var sourceClient = _clientFactory.Create(source.ServerId);
                    var destClient = _clientFactory.Create(dest.ServerId);
                    var data = await sourceClient.ReadChunkAsync(
                        chunk.ChunkId, 0, chunk.Length);
                    await destClient.WriteChunkAsync(
                        chunk.ChunkId, 0, data, version: chunk.Version);
                    await _master.MoveReplicaAsync(
                        chunk.ChunkId, source.ServerId, dest.ServerId);
                    await Task.Delay(TimeSpan.FromSeconds(1), ct);
                }
                catch (Exception ex)
                {
                    _logger.LogWarning(
                        "Failed to move chunk {ChunkId}: {Error}",
                        chunk.ChunkId, ex.Message);
                }
            }
        }
    }
}
        

Rebalancing Configuration

20. Fault Tolerance & Recovery

Fault tolerance is the defining characteristic of a distributed file system. Machines crash, disks fail, networks partition, and software has bugs. The system must continue operating correctly despite any single point of failure and should degrade gracefully during multi-failure scenarios.

Failure Modes and Responses

C#
public class FaultToleranceManager
{
    private readonly IMetadataMaster _master;
    private readonly IChunkServerClientFactory _clients;
    private readonly ILogger<FaultToleranceManager> _logger;

    public async Task ProcessHeartbeatAsync(
        Guid serverId, HeartbeatReport report)
    {
        await _master.UpdateServerHeartbeatAsync(serverId);

        foreach (var chunkStatus in report.ChunkStatuses)
        {
            switch (chunkStatus.Status)
            {
                case ChunkStatus.Healthy:
                    await _master.UpdateChunkHealthAsync(
                        chunkStatus.ChunkId, serverId, ChunkHealth.Good);
                    break;
                case ChunkStatus.Corrupt:
                    _logger.LogError(
                        "Chunk {ChunkId} on server {Server} is corrupt",
                        chunkStatus.ChunkId, serverId);
                    await _master.ReportCorruptChunkAsync(
                        chunkStatus.ChunkId, serverId);
                    break;
                case ChunkStatus.UnderReplicated:
                    await _master.AddToReplicationQueueAsync(
                        chunkStatus.ChunkId);
                    break;
            }
        }
    }

    public async Task DetectDeadServersAsync()
    {
        var servers = await _master.GetAllServersAsync();
        var now = DateTime.UtcNow;
        var heartbeatTimeout = TimeSpan.FromSeconds(9);

        foreach (var server in servers.Where(s => s.Status == ServerStatus.Active))
        {
            if (now - server.LastHeartbeat > heartbeatTimeout)
            {
                _logger.LogError(
                    "Server {Server} missed heartbeat, marking as dead",
                    server.ServerId);

                await _master.SetServerStatusAsync(
                    server.ServerId, ServerStatus.Dead);

                var chunks = await _master.GetChunksOnServerAsync(
                    server.ServerId);

                foreach (var chunk in chunks)
                {
                    var currentReplicas = await _master.GetReplicaCountAsync(
                        chunk.ChunkId);
                    var targetReplicas = chunk.TargetReplication;

                    if (currentReplicas < targetReplicas)
                    {
                        await _master.AddToReplicationQueueAsync(
                            chunk.ChunkId, priority: ReplicationPriority.Urgent);
                    }
                }
            }
        }
    }

    public async Task ScrubChunkChecksumsAsync(CancellationToken ct)
    {
        var servers = await _master.GetActiveServersAsync();

        foreach (var server in servers)
        {
            if (ct.IsCancellationRequested) break;
            var client = _clients.Create(server.ServerId);
            var chunks = await client.GetStoredChunksAsync();

            foreach (var chunkId in chunks)
            {
                if (ct.IsCancellationRequested) break;
                try
                {
                    var result = await client.ReadChunkAsync(chunkId, 0, 0);
                    if (!result.ChecksumValid)
                    {
                        _logger.LogError(
                            "Checksum mismatch on chunk {ChunkId} at {Server}",
                            chunkId, server.ServerId);
                        await _master.ReportCorruptChunkAsync(
                            chunkId, server.ServerId);
                    }
                }
                catch (Exception ex)
                {
                    _logger.LogWarning(
                        "Failed to scrub chunk {ChunkId}: {Error}",
                        chunkId, ex.Message);
                }
                await Task.Delay(TimeSpan.FromMilliseconds(10), ct);
            }
        }
    }
}
        

21. Security, Encryption, ACLs & Compliance

Security in a distributed file system spans three dimensions: data protection (encryption at rest and in transit), access control (who can read/write which files), and compliance (audit logging, data retention, and regulatory requirements).

Encryption Architecture

C#
public class EncryptionManager
{
    private readonly IKeyVault _keyVault;
    private readonly IKeyCache _keyCache;

    public async Task<EncryptedWriteResult> EncryptAndWriteAsync(
        Guid fileId, byte[] plaintext)
    {
        var dek = Aes.Create();
        dek.KeySize = 256;
        dek.GenerateKey();
        dek.GenerateIV();

        byte[] ciphertext;
        using (var encryptor = dek.CreateEncryptor())
        using (var ms = new MemoryStream())
        {
            using (var cs = new CryptoStream(ms, encryptor, CryptoStreamMode.Write))
            {
                await cs.WriteAsync(plaintext);
            }
            ciphertext = ms.ToArray();
        }

        var masterKey = await _keyVault.GetCurrentMasterKeyAsync();
        var wrappedDek = await masterKey.WrapKeyAsync(dek.Key);

        return new EncryptedChunk
        {
            FileId = fileId,
            Ciphertext = ciphertext,
            WrappedDek = wrappedDek,
            Iv = dek.IV,
            Algorithm = "AES-256-GCM",
            KeyVersion = masterKey.Version
        };
    }

    public async Task<byte[]> DecryptReadAsync(EncryptedChunk chunk)
    {
        var masterKey = await _keyVault.GetKeyAsync(chunk.KeyVersion);
        var dekBytes = await masterKey.UnwrapKeyAsync(chunk.WrappedDek);

        using var aes = Aes.Create();
        aes.Key = dekBytes;
        aes.IV = chunk.Iv;

        using var decryptor = aes.CreateDecryptor();
        using var ms = new MemoryStream(chunk.Ciphertext);
        using var cs = new CryptoStream(ms, decryptor, CryptoStreamMode.Read);
        using var result = new MemoryStream();
        await cs.CopyToAsync(result);
        return result.ToArray();
    }
}
        

Access Control Lists (ACLs)

C#
public class AccessControlManager
{
    public async Task<bool> CheckAccessAsync(
        string path, string userId, FileOperation operation)
    {
        var inode = await _resolver.ResolvePathAsync(path);

        var posixPerms = CalculatePosixPermissions(
            inode, userId, await GetUserGroupsAsync(userId));

        if (HasPermission(posixPerms, operation))
            return true;

        if (inode.AclEnabled)
        {
            var acls = await _store.GetAclAsync(inode.InodeId);
            return EvaluateAcl(acls, userId, operation);
        }

        return false;
    }

    private bool EvaluateAcl(AclEntry[] acls, string userId, FileOperation op)
    {
        foreach (var acl in acls)
        {
            if (acl.Type == AclType.User && acl.Who == userId)
                return acl.Permissions.HasFlag(op.ToPermission());

            if (acl.Type == AclType.Group)
            {
                var userGroups = GetUserGroupsAsync(userId)
                    .GetAwaiter().GetResult();
                if (userGroups.Contains(acl.Who))
                {
                    var entryPerms = acl.Permissions & GetMask(acls);
                    return entryPerms.HasFlag(op.ToPermission());
                }
            }

            if (acl.Type == AclType.Other)
                return acl.Permissions.HasFlag(op.ToPermission());
        }

        return false;
    }
}

public record AclEntry(
    AclType Type,
    string Who,
    FilePermissions Permissions,
    bool IsDefault
);

public enum AclType { User, Group, Other, Mask }
        

Security Features Summary

22. Monitoring & Observability

A distributed file system requires comprehensive monitoring to detect issues before they impact users. We monitor four key dimensions: availability (is the system up?), performance (is it fast enough?), capacity (are we running out of space?), and correctness (is data intact?).

Key Metrics

C#
public class MetricsCollector
{
    private readonly IMetricRegistry _registry;

    public void RegisterServerMetrics(ChunkServerInfo server)
    {
        _registry.Gauge("dfs_server_disk_used_bytes",
            () => server.UsedDiskBytes,
            new Tag("server", server.Hostname),
            new Tag("rack", server.RackId));

        _registry.Gauge("dfs_server_disk_total_bytes",
            () => server.TotalDiskBytes,
            new Tag("server", server.Hostname));

        _registry.Gauge("dfs_server_chunk_count",
            () => server.ChunkCount,
            new Tag("server", server.Hostname));
    }

    public void RecordReadLatency(Guid chunkId, TimeSpan latency)
    {
        _registry.Histogram("dfs_read_latency_ms",
            latency.TotalMilliseconds,
            new Tag("chunk", chunkId.ToString()));
    }

    public void RecordWriteLatency(Guid chunkId, TimeSpan latency)
    {
        _registry.Histogram("dfs_write_latency_ms",
            latency.TotalMilliseconds,
            new Tag("chunk", chunkId.ToString()));
    }

    public void RecordMetadataOperation(string operation, TimeSpan latency)
    {
        _registry.Histogram("dfs_metadata_latency_ms",
            latency.TotalMilliseconds,
            new Tag("operation", operation));
    }

    public void IncrementCorruptChunks(string serverId)
    {
        _registry.Counter("dfs_corrupt_chunks_total",
            new Tag("server", serverId));
    }

    public void SetUnderReplicatedCount(int count)
    {
        _registry.Gauge("dfs_under_replicated_chunks", () => count);
    }
}
        

Alerting Rules

23. Cost Estimation

Understanding the cost structure of a distributed file system is essential for capacity planning and architectural decisions. We estimate costs for a production cluster serving 10 PB of logical data with 3x replication and 1 billion files.

Infrastructure Cost Breakdown

Total Monthly Cost Summary

Cost Comparison with Cloud Storage

The economics of self-managed DFS vs cloud storage depend critically on scale. At small scale (under 100 TB), cloud storage is almost always cheaper because you avoid the fixed costs of hardware, power, cooling, and operations staff. At very large scale (over 1 PB), self-managed DFS becomes significantly cheaper because the marginal cost of adding another disk is much lower than the per-GB pricing charged by cloud providers. The crossover point is typically around 500 TB to 1 PB, depending on the organization's existing data center infrastructure and operational expertise.

24. Testing Strategy

Testing a distributed file system is exceptionally challenging because it involves concurrent operations across multiple machines, network failures, disk failures, and timing-dependent race conditions. A comprehensive testing strategy combines unit tests, integration tests, fault injection tests, and chaos engineering.

Test Categories

C#
// Integration test: file upload and download with fault injection
public class FileUploadDownloadTests : IAsyncLifetime
{
    private DfsTestCluster _cluster;

    public async Task InitializeAsync()
    {
        _cluster = await DfsTestCluster.CreateAsync(
            chunkServers: 5,
            replicationFactor: 3);
    }

    [Fact]
    public async Task UploadDownload_LargeFile_CorrectnessPreserved()
    {
        // Arrange
        var testData = GenerateRandomData(256 * 1024 * 1024); // 256 MB
        var path = $"/test/file-{Guid.NewGuid()}.dat";

        // Act: Upload
        await _cluster.Client.UploadFileAsync(path, testData);

        // Assert: Download and verify
        var downloaded = await _cluster.Client.DownloadFileAsync(path);
        Assert.Equal(testData, downloaded);
    }

    [Fact]
    public async Task Upload_DuringServerCrash_DataSurvives()
    {
        // Arrange
        var testData = GenerateRandomData(64 * 1024 * 1024); // 64 MB
        var path = $"/test/file-{Guid.NewGuid()}.dat";

        // Act: Start upload, then crash a server mid-upload
        var uploadTask = _cluster.Client.UploadFileAsync(path, testData);
        await Task.Delay(100); // Let upload start
        await _cluster.CrashChunkServerAsync(serverIndex: 2);
        await uploadTask; // Should complete with remaining replicas

        // Assert: File is readable from surviving replicas
        var downloaded = await _cluster.Client.DownloadFileAsync(path);
        Assert.Equal(testData, downloaded);
    }

    [Fact]
    public async Task ConcurrentReads_WritesConsistent()
    {
        var path = $"/test/file-{Guid.NewGuid()}.dat";
        await _cluster.Client.UploadFileAsync(path, GenerateRandomData(1024));

        // Concurrent reads while writing
        var readTasks = Enumerable.Range(0, 10)
            .Select(_ => _cluster.Client.ReadFileAsync(path))
            .ToList();

        var writeTask = _cluster.Client.AppendToFileAsync(
            path, GenerateRandomData(1024));

        await Task.WhenAll(readTasks.Concat(new[] { writeTask }));

        // All reads should return valid data (either before or after append)
        foreach (var data in readTasks.Select(t => t.Result))
        {
            Assert.True(data.Length >= 1024); // At least original data
        }
    }

    [Fact]
    public async Task MasterFailover_MetadataPreserved()
    {
        // Upload a file
        var testData = GenerateRandomData(1024);
        var path = $"/test/file-{Guid.NewGuid()}.dat";
        await _cluster.Client.UploadFileAsync(path, testData);

        // Crash the metadata master
        await _cluster.CrashMasterAsync();

        // Wait for failover (shadow promotes to master)
        await Task.Delay(TimeSpan.FromSeconds(30));

        // Assert: File is still accessible via new master
        var downloaded = await _cluster.Client.DownloadFileAsync(path);
        Assert.Equal(testData, downloaded);
    }

    public async Task DisposeAsync()
    {
        await _cluster.ShutdownAsync();
    }
}
        

Chaos Engineering Scenarios

25. Interview Q&A Deep Dive

Below are the most common system design interview questions about distributed file systems, along with detailed answers that demonstrate senior-level understanding.

Q1: How does GFS handle concurrent writes to the same chunk?

Answer: GFS uses a primary-secondary model. For each chunk, the master designates one replica as the primary and grants it a lease (typically 60 seconds). All writes go through the primary, which assigns a serial order to all concurrent writes and forwards data to secondaries in that order. If the primary fails, the master detects the missed heartbeat and promotes a secondary to primary. Concurrent writes to different chunks do not need coordination because each chunk has its own primary. The key insight is that serialization happens at the chunk level, not the file level, which enables high write throughput while maintaining per-chunk consistency.

Q2: What happens when the metadata master crashes?

Answer: The metadata master's state is persisted via a Write-Ahead Log (WAL) that records every metadata mutation before it is applied. When the master crashes, the standby (shadow) master reads the WAL, replays all uncommitted entries, and promotes itself to primary. This process takes 15-30 seconds. Clients that had open file handles reconnect to the new master and refresh their chunk location caches. The critical property is that no metadata is lost because the WAL is flushed to disk (or replicated to a quorum) before acknowledging any mutation. During the failover window, file metadata operations are unavailable but data reads/writes can continue using cached chunk locations.

Q3: How do you handle the small file problem?

Answer: The small file problem manifests in two ways: metadata overhead (each small file requires a full chunk record even if it uses a tiny fraction of the chunk) and storage waste (a 1 KB file stored in a 64 MB chunk wastes 64 MB). We address this with three strategies: (1) Bundling: many small files are packed into a single chunk with an internal index, reducing metadata overhead by 5x. (2) Metadata caching: small file metadata is aggressively cached in memory, reducing lookup latency to microseconds. (3) Client-side write buffering: small writes from applications are buffered and flushed as a batch when the buffer fills or the file is closed. For workloads dominated by small files (like photo storage), a dedicated small-file tier with 4 MB chunks and bundling provides the best balance.

Q4: How does erasure coding compare to replication?

Answer: Replication (3x) stores three complete copies, providing durability against any two simultaneous failures. It is simple, fast to recover (copy from any surviving replica), and provides high read throughput (read from nearest replica). The downside is 3x storage cost. Erasure coding (e.g., RS(10,4)) splits data into 10 chunks and computes 4 parity chunks, providing the same durability as 3x replication but using only 1.4x storage. The tradeoffs: higher CPU for encoding/decoding (but modern CPUs handle this easily), slower recovery (must re-encode rather than simply copy), and higher read amplification (must read 10 of 14 chunks). Our hybrid approach uses replication for hot data (frequently accessed) and erasure coding for cold data (archives, backups), achieving the best of both worlds.

Q5: How do you ensure data is not lost during rebalancing?

Answer: Rebalancing follows a copy-then-delete protocol: (1) Read the chunk from the source server. (2) Write it to the destination server. (3) Wait for the destination to acknowledge the write. (4) Update metadata to reflect the new replica location. (5) Only then delete the chunk from the source server. At no point during this process is the number of replicas below the target — the chunk exists on both source and destination simultaneously until the metadata update completes. If the source crashes during the copy, the destination already has the data and we just need to update metadata. If the destination crashes, the source still has the copy and we retry with a different destination. The WAL ensures metadata consistency across all these failure scenarios.

Q6: How do you handle network partitions in the DFS?

Answer: Network partitions are handled at two levels. At the metadata level, the master uses ZooKeeper for leader election, which requires a quorum (majority) of nodes to be reachable. If the master is partitioned from the majority, it steps down and a new master is elected from the majority partition. This means the minority partition loses metadata service but the majority continues operating. At the data level, chunk servers that are partitioned from the master stop receiving lease renewals, so their primary leases expire. Clients in the majority partition can still read/write through other replicas. Clients in the minority partition can only read from local replicas (if they have cached chunk locations) but cannot write because they have no primary. This is consistent with the CP model for metadata and AP model for data reads.

Q7: How does the DFS achieve eleven nines of durability?

Answer: Eleven nines (99.999999999%) means one data loss event per 100,000 years per file. With 3x replication across 3 different racks, the probability of data loss is the product of three independent disk failure probabilities. If each disk has an annual failure rate (AFR) of 2%, and we assume failures are independent across racks: P(loss) = P(disk1 fails) x P(disk2 fails within repair window) x P(disk3 fails within repair window). With a repair window of 1 hour and 3 replicas across different racks, the probability of simultaneous failure is approximately 10^-11 to 10^-13 per year, meeting the eleven nines target. Erasure coding with RS(10,4) provides similar durability with even lower storage cost, but recovery takes longer (must re-encode rather than simply copy).

Q8: How do you test for data corruption (bit rot)?

Answer: Bit rot (silent data corruption) is detected through two mechanisms: (1) Online detection: every read operation verifies the checksum of the chunk data. If a checksum mismatch is detected, the read is served from another replica and the corrupt replica is marked for repair. (2) Offline scrubbing: a background process periodically reads every chunk on every server and verifies its checksum. We run a full scrub on a weekly cycle, ensuring that corruption is detected within 7 days. The scrubber uses a lower I/O priority to avoid impacting foreground operations. When corruption is detected, the master triggers re-replication from a known-good replica and removes the corrupt one. The key insight is that checksums must be computed and stored when the chunk is written, and must be verified on every read, not just during scrubbing.

Q9: How do you handle file locking in a distributed environment?

Answer: File locking in a DFS uses ZooKeeper ephemeral sequential nodes to implement a distributed lock with automatic release on client crash. For exclusive locks (write), the client creates an ephemeral sequential node and checks if it is the lowest-numbered node — if so, it holds the lock. For shared locks (read), the client creates a "read-" prefixed node and holds the lock if no "lock-" (write) node with a lower sequence number exists. This ensures that write locks block all other locks (exclusive) and shared locks are compatible with other shared locks but block writes. The lock is automatically released if the client crashes because ZooKeeper deletes ephemeral nodes when the session expires. The timeout mechanism prevents deadlocks: if a lock cannot be acquired within 30 seconds, the client receives an error rather than waiting indefinitely.

Q10: What is the biggest scalability bottleneck and how do you address it?

Answer: The biggest scalability bottleneck is the metadata master, specifically its ability to handle concurrent metadata operations. With a single master, the maximum namespace operation throughput is limited by the master's CPU (for path resolution) and memory (for storing metadata). At 1 billion files, the metadata is approximately 2.2 TB, which fits in a single server's memory. But the QPS limit for metadata operations is approximately 100K operations/second on a single master. To scale beyond this, we partition the namespace by top-level directory hash across multiple master instances, each responsible for a subset of the namespace. This allows horizontal scaling of metadata throughput. For the data path, there is no bottleneck because clients read/write directly to chunk servers — the master is only involved in the initial metadata lookup, which is cached by the client SDK.

Pre-Interview Checklist

• Understand the GFS architecture: single master, chunk servers, pipelined data flow
• Know the tradeoffs between chunk sizes (64 KB vs 64 MB)
• Explain rack-aware replica placement and why it matters
• Understand 3-way replication vs erasure coding tradeoffs
• Know the close-to-open consistency model and why it was chosen
• Discuss the small file problem and bundling solution
• Understand WAL-based crash recovery for the metadata master
• Explain how garbage collection works in a DFS (mark-and-sweep)
• Know how checksums detect and prevent bit rot
• Discuss encryption at rest (envelope encryption with DEKs) and in transit (TLS 1.3)
• Understand rebalancing: copy-then-delete protocol, throttling, non-disruptive
• Know the cost model and how self-managed compares to cloud storage
• Discuss testing strategies: fault injection, chaos engineering, data integrity verification

Key Numbers to Remember