1. Introduction

Container orchestration is the backbone of modern cloud-native infrastructure. Every major technology company — from Netflix and Spotify to Stripe and Shopify — runs thousands of containers across fleets of machines, and they all rely on orchestration platforms to schedule, network, secure, scale, and heal those workloads automatically. Kubernetes has emerged as the de facto standard, but understanding the internals of how such a system is designed from scratch is what separates a senior engineer from a staff engineer.

In this guide, we will dissect every major subsystem of a Kubernetes-style container orchestration platform. We will begin with the high-level landscape, move into capacity estimation and data modeling, then walk through the control plane and data plane in exhaustive detail. We will cover pod lifecycle management, deployment strategies, service discovery, ingress routing, configuration management, persistent storage, role-based access control, package management via Helm, horizontal and vertical auto-scaling, network policies, observability through Prometheus and Grafana, centralized logging with the EFK stack, multi-cluster federation, cost modeling, testing strategies, and finally interview preparation.

This is not a tutorial on running kubectl apply. This is a deep architectural guide for engineers who want to understand the system at the level required to design, build, debug, and optimize orchestration platforms for production workloads serving millions of requests per second.

2. Container Orchestration Landscape

Before diving into the design, it is important to understand the competitive landscape and why Kubernetes won. The orchestration space has evolved through several generations of tooling, each solving progressively harder problems.

2.1 Historical Context

In the early days of containerization, Docker provided the runtime but had no native multi-host scheduling. Teams wrote custom shell scripts to deploy containers across VMs, which worked until the fleet grew beyond a dozen machines. This pain led to a wave of orchestration tools in the 2014-2016 timeframe.

2.2 Comparison of Orchestration Platforms

2.3 Why Kubernetes Won

Kubernetes succeeded because of three strategic advantages. First, the declarative API model — you describe desired state and the system converges toward it, rather than issuing imperative commands. Second, the extensibility model — Custom Resource Definitions (CRDs) and admission webhooks allow the platform to be extended without forking. Third, the ecosystem — Helm, Istio, Prometheus, ArgoCD, and hundreds of other projects created a gravitational pull that no competitor could match.

Understanding these design principles is critical because when we build our own orchestration system, we will adopt the same declarative, extensible approach that made Kubernetes successful.

3. Requirements

Any production-grade orchestration platform must satisfy a comprehensive set of functional and non-functional requirements. Let us enumerate them systematically.

3.1 Functional Requirements

• Container Scheduling: Automatically place containers onto appropriate nodes based on resource requirements, affinity rules, and constraints.
• Self-Healing: Automatically restart failed containers, reschedule them on healthy nodes, and replace unresponsive instances.
• Service Discovery & Load Balancing: Provide DNS-based service discovery and distribute traffic across container replicas.
• Horizontal Auto-Scaling: Automatically adjust replica counts based on CPU, memory, or custom metrics.
• Rolling Updates & Rollbacks: Deploy new versions incrementally with zero downtime and automatic rollback on failure.
• Storage Orchestration: Automatically mount persistent volumes from various storage backends (NFS, cloud block storage, distributed filesystems).
• Secrets Management: Securely store and inject sensitive configuration (passwords, certificates, API keys) into containers.
• Network Policies: Enforce firewall rules between pods to implement zero-trust networking.
• Multi-Tenancy: Support namespace isolation, resource quotas, and RBAC to serve multiple teams from a single cluster.
• Batch & Cron Workloads: Support one-off batch jobs and scheduled recurring tasks alongside long-running services.

3.2 Non-Functional Requirements

3.3 Capacity Estimation

For a production cluster serving a mid-to-large organization, let us estimate the capacity requirements across all subsystems.

4. Data Model (etcd)

etcd is a distributed, consistent key-value store based on the Raft consensus algorithm. In our orchestration platform, etcd serves as the single source of truth for all cluster state. Every resource — nodes, pods, services, deployments, secrets, configmaps, and more — is stored as a key-value pair in etcd.

4.1 Key Schema Design

The key hierarchy follows a logical namespace structure that enables efficient prefix-based lookups and watches.

/registry/
├── /pods/
│   ├── /default/
│   │   ├── nginx-deployment-7fb96c846b-xk2jz
│   │   └── nginx-deployment-7fb96c846b-abc12
│   └── /kube-system/
│       ├── coredns-5d78c9869d-pxyzq
│       └── etcd-master-0
├── /services/
│   ├── /default/
│   │   └── kubernetes
│   └── /production/
│       └── api-gateway
├── /deployments/
│   └── /default/
│       └── nginx-deployment
├── /nodes/
│   ├── node-01
│   ├── node-02
│   └── node-03
├── /secrets/
│   └── /default/
│       └── db-credentials
├── /configmaps/
│   └── /default/
│       └── app-config
├── /replicasets/
├── /daemonsets/
├── /statefulsets/
├── /serviceaccounts/
├── /roles/
├── /rolebindings/
├── /clusterroles/
└── /clusterrolebindings/

4.2 Resource Object Model

Every resource in the system follows a consistent object model with metadata, specification, and status fields.

// C# representation of a Kubernetes-style resource object
public class KubernetesResource
{
    public string ApiVersion { get; set; }    // e.g., "v1", "apps/v1"
    public string Kind { get; set; }          // e.g., "Pod", "Deployment"
    public ObjectMetadata Metadata { get; set; }
    public object Spec { get; set; }           // Desired state
    public object Status { get; set; }         // Observed state
}

public class ObjectMetadata
{
    public string Uid { get; set; }            // Unique identifier (UUID)
    public string Name { get; set; }           // Human-readable name
    public string Namespace { get; set; }      // Logical isolation
    public Dictionary<string, string> Labels { get; set; }
    public Dictionary<string, string> Annotations { get; set; }
    public DateTime CreationTimestamp { get; set; }
    public long ResourceVersion { get; set; }  // Optimistic concurrency
    public int Generation { get; set; }        // Incremented on spec change
}

4.3 Pod Object Example

{
    "apiVersion": "v1",
    "kind": "Pod",
    "metadata": {
        "name": "nginx-deployment-7fb96c846b-xk2jz",
        "namespace": "production",
        "uid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
        "labels": {
            "app": "nginx",
            "version": "v2.1.0",
            "environment": "production",
            "team": "platform"
        },
        "resourceVersion": "12345678",
        "generation": 3
    },
    "spec": {
        "nodeName": "node-01",
        "containers": [{
            "name": "nginx",
            "image": "nginx:1.25.3",
            "ports": [{"containerPort": 8080}],
            "resources": {
                "requests": { "cpu": "250m", "memory": "256Mi" },
                "limits": { "cpu": "500m", "memory": "512Mi" }
            },
            "livenessProbe": {
                "httpGet": { "path": "/healthz", "port": 8080 },
                "initialDelaySeconds": 10,
                "periodSeconds": 5
            }
        }],
        "restartPolicy": "Always"
    },
    "status": {
        "phase": "Running",
        "podIP": "10.244.1.42",
        "hostIP": "192.168.1.101",
        "conditions": [...],
        "containerStatuses": [...]
    }
}

4.4 etcd Performance Tuning

For production deployments, etcd requires careful tuning to maintain consistent low-latency performance under heavy write loads.

5. Architecture: The Control Plane

The control plane is the brain of the orchestration platform. It consists of several components that together manage the desired state of the cluster, schedule workloads, and respond to changes. A well-designed control plane must be highly available, fault-tolerant, and capable of handling thousands of concurrent API requests.

5.1 High Availability Design

Production control planes run multiple instances of each component across availability zones. The API server sits behind a load balancer, etcd runs as a 3 or 5-node cluster for Raft consensus, and the scheduler and controller manager use leader election so only one instance is active at a time.

// C# representation of the HA control plane configuration
public class ControlPlaneConfig
{
    public ApiServerConfig ApiServer { get; set; }
    public EtcdConfig Etcd { get; set; }
    public SchedulerConfig Scheduler { get; set; }
    public ControllerManagerConfig ControllerManager { get; set; }
}

public class ApiServerConfig
{
    public List<string> Endpoints { get; set; } = new()
    {
        "https://api-server-1.internal:6443",
        "https://api-server-2.internal:6443",
        "https://api-server-3.internal:6443"
    };
    public string AuditLogFile { get; set; } = "/var/log/k8s/audit.log";
    public int MaxRequestsInFlight { get; set; } = 400;
    public int MaxMutatingRequestsInFlight { get; set; } = 200;
    public int WatchCacheSize { get; set; } = 1000;
}

public class EtcdConfig
{
    public List<string> Endpoints { get; set; } = new()
    {
        "https://etcd-1:2379",
        "https://etcd-2:2379",
        "https://etcd-3:2379"
    };
    public string CaFile { get; set; } = "/etc/etcd/ca.crt";
    public string CertFile { get; set; } = "/etc/etcd/server.crt";
    public string KeyFile { get; set; } = "/etc/etcd/server.key";
    public int QuotaBackendBytes { get; set; } = 8_589_934_592;
    public string AutoCompactionRetention { get; set; } = "8h";
}

5.2 Component Interaction Flow

When a user submits a deployment, the following sequence occurs:

1. The API server validates the request against admission webhooks and schema validation.
2. The validated object is written to etcd with a new resource version.
3. The controller manager detects the new deployment and creates a ReplicaSet.
4. The scheduler watches for unscheduled pods and assigns them to nodes.
5. kubelets on assigned nodes watch for pods assigned to them and start containers.
6. The controller manager continuously monitors actual vs. desired state and takes corrective action.

6. API Server & Authentication

The API server is the central hub and the only component that communicates directly with etcd. Every operation in the cluster — whether from kubectl, the dashboard, or internal controllers — flows through the API server. It is the enforcement point for authentication, authorization, and admission control.

6.1 Authentication Pipeline

The API server processes every incoming request through a multi-stage pipeline: Authentication, Authorization, Admission Control, and finally persistence to etcd.

6.2 Authentication Methods

6.3 C# API Server Implementation

// Simplified API Server request processing pipeline
public class ApiServerPipeline
{
    private readonly IAuthenticationHandler _authHandler;
    private readonly IAuthorizationHandler _rbacHandler;
    private readonly IEnumerable<IAdmissionWebhook> _admissionWebhooks;
    private readonly IEtcdStore _etcdStore;

    public async Task<ApiResponse> ProcessRequestAsync(
        ApiRequest request)
    {
        // Stage 1: Authentication
        var authResult = await _authHandler.AuthenticateAsync(request);
        if (!authResult.IsAuthenticated)
        {
            return ApiResponse.Unauthorized(
                "Unable to authenticate the request");
        }

        // Stage 2: Authorization (RBAC)
        var authzResult = await _rbacHandler.AuthorizeAsync(
            authResult.User,
            request.Resource,
            request.Verb);
        if (!authzResult.IsAuthorized)
        {
            return ApiResponse.Forbidden(
                $"User {authResult.User.Name} is not allowed " +
                $"to {request.Verb} {request.Resource}");
        }

        // Stage 3: Admission Control
        var admissionCtx = new AdmissionContext
        {
            User = authResult.User,
            Resource = request.Resource,
            Operation = request.Verb,
            Object = request.Body
        };

        foreach (var webhook in _admissionWebhooks)
        {
            var result = await webhook.AdmitAsync(admissionCtx);
            if (!result.Allowed)
            {
                return ApiResponse.BadRequest(result.Reason);
            }
            // Mutating webhooks can modify the object
            if (result.ModifiedObject != null)
            {
                admissionCtx.Object = result.ModifiedObject;
            }
        }

        // Stage 4: Persist to etcd
        var resourceVersion = await _etcdStore.UpsertAsync(
            request.Key,
            admissionCtx.Object);

        return ApiResponse.Created(new
        {
            metadata = new { resourceVersion }
        });
    }
}

6.4 Watch Mechanism

The API server implements a watch mechanism that allows clients to receive real-time notifications when resources change. This is the foundation of the controller pattern — every controller watches for specific resource types and reacts to changes.

// Watch implementation - controllers subscribe to resource changes
public class ResourceWatcher<T> where T : KubernetesResource
{
    private readonly HttpClient _apiClient;
    private readonly string _resourcePath;
    private long _lastResourceVersion;

    public event EventHandler<WatchEvent<T>> OnResourceChanged;

    public async Task StartWatchingAsync(CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            try
            {
                var url = $"{_resourcePath}?watch=true" +
                    $"&resourceVersion={_lastResourceVersion}";

                using var stream = await _apiClient
                    .GetStreamAsync(url, ct);

                using var reader = new StreamReader(stream);
                while (!reader.EndOfStream)
                {
                    var line = await reader.ReadLineAsync();
                    var watchEvent = JsonSerializer
                        .Deserialize<WatchEvent<T>>(line);

                    _lastResourceVersion =
                        watchEvent.Object.Metadata.ResourceVersion;

                    OnResourceChanged?.Invoke(this, watchEvent);
                }
            }
            catch (Exception ex)
            {
                // Reconnect with exponential backoff
                await Task.Delay(
                    TimeSpan.FromSeconds(Math.Min(30,
                        Math.Pow(2, _retryCount))), ct);
                _retryCount++;
            }
        }
    }
}

7. Scheduler & Controller Manager

7.1 The Scheduler

The scheduler is responsible for assigning pods to nodes. It watches for pods with no nodeName set and runs them through a multi-stage scheduling pipeline to find the best node. This is one of the most performance-critical components because scheduling latency directly impacts scaling speed.

7.2 Scheduling Pipeline

7.3 Scheduler Implementation

public class Scheduler
{
    private readonly IApiClient _apiClient;
    private readonly List<IFilterPlugin> _filters;
    private readonly List<IScorePlugin> _scorers;
    private readonly PriorityQueue<SchedulablePod, int> _queue;

    public async Task SchedulePodAsync(Pod pod)
    {
        // Step 1: Get all available nodes
        var nodes = await _apiClient.ListNodesAsync();

        // Step 2: Filtering - remove ineligible nodes
        var feasibleNodes = new List<Node>();
        foreach (var node in nodes)
        {
            bool passes = true;
            foreach (var filter in _filters)
            {
                if (!await filter.ShouldScheduleAsync(pod, node))
                {
                    passes = false;
                    break;
                }
            }
            if (passes) feasibleNodes.Add(node);
        }

        if (!feasibleNodes.Any())
        {
            // Pod is unschedulable - add to backoff queue
            _queue.Enqueue(new SchedulablePod(pod), priority: 0);
            return;
        }

        // Step 3: Scoring - rank feasible nodes
        var scoredNodes = new List<(Node Node, int Score)>();
        foreach (var node in feasibleNodes)
        {
            int totalScore = 0;
            foreach (var scorer in _scorers)
            {
                totalScore += await scorer.ScoreAsync(pod, node);
            }
            scoredNodes.Add((node, totalScore));
        }

        // Step 4: Select the highest-scoring node
        var selected = scoredNodes
            .OrderByDescending(x => x.Score)
            .First();

        // Step 5: Bind the pod to the selected node
        pod.Spec.NodeName = selected.Node.Metadata.Name;
        await _apiClient.BindPodAsync(pod);
    }
}

// Example filter: Check if node has enough resources
public class ResourceFilter : IFilterPlugin
{
    public async Task<bool> ShouldScheduleAsync(Pod pod, Node node)
    {
        var requested = pod.Spec.Containers
            .Sum(c => c.Resources.Requests.CpuMillicores);
        var available = node.Status.Allocatable.CpuMillicores
            - node.Status.Used.CpuMillicores;

        return available >= requested;
    }
}

// Example scorer: Prefer nodes with least requested resources
public class LeastRequestedScorer : IScorePlugin
{
    public async Task<int> ScoreAsync(Pod pod, Node node)
    {
        var totalCpu = node.Status.Allocatable.CpuMillicores;
        var usedCpu = node.Status.Used.CpuMillicores;
        var requestedCpu = pod.Spec.Containers
            .Sum(c => c.Resources.Requests.CpuMillicores);

        // Score 0-100, higher = less utilized = preferred
        var score = (int)((1.0 - (double)(usedCpu + requestedCpu)
            / totalCpu) * 100);
        return Math.Max(0, Math.Min(100, score));
    }
}

7.4 Controller Manager

The controller manager runs multiple controllers, each responsible for a specific resource type. Every controller follows the same reconciliation loop pattern: watch for resource changes, compare actual state to desired state, and take action to converge.

// Base controller reconciliation loop pattern
public abstract class ReconciliationController<T> where T : KubernetesResource
{
    private readonly IApiClient _apiClient;
    private readonly TimeSpan _resyncPeriod = TimeSpan.FromMinutes(5);

    protected abstract string ResourcePath { get; }
    protected abstract Task<bool> ReconcileAsync(T resource);

    public async Task RunAsync(CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            try
            {
                // List all resources and watch for changes
                var resources = await _apiClient
                    .ListAsync<T>(ResourcePath);

                foreach (var resource in resources)
                {
                    await ReconcileAsync(resource);
                }

                // Watch for future changes
                await WatchAsync(ct);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex,
                    "Reconciliation loop failed, retrying");
                await Task.Delay(_resyncPeriod, ct);
            }
        }
    }

    protected async Task<bool> ReconcileAsync(T resource)
    {
        // Compare desired state with actual state
        var desiredState = GetDesiredState(resource);
        var actualState = await GetActualStateAsync(resource);

        if (StatesMatch(desiredState, actualState))
        {
            return true; // Already in desired state
        }

        // Take corrective action
        return await TakeActionAsync(resource, desiredState, actualState);
    }
}

7.5 Key Controllers

8. kubelet & kube-proxy

While the control plane manages the cluster-level state, kubelet and kube-proxy run on every worker node and manage the node-level operations. Together they form the data plane — the layer that actually runs containers and routes network traffic.

8.1 kubelet

kubelet is the node agent that ensures containers described in pod specs assigned to its node are running and healthy. It communicates with the container runtime (containerd, CRI-O) via the Container Runtime Interface (CRI).

// C# representation of kubelet's core loop
public class KubeletNodeAgent
{
    private readonly IContainerRuntime _runtime;
    private readonly IApiClient _apiClient;
    private readonly IMetricsCollector _metrics;
    private readonly Dictionary<string, ManagedPod> _managedPods;

    public async Task RunAsync(CancellationToken ct)
    {
        // Register this node with the API server
        await RegisterNodeAsync();

        // Main sync loop
        while (!ct.IsCancellationRequested)
        {
            try
            {
                // Get pods assigned to this node
                var desiredPods = await _apiClient
                    .ListPodsForNodeAsync(NodeName);

                // Sync desired state
                await SyncPodsAsync(desiredPods);

                // Report node status
                await ReportNodeStatusAsync();

                // Collect and report metrics
                await ReportMetricsAsync();

                // Run garbage collection
                await GarbageCollectAsync();

                await Task.Delay(TimeSpan.FromSeconds(5), ct);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Kubelet sync loop error");
                await Task.Delay(TimeSpan.FromSeconds(5), ct);
            }
        }
    }

    private async Task SyncPodsAsync(List<Pod> desiredPods)
    {
        var desiredUids = desiredPods
            .Select(p => p.Metadata.Uid).ToHashSet();

        // Kill pods that are no longer desired
        foreach (var (uid, managed) in _managedPods)
        {
            if (!desiredUids.Contains(uid))
            {
                await _runtime.StopContainerAsync(
                    managed.ContainerId);
                _managedPods.Remove(uid);
            }
        }

        // Start pods that are desired but not running
        foreach (var pod in desiredPods)
        {
            if (!_managedPods.ContainsKey(pod.Metadata.Uid))
            {
                await StartPodAsync(pod);
            }
        }
    }

    private async Task StartPodAsync(Pod pod)
    {
        foreach (var container in pod.Spec.Containers)
        {
            // Pull image if not present
            await _runtime.PullImageAsync(container.Image);

            // Create and start container
            var containerId = await _runtime.CreateContainerAsync(
                new ContainerConfig
                {
                    Image = container.Image,
                    Command = container.Command,
                    Args = container.Args,
                    EnvVars = container.Env,
                    Ports = container.Ports,
                    ResourceLimits = container.Resources.Limits,
                    ResourceRequests = container.Resources.Requests,
                    Mounts = GetMounts(pod, container)
                });

            await _runtime.StartContainerAsync(containerId);

            _managedPods[pod.Metadata.Uid] = new ManagedPod
            {
                Pod = pod,
                ContainerId = containerId
            };
        }
    }
}

8.2 kube-proxy

kube-proxy maintains network rules on each node to implement the Service abstraction. When a pod sends traffic to a Service ClusterIP, kube-proxy rules intercept and redirect that traffic to one of the backing pods. There are several proxy modes.

9. Pod Lifecycle & Management

Understanding the complete pod lifecycle is essential for building a reliable orchestration platform. Pods progress through a well-defined set of phases, and each phase transition must be handled correctly by the system.

9.1 Pod Phases

9.2 Pod Conditions

9.3 Probe Types

Probes are health-check mechanisms that kubelet uses to determine container health. Each probe type serves a distinct purpose in the pod lifecycle.

// C# implementation of probe evaluation logic
public class ProbeEvaluator
{
    public async Task<ProbeResult> EvaluateLivenessAsync(
        Container container, IContainerRuntime runtime)
    {
        var probe = container.LivenessProbe;
        if (probe == null) return ProbeResult.Success;

        try
        {
            var result = probe.Type switch
            {
                ProbeType.HttpGet => await EvaluateHttpProbeAsync(
                    probe.HttpGet),
                ProbeType.TcpSocket => await EvaluateTcpProbeAsync(
                    probe.TcpSocket),
                ProbeType.Exec => await EvaluateExecProbeAsync(
                    probe.Exec, runtime),
                ProbeType.Grpc => await EvaluateGrpcProbeAsync(
                    probe.Grpc),
                _ => ProbeResult.Success
            };

            // Update consecutive failure/success counts
            if (result == ProbeResult.Failure)
            {
                container.ConsecutiveFailures++;
                if (container.ConsecutiveFailures >= probe.FailureThreshold)
                {
                    return ProbeResult.Failure; // Trigger restart
                }
                return ProbeResult.Success; // Still within threshold
            }
            else
            {
                container.ConsecutiveFailures = 0;
                return ProbeResult.Success;
            }
        }
        catch (Exception)
        {
            container.ConsecutiveFailures++;
            if (container.ConsecutiveFailures >= probe.FailureThreshold)
                return ProbeResult.Failure;
            return ProbeResult.Success;
        }
    }

    private async Task<ProbeResult> EvaluateHttpProbeAsync(
        HttpGetAction action)
    {
        using var client = new HttpClient { Timeout = TimeSpan.FromSeconds(3) };
        var url = $"http://{action.Host}:{action.Port}{action.Path}";

        try
        {
            var response = await client.GetAsync(url);
            return response.StatusCode >= 200 &&
                   response.StatusCode < 400
                ? ProbeResult.Success
                : ProbeResult.Failure;
        }
        catch
        {
            return ProbeResult.Failure;
        }
    }
}

9.4 Init Containers

Init containers run sequentially before the main application containers start. They are useful for setup tasks such as database migrations, waiting for external services, or downloading configuration files. The pod will not proceed to the Running phase until all init containers complete successfully.

9.5 Pod Disruption Budgets (PDB)

PDBs protect against voluntary disruptions such as node drains, cluster upgrades, and autoscaler events. They specify the minimum number or percentage of pods that must remain available.

{
    "apiVersion": "policy/v1",
    "kind": "PodDisruptionBudget",
    "metadata": {
        "name": "api-server-pdb",
        "namespace": "production"
    },
    "spec": {
        "minAvailable": "60%",
        "selector": {
            "matchLabels": {
                "app": "api-server"
            }
        }
    }
}

10. Deployment & ReplicaSet

Deployments are the most common workload type in Kubernetes. They provide declarative updates for pods and ReplicaSets, enabling zero-downtime rolling updates and automatic rollbacks.

10.1 Deployment Strategy: Rolling Update

During a rolling update, the Deployment controller gradually replaces old pods with new ones, ensuring that the desired number of replicas is always maintained. The two key parameters are maxSurge (how many extra pods can be created) and maxUnavailable (how many pods can be down during the update).

// C# implementation of the Deployment controller's rolling update logic
public class DeploymentController
{
    public async Task<bool> ReconcileDeploymentAsync(
        Deployment deployment)
    {
        // Find the ReplicaSet managed by this deployment
        var replicaSets = await _apiClient
            .ListReplicaSetsAsync(deployment.Metadata.Namespace,
                deployment.Metadata.Labels);

        var currentRs = replicaSets
            .Where(rs => MatchesSelector(deployment, rs))
            .OrderByDescending(rs => rs.Metadata.CreationTimestamp)
            .FirstOrDefault();

        var desiredReplicas = deployment.Spec.Replicas;
        var maxSurge = deployment.Spec.Strategy.RollingUpdate
            ?.MaxSurge ?? 1;
        var maxUnavailable = deployment.Spec.Strategy.RollingUpdate
            ?.MaxUnavailable ?? 0;

        // Check if update is in progress
        if (deployment.Spec.Template.Spec.Image !=
            currentRs?.Spec.Template.Spec.Containers[0].Image)
        {
            // Create new ReplicaSet with updated template
            var newRs = await CreateReplicaSetAsync(deployment,
                deployment.Spec.Template);

            // Scale down old, scale up new
            var totalPods = currentRs.Status.Replicas + newRs.Status.Replicas;
            var desiredTotal = desiredReplicas +
                Math.Min(maxSurge, desiredReplicas);

            // Calculate scaling for both ReplicaSets
            var (oldScale, newScale) = CalculateRollingScales(
                currentRs.Status.Replicas,
                newRs.Status.Replicas,
                desiredReplicas,
                maxSurge,
                maxUnavailable);

            await _apiClient.ScaleReplicaSetAsync(currentRs, oldScale);
            await _apiClient.ScaleReplicaSetAsync(newRs, newScale);

            return false; // Not yet complete
        }

        // Normal scaling (no update in progress)
        await ScaleReplicaSetAsync(currentRs, desiredReplicas);
        return true;
    }

    private (int oldScale, int newScale) CalculateRollingScales(
        int currentReplicas, int newReplicas,
        int desired, int maxSurge, int maxUnavailable)
    {
        int available = currentReplicas + newReplicas;
        int targetAvailable = desired - maxUnavailable;

        int newTarget = Math.Min(
            desired,
            Math.Max(newReplicas, targetAvailable - currentReplicas));

        int oldTarget = Math.Min(
            desired,
            Math.Max(currentReplicas,
                desired + maxSurge - newReplicas));

        return (oldTarget, newTarget);
    }
}

10.2 Rollback Mechanism

When a deployment update fails (detected by pod readiness failures or exceeding the progress deadline), the system automatically reverts to the previous ReplicaSet. The rollback process is the same as a forward update — it is simply a rolling update from the failed template to the previous template.

11. Service & Ingress

Services provide stable network endpoints for a dynamic set of pods. Since pods are ephemeral and can be rescheduled at any time, Services provide a fixed IP address and DNS name that routes traffic to healthy backing pods.

11.1 Service Types

11.2 Service Discovery

// Service DNS resolution within the cluster
// Format: <service-name>.<namespace>.svc.cluster.local
// Example: api-gateway.production.svc.cluster.local

// C# service discovery client
public class ServiceDiscoveryClient
{
    private readonly IDnsResolver _dns;

    public async Task<List<Endpoint>> ResolveServiceAsync(
        string serviceName, string ns)
    {
        // Kubernetes DNS resolves to cluster IP
        var clusterIp = await _dns.ResolveAsync(
            $"{serviceName}.{ns}.svc.cluster.local");

        // kube-proxy rules redirect ClusterIP traffic to pod IPs
        var endpoints = await _apiClient.GetEndpointsAsync(
            serviceName, ns);

        return endpoints.Subsets
            .SelectMany(s => s.Addresses
                .Select(a => new Endpoint
                {
                    Ip = a.Ip,
                    Port = s.Ports.First().Port,
                    Ready = a.Conditions.Ready == true
                }))
            .Where(e => e.Ready)
            .ToList();
    }
}

11.3 Ingress Controllers

Ingress provides HTTP/HTTPS routing at the cluster edge. An Ingress controller (typically NGINX, Traefik, or Envoy) reads Ingress resources and configures routing rules.

{
    "apiVersion": "networking.k8s.io/v1",
    "kind": "Ingress",
    "metadata": {
        "name": "api-ingress",
        "annotations": {
            "nginx.ingress.kubernetes.io/ssl-redirect": "true",
            "nginx.ingress.kubernetes.io/rate-limit": "1000",
            "cert-manager.io/cluster-issuer": "letsencrypt-prod"
        }
    },
    "spec": {
        "tls": [{
            "hosts": ["api.example.com"],
            "secretName": "api-tls-cert"
        }],
        "rules": [{
            "host": "api.example.com",
            "http": {
                "paths": [
                    {
                        "path": "/v1",
                        "pathType": "Prefix",
                        "backend": {
                            "service": {
                                "name": "api-v1",
                                "port": { "number": 8080 }
                            }
                        }
                    },
                    {
                        "path": "/v2",
                        "pathType": "Prefix",
                        "backend": {
                            "service": {
                                "name": "api-v2",
                                "port": { "number": 8080 }
                            }
                        }
                    }
                ]
            }
        }]
    }
}

11.4 Service Mesh Integration

For advanced traffic management, service meshes like Istio or Linkerd inject sidecar proxies alongside each pod. These proxies handle mTLS, load balancing, circuit breaking, retries, and observability without application code changes. The sidecar intercepts all inbound and outbound traffic, creating a transparent proxy layer.

12. ConfigMap & Secrets

Configuration management in containerized environments requires separating configuration from code. ConfigMaps handle non-sensitive configuration while Secrets handle sensitive data such as passwords, certificates, and API keys.

12.1 ConfigMap Structure

{
    "apiVersion": "v1",
    "kind": "ConfigMap",
    "metadata": {
        "name": "app-config",
        "namespace": "production"
    },
    "data": {
        "DATABASE_HOST": "postgres-cluster.internal",
        "DATABASE_PORT": "5432",
        "LOG_LEVEL": "info",
        "FEATURE_FLAG_NEW_UI": "true",
        "nginx.conf": "server { listen 80; ... }"
    }
}

12.2 Secrets Management

Secrets are base64-encoded by default but should be encrypted at rest for production. The encryption at rest feature uses AES-CBC or AES-GCM to encrypt Secret objects before writing them to etcd.

// Secret encryption configuration
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}  # Fallback for reading unencrypted secrets

12.3 Secret Injection Methods

13. Persistent Volumes & Storage

Stateful workloads such as databases, message queues, and file servers require persistent storage that survives pod restarts and rescheduling. The Container Storage Interface (CSI) provides a standard API for storage providers to integrate with the orchestration platform.

13.1 Storage Architecture

13.2 StatefulSet for Stateful Workloads

// C# representation of StatefulSet management
public class StatefulSetController
{
    public async Task<bool> ReconcileStatefulSetAsync(
        StatefulSet sts)
    {
        var pods = await _apiClient.ListPodsAsync(
            sts.Metadata.Namespace,
            sts.Metadata.Labels);

        var healthyPods = pods
            .Where(p => p.Status.Phase == PodPhase.Running)
            .OrderBy(p => p.Metadata.Name)
            .ToList();

        // StatefulSets manage pods sequentially
        var desiredReplicas = sts.Spec.Replicas;

        if (healthyPods.Count < desiredReplicas)
        {
            // Create next pod in order (ordinal index)
            var nextOrdinal = healthyPods.Count;
            var podName = $"{sts.Metadata.Name}-{nextOrdinal}";

            // Ensure PVC exists before creating pod
            var pvcName = $"{sts.Spec.VolumeClaimTemplates[0]
                .Metadata.Name}-{podName}";
            await EnsurePvcExistsAsync(pvcName, sts);

            // Create pod with stable network identity
            await CreateStatefulPodAsync(sts, podName, nextOrdinal);

            return false;
        }

        // Verify ordered readiness
        for (int i = 0; i < healthyPods.Count; i++)
        {
            if (healthyPods[i].Metadata.Name !=
                $"{sts.Metadata.Name}-{i}")
            {
                return false; // Out of order
            }
        }

        return true;
    }
}

13.3 Storage Classes

14. RBAC & Security

Role-Based Access Control (RBAC) is the primary authorization mechanism in Kubernetes. It determines which users, groups, and service accounts can perform which actions on which resources. A well-designed RBAC policy follows the principle of least privilege.

14.1 RBAC Components

14.2 RBAC Example

// Namespace-scoped Role: allows read-only access to pods
{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "Role",
    "metadata": {
        "name": "pod-reader",
        "namespace": "production"
    },
    "rules": [
        {
            "apiGroups": [""],
            "resources": ["pods", "pods/log"],
            "verbs": ["get", "list", "watch"]
        },
        {
            "apiGroups": ["apps"],
            "resources": ["deployments", "replicasets"],
            "verbs": ["get", "list", "watch"]
        }
    ]
}

// RoleBinding: assigns the role to a service account
{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "RoleBinding",
    "metadata": {
        "name": "read-pods-binding",
        "namespace": "production"
    },
    "subjects": [
        {
            "kind": "ServiceAccount",
            "name": "monitoring-agent",
            "namespace": "production"
        }
    ],
    "roleRef": {
        "kind": "Role",
        "name": "pod-reader",
        "apiGroup": "rbac.authorization.k8s.io"
    }
}

14.3 C# RBAC Evaluator

public class RbacAuthorizer
{
    public async Task<AuthorizationResult> AuthorizeAsync(
        UserInfo user, string resource, string verb,
        string namespaceName)
    {
        // Get all ClusterRoles bound to this user
        var clusterRoles = await GetBoundClusterRolesAsync(user);
        foreach (var role in clusterRoles)
        {
            if (role.Matches(resource, verb, namespaceName))
            {
                return AuthorizationResult.Allowed(
                    $"Granted by ClusterRole {role.Name}");
            }
        }

        // Get all namespace-scoped Roles bound to this user
        if (!string.IsNullOrEmpty(namespaceName))
        {
            var nsRoles = await GetBoundRolesAsync(
                user, namespaceName);
            foreach (var role in nsRoles)
            {
                if (role.Matches(resource, verb))
                {
                    return AuthorizationResult.Allowed(
                        $"Granted by Role {role.Name} " +
                        $"in {namespaceName}");
                }
            }
        }

        return AuthorizationResult.Denied(
            $"No RBAC rules allow {verb} on {resource}");
    }
}

public class ClusterRole
{
    public string Name { get; set; }
    public List<PolicyRule> Rules { get; set; }

    public bool Matches(string resource, string verb,
        string requestedNamespace = null)
    {
        return Rules.Any(rule =>
            rule.Verbs.Contains(verb) &&
            rule.Resources.Contains(resource) &&
            (string.IsNullOrEmpty(requestedNamespace) ||
             rule.Namespaces.Contains(requestedNamespace) ||
             rule.Namespaces.Contains("*")));
    }
}

14.4 Security Best Practices

• Pod Security Standards: Use restricted, baseline, or privileged profiles to control what pod configurations are allowed.
• Network Policies: Implement default-deny ingress and egress, then whitelist specific pod-to-pod communication.
• Service Accounts: Disable auto-mounting of service account tokens when pods do not need cluster API access.
• Image Security: Use image scanning (Trivy, Snyk), sign images with Sigstore/Cosign, and enforce image pull policies.
• Runtime Security: Use Falco or Tetragon for runtime threat detection, seccomp profiles for syscall filtering, and AppArmor/SELinux for mandatory access control.
• Supply Chain Security: Implement SLSA framework compliance, use SBOM generation, and verify image provenance.

15. Helm Charts & Package Management

Helm is the package manager for Kubernetes. It provides a templating engine, release management, and a repository system for distributing reusable application definitions. Understanding Helm internals is critical for managing complex deployments at scale.

15.1 Helm Chart Structure

my-chart/
├── Chart.yaml          # Chart metadata
├── Chart.lock          # Dependency lock file
├── values.yaml         # Default configuration values
├── templates/
│   ├── _helpers.tpl    # Template helper functions
│   ├── deployment.yaml
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── configmap.yaml
│   ├── secret.yaml
│   ├── hpa.yaml
│   ├── serviceaccount.yaml
│   └── tests/
│       └── test-connection.yaml
└── charts/             # Sub-chart dependencies

15.2 Helm Template Example

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "mychart.fullname" . }}
  labels:
    {{- include "mychart.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "mychart.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      labels:
        {{- include "mychart.selectorLabels" . | nindent 8 }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          ports:
            - containerPort: {{ .Values.service.targetPort }}
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
          {{- if .Values.livenessProbe.enabled }}
          livenessProbe:
            httpGet:
              path: {{ .Values.livenessProbe.path }}
              port: {{ .Values.service.targetPort }}
            initialDelaySeconds: {{ .Values.livenessProbe.initialDelay }}
            periodSeconds: {{ .Values.livenessProbe.period }}
          {{- end }}
          {{- if .Values.readinessProbe.enabled }}
          readinessProbe:
            httpGet:
              path: {{ .Values.readinessProbe.path }}
              port: {{ .Values.service.targetPort }}
            initialDelaySeconds: {{ .Values.readinessProbe.initialDelay }}
            periodSeconds: {{ .Values.readinessProbe.period }}
          {{- end }}

15.3 Helm Values

# values.yaml - Default configuration
replicaCount: 3

image:
  repository: myregistry.io/api-server
  tag: "v2.1.0"
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80
  targetPort: 8080

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

livenessProbe:
  enabled: true
  path: /healthz
  initialDelay: 10
  period: 5

readinessProbe:
  enabled: true
  path: /ready
  initialDelay: 5
  period: 3

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 20
  targetCPU: 70
  targetMemory: 80

15.4 C# Helm Chart Installer

public class HelmChartInstaller
{
    public async Task<HelmRelease> InstallChartAsync(
        HelmInstallRequest request)
    {
        // Step 1: Template the chart
        var renderedManifests = await RenderChartAsync(
            request.ChartPath, request.Values);

        // Step 2: Apply resource ordering
        var orderedManifests = ApplyResourceOrdering(renderedManifests);

        // Step 3: Apply manifests with server-side apply
        foreach (var manifest in orderedManifests)
        {
            await _apiClient.ServerSideApplyAsync(manifest);
        }

        // Step 4: Record release in Helm storage
        var release = new HelmRelease
        {
            Name = request.ReleaseName,
            Chart = request.ChartPath,
            Revision = 1,
            Status = HelmReleaseStatus.Deployed,
            Values = request.Values,
            Timestamp = DateTime.UtcNow
        };

        await _storage.SaveReleaseAsync(release);
        return release;
    }

    private List<string> ApplyResourceOrdering(
        List<string> manifests)
    {
        var order = new Dictionary<string, int>
        {
            ["Namespace"] = 1,
            ["ServiceAccount"] = 2,
            ["ConfigMap"] = 3,
            ["Secret"] = 4,
            ["PersistentVolumeClaim"] = 5,
            ["Service"] = 6,
            ["Deployment"] = 7,
            ["DaemonSet"] = 7,
            ["StatefulSet"] = 7,
            ["Ingress"] = 8,
            ["PodDisruptionBudget"] = 9,
            ["HorizontalPodAutoscaler"] = 10
        };

        return manifests
            .OrderBy(m =>
            {
                var kind = ExtractKind(m);
                return order.TryGetValue(kind, out var o) ? o : 50;
            })
            .ToList();
    }
}

16. Auto-Scaling (HPA / VPA)

Auto-scaling is essential for optimizing resource utilization and maintaining performance during traffic spikes. The platform supports two primary auto-scaling mechanisms: Horizontal Pod Autoscaler (HPA) for scaling replica counts, and Vertical Pod Autoscaler (VPA) for adjusting resource requests.

16.1 HPA Algorithm

The HPA uses a proportional control algorithm to calculate the desired replica count based on current vs. target metrics.

desiredReplicas = ceil(
    currentReplicas × (
        currentMetricValue / targetMetricValue
    )
)

16.2 HPA Manifest

{
    "apiVersion": "autoscaling/v2",
    "kind": "HorizontalPodAutoscaler",
    "metadata": {
        "name": "api-hpa",
        "namespace": "production"
    },
    "spec": {
        "scaleTargetRef": {
            "apiVersion": "apps/v1",
            "kind": "Deployment",
            "name": "api-server"
        },
        "minReplicas": 3,
        "maxReplicas": 50,
        "metrics": [
            {
                "type": "Resource",
                "resource": {
                    "name": "cpu",
                    "target": {
                        "type": "Utilization",
                        "averageUtilization": 70
                    }
                }
            },
            {
                "type": "Resource",
                "resource": {
                    "name": "memory",
                    "target": {
                        "type": "Utilization",
                        "averageUtilization": 80
                    }
                }
            },
            {
                "type": "Pods",
                "pods": {
                    "metric": {
                        "name": "http_requests_per_second"
                    },
                    "target": {
                        "type": "AverageValue",
                        "averageValue": "1000"
                    }
                }
            }
        ],
        "behavior": {
            "scaleUp": {
                "stabilizationWindowSeconds": 60,
                "policies": [
                    {
                        "type": "Percent",
                        "value": 100,
                        "periodSeconds": 60
                    }
                ]
            },
            "scaleDown": {
                "stabilizationWindowSeconds": 300,
                "policies": [
                    {
                        "type": "Percent",
                        "value": 10,
                        "periodSeconds": 60
                    }
                ]
            }
        }
    }
}

16.3 VPA (Vertical Pod Autoscaler)

VPA adjusts the CPU and memory requests of pods based on historical usage. It recommends or automatically applies resource changes, helping right-size containers without manual tuning.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
      - containerName: api
        minAllowed:
          cpu: 100m
          memory: 128Mi
        maxAllowed:
          cpu: 4
          memory: 8Gi
        controlledResources: ["cpu", "memory"]

17. Network Policies

Network Policies are the Kubernetes mechanism for implementing micro-segmentation and zero-trust networking. By default, all pods in a cluster can communicate with all other pods. Network Policies restrict this by specifying which pod-to-pod communication is allowed.

17.1 Default Deny Pattern

# Deny all ingress traffic in a namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress

17.2 Allow Specific Communication

# Allow frontend to communicate with backend only
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080

17.3 C# Network Policy Evaluator

public class NetworkPolicyEvaluator
{
    private readonly List<NetworkPolicy> _policies;

    public bool IsConnectionAllowed(
        Pod source, Pod destination, int port)
    {
        // Find all policies targeting the destination pod
        var applicablePolicies = _policies
            .Where(p =>
                p.Namespace == destination.Metadata.Namespace &&
                MatchesSelector(p.Spec.PodSelector,
                    destination) &&
                p.Spec.PolicyTypes.Contains("Ingress"))
            .ToList();

        if (!applicablePolicies.Any())
        {
            // No policies = allow all (if no default deny)
            return !_hasDefaultDeny[destination.Metadata.Namespace];
        }

        // Check if any policy allows this connection
        return applicablePolicies.Any(policy =>
            policy.Spec.Ingress.Any(rule =>
                rule.From.Any(sourceRule =>
                    MatchesPodSelector(sourceRule.PodSelector,
                        source)) &&
                rule.Ports.Any(p =>
                    p.Port == port)));
    }
}

18. Monitoring: Prometheus & Grafana

Observability is not optional in production. The monitoring stack provides visibility into cluster health, application performance, and resource utilization. Prometheus collects metrics, Alertmanager handles alerting, and Grafana provides visualization.

18.1 Prometheus Architecture

18.2 Key Metrics to Monitor

18.3 PrometheusRule Example

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cluster-alerts
spec:
  groups:
    - name: cluster-health
      rules:
        - alert: HighAPIServerLatency
          expr: |
            histogram_quantile(0.99,
              rate(apiserver_request_duration_seconds_bucket[5m])
            ) > 1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "API server p99 latency exceeds 1s"
            description: "API server latency has been above 1s
              for 5 minutes. Current value: {{ $value }}s"

        - alert: PodCrashLooping
          expr: |
            increase(kube_pod_container_status_restarts_total[1h]) > 5
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.pod }} is crash looping"

        - alert: NodeDiskSpaceLow
          expr: |
            (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
            / node_filesystem_size_bytes) < 0.2
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Node {{ $labels.instance }} disk space
              below 20%"

18.4 Custom Metrics Adapter

To enable HPA scaling on custom application metrics (not just CPU/memory), the Custom Metrics API adapter translates application-specific Prometheus queries into the metrics API that HPA consumes.

// Custom metrics configuration for HPA
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-metrics-config
data:
  config.yaml: |
    rules:
      - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
        resources:
          overrides:
            namespace: {resource: "namespace"}
            pod: {resource: "pod"}
        name:
          matches: "^(.*)_total$"
          as: "${1}_per_second"
        metricsQuery: 'sum(rate(<>[1m])) by (<.GroupBy>)'

19. Logging: The EFK Stack

Centralized logging aggregates logs from all containers and nodes into a searchable, analyzable system. The EFK stack (Elasticsearch, Fluentd, Kibana) is the most common logging solution, though Loki has gained significant traction as a lighter alternative.

19.1 Logging Architecture

19.2 Fluentd Configuration

# Fluentd configuration for Kubernetes log collection
<source>
  @type tail
  @id in_tail_container_logs
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  exclude_path ["/var/log/containers/fluentd*"]
  read_from_head true
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      keep_time_key true
    </pattern>
    <pattern>
      format regexp
      expression /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
      time_format %Y-%m-%dT%H:%M:%S.%N%:z
    </pattern>
  </parse>
</source>

# Enrich with Kubernetes metadata
<filter kubernetes.**>
  @type kubernetes_metadata
  @id filter_kube_metadata
  kubernetes_url "https://#{ENV['KUBERNETES_SERVICE_HOST']}:#{ENV['KUBERNETES_SERVICE_PORT']}"
  verify_ssl true
  ca_file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  bearer_token_file /var/run/secrets/kubernetes.io/serviceaccount/token
  skip_labels false
  skip_container_metadata false
  skip_master_url true
  skip_namespace_metadata true
</filter>

# Output to Elasticsearch
<match kubernetes.**>
  @type elasticsearch
  @id out_es
  @log_level info
  host elasticsearch.logging.svc.cluster.local
  port 9200
  index_name kubernetes-logs
  type_name _doc
  logstash_format true
  logstash_prefix k8s-logs
  logstash_dateformat %Y.%m.%d
  include_tag_key true
  flush_interval 5s
  <buffer>
    @type file
    path /var/log/fluentd-buffers/kubernetes.buffer
    flush_mode interval
    retry_type exponential_backoff
    flush_interval 5s
    chunk_limit_size 2M
    queue_limit_length 8
    overflow_action block
  </buffer>
</match>

19.3 Log Retention Strategy

20. Multi-Cluster Federation

As organizations scale, a single cluster becomes insufficient for global reach, regulatory compliance, blast radius containment, or cloud provider diversity. Multi-cluster federation distributes workloads across multiple independent clusters.

20.1 Federation Patterns

20.2 C# Cluster Registry

public class MultiClusterManager
{
    private readonly Dictionary<string, IApiClient> _clusterClients;

    public async Task<ClusterDeploymentResult> DeployMultiClusterAsync(
        ApplicationManifest manifest,
        List<ClusterTarget> targets)
    {
        var results = new List<ClusterDeploymentResult>();

        foreach (var target in targets)
        {
            var client = _clusterClients[target.ClusterName];

            try
            {
                // Apply namespace and RBAC if needed
                await EnsureNamespaceAsync(client, target);

                // Apply manifests with cluster-specific overrides
                var adaptedManifest = AdaptManifestForCluster(
                    manifest, target);

                await client.ApplyAsync(adaptedManifest);

                // Verify deployment health
                var health = await WaitForHealthyAsync(
                    client, adaptedManifest);

                results.Add(new ClusterDeploymentResult
                {
                    Cluster = target.ClusterName,
                    Status = DeploymentStatus.Success,
                    Health = health
                });
            }
            catch (Exception ex)
            {
                results.Add(new ClusterDeploymentResult
                {
                    Cluster = target.ClusterName,
                    Status = DeploymentStatus.Failed,
                    Error = ex.Message
                });
            }
        }

        return AggregateResults(results);
    }

    private ApplicationManifest AdaptManifestForCluster(
        ApplicationManifest manifest, ClusterTarget target)
    {
        // Apply cluster-specific resource overrides
        var adapted = manifest.Clone();

        if (target.ResourceProfile == ResourceProfile.HighMemory)
        {
            adapted.SetResourceLimits("memory", "16Gi");
            adapted.SetReplicas(Math.Max(manifest.Replicas, 5));
        }

        // Apply cluster-specific config
        adapted.SetConfigValue("CLUSTER_NAME", target.ClusterName);
        adapted.SetConfigValue("REGION", target.Region);

        return adapted;
    }
}

20.3 Global Load Balancing

Global load balancers distribute traffic across clusters in different regions. They use DNS-based routing (latency-based, weighted, or geolocation) to direct users to the nearest healthy cluster.

21. Cost Estimation

Running a production Kubernetes cluster involves significant infrastructure costs. Understanding these costs is essential for capacity planning and budget management.

21.1 Infrastructure Cost Breakdown

21.2 Cost Optimization Strategies

• Right-Sizing: Use VPA recommendations to right-size pod resource requests. Over-provisioned requests waste cluster capacity.
• Spot/Preemptible Instances: Use spot instances for batch workloads and non-critical services. Savings of 60-80% compared to on-demand.
• Cluster Autoscaler: Automatically remove underutilized nodes during low-traffic periods.
• Bin Packing: The scheduler's LeastRequestedPriority scoring helps pack pods densely onto nodes, reducing the number of active nodes.
• Reserved Instances: Commit to 1-3 year reserved instances for baseline workloads. Savings of 30-50%.

21.3 Cost Monitoring

// C# cost estimation model
public class ClusterCostEstimator
{
    public ClusterCostReport EstimateMonthlyCost(
        ClusterSpec spec)
    {
        var report = new ClusterCostReport();

        // Master node costs
        report.MasterNodes = spec.MasterNodes *
            GetNodeCost(spec.MasterNodeSpec);

        // Worker node costs (separate by pool)
        foreach (var pool in spec.NodePools)
        {
            report.WorkerNodes += pool.Count *
                GetNodeCost(pool.Spec);
        }

        // Storage costs
        report.Storage = spec.PersistentStorageGB *
            0.10m; // $0.10/GB/month for SSD

        // Network costs
        report.NetworkEgress = spec.MonthlyEgressTB *
            85m; // $85/TB egress

        // Monitoring & logging overhead
        report.Overhead = report.Total * 0.12m; // ~12% overhead

        report.Total = report.MasterNodes +
            report.WorkerNodes + report.Storage +
            report.NetworkEgress + report.Overhead;

        return report;
    }
}

22. Testing

Testing an orchestration platform requires a multi-layered strategy that validates correctness at every level — from individual controller logic to end-to-end cluster behavior.

22.1 Testing Pyramid

22.2 Test Categories

22.3 C# Controller Unit Test

public class DeploymentControllerTests
{
    [Fact]
    public async Task RollingUpdate_CreatesNewReplicaSet()
    {
        // Arrange
        var mockApi = new Mock<IApiClient>();
        mockApi.Setup(x => x.ListReplicaSetsAsync(
            It.IsAny<string>(), It.IsAny<Dictionary<string, string>>()))
            .ReturnsAsync(new List<ReplicaSet>
            {
                new ReplicaSet
                {
                    Metadata = new ObjectMetadata
                    {
                        Name = "app-v1",
                        Labels = new Dictionary<string, string>
                        {
                            ["app"] = "web"
                        }
                    },
                    Spec = new ReplicaSetSpec
                    {
                        Replicas = 5,
                        Template = new PodTemplate
                        {
                            Containers = new List<Container>
                            {
                                new() { Image = "app:v1" }
                            }
                        }
                    },
                    Status = new ReplicaSetStatus { Replicas = 5 }
                }
            });

        var deployment = new Deployment
        {
            Metadata = new ObjectMetadata
            {
                Name = "app",
                Namespace = "default"
            },
            Spec = new DeploymentSpec
            {
                Replicas = 5,
                Strategy = new DeploymentStrategy
                {
                    Type = RollingUpdate,
                    RollingUpdate = new RollingUpdateStrategy
                    {
                        MaxSurge = 1,
                        MaxUnavailable = 0
                    }
                },
                Template = new PodTemplate
                {
                    Containers = new List<Container>
                    {
                        new() { Image = "app:v2" }
                    }
                }
            }
        };

        var controller = new DeploymentController(
            mockApi.Object, Mock.Of<ILogger>());

        // Act
        var result = await controller.ReconcileDeploymentAsync(
            deployment);

        // Assert
        Assert.False(result); // Not yet complete
        mockApi.Verify(x => x.CreateReplicaSetAsync(
            It.Is<ReplicaSet>(rs =>
                rs.Spec.Template.Containers[0].Image == "app:v2")),
            Times.Once);
    }
}

22.4 Chaos Engineering Tests

• Node Failure: Kill a worker node and verify pods are rescheduled within 30 seconds.
• etcd Leader Loss: Kill the etcd leader and verify the cluster recovers within 10 seconds with no data loss.
• Network Partition: Simulate network isolation between two nodes and verify pod-to-pod communication degrades gracefully.
• Disk Pressure: Fill the disk on a worker node and verify kubelet evicts pods before the node becomes unresponsive.
• API Server Overload: Send 10,000 concurrent API requests and verify the server handles them with acceptable latency.

23. Interview Q&A

Q1: How does the Kubernetes scheduler decide which node to place a pod on?

The scheduler uses a two-phase approach: filtering and scoring. In the filtering phase, it eliminates nodes that cannot run the pod due to insufficient resources, unmet node selectors, taint/toleration mismatches, or volume binding constraints. In the scoring phase, it evaluates the remaining feasible nodes using a set of scoring plugins — LeastRequestedPriority, BalancedResourceAllocation, NodeAffinityPriority, InterPodAffinityPriority, and others. Each plugin assigns a score (0-100), and the node with the highest aggregate score is selected. If there is a tie, the node is chosen arbitrarily.

Q2: What happens when an etcd node fails in a 3-node cluster?

Raft consensus requires a majority (quorum) of nodes to be available. With 3 nodes, quorum is 2. If 1 node fails, the remaining 2 nodes maintain quorum and continue operating normally. The failed node's data will be replicated when it recovers. If 2 nodes fail simultaneously, the cluster loses quorum and becomes read-only — no new writes can be accepted. This is why production clusters run 3 or 5 etcd nodes. With 5 nodes, quorum is 3, so the cluster can tolerate 2 simultaneous failures.

Q3: Explain the difference between a Deployment, ReplicaSet, DaemonSet, and StatefulSet.

A Deployment manages ReplicaSets and provides declarative rolling updates. A ReplicaSet ensures a specified number of pod replicas are running at any time. A DaemonSet ensures exactly one pod runs on every node (or a subset of nodes matching a selector) — used for node-level agents like log collectors and monitoring exporters. A StatefulSet provides ordered, sequential pod management with stable network identities and persistent storage — designed for stateful workloads like databases, where each pod needs a unique, persistent identity and ordered startup/shutdown.

Q4: How would you handle a scenario where pods keep getting OOMKilled?

First, check if the container's memory limit is too low for its actual usage by examining container_memory_working_set_bytes vs. kube_pod_container_resource_limits. Second, profile the application for memory leaks using heap dumps. Third, consider increasing the memory limit in the pod spec. Fourth, check if VPA is recommending higher memory requests. Fifth, review if there are many sidecar containers sharing the same memory limit. Finally, check if the node itself is under memory pressure and kubelet is evicting pods proactively. The key insight is distinguishing between container-level OOMKill (memory limit exceeded) and node-level eviction (node runs out of memory).

Q5: Design a zero-downtime deployment strategy for a stateful application (e.g., a database).

Stateful applications cannot use standard rolling updates because data must be migrated and replicas must be synchronized before old pods are terminated. The approach is: (1) Use a StatefulSet with ordered deployment strategy. (2) Implement readiness probes that verify the new replica has caught up with the primary. (3) Use a custom controller or operator that performs a "scale up, sync, verify, scale down" sequence. (4) Ensure PersistentVolumeClaims are not deleted during the update. (5) Use a Service with a headless configuration for stable DNS. (6) Implement backup before the upgrade and verify restore capability. (7) Use PodDisruptionBudgets to prevent simultaneous termination of multiple replicas.

Q6: How does Kubernetes handle network connectivity between pods on different nodes?

When a pod on Node A sends traffic to a pod on Node B, the packet first hits the CNI plugin's virtual network interface on Node A. The CNI plugin has configured routing rules that know Node B's pod CIDR is reachable via Node B's IP address. The packet is encapsulated (in overlay networks like Flannel/VXLAN) or routed directly (in routed networks like Calico) to Node B. On Node B, the CNI plugin de-encapsulates (if applicable) and delivers the packet to the target pod's network namespace. The kube-proxy component handles Service-level routing by setting up iptables/IPVS rules that intercept traffic to Service ClusterIPs and redirect to backing pod IPs.

Q7: What is the control plane failure domain, and how do you minimize blast radius?

The control plane failure domain is the scope of impact when a control plane component fails. If the API server goes down, no new workloads can be deployed and no existing workloads can be modified — but already-running pods continue operating. If etcd loses quorum, the entire cluster becomes unmanageable. To minimize blast radius: (1) Run 3+ API server replicas across availability zones. (2) Run 3 or 5 etcd nodes with NVMe storage. (3) Use PodDisruptionBudgets to ensure workload availability during control plane maintenance. (4) Implement cluster federation so a single cluster failure does not affect the entire platform. (5) Use multiple clusters for different failure domains (region, cloud provider). (6) Implement circuit breakers in client applications so API server failures do not cascade.

Q8: How would you debug a pod stuck in "ContainerCreating" state?

The debugging steps are: (1) Run kubectl describe pod to see the Events section, which typically reveals the root cause. (2) Common causes include: image pull failures (check image name, tag, registry authentication), volume mount failures (PVC not bound, NFS server unreachable), network plugin errors (CNI configuration issues), and resource constraints (insufficient CPU/memory on the node). (3) Check node conditions with kubectl describe node for MemoryPressure, DiskPressure, or PIDPressure. (4) Check kubelet logs on the assigned node: journalctl -u kubelet. (5) Check container runtime logs: crictl logs or docker logs. (6) Verify the service account token and image pull secrets are valid.

Q9: Explain the reconciliation pattern and why Kubernetes uses it instead of imperative commands.

The reconciliation pattern is the heart of Kubernetes' declarative model. Instead of issuing imperative commands ("create 5 pods"), you declare desired state ("there should be 5 pods with this spec"), and controllers continuously reconcile actual state with desired state. This provides several critical properties: (1) Self-healing — if a pod crashes, the controller detects the discrepancy and recreates it. (2) Convergence — multiple concurrent changes are handled gracefully because the system always converges to the declared state. (3) Idempotency — applying the same manifest twice produces the same result. (4) Auditability — the desired state is explicitly declared and version-controlled. (5) Simplified operations — operators declare intent, not steps, making the system easier to reason about at scale.

Q10: How do you implement multi-tenancy in a Kubernetes cluster?

Multi-tenancy requires isolation at multiple layers: (1) Namespace isolation: Each tenant gets a dedicated namespace. (2) RBAC: Tenants can only manage resources in their namespace via RoleBindings. (3) Resource Quotas: Limit CPU, memory, storage, and object counts per namespace. (4) Limit Ranges: Set default and maximum resource requests per container. (5) Network Policies: Default-deny ingress/egress per namespace, whitelist only necessary cross-namespace communication. (6) Admission Controllers: Validate that tenants cannot create resources outside their allowed configurations. (7) Pod Security Standards: Enforce baseline or restricted profiles per namespace. (8) Separate logging and monitoring: Each tenant sees only their own metrics and logs via RBAC-scoped dashboards.

24. Conclusion

Building a Kubernetes-style container orchestration platform is one of the most complex and rewarding engineering challenges in modern infrastructure. The system touches every domain of distributed systems — consensus algorithms, scheduling theory, network programming, storage systems, security, and observability.

The key takeaways from this guide are: (1) The declarative, reconciliation-based architecture is the foundation that makes everything else work. (2) etcd is the single point of criticality — treat it with the highest level of care. (3) The API server is the enforcement point for all security policies — invest heavily in its hardening. (4) The scheduler is a performance-critical path — benchmark and tune it for your workload patterns. (5) Observability is not optional — you cannot operate what you cannot see.

Whether you are building your own orchestration platform, contributing to Kubernetes, or simply operating clusters at scale, understanding these internals will make you a significantly more effective platform engineer. The questions in the interview section represent the depth of understanding expected at staff-level positions at major technology companies.

The journey from a single container on a laptop to a multi-cluster orchestration platform spanning hundreds of nodes across multiple regions is one of increasing abstraction and automation. Each layer of the system — from the container runtime to the scheduler to the API server — exists because manual operations cannot scale. The tools and patterns we have explored provide the foundation for building infrastructure that serves millions of users without requiring a correspondingly large operations team.

As the cloud-native ecosystem continues to evolve, with WebAssembly runtimes, AI-driven scheduling, and edge computing pushing the boundaries of what orchestration platforms can do, the fundamental principles covered here — declarative state, reconciliation loops, and extensible APIs — will remain the bedrock of container orchestration for years to come. Investing in a deep understanding of these principles will pay dividends throughout your career in platform engineering and distributed systems architecture.