1. Introduction & Why CI/CD is Hard

Continuous Integration and Continuous Delivery (CI/CD) is the backbone of modern software engineering. Every time a developer pushes code, an intricate system of machines, queues, sandboxes, and orchestration engines springs into action — compiling code, running tests, building containers, scanning for vulnerabilities, signing artifacts, and deploying to production. Systems like GitHub Actions, Jenkins, GitLab CI, CircleCI, and Travis CI handle millions of pipeline runs per day, and designing one from scratch is one of the most fascinating system design challenges in DevOps.

The fundamental difficulty of CI/CD lies in the intersection of multiple hard problems: distributed systems (runners across data centers and clouds), security (executing untrusted code safely), scheduling (efficiently allocating compute resources), state management (tracking the lifecycle of thousands of concurrent pipeline runs), and extensibility (supporting arbitrary user-defined workflows). Getting any one of these right is hard. Getting all of them right simultaneously is what separates a toy CI/CD script from a production-grade platform.

The Evolution of CI/CD

The journey from manual deployments to fully automated pipelines is instructive. In the early 2000s, tools like CruiseControl and Ant provided basic build automation. Jenkins (originally Hudson) popularized the concept of a self-hosted build server with a plugin ecosystem. The 2010s saw the rise of hosted services — Travis CI, CircleCI, and GitLab CI — that abstracted away runner management. GitHub Actions, launched in 2019, combined marketplace extensibility with native SCM integration. Each generation solved the previous generation's pain points while introducing new challenges around scale, security, and cost.

Today, a modern CI/CD platform must support monorepos with thousands of packages, polyglot build environments, ephemeral container-based runners, signed and attested supply chains, multi-cloud deployments, and compliance frameworks like SOC 2 and FedRAMP. This article designs such a system from the ground up.

2. Functional & Non-Functional Requirements

Functional Requirements

• Pipeline Definition: Users define pipelines in YAML files stored in their repositories (e.g., .github/workflows/main.yml).
• Trigger Mechanisms: Pipelines trigger on push, pull request, schedule (cron), manual dispatch, release, and webhook events.
• Job Orchestration: DAG-based execution with dependency management, parallel jobs, conditional execution, and reusable workflows.
• Runner Management: Support for both hosted and self-hosted runners with automatic scaling.
• Container Execution: Each step can run in its own container with configurable Docker images.
• Artifact Management: Upload, download, and share artifacts between jobs and across pipeline runs.
• Secret Management: Encrypted secrets at organization, repository, and environment levels.
• Deployment Environments: Named environments with protection rules, required reviewers, and deployment branches.
• Marketplace: Reusable actions published by the community, versioned and discoverable.
• Notifications: Integration with Slack, email, webhooks, and status checks on commits and PRs.
• Audit Trail: Complete logging of who did what, when, and with what secrets.
• RBAC: Fine-grained permissions for organizations, teams, and individual repositories.

Non-Functional Requirements

3. Capacity Estimation & Sizing

Let's size the system for a GitHub-scale platform:

Storage Estimates

• Workflow YAML files: ~2KB average x 150M runs/day = ~300 GB/day of definitions (mostly deduplicated via git SHA references)
• Build logs: ~500KB average x 150M runs = ~75 TB/day (compressed, 90-day retention = ~6.75 PB)
• Artifacts: Highly variable, ~10 GB average per active repo per day
• Metadata (run records, status, etc.): ~1KB per run = ~150 GB/day

Network Estimates

At peak, the system handles approximately 100 Gbps of combined traffic: webhook ingest from Git providers (~10 Gbps), artifact upload/download from runners (~60 Gbps), log streaming (~20 Gbps), and API traffic (~10 Gbps). Runner provisioning (VM/container startup) adds burst capacity requirements that are 2-3x the steady-state average.

4. High-Level Architecture Overview

The CI/CD platform consists of the following major subsystems, each independently scalable:

Component Responsibilities

5. Pipeline Definition & YAML Syntax

A CI/CD pipeline definition language must balance expressiveness with safety. GitHub Actions uses YAML with a specific schema. Let's design a similar system and implement the parser in C#.

YAML Schema Design

yaml# .github/workflows/ci.yml
name: CI Pipeline
on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * 1-5'
  workflow_dispatch:
    inputs:
      environment:
        description: 'Target environment'
        required: true
        default: 'staging'
        type: choice
        options: [staging, production]

env:
  DOTNET_VERSION: '8.0.x'
  REGISTRY: ghcr.io

permissions:
  contents: read
  packages: write
  id-token: write

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    outputs:
      version: ${{ steps.version.outputs.tag }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: ${{ env.DOTNET_VERSION }}
      - name: Build
        run: dotnet build --no-restore -c Release
      - name: Test
        run: dotnet test --no-build -c Release

  security-scan:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Trivy
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'

  deploy-staging:
    needs: [build, security-scan]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment:
      name: staging
      url: https://staging.example.com
    steps:
      - name: Deploy to Staging
        run: echo "Deploying to staging"

  deploy-production:
    needs: [build, security-scan, deploy-staging]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment:
      name: production
      url: https://example.com
    steps:
      - name: Deploy to Production
        run: echo "Canary deployment"

C# Pipeline Parser Implementation

C#using YamlDotNet.Serialization;
using YamlDotNet.Serialization.NamingConventions;
using System.ComponentModel.DataAnnotations;

namespace CICDPlatform.Pipeline;

public class WorkflowDefinition
{
    [Required] public string Name { get; set; } = string.Empty;
    [Required] public TriggerConfig On { get; set; } = new();
    public Dictionary<string, string> Env { get; set; } = new();
    public PermissionsConfig? Permissions { get; set; }
    public ConcurrencyConfig? Concurrency { get; set; }
    [Required] public Dictionary<string, JobDefinition> Jobs { get; set; } = new();
}

public class TriggerConfig
{
    public PushTrigger? Push { get; set; }
    public PullRequestTrigger? PullRequest { get; set; }
    public ScheduleTrigger[]? Schedule { get; set; }
    public WorkflowDispatchTrigger? WorkflowDispatch { get; set; }
}

public class PushTrigger
{
    public string[]? Branches { get; set; }
    public string[]? BranchesIgnore { get; set; }
    public string[]? Paths { get; set; }
    public string[]? PathsIgnore { get; set; }
    public string[]? Tags { get; set; }
}

public class ScheduleTrigger
{
    [Required] public string Cron { get; set; } = string.Empty;
}

public class JobDefinition
{
    public string? Name { get; set; }
    [Required] public string RunsOn { get; set; } = string.Empty;
    public string[]? Needs { get; set; }
    public string? If { get; set; }
    public Dictionary<string, string> Env { get; set; } = new();
    public List<StepDefinition> Steps { get; set; } = new();
    public int TimeoutMinutes { get; set; } = 360;
    public string? Container { get; set; }
    public StrategyConfig? Strategy { get; set; }
    public EnvironmentConfig? Environment { get; set; }
}

public class StepDefinition
{
    public string? Name { get; set; }
    public string? Uses { get; set; }
    public string? Run { get; set; }
    public string? Id { get; set; }
    public string? If { get; set; }
    public Dictionary<string, string> With { get; set; } = new();
    public string? WorkingDirectory { get; set; }
    public string? Shell { get; set; }
    public bool ContinueOnError { get; set; }
    public int TimeoutMinutes { get; set; }
}

public class StrategyConfig
{
    public MatrixConfig Matrix { get; set; } = new();
    public bool FailFast { get; set; } = true;
    public int MaxParallel { get; set; } = 4;
}

public class MatrixConfig
{
    public Dictionary<string, string[]> Include { get; set; } = new();
    public Dictionary<string, string[]> Exclude { get; set; } = new();
}

public class PipelineParser
{
    public async Task<WorkflowDefinition> ParseAsync(string yamlContent)
    {
        var deserializer = new DeserializerBuilder()
            .WithNamingConvention(NullNamingConvention.Instance)
            .Build();
        var definition = deserializer.Deserialize<WorkflowDefinition>(yamlContent);
        if (definition == null)
            throw new InvalidOperationException("Failed to parse workflow YAML");
        Validate(definition);
        ExpandMatrixJobs(definition);
        return definition;
    }

    private void Validate(WorkflowDefinition definition)
    {
        var context = new ValidationContext(definition);
        var results = new List<ValidationResult>();
        if (!Validator.TryValidateObject(definition, context, results, true))
        {
            var errors = string.Join("; ", results.Select(r => r.ErrorMessage));
            throw new ValidationException($"Workflow validation failed: {errors}");
        }
        var graph = BuildDependencyGraph(definition.Jobs);
        if (HasCycle(graph))
            throw new ValidationException("Job dependency graph contains a cycle");
    }

    private Dictionary<string, List<string>> BuildDependencyGraph(
        Dictionary<string, JobDefinition> jobs)
    {
        var graph = new Dictionary<string, List<string>>();
        foreach (var (jobId, job) in jobs)
            graph[jobId] = job.Needs?.ToList() ?? new List<string>();
        return graph;
    }

    private bool HasCycle(Dictionary<string, List<string>> graph)
    {
        var visited = new HashSet<string>();
        var stack = new HashSet<string>();
        foreach (var node in graph.Keys)
            if (DfsCycle(node, graph, visited, stack)) return true;
        return false;
    }

    private bool DfsCycle(string node, Dictionary<string, List<string>> graph,
        HashSet<string> visited, HashSet<string> stack)
    {
        if (stack.Contains(node)) return true;
        if (visited.Contains(node)) return false;
        visited.Add(node);
        stack.Add(node);
        if (graph.TryGetValue(node, out var neighbors))
            foreach (var neighbor in neighbors)
                if (graph.ContainsKey(neighbor) &&
                    DfsCycle(neighbor, graph, visited, stack)) return true;
        stack.Remove(node);
        return false;
    }

    private void ExpandMatrixJobs(WorkflowDefinition definition) { /* ... */ }
}

6. Trigger Mechanisms

Triggers are the entry point of every pipeline run. A robust trigger system must handle deduplication, filtering, rate limiting, and idempotency.

Webhook-Based Triggers (Push, PR, Release)

When a developer pushes code, the Git provider sends an HTTP POST to our webhook endpoint with commit details, branch, author, and changed files.

C#public class WebhookProcessor
{
    private readonly IPipelineRepository _pipelineRepo;
    private readonly IScheduler _scheduler;
    private readonly IDistributedCache _cache;

    public async Task<WebhookResult> ProcessPushAsync(
        PushWebhookPayload payload, string provider)
    {
        // Step 1: Deduplicate
        var dedupKey = $"webhook:{provider}:{payload.DeliveryId}";
        if (await _cache.GetStringAsync(dedupKey) != null)
            return WebhookResult.Duplicate;
        await _cache.SetStringAsync(dedupKey, "processed",
            new DistributedCacheEntryOptions
            { AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(1) });

        // Step 2: Find matching workflow files
        var workflows = await _pipelineRepo.GetWorkflowsForRepoAsync(
            payload.RepositoryId, payload.BeforeSha);

        var triggeredRuns = new List<PipelineRun>();

        foreach (var workflow in workflows)
        {
            if (!ShouldTrigger(workflow, payload)) continue;

            // Step 3: Check concurrency limits
            var concurrencyGroup = ResolveConcurrencyGroup(workflow, payload);
            if (concurrencyGroup != null)
            {
                var existing = await _scheduler
                    .GetActiveRunInGroupAsync(concurrencyGroup);
                if (existing != null &&
                    workflow.Concurrency?.CancelInProgress == true)
                    await _scheduler.CancelRunAsync(existing.Id);
            }

            // Step 4: Create and enqueue pipeline run
            var run = new PipelineRun
            {
                Id = Guid.NewGuid(),
                WorkflowId = workflow.Id,
                RepositoryId = payload.RepositoryId,
                TriggerType = TriggerType.Push,
                HeadSha = payload.HeadSha,
                Branch = payload.Branch,
                Author = payload.Author,
                CreatedAt = DateTime.UtcNow,
                Status = RunStatus.Queued,
                EventPayload = JsonSerializer.Serialize(payload)
            };
            await _pipelineRepo.CreateRunAsync(run);
            await _scheduler.EnqueueAsync(run);
            triggeredRuns.Add(run);
        }
        return WebhookResult.Processed(triggeredRuns.Count);
    }

    private bool ShouldTrigger(WorkflowDefinition workflow,
        PushWebhookPayload payload)
    {
        var trigger = workflow.On.Push;
        if (trigger == null) return false;
        if (trigger.Branches != null && !trigger.Branches
            .Any(b => MatchesGlob(b, payload.Branch))) return false;
        if (trigger.BranchesIgnore != null && trigger.BranchesIgnore
            .Any(b => MatchesGlob(b, payload.Branch))) return false;
        if (trigger.Paths != null && !trigger.Paths
            .Any(p => payload.ChangedFiles.Any(f => MatchesGlob(p, f))))
            return false;
        return true;
    }

    private bool MatchesGlob(string pattern, string value) =>
        GlobMatcher.IsMatch(pattern, value);
}

Scheduled Triggers (Cron)

Scheduled triggers require a cron scheduler service that evaluates cron expressions and enqueues runs at the appropriate times.

C#public class CronSchedulerService : BackgroundService
{
    private readonly IPipelineRepository _pipelineRepo;
    private readonly IScheduler _scheduler;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            var now = DateTime.UtcNow;
            var windowStart = now - TimeSpan.FromMinutes(1);
            var scheduledWorkflows = await _pipelineRepo
                .GetScheduledWorkflowsAsync(windowStart, now);

            foreach (var workflow in scheduledWorkflows)
            {
                foreach (var schedule in workflow.On.Schedule!)
                {
                    if (ShouldRunNow(schedule.Cron, now))
                    {
                        var run = new PipelineRun
                        {
                            Id = Guid.NewGuid(),
                            WorkflowId = workflow.Id,
                            RepositoryId = workflow.RepositoryId,
                            TriggerType = TriggerType.Schedule,
                            HeadSha = await _pipelineRepo
                                .GetLatestShaAsync(workflow.RepositoryId,
                                    workflow.DefaultBranch),
                            CreatedAt = now,
                            Status = RunStatus.Queued
                        };
                        await _pipelineRepo.CreateRunAsync(run);
                        await _scheduler.EnqueueAsync(run);
                    }
                }
            }
            await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
        }
    }

    private bool ShouldRunNow(string cron, DateTime now)
    {
        var parts = cron.Split(' ');
        var minute = int.Parse(parts[0]);
        var hour = parts[1] == "*" || int.Parse(parts[1]) == now.Hour;
        var dom = parts[2] == "*" || int.Parse(parts[2]) == now.Day;
        var month = parts[3] == "*" || int.Parse(parts[3]) == now.Month;
        var dow = parts[4] == "*" || parts[4].Split(',')
            .Contains(((int)now.DayOfWeek).ToString());
        return minute == now.Minute && hour && dom && month && dow;
    }
}

Trigger Summary

7. DAG-Based Job Orchestration

CI/CD pipelines are naturally modeled as Directed Acyclic Graphs (DAGs), where nodes are jobs and edges are dependencies. The needs keyword defines these dependencies.

C#public class DAGScheduler
{
    private readonly IStateService _stateService;
    private readonly IRunnerManager _runnerManager;

    public async Task ScheduleRunAsync(PipelineRun run,
        WorkflowDefinition workflow)
    {
        var dag = BuildDAG(workflow.Jobs);
        foreach (var jobId in dag.Nodes)
            await _stateService.InitializeJobStateAsync(
                run.Id, jobId, JobStatus.Pending);
        await DispatchReadyJobsAsync(run, dag);
    }

    public async Task OnJobCompletedAsync(Guid runId, string jobId,
        JobResult result)
    {
        await _stateService.UpdateJobStatusAsync(runId, jobId,
            result.Success ? JobStatus.Completed : JobStatus.Failed);

        var currentRun = await _stateService.GetRunAsync(runId);
        var currentWorkflow = await LoadWorkflowAsync(currentRun);
        var currentDag = BuildDAG(currentWorkflow.Jobs);
        await DispatchReadyJobsAsync(currentRun, currentDag);

        if (await _stateService.AllJobsCompletedAsync(runId))
        {
            var allPassed = await _stateService.AllJobsSucceededAsync(runId);
            await _stateService.UpdateRunStatusAsync(runId,
                allPassed ? RunStatus.Completed : RunStatus.Failed);
        }
    }

    private async Task DispatchReadyJobsAsync(PipelineRun run, DAG dag)
    {
        var jobStates = await _stateService.GetJobStatesAsync(run.Id);
        foreach (var jobId in dag.Nodes)
        {
            var state = jobStates.GetValueOrDefault(jobId);
            if (state?.Status != JobStatus.Pending) continue;

            var deps = dag.GetDependencies(jobId);
            var depsCompleted = deps.All(d =>
                jobStates.GetValueOrDefault(d)?.Status ==
                    JobStatus.Completed);
            var depsSucceeded = deps.All(d =>
                jobStates.GetValueOrDefault(d)?.Status ==
                    JobStatus.Completed &&
                jobStates.GetValueOrDefault(d)?.Result?.Success == true);

            if (!depsCompleted) continue;
            if (!depsSucceeded && !HasAlwaysCondition(run, jobId))
                continue;
            if (!EvaluateCondition(run, jobId)) continue;

            await _runnerManager.DispatchJobAsync(run, jobId);
        }
    }

    private DAG BuildDAG(Dictionary<string, JobDefinition> jobs)
    {
        var dag = new DAG();
        foreach (var (jobId, job) in jobs)
        {
            dag.AddNode(jobId);
            if (job.Needs != null)
                foreach (var dep in job.Needs)
                    dag.AddEdge(dep, jobId);
        }
        return dag;
    }

    private bool HasAlwaysCondition(PipelineRun run, string jobId) => false;
    private bool EvaluateCondition(PipelineRun run, string jobId) => true;
}

public class DAG
{
    public List<string> Nodes { get; } = new();
    public Dictionary<string, List<string>> AdjacencyList { get; } = new();

    public void AddNode(string node)
    {
        if (!Nodes.Contains(node)) Nodes.Add(node);
        if (!AdjacencyList.ContainsKey(node))
            AdjacencyList[node] = new List<string>();
    }

    public void AddEdge(string from, string to) =>
        AdjacencyList[from].Add(to);

    public List<string> GetDependencies(string node) =>
        AdjacencyList
            .Where(kvp => kvp.Value.Contains(node))
            .Select(kvp => kvp.Key)
            .ToList();

    public List<string> TopologicalSort()
    {
        var visited = new HashSet<string>();
        var result = new List<string>();
        foreach (var node in Nodes)
            if (!visited.Contains(node))
                Dfs(node, visited, result);
        result.Reverse();
        return result;
    }

    private void Dfs(string node, HashSet<string> visited,
        List<string> result)
    {
        if (visited.Contains(node)) return;
        visited.Add(node);
        foreach (var neighbor in AdjacencyList[node])
            Dfs(neighbor, visited, result);
        result.Add(node);
    }
}

8. API Design

The CI/CD platform exposes both a REST API for CRUD operations and a real-time WebSocket/gRPC API for status updates.

Core API Endpoints

GraphQL API

graphqltype Query {
    repository(owner: String!, name: String!): Repository
}

type Repository {
    name: String!
    workflows(first: Int, after: String): WorkflowConnection!
    runs(first: Int, after: String, status: RunStatus): RunConnection!
}

type Run {
    id: ID!
    name: String!
    status: RunStatus!
    conclusion: RunConclusion
    createdAt: DateTime!
    headBranch: String!
    headSha: String!
    jobs(first: Int, after: String): JobConnection!
    artifacts(first: Int, after: String): ArtifactConnection!
}

type Job {
    id: ID!
    name: String!
    status: RunStatus!
    conclusion: RunConclusion
    steps(first: Int, after: String): StepConnection!
    runner: Runner
}

enum RunStatus {
    QUEUED
    IN_PROGRESS
    COMPLETED
    WAITING
    PENDING
}

enum RunConclusion {
    SUCCESS
    FAILURE
    CANCELLED
    SKIPPED
    TIMED_OUT
}

9. Runner & Agent Management

Runners are the machines (VMs, containers, or bare-metal servers) that execute pipeline steps. Runner management is one of the most operationally complex parts of a CI/CD platform.

Cloud Runner Provisioning

C#public interface IRunnerProvisioner
{
    Task<RunnerInstance> ProvisionRunnerAsync(RunnerRequest request);
    Task DecommissionRunnerAsync(string runnerId);
}

public class CloudRunnerProvisioner : IRunnerProvisioner
{
    private readonly ICloudProvider _cloudProvider;
    private readonly RunnerImageCache _imageCache;

    public async Task<RunnerInstance> ProvisionRunnerAsync(RunnerRequest request)
    {
        var startTime = DateTime.UtcNow;

        // Check for idle runner
        var idleRunner = await FindIdleRunnerAsync(request);
        if (idleRunner != null)
        {
            idleRunner.Status = RunnerStatus.Claimed;
            return idleRunner;
        }

        // Provision new VM
        var image = await _imageCache.GetOrBuildImageAsync(
            request.Image, request.OS);
        var vmSpec = SelectVMSize(request);

        var instance = await _cloudProvider.CreateVMAsync(new VMRequest
        {
            ImageId = image.Id,
            Size = vmSpec,
            Labels = request.Labels,
            Metadata = new Dictionary<string, string>
            {
                ["cicd-run-id"] = request.RunId.ToString(),
                ["cicd-job-id"] = request.JobId,
                ["cicd-org"] = request.OrganizationId
            },
            UserData = GenerateUserData(request)
        });

        // Wait for agent to register
        var agent = await WaitForAgentRegistrationAsync(
            instance.Id, TimeSpan.FromMinutes(3));

        var provisionTime = DateTime.UtcNow - startTime;
        return new RunnerInstance
        {
            Id = instance.Id,
            AgentId = agent.Id,
            Status = RunnerStatus.Ready,
            OS = request.OS,
            Labels = request.Labels,
            ProvisionedAt = DateTime.UtcNow
        };
    }

    private string[] GenerateUserData(RunnerRequest request)
    {
        var token = GenerateRunnerToken(request);
        return new[]
        {
            "#!/bin/bash",
            "set -e",
            $"curl -sL https://packages.example.com/install-runner.sh | bash -s -- --token {token}",
            "echo 'Runner started' > /var/log/cicd-runner.log"
        };
    }

    private RunnerToken GenerateRunnerToken(RunnerRequest request) =>
        new() { Value = "encrypted-token" };
}

Runner Auto-Scaling

C#public class RunnerAutoScaler : BackgroundService
{
    private readonly IRunnerProvisioner _provisioner;
    private readonly IRunnerRegistry _registry;

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            var stats = await _registry.GetPoolStatsAsync();
            var utilization = stats.Total == 0 ? 0 :
                (double)(stats.Total - stats.Idle) / stats.Total;

            if (utilization > 0.8 && stats.QueuedJobs > stats.Idle)
            {
                var toProvision = Math.Min(
                    stats.QueuedJobs - stats.Idle,
                    stats.MaxCapacity - stats.Total);
                // Scale up runners...
            }

            if (utilization < 0.2 && stats.Idle > 50)
            {
                var toDecommission = (stats.Idle - 25) / 2;
                // Scale down idle runners...
            }

            await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
        }
    }
}

Self-Hosted vs GitHub-Hosted Runners

10. Container-Based Execution & Step Isolation

Modern CI/CD platforms execute each step in its own container, providing strong isolation between steps and between different users' jobs.

Container Execution Engine

C#public class ContainerExecutor : IStepExecutor
{
    private readonly IDockerClient _docker;
    private readonly IArtifactService _artifacts;
    private readonly ILogCollector _logs;

    public async Task<StepResult> ExecuteStepAsync(
        ExecutionContext context, StepDefinition step)
    {
        var containerConfig = new CreateContainerParameters
        {
            Image = ResolveImage(step, context),
            WorkingDirectory = step.WorkingDirectory ?? "/workspace",
            Env = BuildEnvironmentVariables(context, step),
            HostConfig = new HostConfig
            {
                Binds = new List<string>
                {
                    $"{context.WorkspacePath}:/workspace:rw",
                    $"{context.ArtifactsPath}:/artifacts:rw",
                    $"{context.TempPath}:/tmp:rw"
                },
                // Resource limits
                Memory = 4L * 1024 * 1024 * 1024,   // 4 GB
                NanoCpus = 4_000_000_000,             // 4 cores
                // Security: drop all capabilities
                CapDrop = new[] { "ALL" },
                CapAdd = new[] { "NET_BIND_SERVICE" },
                SecurityOpt = new[] { "no-new-privileges:true" }
            },
            Cmd = new[] { "/bin/sh", "-c", step.Run ?? "" },
            Labels = new Dictionary<string, string>
            {
                ["cicd-run-id"] = context.RunId.ToString(),
                ["cicd-job-id"] = context.JobId,
                ["cicd-step"] = step.Id ?? "unnamed"
            }
        };

        await EnsureImageAvailableAsync(containerConfig.Image);
        var container = await _docker.Containers
            .CreateContainerAsync(containerConfig);
        await _docker.Containers.StartContainerAsync(
            container.ID, new ContainerStartParameters());

        // Stream logs in real-time
        var logStream = await _docker.Containers.GetContainerLogsAsync(
            container.ID,
            new ContainerLogsParameters
            { ShowStdout = true, ShowStderr = true, Follow = true });
        var logTask = _logs.StreamLogsAsync(context, step.Id, logStream);

        var exitCode = await WaitForContainerAsync(
            container.ID, step.TimeoutMinutes);
        await logTask;

        // Collect artifacts if specified
        if (step.With.TryGetValue("path", out var artifactPath))
            await _artifacts.UploadArtifactAsync(context, step, artifactPath);

        await _docker.Containers.RemoveContainerAsync(
            container.ID, new ContainerRemoveParameters { Force = true });

        return new StepResult
        {
            Success = exitCode == 0,
            ExitCode = exitCode,
            Duration = DateTime.UtcNow - context.StepStartTime
        };
    }
}

Step Isolation Model

11. Artifact Management & Caching

Artifacts are files generated during a pipeline run — build outputs, test reports, binary packages. Caching speeds up repeated operations by storing intermediate results.

Artifact Storage Architecture

C#public class ArtifactService : IArtifactService
{
    private readonly IBlobStorage _storage;
    private readonly IArtifactRepository _repository;
    private readonly ICompressionService _compression;

    public async Task<ArtifactManifest> UploadArtifactAsync(
        ArtifactUploadRequest request)
    {
        // Scan for secrets in artifacts
        var scanResult = await ScanForSecretsAsync(request.SourcePath);
        if (scanResult.FoundSecrets)
            throw new SecurityException(
                $"Artifact contains detected secrets: {string.Join(", ", scanResult.Findings)}");

        // Compress the artifact
        var archivePath = await _compression.CompressDirectoryAsync(request.SourcePath);

        // Content-addressable hash for deduplication
        var contentHash = await ComputeContentHashAsync(archivePath);
        var blobKey = $"artifacts/{request.RunId}/{request.Name}/{contentHash}";

        // Upload with dedup check
        var existing = await _storage.GetBlobMetadataAsync(blobKey);
        if (existing == null)
        {
            await _storage.UploadBlobAsync(blobKey, archivePath, new BlobMetadata
            {
                RunId = request.RunId,
                ArtifactName = request.Name,
                CreatedAt = DateTime.UtcNow,
                ExpiresAt = DateTime.UtcNow.AddDays(request.RetentionDays),
                SizeBytes = new FileInfo(archivePath).Length
            });
        }

        var manifest = new ArtifactManifest
        {
            Id = Guid.NewGuid(),
            RunId = request.RunId,
            Name = request.Name,
            BlobKey = blobKey,
            ContentHash = contentHash,
            SizeBytes = new FileInfo(archivePath).Length,
            RetentionDays = request.RetentionDays,
            CreatedAt = DateTime.UtcNow
        };
        await _repository.CreateArtifactAsync(manifest);
        return manifest;
    }
}

Caching Strategy

C#public class CacheService : ICacheService
{
    private readonly IBlobStorage _storage;

    public async Task<bool> RestoreCacheAsync(CacheRestoreRequest request)
    {
        var keys = GenerateKeys(request);
        // Try keys from most specific to least specific
        foreach (var key in keys)
        {
            var blobPath = $"cache/{request.RepositoryId}/{key}.tar.gz";
            var blob = await _storage.GetBlobMetadataAsync(blobPath);
            if (blob != null)
            {
                var archivePath = Path.GetTempFileName();
                await _storage.DownloadBlobAsync(blobPath, archivePath);
                await ExtractCacheAsync(archivePath, request.DestinationPath);
                File.Delete(archivePath);
                return true;
            }
        }
        return false;
    }

    public async Task SaveCacheAsync(CacheSaveRequest request)
    {
        var key = GeneratePrimaryKey(request);
        var blobPath = $"cache/{request.RepositoryId}/{key}.tar.gz";
        var existing = await _storage.GetBlobMetadataAsync(blobPath);
        if (existing != null) return; // Already cached

        var archivePath = await CompressCacheAsync(request.SourcePath);
        await _storage.UploadBlobAsync(blobPath, archivePath, new BlobMetadata
        {
            SizeBytes = new FileInfo(archivePath).Length,
            ExpiresAt = DateTime.UtcNow.AddDays(7)
        });
        await EnforceCacheLimitAsync(request.RepositoryId);
        File.Delete(archivePath);
    }

    private async Task EnforceCacheLimitAsync(string repoId)
    {
        var caches = await GetCachesAsync(repoId);
        var totalSize = caches.Sum(c => c.SizeBytes);
        var maxSize = 10L * 1024 * 1024 * 1024; // 10 GB
        if (totalSize > maxSize)
        {
            var toEvict = caches
                .OrderBy(c => c.LastAccessedAt)
                .TakeWhile(c => { totalSize -= c.SizeBytes; return totalSize > maxSize; })
                .ToList();
            foreach (var cache in toEvict)
                await _storage.DeleteBlobAsync(cache.BlobPath);
        }
    }
}

Cache Key Strategy

12. Secret Management

CI/CD pipelines need access to secrets — API keys, passwords, SSH keys — but these must be protected from leakage through logs, artifacts, and unauthorized access.

Secret Encryption Architecture

C#public class SecretManager
{
    private readonly IKeyManagementService _kms;
    private readonly IEncryptionService _encryption;
    private readonly IAuditLogger _auditLogger;

    public async Task<EncryptedSecret> SetSecretAsync(
        SecretSetRequest request, ClaimsPrincipal user)
    {
        await ValidatePermissionAsync(user, request.Scope);
        var dek = await _kms.GetOrCreateDEKAsync(request.OrganizationId);

        // Encrypt with AES-256-GCM
        var encrypted = _encryption.Encrypt(
            Encoding.UTF8.GetBytes(request.Value), dek);

        var secret = new EncryptedSecret
        {
            Id = Guid.NewGuid(),
            Name = request.Name,
            EncryptedValue = encrypted.Ciphertext,
            IV = encrypted.IV,
            AuthTag = encrypted.AuthTag,
            KMSKeyVersion = dek.Version,
            Scope = request.Scope,
            CreatedAt = DateTime.UtcNow
        };
        await _repository.UpsertSecretAsync(secret);

        await _auditLogger.LogAsync(new AuditEntry
        {
            Action = "secret.set",
            Actor = user.Identity?.Name ?? "unknown",
            Metadata = new Dictionary<string, string>
            { ["secret_name"] = request.Name }
            // NEVER log the secret value itself!
        });
        return secret;
    }

    public async Task<Dictionary<string, string>> GetSecretsForJobAsync(
        Guid runId, string jobId, string repoId, string? envName)
    {
        var secrets = new Dictionary<string, string>();
        // Precedence: Environment > Repository > Organization
        if (envName != null)
        {
            var env = await _repository.GetSecretsAsync(
                ScopeType.Environment, envName);
            foreach (var s in env)
                secrets[s.Name] = await DecryptSecretAsync(s);
        }
        var repo = await _repository.GetSecretsAsync(
            ScopeType.Repository, repoId);
        foreach (var s in repo)
            if (!secrets.ContainsKey(s.Name))
                secrets[s.Name] = await DecryptSecretAsync(s);
        var org = await _repository.GetSecretsAsync(
            ScopeType.Organization, GetOrgId(repoId));
        foreach (var s in org)
            if (!secrets.ContainsKey(s.Name))
                secrets[s.Name] = await DecryptSecretAsync(s);
        return secrets;
    }
}

public class SecretMaskingService
{
    private readonly List<string> _knownSecrets = new();

    public void RegisterSecret(string name, string value)
    {
        _knownSecrets.Add(value);
        _knownSecrets.Add(Convert.ToHexString(
            Encoding.UTF8.GetBytes(value)));
        _knownSecrets.Add(Convert.ToBase64String(
            Encoding.UTF8.GetBytes(value)));
    }

    public string MaskLogOutput(string logLine)
    {
        var masked = logLine;
        foreach (var secret in _knownSecrets)
            masked = masked.Replace(secret, "***");
        return masked;
    }
}

13. Matrix Builds & Parallel Execution

Matrix builds run the same job across multiple configurations in parallel — different OS versions, language versions, or parameter combinations.

Matrix Expansion Algorithm

C#public class MatrixExpander
{
    public List<MatrixCombination> Expand(MatrixConfig matrix)
    {
        var result = new List<Dictionary<string, string>> { new() };
        // Cartesian product of all dimensions
        foreach (var (dimension, values) in matrix.Include)
        {
            var expanded = new List<Dictionary<string, string>>();
            foreach (var combo in result)
            {
                foreach (var value in values)
                {
                    var newCombo = new Dictionary<string, string>(combo)
                    { [dimension] = value };
                    expanded.Add(newCombo);
                }
            }
            result = expanded;
        }
        // Apply exclusions
        if (matrix.Exclude?.Count > 0)
        {
            result = result.Where(combo =>
                !matrix.Exclude.Any(exclude =>
                    exclude.All(kvp =>
                        combo.TryGetValue(kvp.Key, out var val) &&
                        val == kvp.Value)))
                .ToList();
        }
        return result.Select(c =>
            new MatrixCombination { Values = c }).ToList();
    }
}

Matrix Build Example

yamlstrategy:
  matrix:
    os: [ubuntu-latest, windows-latest, macos-latest]
    dotnet: ['6.0.x', '7.0.x', '8.0.x']
    exclude:
      - os: macos-latest
        dotnet: '6.0.x'
    include:
      - os: ubuntu-latest
        dotnet: '9.0.x'
        experimental: true
  max-parallel: 4
  fail-fast: false

This generates 9 matrix combinations running in parallel. max-parallel: 4 limits concurrency, and fail-fast: false ensures all combinations complete.

14. Conditional Steps & Reusable Workflows

Conditional execution allows steps to be skipped based on runtime context. Reusable workflows enable DRY pipeline definitions across repositories.

Conditional Expression Engine

C#public class ConditionEvaluator
{
    public bool Evaluate(string expression, ExecutionContext context)
    {
        var tokens = Tokenize(expression);
        var ast = ParseExpression(tokens);
        return EvaluateNode(ast, context);
    }

    private bool EvaluateNode(ASTNode node, ExecutionContext context) =>
        node.Type switch
        {
            ASTNodeType.Literal => (bool)node.Value,
            ASTNodeType.Not => !EvaluateNode(node.Left!, context),
            ASTNodeType.And =>
                EvaluateNode(node.Left!, context) &&
                EvaluateNode(node.Right!, context),
            ASTNodeType.Or =>
                EvaluateNode(node.Left!, context) ||
                EvaluateNode(node.Right!, context),
            ASTNodeType.FunctionCall =>
                EvaluateFunction(node, context),
            ASTNodeType.Comparison =>
                EvaluateComparison(node, context),
            _ => throw new ArgumentException($"Unknown: {node.Type}")
        };

    private bool EvaluateFunction(ASTNode node,
        ExecutionContext context) =>
        node.FunctionName switch
        {
            "always" => true,
            "cancelled" =>
                context.RunStatus == RunStatus.Cancelled,
            "success" =>
                context.RunStatus == RunStatus.Completed &&
                context.AllJobsSucceeded,
            "failure" =>
                context.RunStatus == RunStatus.Failed ||
                context.AnyJobFailed,
            "contains" =>
                context.GetVariable(node.Args[0])?.ToString()
                    .Contains(context.GetVariable(
                        node.Args[1])?.ToString() ?? "") ?? false,
            _ => throw new ArgumentException(
                $"Unknown function: {node.FunctionName}")
        };
}

Reusable Workflows

yaml# Caller workflow
jobs:
  deploy:
    uses: my-org/.github/.github/workflows/deploy-template@main
    with:
      environment: production
      image-tag: ${{ github.sha }}
    secrets:
      DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }}

# Reusable workflow (deploy-template.yml)
name: Deploy Template
on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
      image-tag:
        required: true
        type: string
    secrets:
      DEPLOY_TOKEN:
        required: true
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ inputs.environment }}
    steps:
      - name: Deploy
        run: echo "Deploying ${{ inputs.image-tag }}"

15. Actions Marketplace & Ecosystem

The marketplace transforms a CI/CD platform from a tool into an ecosystem, allowing developers to publish reusable actions.

Action Reference Formats

Action Types

• JavaScript Action: Runs in Node.js, fast startup, native API access
• Docker Action: Runs in a container, any language, heavier but flexible
• Composite Action: Chains multiple steps together, lightweight
• Reusable Workflow: Full pipeline template with multiple jobs

16. Security, Sandboxing & Supply Chain Attacks

Security is the paramount concern in CI/CD. The platform executes untrusted code, handles secrets, and has access to production infrastructure.

Threat Model

Sandboxing Architecture

C#public class SandboxedRunner : IRunner
{
    private readonly IFirecrackerMicroVM _microVM;

    public async Task<RunResult> ExecuteInSandboxAsync(RunRequest request)
    {
        var vmConfig = new MicroVMConfig
        {
            VcpuCount = request.CpuCount,
            MemoryMb = request.MemoryMb,
            KernelImage = "vmlinux-5.10",
            RootFsImage = await PrepareRootFsAsync(request.OS, request.Image),
            NetworkConfig = new NetworkConfig
            {
                AllowOutbound = request.Permissions.Network ?? false,
                EgressFilter = new EgressFilter
                {
                    AllowedHosts = await GetAllowedHostsAsync(request.RepositoryId),
                    BlockedCIDRs = new[] {
                        "10.0.0.0/8", "172.16.0.0/12",
                        "192.168.0.0/16", "169.254.0.0/16"
                    }
                }
            },
            EnableSnapshot = true
        };

        var vm = await _microVM.CreateAsync(vmConfig);
        await vm.MountFileSystemAsync("/workspace", request.WorkspacePath);
        await vm.InjectSecretsAsync(request.Secrets);

        var agent = await vm.StartAgentAsync(new AgentConfig
        {
            RunId = request.RunId,
            Steps = request.Steps,
            Timeout = request.Timeout
        });

        var result = await agent.WaitForCompletionAsync();
        await vm.DestroyAsync();
        return result;
    }
}

SLSA Framework Levels

17. Deployment Strategies

The CI/CD platform must support multiple deployment strategies, each with different risk profiles and rollback capabilities.

Blue-Green Deployment

C#public class BlueGreenDeployer : IDeploymentStrategy
{
    private readonly ILoadBalancer _loadBalancer;
    private readonly IContainerOrchestrator _orchestrator;

    public async Task<DeploymentResult> DeployAsync(DeploymentRequest request)
    {
        var currentSlot = await _loadBalancer.GetActiveSlotAsync();
        var targetSlot = currentSlot == Slot.Blue ? Slot.Green : Slot.Blue;

        // Deploy to inactive slot
        await _orchestrator.DeployAsync(targetSlot, request.ImageTag, request.Config);
        await _orchestrator.WaitForReadyAsync(targetSlot, TimeSpan.FromMinutes(5));

        // Run smoke tests
        var smokeResult = await RunSmokeTestsAsync(targetSlot);
        if (!smokeResult.Success)
        {
            await _orchestrator.ScaleDownAsync(targetSlot);
            return DeploymentResult.Failed("Smoke tests failed");
        }

        // Switch traffic
        await _loadBalancer.SwitchTrafficAsync(targetSlot);

        // Keep old slot warm for quick rollback (15 min)
        await Task.Delay(TimeSpan.FromMinutes(15));

        // Decommission old slot
        await _orchestrator.ScaleDownAsync(currentSlot);
        return DeploymentResult.Success();
    }
}

Canary Deployment

C#public class CanaryDeployer : IDeploymentStrategy
{
    private readonly ILoadBalancer _loadBalancer;
    private readonly IMetricsCollector _metrics;

    public async Task<DeploymentResult> DeployAsync(DeploymentRequest request)
    {
        var stages = new[]
        {
            new CanaryStage { Weight = 1, Duration = TimeSpan.FromMinutes(5) },
            new CanaryStage { Weight = 5, Duration = TimeSpan.FromMinutes(10) },
            new CanaryStage { Weight = 25, Duration = TimeSpan.FromMinutes(15) },
            new CanaryStage { Weight = 100, Duration = TimeSpan.Zero }
        };

        await _loadBalancer.DeployCanaryAsync(request.ImageTag);

        foreach (var stage in stages)
        {
            await _loadBalancer.SetCanaryWeightAsync(stage.Weight);
            var baseline = await _metrics.GetBaselineMetricsAsync(TimeSpan.FromMinutes(2));
            var canary = await _metrics.GetCanaryMetricsAsync(stage.Duration);

            // Check for error rate regression (>10% increase)
            if (canary.ErrorRate > baseline.ErrorRate * 1.1)
            {
                await _loadBalancer.RemoveCanaryAsync();
                return DeploymentResult.Failed($"Error regression at {stage.Weight}%");
            }
            // Check for latency regression (>20% increase)
            if (canary.P99Latency > baseline.P99Latency * 1.2)
            {
                await _loadBalancer.RemoveCanaryAsync();
                return DeploymentResult.Failed($"Latency regression at {stage.Weight}%");
            }
        }

        await _loadBalancer.PromoteCanaryAsync();
        return DeploymentResult.Success();
    }
}

Rolling Deployment

C#public class RollingDeployer : IDeploymentStrategy
{
    public async Task<DeploymentResult> DeployAsync(DeploymentRequest request)
    {
        var instances = await _orchestrator.GetInstancesAsync();
        var batchSize = Math.Max(1, (int)(instances.Count * 0.2)); // 20% at a time

        for (var i = 0; i < instances.Count; i += batchSize)
        {
            var batch = instances.Skip(i).Take(batchSize).ToList();
            foreach (var instance in batch)
            {
                await _orchestrator.UpdateInstanceAsync(instance.Id, request.ImageTag);
                await _healthChecker.WaitForHealthyAsync(instance.Id);
            }

            var metrics = await _metrics.GetRollingMetricsAsync(batch);
            if (metrics.ErrorRate > 0.05) // 5% threshold
            {
                foreach (var instance in batch)
                    await _orchestrator.UpdateInstanceAsync(
                        instance.Id, request.PreviousImageTag);
                return DeploymentResult.Failed($"Batch {i / batchSize + 1} failed");
            }
        }
        return DeploymentResult.Success();
    }
}

Strategy Comparison

18. Environment Management & Rollback

Environments provide logical groupings for deployment targets with protection rules that prevent accidental deployments.

C#public class EnvironmentService
{
    private readonly IEnvironmentRepository _repo;
    private readonly IApprovalService _approvalService;
    private readonly INotificationService _notifications;

    public async Task<EnvironmentDeployment> RequestDeploymentAsync(
        DeploymentRequest request, ClaimsPrincipal user)
    {
        var environment = await _repo.GetEnvironmentAsync(
            request.RepositoryId, request.EnvironmentName);

        // Branch protection check
        if (environment.DeploymentBranchPolicy != null)
        {
            if (!MatchesBranchPolicy(request.Branch, environment.DeploymentBranchPolicy))
                throw new DeploymentDeniedException(
                    $"Branch '{request.Branch}' not allowed for '{request.EnvironmentName}'");
        }

        // Required reviewers check
        if (environment.RequiredReviewers?.Count > 0)
        {
            var approval = await _approvalService.RequestApprovalAsync(new ApprovalRequest
            {
                Environment = request.EnvironmentName,
                RequestedBy = user.Identity?.Name ?? "unknown",
                ImageTag = request.ImageTag,
                Reviewers = environment.RequiredReviewers,
                ExpiresAt = DateTime.UtcNow.AddHours(24)
            });
            if (!approval.Approved)
                return new EnvironmentDeployment
                {
                    Status = DeploymentStatus.WaitingForApproval,
                    Message = $"Waiting for: {string.Join(", ", environment.RequiredReviewers)}"
                };
        }

        // Wait timer
        if (environment.WaitTimerMinutes > 0)
            await Task.Delay(TimeSpan.FromMinutes(environment.WaitTimerMinutes));

        return await ExecuteDeploymentAsync(request, environment);
    }

    public async Task<RollbackResult> RollbackAsync(
        string envName, string repoId, ClaimsPrincipal user)
    {
        var env = await _repo.GetEnvironmentAsync(repoId, envName);
        var lastSuccess = env.DeploymentHistory
            .Where(d => d.Status == DeploymentStatus.Succeeded)
            .OrderByDescending(d => d.CompletedAt)
            .FirstOrDefault();

        if (lastSuccess == null)
            throw new InvalidOperationException("No successful deployment to rollback to");

        var result = await ExecuteDeploymentAsync(new DeploymentRequest
        {
            RepositoryId = repoId,
            EnvironmentName = envName,
            ImageTag = lastSuccess.ImageTag,
            IsRollback = true
        }, env);

        await _notifications.SendAsync(new RollbackNotification
        {
            Environment = envName,
            FromImage = env.CurrentImageTag,
            ToImage = lastSuccess.ImageTag,
            InitiatedBy = user.Identity?.Name ?? "unknown"
        });

        return new RollbackResult
        {
            Success = true,
            PreviousImage = env.CurrentImageTag,
            RolledBackTo = lastSuccess.ImageTag
        };
    }
}

19. Notification Integration & Audit Trail

Notifications keep teams informed. The audit trail provides a complete, tamper-proof record for compliance and forensics.

C#public class NotificationService : INotificationService
{
    private readonly List<INotificationChannel> _channels;

    public NotificationService()
    {
        _channels = new List<INotificationChannel>
        {
            new SlackChannel(), new EmailChannel(),
            new WebhookChannel(), new StatusCheckChannel(),
            new TeamsChannel()
        };
    }

    public async Task SendRunNotificationAsync(
        PipelineRun run, RunNotificationEvent evt)
    {
        var notification = BuildNotification(run, evt);
        var subscribers = await GetSubscribersAsync(run.RepositoryId, evt);

        foreach (var subscriber in subscribers)
        {
            var channel = _channels.FirstOrDefault(c =>
                c.SupportsChannel(subscriber.ChannelType));
            if (channel != null)
            {
                try { await channel.SendAsync(subscriber, notification); }
                catch (Exception ex)
                {
                    _logger.LogError(ex, "Failed to notify {Sub}", subscriber.Id);
                }
            }
        }

        await UpdateCommitStatusAsync(run, evt);
        if (run.PullRequestNumber.HasValue)
            await UpdatePRStatusAsync(run, evt);
    }
}

public class AuditTrailService
{
    private readonly IAuditLogStore _logStore;

    public async Task LogAsync(AuditEntry entry)
    {
        var record = new AuditRecord
        {
            Id = Guid.NewGuid(),
            Timestamp = DateTime.UtcNow,
            Action = entry.Action,
            Actor = entry.Actor,
            ActorIP = entry.ActorIP,
            Scope = entry.Scope,
            ScopeId = entry.ScopeId,
            Metadata = entry.Metadata,
            PreviousHash = await _logStore.GetLatestHashAsync(entry.ScopeId)
        };
        record.CurrentHash = ComputeHash(record);
        await _logStore.AppendAsync(record);
        await PublishToSIEMAsync(record);
    }

    public async Task<bool> VerifyIntegrityAsync(
        string scopeId, DateTime from, DateTime to)
    {
        var records = await _logStore.GetRecordsAsync(scopeId, from, to);
        string? prevHash = null;
        foreach (var record in records)
        {
            if (record.PreviousHash != prevHash) return false;
            var expected = ComputeHash(record with { PreviousHash = prevHash });
            if (record.CurrentHash != expected) return false;
            prevHash = record.CurrentHash;
        }
        return true;
    }

    private string ComputeHash(AuditRecord r) =>
        Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(
            $"{r.Timestamp}{r.Action}{r.Actor}{r.PreviousHash}")));
}

Audit Log Schema

20. RBAC & Billing for Compute Time

Enterprise CI/CD platforms require fine-grained access control and usage-based billing.

C#public enum Permission
{
    Actions_Read, Actions_Write,
    Secrets_Read, Secrets_Write,
    Variables_Read, Variables_Write,
    Environments_Read, Environments_Write,
    Workflow_Dispatch, Workflow_Cancel, Workflow_Rerun,
    Runner_Read, Runner_Register, Runner_Remove,
    Org_Actions_Read, Org_Actions_Write,
    Org_Secrets_Read, Org_Secrets_Write,
    Org_Billing_Read, Org_Members_Manage,
    Marketplace_Publish, Marketplace_Manage
}

public class RBACService
{
    private readonly IPermissionRepository _repo;

    public async Task<bool> HasPermissionAsync(
        ClaimsPrincipal user, Permission permission, ResourceScope scope)
    {
        var userPerms = await _repo.GetUserPermissionsAsync(
            user.Identity?.Name ?? "", scope);
        if (userPerms.Contains(permission)) return true;

        var teams = await _repo.GetUserTeamsAsync(
            user.Identity?.Name ?? "", scope);
        foreach (var team in teams)
        {
            var teamPerms = await _repo.GetTeamPermissionsAsync(team.Id, scope);
            if (teamPerms.Contains(permission)) return true;
        }

        if (scope.Parent != null)
            return await HasPermissionAsync(user, permission, scope.Parent);
        return false;
    }
}

public static class Roles
{
    public static readonly Role Admin = new("admin",
        Enum.GetValues<Permission>().ToHashSet());

    public static readonly Role Maintainer = new("maintainer",
        new HashSet<Permission>
        {
            Permission.Actions_Read, Permission.Actions_Write,
            Permission.Secrets_Read, Permission.Secrets_Write,
            Permission.Variables_Read, Permission.Variables_Write,
            Permission.Environments_Read, Permission.Environments_Write,
            Permission.Runner_Read, Permission.Runner_Register,
            Permission.Workflow_Dispatch,
            Permission.Workflow_Cancel, Permission.Workflow_Rerun
        });

    public static readonly Role Developer = new("developer",
        new HashSet<Permission>
        {
            Permission.Actions_Read, Permission.Secrets_Read,
            Permission.Variables_Read, Permission.Environments_Read,
            Permission.Workflow_Dispatch, Permission.Workflow_Rerun
        });

    public static readonly Role Viewer = new("viewer",
        new HashSet<Permission>
        {
            Permission.Actions_Read,
            Permission.Secrets_Read, Permission.Variables_Read
        });
}

Pricing Comparison

21. Artifact Signing & Provenance

Artifact signing provides cryptographic proof that an artifact was built by a specific pipeline from a specific source commit.

C#public class ArtifactSigningService
{
    private readonly ISigningKeyProvider _keyProvider;
    private readonly IProvenanceGenerator _provenanceGen;

    public async Task<SignedArtifact> SignArtifactAsync(
        Artifact artifact, PipelineRun run)
    {
        // Generate SLSA provenance attestation
        var provenance = await _provenanceGen.GenerateAsync(new ProvenanceInput
        {
            BuildType = "github-actions",
            SourceUri = $"git+https://github.com/{run.RepositoryFullName}@{run.HeadSha}",
            BuilderId = "https://github.com/actions/runner",
            BuildConfig = run.WorkflowYaml,
            Materials = run.InputArtifacts.Select(a => new Material
            { Uri = a.Uri, Digest = a.SHA256 }).ToList()
        });

        // Sign with Sigstore/Cosign (keyless signing via OIDC)
        var signature = await SignWithCosignAsync(artifact, provenance);

        // Upload to OCI registry
        await UploadToOCIAsync(artifact, signature, provenance);

        return new SignedArtifact
        {
            Artifact = artifact,
            Signature = signature,
            Provenance = provenance,
            SignedAt = DateTime.UtcNow
        };
    }

    private async Task<Signature> SignWithCosignAsync(
        Artifact artifact, Provenance provenance)
    {
        var certificate = await _keyProvider.GetOIDCCertificateAsync(new OIDCRequest
        {
            Issuer = "https://token.actions.githubusercontent.com",
            Audience = "sigstore",
            Claims = new Dictionary<string, string>
            {
                ["repository"] = artifact.RepositoryFullName,
                ["workflow_ref"] = artifact.WorkflowRef,
                ["sha"] = artifact.SourceSHA
            }
        });

        var payload = Encoding.UTF8.GetBytes(artifact.SHA256);
        var sigBytes = await _keyProvider.SignAsync(certificate.Key, payload);

        return new Signature
        {
            Content = Convert.ToBase64String(sigBytes),
            Certificate = certificate.PEM,
            Bundle = await CreateBundleAsync(artifact, sigBytes, certificate)
        };
    }
}

22. Monitoring & Observability

A CI/CD platform requires deep observability to maintain reliability, detect anomalies, and optimize costs.

Key Metrics

Alerting Rules

yaml# Prometheus alerting rules for CI/CD platform
groups:
  - name: cicd-critical
    rules:
      - alert: HighWebhookLatency
        expr: histogram_quantile(0.99,
          rate(webhook_processing_duration_seconds_bucket[5m]))
          > 5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Webhook processing latency exceeds 5s"

      - alert: RunnerPoolExhausted
        expr: (cicd_runners_busy / cicd_runners_total) > 0.95
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Runner pool utilization above 95%"

      - alert: HighJobFailureRate
        expr: rate(cicd_jobs_failed_total[1h]) /
          rate(cicd_jobs_total[1h]) > 0.1
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Job failure rate above 10%"

      - alert: SecretMaskingFailure
        expr: cicd_secret_masking_failures_total > 0
        labels:
          severity: critical
        annotations:
          summary: "CRITICAL: Secret masking failure detected"

      - alert: AuditChainBroken
        expr: cicd_audit_integrity_check_failed > 0
        labels:
          severity: critical
        annotations:
          summary: "CRITICAL: Audit trail integrity check failed"

23. Compliance & Governance

Enterprise customers in regulated industries require compliance with SOC 2, ISO 27001, FedRAMP, and HIPAA.

Compliance Requirements Matrix

C#public class ComplianceService
{
    public async Task<ComplianceReport> GenerateReportAsync(string organizationId)
    {
        var report = new ComplianceReport
        {
            OrganizationId = organizationId,
            GeneratedAt = DateTime.UtcNow,
            Checks = new List<ComplianceCheck>()
        };

        var repos = await GetReposAsync(organizationId);

        // Check: All repos have branch protection
        var unprotected = repos.Where(r => !r.HasBranchProtection).ToList();
        report.Checks.Add(new ComplianceCheck
        {
            Name = "Branch Protection",
            Status = unprotected.Count == 0 ? ComplianceStatus.Pass : ComplianceStatus.Fail,
            Details = $"{unprotected.Count}/{repos.Count} repos missing branch protection"
        });

        // Check: Secret scanning enabled
        var noScan = repos.Where(r => !r.SecretScanningEnabled).ToList();
        report.Checks.Add(new ComplianceCheck
        {
            Name = "Secret Scanning",
            Status = noScan.Count == 0 ? ComplianceStatus.Pass : ComplianceStatus.Warning,
            Details = $"{noScan.Count} repos without secret scanning"
        });

        // Check: Audit trail integrity
        var integrityValid = await _auditTrail.VerifyIntegrityAsync(
            organizationId,
            DateTime.UtcNow.AddDays(-30), DateTime.UtcNow);
        report.Checks.Add(new ComplianceCheck
        {
            Name = "Audit Trail Integrity",
            Status = integrityValid ? ComplianceStatus.Pass : ComplianceStatus.Critical,
            Details = integrityValid ? "30-day chain integrity verified" : "CHAIN BROKEN"
        });

        // Check: Required reviewers on production environments
        var noReviewers = await FindEnvironmentsWithoutReviewersAsync(organizationId);
        report.Checks.Add(new ComplianceCheck
        {
            Name = "Deployment Approval",
            Status = noReviewers.Count == 0 ? ComplianceStatus.Pass : ComplianceStatus.Fail,
            Details = $"{noReviewers.Count} environments without required reviewers"
        });

        return report;
    }
}

24. Cost Estimation

Understanding cost structure is crucial for capacity planning and pricing decisions.

Infrastructure Cost Breakdown

Cost Optimization Strategies

• Spot/Preemptible VMs: 60-70% cost reduction for runner compute. Implement automatic retry on preemption.
• Intelligent caching: Higher cache hit rates reduce dependency downloads and per-job duration.
• Path-based triggering: Only run workflows when relevant files change.
• Timeout enforcement: Aggressive timeouts prevent runaway jobs.
• Idle runner shutdown: Scale down idle runners quickly.
• Log compression and retention: Compress aggressively and enforce retention. 90% of logs are never read after 7 days.

C#public class CostEstimator
{
    public JobCost EstimateJobCost(JobRequest request)
    {
        var machineType = request.RunnerType switch
        {
            "ubuntu-latest"  => new MachineCost { CpuPerHour = 0.08m, MemoryGbPerHour = 0.04m },
            "windows-latest" => new MachineCost { CpuPerHour = 0.16m, MemoryGbPerHour = 0.08m },
            "macos-latest"   => new MachineCost { CpuPerHour = 0.80m, MemoryGbPerHour = 0.40m },
            _ => new MachineCost { CpuPerHour = 0.08m, MemoryGbPerHour = 0.04m }
        };

        var durationHours = (decimal)request.ExpectedDurationMinutes / 60m;
        var computeCost = (machineType.CpuPerHour +
            machineType.MemoryGbPerHour * request.MemoryGb) * durationHours;
        var storageCost = request.ExpectedArtifactSizeGb * 0.005m;
        var networkCost = request.ExpectedTransferGb * 0.09m;

        return new JobCost
        {
            ComputeCost = computeCost,
            StorageCost = storageCost,
            NetworkCost = networkCost,
            TotalEstimatedCost = computeCost + storageCost + networkCost
        };
    }
}

25. Testing Strategy

Testing a CI/CD platform is uniquely challenging — you're testing a system that runs other systems.

Testing Layers

Key Test Scenarios

C#public class PipelineParserTests
{
    [Fact]
    public async Task Parse_ValidWorkflow_ReturnsCorrectStructure()
    {
        var yaml = @"
name: CI
on:
  push:
    branches: [main]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: echo hello
";
        var parser = new PipelineParser();
        var result = await parser.ParseAsync(yaml);

        Assert.Equal("CI", result.Name);
        Assert.Single(result.Jobs);
        Assert.True(result.Jobs.ContainsKey("build"));
        Assert.Equal("ubuntu-latest", result.Jobs["build"].RunsOn);
        Assert.Equal(2, result.Jobs["build"].Steps.Count);
    }

    [Fact]
    public async Task Parse_CyclicDependencies_ThrowsValidation()
    {
        var yaml = @"
name: Cyclic
on: {push: {branches: [main]}}
jobs:
  a:
    runs-on: ubuntu-latest
    needs: [b]
    steps: [{run: echo a}]
  b:
    runs-on: ubuntu-latest
    needs: [a]
    steps: [{run: echo b}]
";
        var parser = new PipelineParser();
        await Assert.ThrowsAsync<ValidationException>(
            () => parser.ParseAsync(yaml));
    }

    [Fact]
    public void MatrixExpand_CartesianProduct_GeneratesAllCombinations()
    {
        var matrix = new MatrixConfig
        {
            Include = new Dictionary<string, string[]>
            {
                ["os"] = new[] { "ubuntu", "windows" },
                ["dotnet"] = new[] { "6.0", "7.0", "8.0" }
            }
        };
        var expander = new MatrixExpander();
        var result = expander.Expand(matrix);
        Assert.Equal(6, result.Count);
    }
}

public class DAGSchedulerTests
{
    [Fact]
    public void TopologicalSort_LinearDependencies_ReturnsCorrectOrder()
    {
        var dag = new DAG();
        dag.AddEdge("build", "test");
        dag.AddEdge("test", "deploy");
        var sorted = dag.TopologicalSort();
        Assert.Equal(new[] { "build", "test", "deploy" }, sorted);
    }

    [Fact]
    public void TopologicalSort_ParallelJobs_BothAppear()
    {
        var dag = new DAG();
        dag.AddEdge("build", "test-unit");
        dag.AddEdge("build", "test-integration");
        dag.AddEdge("test-unit", "deploy");
        dag.AddEdge("test-integration", "deploy");
        var sorted = dag.TopologicalSort();
        Assert.Equal("build", sorted[0]);
        Assert.Equal("deploy", sorted[^1]);
    }
}

public class SecretMaskingTests
{
    [Fact]
    public void MaskLogOutput_PlainTextSecret_IsMasked()
    {
        var masking = new SecretMaskingService();
        masking.RegisterSecret("token", "sk-abc123xyz");
        var masked = masking.MaskLogOutput("Using token sk-abc123xyz for auth");
        Assert.Equal("Using token *** for auth", masked);
    }
}

Canary Testing for Platform Changes

When updating the platform itself, use canary deployment: run the new version alongside the old, route a small percentage of executions to it, and monitor for regressions. Maintain "golden repositories" — representative projects in different languages — and run them on every platform change, comparing outputs against known-good baselines.

26. Codebase Structure

The CI/CD platform is organized as a set of microservices with shared libraries.

textcicd-platform/
├── src/
│   ├── api-gateway/                    # REST/GraphQL API
│   │   ├── Controllers/
│   │   ├── Middleware/
│   │   └── Graphql/
│   ├── webhook-receiver/               # Git webhook processing
│   │   ├── Handlers/
│   │   └── Validators/
│   ├── pipeline-parser/                # YAML parsing, validation
│   │   ├── Schema/
│   │   └── Validators/
│   ├── scheduler/                      # DAG scheduling, dispatch
│   │   ├── DAG/
│   │   ├── Queue/
│   │   └── Dispatch/
│   ├── runner-manager/                 # Runner provisioning
│   │   ├── CloudProviders/
│   │   ├── SelfHosted/
│   │   └── AutoScaler/
│   ├── runner-agent/                   # Runs inside runner VMs
│   │   ├── Executor/
│   │   ├── Container/
│   │   └── LogCollector/
│   ├── state-service/                  # Run/job lifecycle
│   │   ├── StateMachine/
│   │   └── Events/
│   ├── artifact-service/               # Artifact storage
│   ├── cache-service/                  # Build cache
│   ├── secret-service/                 # Secret management
│   ├── notification-service/           # Slack, email, webhooks
│   ├── marketplace/                    # Action marketplace
│   ├── signing-service/                # Artifact signing
│   ├── compliance-service/             # Audit, RBAC
│   └── billing-service/                # Usage tracking
├── shared/
│   ├── CICD.Contracts/                 # Shared DTOs, enums
│   ├── CICD.Storage/                   # Database, blob access
│   ├── CICD.Security/                  # Encryption, auth
│   └── CICD.Monitoring/                # Metrics, tracing
├── infrastructure/
│   ├── terraform/                      # Infrastructure as code
│   ├── kubernetes/                     # K8s manifests
│   └── docker/                         # Dockerfiles
└── tests/
    ├── unit-tests/
    ├── integration-tests/
    ├── e2e-tests/
    └── load-tests/

Technology Stack

27. Interview Q&A Deep Dive

These are the most commonly asked questions about CI/CD system design in senior+ engineering interviews, along with comprehensive answers.

Q1: How would you design a CI/CD system that handles 1 million pipeline runs per day?

Q2: How do you ensure security when executing untrusted code from external PRs?

Q3: How do you handle the tension between fast builds and cost optimization?

Q4: How would you design the DAG scheduler for job orchestration?

Q5: How do you implement effective secret management in CI/CD?

Q6: How do you prevent supply chain attacks in CI/CD?

Q7: How do you handle monorepo builds efficiently?

Q8: How would you design the runner auto-scaling system?

Q9: How do you implement rollback strategies in CI/CD?

Q10: How do you estimate the cost of running a CI/CD platform?

Q11: How do you handle concurrency and prevent conflicts?

Q12: How do you monitor and debug pipeline failures at scale?