system-design46 min read

How to Design Feature Flag & Experimentation Platform like LaunchDarkly - A Senior+ Guide | Ayodhyya

How to Design Feature Flag & Experimentation Platform like LaunchDarkly

Building feature flags, A/B testing, and progressive rollouts for engineering teams at scale

System DesignFeature FlagsA/B TestingLaunchDarklyProgressive RolloutSenior+ Guide | Published: July 14, 2026

1. Introduction - Why Feature Flags Matter

Feature flags (also called feature toggles, feature switches, or feature gates) are one of the most powerful techniques in modern software engineering. They allow teams to decouple code deployment from feature release, enabling safer, faster, and more controlled rollouts. When combined with experimentation and A/B testing, feature flags become the backbone of a data-driven product development culture.

LaunchDarkly, founded in 2014, pioneered the commercial feature management space and now serves over 2,500 customers including IBM, Atlassian, NBC, and Microsoft. The platform processes an astounding 10+ trillion flag evaluations per day, with sub-millisecond latency for SDK evaluations. Split.io (acquired by Harness) built a feature delivery platform tightly integrated with experimentation, while Flagsmith and Unleash offer open-source alternatives.

The global feature flag management market was valued at $1.3 billion in 2025 and is projected to reach $4.8 billion by 2030, growing at a CAGR of 29.5%. This explosive growth is driven by the adoption of DevOps practices, continuous delivery, and the need for data-driven feature releases.

10T+
Daily flag evaluations (LaunchDarkly)
<1ms
SDK evaluation latency (p99)
99.99%
Platform uptime SLA
2,500+
Enterprise customers

In this comprehensive system design guide, we will architect a feature flag and experimentation platform from scratch, covering flag management, real-time streaming, SDK design, A/B testing, statistical analysis, progressive rollouts, and the data pipeline. This guide is aimed at senior+ engineers preparing for system design interviews or building similar platforms in production.

What makes this platform challenging?
  • Scale: Trillions of evaluations per day with sub-millisecond latency
  • Consistency: All SDK instances must see the same flag state within seconds of a change
  • Accuracy: Experimentation requires deterministic bucketing and statistical rigor
  • Availability: SDKs must continue working even if the platform is down
  • Security: Customer data never leaves their infrastructure; HIPAA/SOC2 compliance

2. Functional & Non-Functional Requirements

Functional Requirements

#RequirementDescription
FR1Flag CRUD OperationsCreate, read, update, delete feature flags with metadata, tags, and descriptions
FR2Targeting RulesDefine rules to target users by attributes (country, plan, age, custom attributes)
FR3Percentage RolloutsGradually roll out features to a percentage of users with consistent bucketing
FR4Multi-Variate FlagsSupport boolean, string, number, JSON variants with multiple values
FR5A/B TestingCreate experiments with control/treatment groups and track metrics
FR6Progressive DeliveryCanary deployments, ring-based rollouts, and scheduled releases
FR7Audit LoggingTrack all flag changes with who, when, what, and diff
FR8SDK DistributionClient-side, server-side, and edge SDKs for multiple platforms
FR9Real-Time UpdatesPush flag changes to SDKs within 2 seconds via SSE/WebSocket
FR10IntegrationsJira, GitHub, Slack, Datadog, PagerDuty integrations
FR11Flag SchedulingSchedule flag changes for future dates and time windows
FR12Approval WorkflowsRequire approvals before flag changes in production environments

Non-Functional Requirements

#RequirementTarget
NFR1Latency (SDK evaluation)<1ms at p99, <0.5ms at p50
NFR2Availability99.99% for SDK evaluation path; 99.9% for management API
NFR3ConsistencyFlag propagation to all SDKs within 2 seconds globally
NFR4ScalabilitySupport 10+ trillion evaluations/day, 1M+ concurrent SDK connections
NFR5DurabilityNo flag state lost on platform failure; SDKs cache last-known-good state
NFR6SecurityTLS everywhere, SDK keys scoped by environment, SOC2 Type II compliant
NFR7Offline ModeSDKs function with cached flag state when platform is unreachable
NFR8Multi-TenancyComplete isolation between customer organizations and projects

3. Capacity Estimation

Let us estimate the system scale based on industry benchmarks and LaunchDarkly published metrics:

Flag & Customer Scale

EntityCountCalculation
Organizations5,000Enterprise + SMB customers
Projects per org10 avg5,000 x 10 = 50,000 projects
Environments per project4 avgDev, Staging, Beta, Production
Flags per project200 avg50,000 x 200 = 10M flags
Active flags5M~50% of all flags are active
SDK instances (server)500KMicroservices across all customers
SDK instances (client)50MMobile + web browser clients

Evaluation Volume

Calculations:

  • Total evaluations per day: 12 trillion (12 x 10^12)
  • Evaluations per second: ~140M (peak: 500M/s)
  • Average flags evaluated per context: 50 (batch evaluation)
  • Unique evaluation requests per second: ~2.8M
  • Data per evaluation response: ~200 bytes
  • Bandwidth for evaluation responses: ~560 MB/s (peak: 2 GB/s)

Storage Estimation

Data TypeSize per RecordAnnual VolumeTotal Storage
Flag configurations5 KB10M flags (snapshot)~50 GB
Evaluation events200 bytes12T/day x 365~876 TB/year
Audit logs1 KB10M changes/day~3.6 TB/year
Experiment metrics500 bytes1B events/day~182 TB/year
User contexts500 bytes100M users~50 GB
Note: The evaluation event volume is the largest data challenge. We use streaming aggregation (count-min sketch, HyperLogLog) to reduce raw event storage by 99.9% while maintaining statistical accuracy.

4. Data Model

The data model must support multi-tenancy, hierarchical organization, and complex targeting rules:

erDiagram ORGANIZATION ||--o{ PROJECT : contains PROJECT ||--o{ ENVIRONMENT : has PROJECT ||--o{ FLAG : defines ENVIRONMENT ||--o{ FLAG_STATE : has FLAG ||--o{ TARGETING_RULE : has rules FLAG ||--o{ VARIATION : has variations FLAG ||--o{ EXPERIMENT : can run EXPERIMENT ||--o{ METRIC : measures ORGANIZATION { uuid id PK string name string plan json settings } PROJECT { uuid id PK uuid org_id FK string name string key } ENVIRONMENT { uuid id PK uuid project_id FK string name string key boolean critical } FLAG { uuid id PK uuid project_id FK string key string name string flag_type boolean archived } VARIATION { uuid id PK uuid flag_id FK string key string value } TARGETING_RULE { uuid id PK uuid flag_id FK int priority json conditions string variation_key } EXPERIMENT { uuid id PK uuid flag_id FK string hypothesis float traffic_percentage string status } METRIC { uuid id PK uuid experiment_id FK string name string event_key }

Flag States per Environment

FieldTypeDescription
flag_idUUIDReference to the flag definition
environment_idUUIDReference to the environment
enabledBooleanWhether the flag is enabled in this environment
default_variationStringVariation returned when no rules match
targetsJSONIndividual user targeting (on/off variations)
rulesJSON ArrayOrdered targeting rules with conditions
fallthroughJSONDefault rule when no targeting matches
prerequisitesJSON ArrayPrerequisite flags that must be on
scheduled_changesJSON ArrayPending future state changes

5. API Design

Management API (Dashboard & Admin)

POST   /v1/orgs                          // Create organization
GET    /v1/orgs/{orgId}                   // Get organization details
PATCH  /v1/orgs/{orgId}                   // Update organization
POST   /v1/orgs/{orgId}/projects          // Create project
GET    /v1/orgs/{orgId}/projects          // List projects
POST   /v1/projects/{projectKey}/envs     // Create environment
GET    /v1/projects/{projectKey}/envs     // List environments
POST   /v1/envs/{envKey}/clone            // Clone environment
POST   /v1/projects/{projectKey}/flags     // Create flag
GET    /v1/projects/{projectKey}/flags     // List flags (paginated)
GET    /v1/flags/{flagKey}                 // Get flag with all env states
PATCH  /v1/flags/{flagKey}                 // Update flag metadata
DELETE /v1/flags/{flagKey}                 // Archive flag
GET    /v1/flags/{flagKey}/envs/{envKey}            // Get flag state
PATCH  /v1/flags/{flagKey}/envs/{envKey}            // Update flag state
POST   /v1/flags/{flagKey}/envs/{envKey}/rules       // Add targeting rule
DELETE /v1/flags/{flagKey}/envs/{envKey}/rules/{id}  // Remove rule
POST   /v1/flags/{flagKey}/envs/{envKey}/experiments   // Create experiment
GET    /v1/experiments/{experimentId}/results           // Get experiment results
GET    /v1/audit-log?projectKey={pk}&since={date}      // Get audit entries

Client-Side SDK API

public interface IFeatureFlagClientSDK
{
    bool BoolVariation(string flagKey, bool defaultValue, UserContext context);
    string StringVariation(string flagKey, string defaultValue, UserContext context);
    int IntVariation(string flagKey, int defaultValue, UserContext context);
    double DoubleVariation(string flagKey, double defaultValue, UserContext context);
    T JsonVariation<T>(string flagKey, T defaultValue, UserContext context);
    Dictionary<string, FlagValue> AllFlags(UserContext context);
    void Identify(UserContext context);
    void Clear();
    void Track(string eventName, UserContext context, Dictionary<string, object> data);
    Task<bool> WaitForInitialization(TimeSpan timeout);
    void Close();
}

Server-Side SDK API

public interface IFeatureFlagServerSDK
{
    EvaluationResult Evaluate(string flagKey, UserContext context);
    Dictionary<string, EvaluationResult> EvaluateAll(UserContext context);
    Flag GetFlag(string flagKey);
    IReadOnlyList<Flag> GetAllFlags();
    void Track(Event event);
    ClientStatus Status { get; }
    event EventHandler<FlagChangedEventArgs> FlagChanged;
}

6. High-Level Architecture

The platform follows a control plane + data plane separation. The control plane manages flag configurations and experiments, while the data plane handles flag evaluations and event collection at the edge.

graph TB subgraph ClientApps[Client Applications] WEB[Web App JS SDK] MOB[Mobile App iOS Android] SRV[Backend Service .NET Java Go] EDGE_SDK[Edge Worker Cloudflare] end subgraph EdgeLayer[Edge Layer Data Plane] CDN[CDN Edge] STREAM_SSE[SSE Gateway] EVAL_EDGE[Edge Evaluation] EVENT_COLLECTOR[Event Collector] end subgraph ControlPlane[Control Plane] LB[Load Balancer] API_GW[API Gateway] FLAG_MGMT[Flag Management Service] TARGETING[Targeting Rules Engine] EXPERIMENT[Experimentation Engine] SCHEDULER[Scheduler] WEBHOOK[Webhook Dispatch] end subgraph StreamingLayer[Streaming Layer] KAFKA[Apache Kafka] REDIS_STREAM[Redis Pub Sub] SSE_SERVER[SSE Server Cluster] end subgraph DataLayer[Data Layer] PG_PRIMARY[PostgreSQL Primary] PG_REPLICA[PostgreSQL Replica] REDIS_CACHE[Redis Cluster] ELASTIC[Elasticsearch] end subgraph Metrics[Metrics and Analytics] KAFKA_Metrics[Kafka Pipeline] CLICKHOUSE[ClickHouse] DASHBOARD[Analytics Dashboard] end WEB --> CDN MOB --> CDN SRV --> STREAM_SSE EDGE_SDK --> CDN CDN --> EVAL_EDGE CDN --> STREAM_SSE STREAM_SSE --> REDIS_STREAM SSE_SERVER --> REDIS_STREAM LB --> API_GW API_GW --> FLAG_MGMT API_GW --> TARGETING API_GW --> EXPERIMENT API_GW --> SCHEDULER FLAG_MGMT --> PG_PRIMARY TARGETING --> PG_PRIMARY PG_PRIMARY --> PG_REPLICA FLAG_MGMT --> REDIS_CACHE EVAL_EDGE --> REDIS_CACHE EVENT_COLLECTOR --> KAFKA_Metrics KAFKA_Metrics --> CLICKHOUSE CLICKHOUSE --> DASHBOARD FLAG_MGMT --> KAFKA KAFKA --> REDIS_STREAM FLAG_MGMT --> WEBHOOK

Architecture Principles

  • Control Plane / Data Plane Separation: The control plane operates independently from the data plane. SDK evaluations continue even if the control plane is down.
  • SDK-Side Evaluation: All flag evaluation happens locally within the SDK. No network calls needed.
  • Push-Based Distribution: Flag changes are pushed to SDKs via SSE/WebSocket, ensuring near-instant propagation.
  • Eventual Consistency with Bounded Staleness: SDKs may lag behind by at most 2 seconds in normal operation.

7. Flag Evaluation Engine

The flag evaluation engine is the heart of the system. It takes a flag key, user context, and configuration, then returns the correct variation. The engine must be deterministic, fast, and consistent across all SDK instances.

Evaluation Flow

flowchart TD START[Evaluate Flag] --> VALIDATE{Valid context?} VALIDATE -->|No| DEFAULT_OFF[Return Default INVALID_CONTEXT] VALIDATE -->|Yes| ENABLED{Flag enabled?} ENABLED -->|No| DEFAULT_OFF2[Return Default FLAG_DISABLED] ENABLED -->|Yes| PREREQ{Has prerequisites?} PREREQ -->|Yes| PREREQ_CHECK{Prerequisite match?} PREREQ_CHECK -->|No| PREREQ_FAIL[Return Default PREREQUISITE_FAILED] PREREQ_CHECK -->|Yes| INDIVIDUAL PREREQ -->|No| INDIVIDUAL{User in target list?} INDIVIDUAL -->|Yes| TARGETED[Return Targeted Variation] INDIVIDUAL -->|No| RULES[Iterate Targeting Rules] RULES --> RULE_CHECK{Rule conditions match?} RULE_CHECK -->|Yes| RULE_VARIATION{Has percentage rollout?} RULE_CHECK -->|No| NEXT_RULE{More rules?} NEXT_RULE -->|Yes| RULES NEXT_RULE -->|No| FALLTHROUGH[Fallthrough Rule] RULE_VARIATION -->|Yes| BUCKET[Bucket User MurmurHash3] RULE_VARIATION -->|No| RULE_VAR[Return Rule Variation] BUCKET --> PERCENTAGE[Return Variation by Percentage] FALLTHROUGH --> FT_CHECK{Fallthrough has rollout?} FT_CHECK -->|Yes| BUCKET_FT[Bucket User] FT_CHECK -->|No| FT_DEFAULT[Return Default Variation] BUCKET_FT --> FT_RESULT[Return Variation FALLTHROUGH] style START fill:#0088ff,color:white style TARGETED fill:#10b981,color:white style PERCENTAGE fill:#10b981,color:white style FT_RESULT fill:#10b981,color:white

Deterministic Bucketing Algorithm

For percentage rollouts, we use a deterministic hash function that consistently assigns users to buckets:

flowchart LR A["User Key + Flag Key"] --> B["Generate Seed"] B --> C["MurmurHash3 128-bit"] C --> D["Extract 0-9999 range"] D --> E{"Value < Rollout % ?"} E -->|Yes| F["Treatment Variation"] E -->|No| G["Control Variation"] style C fill:#0088ff,color:white style F fill:#10b981,color:white style G fill:#f59e0b,color:white

The bucketing algorithm uses a composite key of user_key + flag_key + salt to compute a deterministic hash. The hash maps to a 0-9999 range for 0.01% granularity.

Evaluation Example

var context = new UserContext
{
    Key = "user-12345",
    Custom = new Dictionary<string, object>
    {
        ["country"] = "US",
        ["plan"] = "enterprise"
    }
};
var result = sdk.Evaluate("new-checkout-flow", context);
// Trace:
// 1. Flag enabled? Yes
// 2. Individual targeting? No match
// 3. Rule 1: country == "US" AND plan == "enterprise" -> "treatment-v2" (100%)
// 4. Result: "treatment-v2", Reason: RULE_MATCH

8. SDK Architecture

The SDK is the primary interface between customer applications and the feature flag platform.

graph TB subgraph SDK[SDK Instance] EVAL_ENGINE[Evaluation Engine] FLAG_CACHE[In-Memory Flag Cache] EVENT_QUEUE[Event Queue] STREAM_CONN[Streaming Connection] PERSIST_CACHE[Persistent Cache] STAT_COL[Statistics Collector] end subgraph Init[Initialization] LOAD_PERSIST[Load Persistent Cache] FETCH_FLAGS[Fetch Flags from API] MERGE[Merge into Memory] OPEN_STREAM[Open SSE Stream] end subgraph Runtime[Runtime] INCOMING_FLAG[Flag Change Event] UPDATE_FLAG[Update Cache] EMIT_EVENT[Emit Change Event] BATCH_SEND[Batch Send Events] PERIODIC_POLL[Periodic Poll] end LOAD_PERSIST --> MERGE FETCH_FLAGS --> MERGE MERGE --> OPEN_STREAM OPEN_STREAM --> STREAM_CONN STREAM_CONN --> INCOMING_FLAG INCOMING_FLAG --> UPDATE_FLAG UPDATE_FLAG --> EMIT_EVENT EVAL_ENGINE --> FLAG_CACHE EVAL_ENGINE --> EVENT_QUEUE EVENT_QUEUE --> BATCH_SEND PERIODIC_POLL --> FETCH_FLAGS STAT_COL --> EVENT_QUEUE style EVAL_ENGINE fill:#0088ff,color:white style FLAG_CACHE fill:#10b981,color:white style STREAM_CONN fill:#8b5cf6,color:white

SDK Initialization Sequence

sequenceDiagram participant App as Application participant SDK as Feature SDK participant Cache as Persistent Cache participant API as Feature API participant Stream as SSE Stream App->>SDK: new FeatureSDK(sdkKey) SDK->>Cache: Load cached flags Cache-->>SDK: Cached flag state SDK->>SDK: Initialize evaluation engine par Parallel Init SDK->>API: GET /v1/sdk/flags API-->>SDK: All flags SDK->>SDK: Merge into memory SDK->>Stream: Open SSE connection Stream-->>SDK: Connected end SDK-->>App: Ready event Note over Stream: Later Stream->>SDK: Flag change event SDK->>SDK: Update cache SDK-->>App: FlagChanged event

SDK Types Comparison

FeatureClient-SideServer-SideEdge
Evaluation LocationIn-browserIn-processEdge worker
AuthenticationClient keyServer keyEdge key
Flag DistributionSSE streamingSSE + pollingEdge KV
Persistent CachelocalStorageFile systemEdge KV
Evaluation Latency<1ms<1ms<5ms
Event SendingBatched 30sBatched 60sAggregated
SecurityRules exposedRules hiddenTrusted edge

9. Real-Time Flag Streaming

Real-time flag propagation is critical. When a developer toggles a flag, all connected SDKs must receive the update within 2 seconds. We use Server-Sent Events (SSE) as the primary protocol.

sequenceDiagram participant Dashboard as Dashboard participant API as Flag Management API participant DB as PostgreSQL participant Redis as Redis Pub/Sub participant SSE1 as SSE Gateway US participant SSE2 as SSE Gateway EU participant SDK1 as SDK US participant SDK2 as SDK EU Dashboard->>API: PATCH /flags/checkout API->>DB: Update flag state DB-->>API: Updated API->>Redis: PUBLISH flag:checkout par Parallel Distribution Redis->>SSE1: SUBSCRIBE SSE1->>SDK1: SSE flag change Redis->>SSE2: SUBSCRIBE SSE2->>SDK2: SSE flag change end SDK1->>SDK1: Update cache SDK2->>SDK2: Update cache

SSE Message Protocol

// SSE stream format for flag updates
// Connection: GET /v1/stream?sdk-key=xxx

// Initial connection
event: connected
data: {"version":42,"flags":1250,"interval":30000}

// Individual flag change
event: patch
data: {"flag":"new-checkout","version":43,"data":{"enabled":true,"variation":"treatment"}}

// Flag deletion
event: delete
data: {"flag":"old-feature","version":44}

// Full sync (reconnect)
event: sync
data: {"flags":{"new-checkout":{...}},"version":45}

// Keep-alive
: keepalive

Reconnection Strategy

Exponential backoff with jitter:

  • Attempt 1: 1 second
  • Attempt 2: 2 seconds + random(0-1s) jitter
  • Attempt 3: 4 seconds + random(0-2s) jitter
  • Attempt 4: 8 seconds + random(0-4s) jitter
  • Max delay: 30 seconds
  • On reconnect: send Last-Event-ID header for replay

The server maintains a flag change log (last 1000 changes) per stream for replay during reconnection.

10. A/B Testing & Experimentation Framework

An experimentation framework transforms feature flags from a deployment tool into a decision-making engine.

flowchart TB HYPOTHESIS[1. Formulate Hypothesis] --> DESIGN[2. Design Experiment] DESIGN --> CONFIGURE[3. Configure Flag & Variations] CONFIGURE --> VALIDATE[4. Validate Setup] VALIDATE --> RUN[5. Run Experiment] RUN --> COLLECT[6. Collect Data] COLLECT --> ANALYZE[7. Analyze Results] ANALYZE --> DECIDE[8. Make Decision] DECIDE -->|Significant| ROLLOUT[9. Rollout Winner] DECIDE -->|Not Significant| ITERATE[Iterate and Re-run] ITERATE --> HYPOTHESIS style HYPOTHESIS fill:#0088ff,color:white style RUN fill:#10b981,color:white style ANALYZE fill:#8b5cf6,color:white style ROLLOUT fill:#10b981,color:white

Experiment Configuration

FieldDescriptionExample
experiment_keyUnique identifier"checkout-redesign-v2"
hypothesisExpected outcome"Redesign increases conversion by 10%"
flag_keyControlling flag"new-checkout-flow"
variationsControl + treatment["control", "v2-minimal", "v2-steps"]
traffic_allocation% of users in experiment50%
primary_metricMain success metric"checkout_completed"
guardrail_metricsMust not regress["error_rate", "page_load_time"]
minimum_sample_sizeRequired per variation10,000
significance_levelStatistical threshold0.05 (95% confidence)

Event Tracking Pipeline

flowchart LR APP["Application"] -->|"sdk.track(event)"| SDK[SDK] SDK --> BATCH[Event Batch 30s] BATCH --> INGEST[Event Ingestion API] INGEST --> KAFKA[Kafka] KAFKA --> AGG[Streaming Aggregation] AGG --> CH[ClickHouse] CH --> RESULTS[Experiment Results API] style SDK fill:#0088ff,color:white style KAFKA fill:#8b5cf6,color:white style CH fill:#10b981,color:white

11. Statistical Analysis

The experimentation engine supports both Frequentist and Bayesian analysis approaches.

Frequentist vs Bayesian

AspectFrequentistBayesian
Core Conceptp-value, confidence intervalsPosterior probability distributions
Sample SizePre-computed requirementCan peek at any time
Interpretation95% chance result not due to chance94% probability treatment is better
GuardrailsRequires Bonferroni correctionNaturally handles multiple comparisons
Best ForLarge samples, fixed hypothesesEarly-stage, quick decisions
Stopping RuleFixed-horizon onlyContinuous monitoring allowed

Sample Size Formula: n = (Z_{1-a/2} + Z_{1-b})^2 x 2s^2 / d^2

  • Z_{1-a/2} = 1.96 for 95% confidence
  • Z_{1-b} = 0.84 for 80% power
  • s = Standard deviation of metric
  • d = Minimum detectable effect (e.g., 5% lift)

For s=1.0 and 5% lift: n ~ 15,700 per variation.

Streaming Statistics

For real-time dashboards, we use streaming statistical algorithms:

  • Count-Min Sketch: Approximate event counting with bounded error
  • T-Digest: Streaming quantile estimation for metrics like revenue
  • HyperLogLog: Unique user counting with 0.81% standard error
  • Streaming Variance: Welford's algorithm for running mean and variance

12. Progressive Rollout & Canary Deployments

Progressive rollout gradually exposes a feature to increasing percentages of users while monitoring for issues.

gantt title Progressive Rollout Timeline dateFormat YYYY-MM-DD axisFormat %b %d section Canary 1% 1% Traffic :canary, 2026-07-14, 1d Monitor :crit, m1, 2026-07-14, 1d section Early Adopters 5% 5% Traffic :early, 2026-07-15, 2d Monitor :crit, m2, 2026-07-15, 2d section Gradual 10% Traffic :g1, 2026-07-17, 2d 25% Traffic :g2, 2026-07-19, 2d 50% Traffic :g3, 2026-07-21, 2d section Full Rollout 100% Traffic :full, 2026-07-23, 2d Cleanup :cleanup, 2026-07-25, 3d

Rollout Configuration

StageTrafficDurationAuto-AdvanceRollback Trigger
Canary1%24 hoursYesError rate > 0.1% increase
Early Adopters5%48 hoursYesError rate > 0.05% increase
Partial10%48 hoursManualGuardrail metric regression
Growth25-50%72 hours eachManualError rate, latency, conversion
Full100%7 daysNoAny critical alert

Automatic Rollback

// Rollout monitor - runs as background service
public class RolloutMonitor : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            var activeRollouts = await _rolloutService.GetActiveRollouts();
            foreach (var rollout in activeRollouts)
            {
                var metrics = await _metricsService
                    .GetGuardrailMetrics(rollout.FlagKey, rollout.Environment);
                foreach (var guardrail in rollout.Guardrails)
                {
                    var current = metrics[guardrail.MetricName];
                    var baseline = await _baselineService
                        .GetBaseline(guardrail.MetricName, rollout.Environment);
                    if (current > baseline * (1 + guardrail.Threshold))
                    {
                        await _flagService.Rollback(rollout.FlagKey,
                            rollout.Environment, rollout.PreviousVariation);
                        await _alertService.Send(new RollbackAlert
                        {
                            FlagKey = rollout.FlagKey,
                            Metric = guardrail.MetricName,
                            Current = current,
                            Baseline = baseline
                        });
                    }
                }
            }
            await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
        }
    }
}

13. Flag Lifecycle Management

Feature flags have a lifecycle that must be managed to prevent flag debt.

stateDiagram-v2 [*] --> Draft: Create Flag Draft --> InReview: Submit for Review InReview --> Approved: Review Passed InReview --> Draft: Changes Requested Approved --> Inactive: Deploy to Dev Inactive --> Testing: Enable in Staging Testing --> CanaryRollout: Enable in Production 1% CanaryRollout --> PartialRollout: Monitoring OK CanaryRollout --> RolledBack: Guardrail Breach PartialRollout --> FullRollout: All Stages Pass PartialRollout --> RolledBack: Guardrail Breach FullRollout --> Cleanup: Flag No Longer Needed RolledBack --> Draft: Investigate Fix Cleanup --> Archived: Remove from Code Archived --> [*]

Flag Cleanup Automation

  1. Day 60: Warning notification to the flag owner
  2. Day 90: Flag marked as cleanup_suggested; creates Jira ticket
  3. Day 120: Escalates to engineering manager; flag becomes read-only
  4. Day 180: Flag auto-archived; cleanup ticket priority escalated

14. Audit Logging & Change Tracking

Every action on the platform must be logged for compliance (SOC2, HIPAA, FedRAMP).

FieldTypeDescription
audit_idUUIDUnique identifier
timestampISO 8601When the action occurred
actorObject{ id, email, name, ip_address, user_agent }
actionEnumCREATE, UPDATE, DELETE, ENABLE, DISABLE, APPROVE, ROLLBACK
resource_typeStringFLAG, ENVIRONMENT, PROJECT, EXPERIMENT
resource_idUUIDID of the affected resource
environmentStringWhich environment was affected
changesJSONDiff of before/after state (JSON Patch)
metadataJSONAdditional context (Jira ticket, PR link)

Audit Log Storage Tiers

  • Hot (Elasticsearch): Last 90 days - fast search, real-time queries
  • Warm (S3 Parquet): 90 days to 2 years - compressed, queryable via Athena
  • Cold (Glacier): 2+ years - compliance archive

15. Multi-Variate & Complex Targeting

Beyond simple boolean flags, multi-variate flags allow testing multiple variations simultaneously.

Complex Targeting Rules Example

// Example: Multi-variate flag with complex targeting
public static class TargetingRuleExample
{
    public static FlagConfiguration ConfigureHomepageLayout()
    {
        return new FlagConfiguration
        {
            Key = "homepage-layout",
            Type = FlagType.String,
            DefaultVariation = "classic",
            Variations = new[]
            {
                new Variation { Key = "classic", Value = "classic" },
                new Variation { Key = "modern", Value = "modern" },
                new Variation { Key = "minimal", Value = "minimal" },
                new Variation { Key = "beta", Value = "beta" }
            },
            Rules = new[]
            {
                // Rule 1: Internal employees always get beta
                new TargetingRule
                {
                    Priority = 1,
                    Conditions = new[]
                    {
                        new Condition { Attribute = "email",
                            Op = "endsWith", Values = ["@company.com"] }
                    },
                    Variation = "beta"
                },
                // Rule 2: Premium users in US get modern
                new TargetingRule
                {
                    Priority = 2,
                    Conditions = new[]
                    {
                        new Condition { Attribute = "plan",
                            Op = "in", Values = ["premium", "enterprise"] },
                        new Condition { Attribute = "country",
                            Op = "in", Values = ["US", "CA"] }
                    },
                    Variation = "modern"
                },
                // Rule 3: 30% rollout of minimal to everyone else
                new TargetingRule
                {
                    Priority = 3,
                    PercentageRollout = new[]
                    {
                        new PercentageVariation { Variation = "minimal", Weight = 30 },
                        new PercentageVariation { Variation = "classic", Weight = 70 }
                    }
                }
            }
        };
    }
}

Supported Operators

OperatorDescriptionExample
inValue is in setcountry in ["US", "CA", "UK"]
notInValue not in setplan notIn ["free", "trial"]
endsWithString ends withemail endsWith ["@company.com"]
startsWithString starts withversion startsWith ["2."]
containsString containsbrowser contains ["Chrome"]
greaterThanNumeric greaterage greaterThan ["18"]
lessThanNumeric lessscore lessThan ["50"]
betweenNumeric betweenversion between ["1.0", "2.0"]
existsAttribute existsbeta_tester exists
beforeDate beforesignupDate before ["2025-01-01"]
afterDate aftersignupDate after ["2024-06-01"]

16. Integration Ecosystem

Feature flag platforms thrive when connected to the developer's existing tools.

graph LR subgraph SourceControl[Source Control] GH[GitHub] GL[GitLab] end subgraph ProjectMgmt[Project Management] JIRA[Jira] LINEAR[Linear] end subgraph Comms[Communication] SLACK[Slack] TEAMS[MS Teams] end subgraph Monitoring[Monitoring] DD[Datadog] NR[New Relic] PD[PagerDuty] end subgraph CICD[CI/CD] GHA[GitHub Actions] JENKINS[Jenkins] end subgraph Platform[Feature Platform] FP[Webhook Engine] end FP --> GH FP --> GL FP --> JIRA FP --> LINEAR FP --> SLACK FP --> TEAMS FP --> DD FP --> NR FP --> PD FP --> GHA FP --> JENKINS style FP fill:#0088ff,color:white

Webhook Events

EventDescriptionKey Fields
flag.createdNew flag createdflag_key, project, created_by
flag.updatedConfig changedflag_key, environment, changes
flag.enabledTurned onflag_key, environment, variation
flag.disabledTurned offflag_key, environment
experiment.startedExperiment launchedexperiment_key, variations
experiment.concludedResults readyexperiment_key, winner, confidence
rollout.alertGuardrail breachflag_key, metric, threshold
approval.requestedAwaiting approvalflag_key, environment

17. Performance & Latency Optimization

Sub-millisecond evaluation latency is a core requirement.

Optimization Layers

LayerOptimizationImpact
In-MemoryFlags stored as flat byte array; zero-copy<0.1ms
CompressionBrotli compression; delta updates90% less bandwidth
HashingMurmurHash3 (hardware-optimized)<0.01ms per hash
BatchingMultiple evaluations in single callAmortized overhead
Edge CachingFlag snapshots at CDN edge<5ms TTFB
Persistent CacheDisk cache avoids cold start0ms startup
SSE ReconnectLast-Event-ID replayInstant recovery

Benchmark Comparison

Platformp50 Latencyp99 LatencyPropagation
LaunchDarkly<0.5ms<1ms<2 seconds
Split.io<1ms<3ms<3 seconds
Flagsmith<2ms<10ms<5 seconds
Unleash<3ms<15ms<10 seconds
Our Platform<0.5ms<1ms<2 seconds

18. Data Pipeline for Metrics & Insights

flowchart TB subgraph Sources[Data Sources] SDK_EVENTS[SDK Evaluation Events] TRACK_EVENTS[Custom Tracking Events] API_EVENTS[API Audit Events] end subgraph Ingestion[Ingestion Layer] EVENT_API[Event Ingestion API] KAFKA_COLLECT[Kafka Cluster] EVENT_SCHEMA[Schema Registry] end subgraph Processing[Stream Processing] FLINK[Apache Flink] COUNTER_SKETCH[Count-Min Sketch] HLL[HyperLogLog] end subgraph Storage[Storage] CLICKHOUSE[ClickHouse Analytics] S3_LAKE[S3 Data Lake] REDIS_AGG[Redis Live Aggregates] end subgraph Serving[Serving] METRICS_API[Metrics API] DASHBOARD[Analytics Dashboard] EXPORT[Data Export] end SDK_EVENTS --> EVENT_API TRACK_EVENTS --> EVENT_API EVENT_API --> KAFKA_COLLECT KAFKA_COLLECT --> EVENT_SCHEMA KAFKA_COLLECT --> FLINK FLINK --> COUNTER_SKETCH FLINK --> HLL FLINK --> CLICKHOUSE FLINK --> REDIS_AGG KAFKA_COLLECT --> S3_LAKE CLICKHOUSE --> METRICS_API CLICKHOUSE --> DASHBOARD S3_LAKE --> EXPORT REDIS_AGG --> METRICS_API style KAFKA_COLLECT fill:#8b5cf6,color:white style FLINK fill:#0088ff,color:white style CLICKHOUSE fill:#10b981,color:white

Data Retention Policy

Data TypeHot (Queryable)Warm (Compressed)Cold (Archive)
Evaluation Events7 days (ClickHouse)90 days (S3 Parquet)1 year (Glacier)
Custom Events30 days (ClickHouse)1 year (S3 Parquet)3 years (Glacier)
Audit Logs90 days (Elasticsearch)2 years (S3 Parquet)7 years (Glacier)
Aggregated Metrics30 days (Redis)2 years (ClickHouse)Indefinite (S3)
Experiment Results90 days (ClickHouse)Indefinite (S3)N/A

19. Database Design

PostgreSQL Schema (Core Tables)

public class DatabaseSchema
{
    // Organizations
    // CREATE TABLE organizations (
    //     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    //     name VARCHAR(255) NOT NULL,
    //     key VARCHAR(100) UNIQUE NOT NULL,
    //     plan VARCHAR(50) NOT NULL DEFAULT 'free',
    //     settings JSONB DEFAULT '{}',
    //     created_at TIMESTAMPTZ DEFAULT NOW(),
    //     version INTEGER DEFAULT 1
    // );

    // Projects
    // CREATE TABLE projects (
    //     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    //     org_id UUID REFERENCES organizations(id),
    //     name VARCHAR(255) NOT NULL,
    //     key VARCHAR(100) UNIQUE NOT NULL,
    //     created_at TIMESTAMPTZ DEFAULT NOW(),
    //     version INTEGER DEFAULT 1
    // );

    // Feature Flags
    // CREATE TABLE flags (
    //     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    //     project_id UUID REFERENCES projects(id),
    //     key VARCHAR(255) NOT NULL,
    //     name VARCHAR(255) NOT NULL,
    //     flag_type VARCHAR(20) DEFAULT 'boolean',
    //     tags TEXT[] DEFAULT '{}',
    //     archived BOOLEAN DEFAULT false,
    //     created_at TIMESTAMPTZ DEFAULT NOW(),
    //     version INTEGER DEFAULT 1,
    //     UNIQUE(project_id, key)
    // );

    // Flag States (per environment)
    // CREATE TABLE flag_states (
    //     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    //     flag_id UUID REFERENCES flags(id),
    //     environment_id UUID REFERENCES environments(id),
    //     enabled BOOLEAN DEFAULT false,
    //     default_variation VARCHAR(255),
    //     targets JSONB DEFAULT '[]',
    //     rules JSONB DEFAULT '[]',
    //     fallthrough JSONB DEFAULT '{}',
    //     version INTEGER DEFAULT 1,
    //     UNIQUE(flag_id, environment_id)
    // );

    // Variations
    // CREATE TABLE variations (
    //     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    //     flag_id UUID REFERENCES flags(id),
    //     key VARCHAR(255) NOT NULL,
    //     value TEXT NOT NULL,
    //     name VARCHAR(255),
    //     UNIQUE(flag_id, key)
    // );

    // Experiments
    // CREATE TABLE experiments (
    //     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    //     flag_id UUID REFERENCES flags(id),
    //     environment_id UUID REFERENCES environments(id),
    //     key VARCHAR(255) NOT NULL,
    //     hypothesis TEXT,
    //     traffic_percentage DECIMAL(5,2) DEFAULT 100.00,
    //     status VARCHAR(20) DEFAULT 'draft',
    //     start_time TIMESTAMPTZ,
    //     end_time TIMESTAMPTZ,
    //     version INTEGER DEFAULT 1
    // );

    // Audit Log
    // CREATE TABLE audit_logs (
    //     id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    //     timestamp TIMESTAMPTZ DEFAULT NOW(),
    //     actor_id UUID,
    //     actor_email VARCHAR(255),
    //     action VARCHAR(50) NOT NULL,
    //     resource_type VARCHAR(50) NOT NULL,
    //     resource_id UUID,
    //     environment VARCHAR(100),
    //     changes JSONB,
    //     metadata JSONB
    // );

    // Event Sourcing Table
    // CREATE TABLE flag_events (
    //     id BIGSERIAL PRIMARY KEY,
    //     flag_id UUID NOT NULL,
    //     environment_id UUID NOT NULL,
    //     event_type VARCHAR(50) NOT NULL,
    //     payload JSONB NOT NULL,
    //     version INTEGER NOT NULL,
    //     created_at TIMESTAMPTZ DEFAULT NOW(),
    //     UNIQUE(flag_id, environment_id, version)
    // );
}

Optimistic Concurrency Control

public class FlagStateService
{
    public async Task<FlagState> UpdateFlagState(
        string flagKey, string envKey, FlagStateUpdate update, int expectedVersion)
    {
        using var connection = await _connectionFactory.CreateConnection();
        await connection.OpenAsync();
        using var transaction = await connection.BeginTransactionAsync();
        try
        {
            var rowsAffected = await connection.ExecuteAsync(@"
                UPDATE flag_states
                SET enabled = @Enabled,
                    default_variation = @DefaultVariation,
                    targets = @Targets::jsonb,
                    rules = @Rules::jsonb,
                    version = version + 1,
                    updated_at = NOW()
                WHERE flag_id = (SELECT id FROM flags WHERE key = @FlagKey)
                  AND environment_id = (SELECT id FROM environments WHERE key = @EnvKey)
                  AND version = @ExpectedVersion",
                new {
                    update.Enabled, update.DefaultVariation,
                    Targets = JsonSerializer.Serialize(update.Targets),
                    Rules = JsonSerializer.Serialize(update.Rules),
                    FlagKey = flagKey, EnvKey = envKey,
                    ExpectedVersion = expectedVersion
                }, transaction);

            if (rowsAffected == 0)
            {
                throw new ConcurrencyConflictException(
                    $"Flag '{flagKey}' was modified by another user. " +
                    $"Expected version {expectedVersion}. Please refresh and retry.");
            }

            // Record event for event sourcing
            await connection.ExecuteAsync(@"
                INSERT INTO flag_events
                    (flag_id, environment_id, event_type, payload, version)
                VALUES ((SELECT id FROM flags WHERE key = @FlagKey),
                        (SELECT id FROM environments WHERE key = @EnvKey),
                        'state_updated', @Payload::jsonb, @Version)",
                new {
                    FlagKey = flagKey, EnvKey = envKey,
                    Payload = JsonSerializer.Serialize(update),
                    Version = expectedVersion + 1
                }, transaction);

            await transaction.CommitAsync();

            // Publish change to streaming layer
            await _eventBus.PublishAsync(new FlagStateChangedEvent
            {
                FlagKey = flagKey,
                EnvironmentKey = envKey,
                Version = expectedVersion + 1,
                Timestamp = DateTime.UtcNow
            });

            return await GetFlagState(flagKey, envKey);
        }
        catch
        {
            await transaction.RollbackAsync();
            throw;
        }
    }
}

20. Caching Strategy

graph TB subgraph L1[Level 1 In-Memory SDK] MEM[In-Memory Cache O1 lookup] end subgraph L2[Level 2 Server-Side] REDIS[Redis Cluster Flag snapshots TTL 5min] end subgraph L3[Level 3 Persistent SDK] DISK[Disk Cache SQLite File] end subgraph L4[Level 4 Database] PG[PostgreSQL Source of truth] end MEM -->|Miss| REDIS REDIS -->|Miss| PG REDIS -->|Write-Through| MEM PG -->|Invalidation| REDIS DISK -->|Startup| MEM MEM -->|Periodic Sync| DISK style MEM fill:#0088ff,color:white style REDIS fill:#ef4444,color:white style DISK fill:#10b981,color:white

Cache Invalidation Strategy

TriggerMethodPropagation
Flag state updatedRedis invalidation + SSE push<2 seconds
Flag created/archivedProject-level invalidation<5 seconds
SDK reconnectsFull snapshot from APIImmediate
SDK cold startDisk cache then API fallbackDisk 0ms, API <100ms
Redis failureDirect DB queryImmediate (slower)

21. Multi-Region Design

graph TB subgraph GSLB[Global Load Balancer] GLB[Cloudflare/F5] end subgraph USEAST[US-EAST Region] API_US[API Servers] PG_US[PostgreSQL Primary] REDIS_US[Redis Cluster] SSE_US[SSE Gateway] end subgraph EUWEST[EU-WEST Region] API_EU[API Servers] PG_EU[PostgreSQL Replica] REDIS_EU[Redis Cluster] SSE_EU[SSE Gateway] end subgraph APAC[APAC Region] API_AP[API Servers] PG_AP[PostgreSQL Replica] REDIS_AP[Redis Cluster] SSE_AP[SSE Gateway] end GLB --> API_US GLB --> API_EU GLB --> API_AP PG_US -->|Async Replication| PG_EU PG_US -->|Async Replication| PG_AP REDIS_US -->|CRDT Sync| REDIS_EU REDIS_US -->|CRDT Sync| REDIS_AP style GLB fill:#0088ff,color:white

Regional Data Strategy

  • Flag Configurations: Written to primary (US-EAST), async replicated to EU/APAC. Lag <500ms.
  • Evaluation Events: Collected locally, streamed to regional Kafka, aggregated regionally.
  • SDK Streaming: Each region has its own SSE gateway. SDKs connect via GeoDNS.
  • Failover: Global load balancer routes to nearest healthy region on outage.

22. Cost Estimation

ComponentSpecMonthly Cost
PostgreSQL (Primary + 3 Replicas)r6g.2xlarge, 1TB gp3$6,000
Redis Cluster (6 nodes)r6g.xlarge, 256GB$4,800
Kafka (6 brokers)m5.2xlarge, 2TB EBS$5,400
ClickHouse Cluster (3 nodes)r6g.2xlarge, 2TB$4,500
SSE/WebSocket Servers (30)c5.large$3,200
API Servers (20)c5.xlarge$4,200
Event Ingestion (15)c5.xlarge$3,150
Elasticsearch (3 nodes)r5.xlarge, 1TB$2,400
S3 Storage (500TB)Standard + Glacier$12,000
CloudFront / Cloudflare50TB/month transfer$4,000
MonitoringDatadog/Grafana$5,000
Multi-Region ReplicationEU + APAC$15,000
Total~$69,650/month

Revenue Model:

  • Free tier: 1,000 flags, 50K evaluations/month, 2 users
  • Pro ($99/month): 10,000 flags, 5M evaluations/month, 10 users
  • Enterprise ($499/month): Unlimited flags, 100M evaluations/month
  • At 1,000 customers averaging $300/month: $300K MRR = $3.6M ARR

23. Interview Q&A

Q1: How do you ensure all SDK instances see the same flag value for a given user?

Answer: We use deterministic bucketing with a composite hash of user_key + flag_key + flag_salt using MurmurHash3. Since all SDKs have the same flag configuration (pushed via SSE) and use the same hash function with the same seed, they independently compute the same result. The hash maps to a 0-9999 range for 0.01% rollout precision. This guarantees a user sees the same variation across all SDK instances without coordination.

Q2: What happens if the feature flag platform goes down?

Answer: SDKs operate in "last known good" mode. Flags are persisted to disk during initialization. If the streaming connection drops, the SDK uses the in-memory cache. If the SDK restarts and cannot reach the platform, it loads from the persistent cache. We implement graceful degradation — a flag that was on before the outage stays on. The platform being down never causes features to accidentally turn off.

Q3: How do you propagate flag changes to millions of SDK instances within 2 seconds?

Answer: We use a multi-tier streaming architecture: (1) Flag change is written to the database, (2) Published to Redis Pub/Sub, (3) Propagated to regional SSE gateways, (4) SSE gateways push to all connected SDKs. This fan-out reaches millions of SDKs within 2 seconds. On reconnect, SDKs send Last-Event-ID for replay from the server's buffer (last 1000 changes).

Q4: How does percentage-based rollout work consistently across millions of users?

Answer: We use deterministic hash-based bucketing. For each user-flag pair: MurmurHash3(user_key + flag_key + salt) % 10000. If the result is less than rollout_percentage * 100, the user gets treatment. This ensures: (1) Each user always gets the same variation, (2) Distribution matches target within +/-0.1%, (3) Changing percentage only affects boundary users.

Q5: How do you prevent a poorly designed experiment from hurting the business?

Answer: We implement guardrail metrics and automated rollback. Every experiment specifies guardrail metrics (error rate, page load time, revenue per session) with thresholds. If any guardrail breaches (e.g., error rate +0.1%), the system: (1) Disables the experiment flag, (2) Reverts all users to control, (3) Alerts the owner. We also enforce minimum duration (7+ days) and sample size.

Q6: Explain the trade-offs between client-side and server-side SDK evaluation.

Answer: Client-side evaluates in the browser/device — all targeting rules are visible to the client. Good for non-sensitive flags, uses SSE streaming, stores in localStorage. Server-side evaluates in your backend — all rules hidden. Has access to server-side attributes (IP, internal data), uses SSE + polling, caches in Redis. Key trade-off: security vs latency.

Q7: How would you handle 500 million evaluations per second at peak?

Answer: Evaluation must be entirely local. The SDK loads all configurations into an in-memory hash map. Optimizations: (1) Pre-compiled byte arrays for zero-copy deserialization, (2) Decision tree compilation at load time, (3) Hardware-optimized MurmurHash3 (AES-NI), (4) Bulk evaluation in single SDK call, (5) Each SDK instance is isolated with no shared state. 500M/s distributed across millions of instances means each handles only a few hundred per second.

Q8: How do you handle targeting rules that reference missing attributes?

Answer: When a rule references an attribute not in the user context, the rule is treated as not matching and evaluation proceeds to the next rule. The SDK reports ATTRIBUTE_MISSING in the evaluation event for analytics. For server-side SDKs, the application can enrich context before evaluation. For client-side, we recommend using only reliably available attributes.

Q9: What is the difference between a feature flag and an experiment?

Answer: A feature flag toggles features for deployment control. An experiment adds metric tracking, traffic splitting, and statistical analysis. Differences: (1) Experiments require metric events, (2) Defined duration and sample size, (3) Deterministic bucketing for consistency, (4) Statistical results (p-values, confidence intervals). A flag can exist without an experiment, but an experiment always needs a flag.

Q10: How do you handle flag dependencies (prerequisite flags)?

Answer: Prerequisite flags allow one flag to depend on another being enabled. The evaluation engine checks prerequisites first: if the prerequisite is off, the dependent flag returns a prerequisite-failed default. We validate the dependency graph is acyclic at creation time. The SDK resolves prerequisites locally during evaluation.

Q11: How would you migrate a customer from their existing config flags?

Answer: Our migration toolkit: (1) Scans codebase for existing patterns (@FeatureFlag, env vars, config files), (2) Generates mapping to our data model, (3) Creates flags via Management API, (4) Provides replacement code snippets. We support a hybrid mode where the SDK references both platform flags and legacy sources, enabling incremental migration.

Q12: How do you prevent evaluation from becoming a bottleneck in high-throughput services?

Answer: Strategies: (1) Batch evaluation — all flags in one call, (2) Lazy evaluation — only used code paths, (3) Pre-computation — evaluate at identify-time, (4) Local-only — zero I/O, zero network, (5) Async events — queued, never blocking evaluation, (6) Tiny SDK — <50KB compiled. Target: <0.1ms overhead per service.

24. Full C# Implementation

Below is a complete, production-quality C# implementation of a feature flag SDK with evaluation engine, streaming client, and experiment tracking.

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Text;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Threading;
using System.Threading.Tasks;
using System.Timers;

namespace FeatureFlagPlatform.SDK
{
    // ═══════════════════════════════════════════════════════════
    // SECTION 1: Core Models
    // ═══════════════════════════════════════════════════════════

    public enum FlagType { Boolean, String, Number, Json }

    public enum EvaluationReason
    {
        FlagDisabled, NotFound, InvalidContext, IndividualTarget,
        RuleMatch, PercentageRollout, Fallthrough, PrerequisiteFailed, Error
    }

    public class UserContext
    {
        public string Key { get; set; }
        public string Name { get; set; }
        public string Email { get; set; }
        public string Ip { get; set; }
        public string Country { get; set; }
        public Dictionary<string, object> Custom { get; set; } = new();

        public object GetAttribute(string attribute)
        {
            return attribute?.ToLowerInvariant() switch
            {
                "key" or "userkey" or "user_key" => Key,
                "name" => Name,
                "email" => Email,
                "ip" or "ipaddress" => Ip,
                "country" or "countrycode" => Country,
                _ => Custom.TryGetValue(attribute, out var val) ? val : null
            };
        }
    }

    public class Variation
    {
        public string Key { get; set; }
        public string Value { get; set; }
        public string Name { get; set; }
    }

    public class TargetingCondition
    {
        public string Attribute { get; set; }
        public string Operator { get; set; }
        public List<string> Values { get; set; } = new();
    }

    public class PercentageVariation
    {
        public string VariationKey { get; set; }
        public int Weight { get; set; }
    }

    public class TargetingRule
    {
        public int Priority { get; set; }
        public List<TargetingCondition> Conditions { get; set; } = new();
        public string VariationKey { get; set; }
        public List<PercentageVariation> PercentageRollout { get; set; }
    }

    public class PrerequisiteFlag
    {
        public string FlagKey { get; set; }
        public string VariationValue { get; set; }
    }

    public class FlagConfiguration
    {
        public string Key { get; set; }
        public string Name { get; set; }
        public FlagType Type { get; set; }
        public bool Enabled { get; set; }
        public List<Variation> Variations { get; set; } = new();
        public string DefaultVariation { get; set; }
        public string OffVariation { get; set; }
        public List<TargetingRule> Rules { get; set; } = new();
        public TargetingRule Fallthrough { get; set; }
        public List<PrerequisiteFlag> Prerequisites { get; set; } = new();
        public Dictionary<string, string> IndividualTargets { get; set; } = new();
        public string Salt { get; set; }
        public int Version { get; set; }
        public DateTime LastUpdated { get; set; }
    }

    public class EvaluationResult
    {
        public string FlagKey { get; set; }
        public string VariationKey { get; set; }
        public string VariationValue { get; set; }
        public bool Enabled { get; set; }
        public EvaluationReason Reason { get; set; }
        public double LatencyMs { get; set; }
        public DateTime Timestamp { get; set; }
    }

    public class FlagChangeEvent : EventArgs
    {
        public string FlagKey { get; set; }
        public int NewVersion { get; set; }
    }

    // ═══════════════════════════════════════════════════════════
    // SECTION 2: Hashing and Bucketing
    // ═══════════════════════════════════════════════════════════

    public static class MurmurHash3
    {
        public static uint Compute(string key, uint seed = 0)
        {
            byte[] data = Encoding.UTF8.GetBytes(key);
            uint hash = seed;
            int length = data.Length;
            int index = 0;

            while (length >= 4)
            {
                uint k = BitConverter.ToUInt32(data, index);
                k *= 0xcc9e2d51;
                k = RotateLeft(k, 15);
                k *= 0x1b873593;
                hash ^= k;
                hash = RotateLeft(hash, 13);
                hash = hash * 5 + 0xe6546b64;
                index += 4;
                length -= 4;
            }

            uint tail = 0;
            switch (length)
            {
                case 3: tail ^= (uint)data[index + 2] << 16; goto case 2;
                case 2: tail ^= (uint)data[index + 1] << 8; goto case 1;
                case 1:
                    tail ^= data[index];
                    tail *= 0xcc9e2d51;
                    tail = RotateLeft(tail, 15);
                    tail *= 0x1b873593;
                    hash ^= tail;
                    break;
            }

            hash ^= (uint)data.Length;
            hash ^= hash >> 16;
            hash *= 0x85ebca6b;
            hash ^= hash >> 13;
            hash *= 0xc2b2ae35;
            hash ^= hash >> 16;
            return hash;
        }

        private static uint RotateLeft(uint value, int count)
            => (value << count) | (value >> (32 - count));

        public static int Bucket(string userKey, string flagKey, string salt)
        {
            string composite = $"{userKey}:{flagKey}:{salt}";
            uint hash = Compute(composite);
            return (int)(hash % 10000);
        }
    }

    // ═══════════════════════════════════════════════════════════
    // SECTION 3: Flag Evaluation Engine
    // ═══════════════════════════════════════════════════════════

    public class EvaluationEngine
    {
        private readonly ConcurrentDictionary<string, FlagConfiguration> _flags = new();

        public void LoadFlags(IEnumerable<FlagConfiguration> flags)
        {
            foreach (var flag in flags)
                _flags[flag.Key] = flag;
        }

        public void UpdateFlag(FlagConfiguration flag)
            => _flags[flag.Key] = flag;

        public void RemoveFlag(string flagKey)
            => _flags.TryRemove(flagKey, out _);

        public EvaluationResult Evaluate(string flagKey, UserContext context)
        {
            var sw = System.Diagnostics.Stopwatch.StartNew();
            try
            {
                if (context == null || string.IsNullOrEmpty(context.Key))
                    return CreateResult(flagKey, EvaluationReason.InvalidContext, sw);

                if (!_flags.TryGetValue(flagKey, out var flag))
                    return CreateResult(flagKey, EvaluationReason.NotFound, sw);

                if (!flag.Enabled)
                    return CreateResult(flagKey, EvaluationReason.FlagDisabled, sw,
                        variation: flag.OffVariation);

                // Check prerequisites
                if (flag.Prerequisites?.Count > 0)
                {
                    foreach (var prereq in flag.Prerequisites)
                    {
                        if (_flags.TryGetValue(prereq.FlagKey, out var prereqFlag))
                        {
                            var prereqResult = Evaluate(prereq.FlagKey, context);
                            if (prereqResult.VariationValue != prereq.VariationValue)
                                return CreateResult(flagKey,
                                    EvaluationReason.PrerequisiteFailed, sw);
                        }
                        else
                            return CreateResult(flagKey,
                                EvaluationReason.PrerequisiteFailed, sw);
                    }
                }

                // Check individual targets
                if (flag.IndividualTargets?.TryGetValue(context.Key, out var targetVar) == true)
                    return CreateResult(flagKey, EvaluationReason.IndividualTarget, sw,
                        variation: targetVar);

                // Evaluate targeting rules
                if (flag.Rules != null)
                {
                    foreach (var rule in flag.Rules.OrderBy(r => r.Priority))
                    {
                        if (EvaluateConditions(rule.Conditions, context))
                        {
                            if (rule.PercentageRollout?.Count > 0)
                            {
                                var bucket = MurmurHash3.Bucket(
                                    context.Key, flagKey, flag.Salt);
                                int cumulative = 0;
                                foreach (var pv in rule.PercentageRollout)
                                {
                                    cumulative += pv.Weight;
                                    if (bucket < cumulative)
                                        return CreateResult(flagKey,
                                            EvaluationReason.RuleMatch, sw,
                                            variation: pv.VariationKey);
                                }
                                return CreateResult(flagKey, EvaluationReason.RuleMatch,
                                    sw, variation: rule.PercentageRollout.Last().VariationKey);
                            }
                            return CreateResult(flagKey, EvaluationReason.RuleMatch, sw,
                                variation: rule.VariationKey);
                        }
                    }
                }

                // Fallthrough
                if (flag.Fallthrough != null)
                {
                    if (flag.Fallthrough.PercentageRollout?.Count > 0)
                    {
                        var bucket = MurmurHash3.Bucket(context.Key, flagKey, flag.Salt);
                        int cumulative = 0;
                        foreach (var pv in flag.Fallthrough.PercentageRollout)
                        {
                            cumulative += pv.Weight;
                            if (bucket < cumulative)
                                return CreateResult(flagKey,
                                    EvaluationReason.Fallthrough, sw,
                                    variation: pv.VariationKey);
                        }
                        return CreateResult(flagKey, EvaluationReason.Fallthrough, sw,
                            variation: flag.Fallthrough.PercentageRollout.Last().VariationKey);
                    }
                    return CreateResult(flagKey, EvaluationReason.Fallthrough, sw,
                        variation: flag.Fallthrough.VariationKey);
                }

                return CreateResult(flagKey, EvaluationReason.Fallthrough, sw,
                    variation: flag.DefaultVariation);
            }
            catch (Exception)
            {
                sw.Stop();
                return new EvaluationResult
                {
                    FlagKey = flagKey, VariationKey = "error",
                    VariationValue = "error", Reason = EvaluationReason.Error,
                    LatencyMs = sw.Elapsed.TotalMilliseconds,
                    Timestamp = DateTime.UtcNow
                };
            }
        }

        private bool EvaluateConditions(
            List<TargetingCondition> conditions, UserContext context)
        {
            if (conditions == null || conditions.Count == 0) return true;
            foreach (var condition in conditions)
            {
                var attrValue = context.GetAttribute(condition.Attribute);
                if (!EvaluateCondition(condition, attrValue)) return false;
            }
            return true;
        }

        private bool EvaluateCondition(TargetingCondition condition, object attrValue)
        {
            var strValue = attrValue?.ToString() ?? "";
            var opValues = condition.Values ?? new List<string>();
            return condition.Operator?.ToLowerInvariant() switch
            {
                "in" => opValues.Any(v =>
                    string.Equals(v, strValue, StringComparison.OrdinalIgnoreCase)),
                "notin" or "not_in" => !opValues.Any(v =>
                    string.Equals(v, strValue, StringComparison.OrdinalIgnoreCase)),
                "startswith" or "starts_with" => opValues.Any(v =>
                    strValue.StartsWith(v, StringComparison.OrdinalIgnoreCase)),
                "endswith" or "ends_with" => opValues.Any(v =>
                    strValue.EndsWith(v, StringComparison.OrdinalIgnoreCase)),
                "contains" => opValues.Any(v =>
                    strValue.Contains(v, StringComparison.OrdinalIgnoreCase)),
                "greaterthan" or "greater_than" =>
                    double.TryParse(strValue, out var num) &&
                    double.TryParse(opValues.FirstOrDefault(), out var thr) && num > thr,
                "lessthan" or "less_than" =>
                    double.TryParse(strValue, out var num) &&
                    double.TryParse(opValues.FirstOrDefault(), out var thr) && num < thr,
                "exists" => attrValue != null,
                "notexists" or "not_exists" => attrValue == null,
                _ => false
            };
        }

        private EvaluationResult CreateResult(
            string flagKey, EvaluationReason reason,
            System.Diagnostics.Stopwatch sw, string variation = null)
        {
            sw.Stop();
            string varKey = variation ?? "off";
            string varValue = varKey;
            if (_flags.TryGetValue(flagKey, out var flag))
            {
                var match = flag.Variations?.FirstOrDefault(v => v.Key == varKey);
                if (match != null) varValue = match.Value;
            }
            return new EvaluationResult
            {
                FlagKey = flagKey, VariationKey = varKey,
                VariationValue = varValue, Enabled = flag?.Enabled ?? false,
                Reason = reason, LatencyMs = sw.Elapsed.TotalMilliseconds,
                Timestamp = DateTime.UtcNow
            };
        }
    }
}
    // ═══════════════════════════════════════════════════════════
    // SECTION 4: SSE Streaming Client
    // ═══════════════════════════════════════════════════════════

    public class SseStreamingClient : IDisposable
    {
        private readonly HttpClient _httpClient;
        private readonly string _streamUrl;
        private readonly string _sdkKey;
        private CancellationTokenSource _cts;
        private Task _streamTask;
        private int _reconnectDelay = 1000;
        private const int MaxReconnectDelay = 30000;
        private string _lastEventId;

        public event EventHandler<FlagConfiguration> FlagChanged;
        public event EventHandler<FlagConfiguration> FlagDeleted;
        public event EventHandler Connected;
        public event EventHandler<Exception> Disconnected;

        public bool IsConnected { get; private set; }

        public SseStreamingClient(string baseUrl, string sdkKey)
        {
            _httpClient = new HttpClient { Timeout = TimeSpan.FromMilliseconds(-1) };
            _streamUrl = $"{baseUrl}/v1/stream";
            _sdkKey = sdkKey;
        }

        public Task StartAsync(CancellationToken cancellationToken = default)
        {
            _cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
            _streamTask = StreamLoopAsync(_cts.Token);
            return Task.CompletedTask;
        }

        private async Task StreamLoopAsync(CancellationToken ct)
        {
            while (!ct.IsCancellationRequested)
            {
                try
                {
                    var request = new HttpRequestMessage(HttpMethod.Get, _streamUrl);
                    request.Headers.Add("Authorization", _sdkKey);
                    request.Headers.Add("Accept", "text/event-stream");
                    request.Headers.Add("Cache-Control", "no-cache");
                    if (!string.IsNullOrEmpty(_lastEventId))
                        request.Headers.Add("Last-Event-ID", _lastEventId);

                    var response = await _httpClient.SendAsync(
                        request, HttpCompletionOption.ResponseHeadersRead, ct);
                    response.EnsureSuccessStatusCode();

                    IsConnected = true;
                    _reconnectDelay = 1000;
                    Connected?.Invoke(this, EventArgs.Empty);

                    using var stream = await response.Content.ReadAsStreamAsync(ct);
                    using var reader = new StreamReader(stream);

                    string eventType = null, eventData = null, eventId = null;

                    while (!ct.IsCancellationRequested && !reader.EndOfStream)
                    {
                        var line = await reader.ReadLineAsync();
                        if (string.IsNullOrEmpty(line))
                        {
                            if (!string.IsNullOrEmpty(eventData))
                            {
                                ProcessEvent(eventType, eventData, eventId);
                                eventId = null;
                                eventType = null;
                                eventData = null;
                            }
                        }
                        else if (line.StartsWith("event:"))
                            eventType = line.Substring(6).Trim();
                        else if (line.StartsWith("data:"))
                            eventData = line.Substring(5).Trim();
                        else if (line.StartsWith("id:"))
                            eventId = line.Substring(3).Trim();
                    }
                }
                catch (Exception ex) when (!ct.IsCancellationRequested)
                {
                    IsConnected = false;
                    Disconnected?.Invoke(this, ex);
                    await Task.Delay(_reconnectDelay, ct);
                    _reconnectDelay = Math.Min(_reconnectDelay * 2, MaxReconnectDelay);
                    _reconnectDelay += Random.Shared.Next(0, _reconnectDelay / 2);
                }
            }
        }

        private void ProcessEvent(string eventType, string data, string eventId)
        {
            if (!string.IsNullOrEmpty(eventId)) _lastEventId = eventId;
            switch (eventType)
            {
                case "patch":
                    var patch = JsonSerializer.Deserialize<FlagPatch>(data);
                    if (patch != null)
                        FlagChanged?.Invoke(this, new FlagConfiguration
                        {
                            Key = patch.FlagKey,
                            Enabled = patch.Data?.Enabled ?? false,
                            Version = patch.Version,
                            LastUpdated = DateTime.UtcNow
                        });
                    break;
                case "delete":
                    var del = JsonSerializer.Deserialize<FlagDelete>(data);
                    if (del?.FlagKey != null)
                        FlagDeleted?.Invoke(this,
                            new FlagConfiguration { Key = del.FlagKey });
                    break;
            }
        }

        public void Dispose()
        {
            _cts?.Cancel();
            _streamTask?.Wait(TimeSpan.FromSeconds(5));
            _httpClient?.Dispose();
        }

        private class FlagPatch
        {
            [JsonPropertyName("flag")] public string FlagKey { get; set; }
            [JsonPropertyName("version")] public int Version { get; set; }
            [JsonPropertyName("data")] public PatchData Data { get; set; }
        }
        private class PatchData
        {
            [JsonPropertyName("enabled")] public bool? Enabled { get; set; }
            [JsonPropertyName("variation")] public string Variation { get; set; }
        }
        private class FlagDelete
        {
            [JsonPropertyName("flag")] public string FlagKey { get; set; }
        }
    }

    // ═══════════════════════════════════════════════════════════
    // SECTION 5: Event Tracker for experimentation
    // ═══════════════════════════════════════════════════════════

    public class EventTracker : IDisposable
    {
        private readonly ConcurrentQueue<TrackingEvent> _eventQueue = new();
        private readonly System.Timers.Timer _flushTimer;
        private readonly HttpClient _httpClient;
        private readonly string _ingestUrl;
        private readonly string _sdkKey;

        public EventTracker(string baseUrl, string sdkKey,
            TimeSpan? flushInterval = null)
        {
            _httpClient = new HttpClient();
            _ingestUrl = $"{baseUrl}/v1/events";
            _sdkKey = sdkKey;
            _flushTimer = new System.Timers.Timer(
                (flushInterval ?? TimeSpan.FromSeconds(30)).TotalMilliseconds);
            _flushTimer.Elapsed += async (s, e) => await FlushEventsAsync();
            _flushTimer.AutoReset = true;
            _flushTimer.Start();
        }

        public void Track(string eventName, UserContext context,
            Dictionary<string, object> data = null)
        {
            _eventQueue.Enqueue(new TrackingEvent
            {
                EventType = eventName, UserKey = context.Key,
                Timestamp = DateTime.UtcNow,
                Data = data ?? new Dictionary<string, object>()
            });
        }

        public void TrackEvaluation(EvaluationResult result, UserContext context)
        {
            _eventQueue.Enqueue(new TrackingEvent
            {
                EventType = "$flag_evaluation", UserKey = context.Key,
                Timestamp = result.Timestamp,
                Data = new Dictionary<string, object>
                {
                    ["flag"] = result.FlagKey,
                    ["variation"] = result.VariationKey,
                    ["enabled"] = result.Enabled,
                    ["reason"] = result.Reason.ToString(),
                    ["latencyMs"] = result.LatencyMs
                }
            });
        }

        public async Task FlushEventsAsync()
        {
            var events = new List<TrackingEvent>();
            while (_eventQueue.TryDequeue(out var evt) && events.Count < 500)
                events.Add(evt);
            if (events.Count == 0) return;
            try
            {
                var payload = JsonSerializer.Serialize(new { events });
                var content = new StringContent(payload,
                    Encoding.UTF8, "application/json");
                content.Headers.Add("Authorization", _sdkKey);
                await _httpClient.PostAsync(_ingestUrl, content);
            }
            catch
            {
                foreach (var evt in events)
                    _eventQueue.Enqueue(evt);
            }
        }

        public void Dispose()
        {
            _flushTimer?.Stop();
            _flushTimer?.Dispose();
            FlushEventsAsync().GetAwaiter().GetResult();
        }

        private class TrackingEvent
        {
            public string EventType { get; set; }
            public string UserKey { get; set; }
            public DateTime Timestamp { get; set; }
            public Dictionary<string, object> Data { get; set; }
        }
    }

    // ═══════════════════════════════════════════════════════════
    // SECTION 6: Persistent Cache
    // ═══════════════════════════════════════════════════════════

    public class PersistentFlagCache
    {
        private readonly string _cacheDirectory;

        public PersistentFlagCache(string cacheDirectory)
        {
            _cacheDirectory = cacheDirectory;
            Directory.CreateDirectory(_cacheDirectory);
        }

        public async Task SaveFlagsAsync(string environmentKey,
            Dictionary<string, FlagConfiguration> flags)
        {
            var filePath = GetFilePath(environmentKey);
            var data = JsonSerializer.Serialize(flags,
                new JsonSerializerOptions { WriteIndented = false });
            await File.WriteAllTextAsync(filePath, data);
        }

        public async Task<Dictionary<string, FlagConfiguration>>
            LoadFlagsAsync(string environmentKey)
        {
            var filePath = GetFilePath(environmentKey);
            if (!File.Exists(filePath))
                return new Dictionary<string, FlagConfiguration>();
            var data = await File.ReadAllTextAsync(filePath);
            return JsonSerializer.Deserialize<
                Dictionary<string, FlagConfiguration>>(data)
                ?? new Dictionary<string, FlagConfiguration>();
        }

        private string GetFilePath(string environmentKey)
        {
            var safeName = Convert.ToBase64String(
                Encoding.UTF8.GetBytes(environmentKey)).Replace('/', '_');
            return Path.Combine(_cacheDirectory, $"flags_{safeName}.json");
        }
    }

    // ═══════════════════════════════════════════════════════════
    // SECTION 7: Experiment Analyzer
    // ═══════════════════════════════════════════════════════════

    public class ExperimentAnalyzer
    {
        public ExperimentResult Analyze(
            List<VariantMetrics> control,
            List<VariantMetrics> treatment,
            double significanceLevel = 0.05)
        {
            var controlMean = control.Average(m => m.Value);
            var treatmentMean = treatment.Average(m => m.Value);
            var controlVar = CalculateVariance(control, controlMean);
            var treatmentVar = CalculateVariance(treatment, treatmentMean);
            var n1 = control.Count;
            var n2 = treatment.Count;
            var se = Math.Sqrt(controlVar / n1 + treatmentVar / n2);
            if (se == 0) return new ExperimentResult { IsSignificant = false };

            var tStat = (treatmentMean - controlMean) / se;
            var df = Math.Pow(controlVar / n1 + treatmentVar / n2, 2) /
                (Math.Pow(controlVar / n1, 2) / (n1 - 1) +
                 Math.Pow(treatmentVar / n2, 2) / (n2 - 1));
            var pValue = CalculatePValue(Math.Abs(tStat), (int)df);
            var lift = controlMean != 0
                ? ((treatmentMean - controlMean) / controlMean) * 100 : 0;

            return new ExperimentResult
            {
                ControlMean = controlMean, TreatmentMean = treatmentMean,
                Lift = lift, TStatistic = tStat, PValue = pValue,
                DegreesOfFreedom = (int)df,
                IsSignificant = pValue < significanceLevel,
                ConfidenceLevel = (1 - pValue) * 100,
                SampleSizeControl = n1, SampleSizeTreatment = n2
            };
        }

        private double CalculateVariance(
            List<VariantMetrics> metrics, double mean)
            => metrics.Sum(m => Math.Pow(m.Value - mean, 2)) / (metrics.Count - 1);

        private double CalculatePValue(double tStat, int df)
        {
            var x = df / (df + tStat * tStat);
            return IncompleteBeta(df / 2.0, 0.5, x);
        }

        private double IncompleteBeta(double a, double b, double x)
        {
            if (x == 0) return 0;
            if (x == 1) return 1;
            var bt = Math.Exp(LogGamma(a + b) - LogGamma(a) - LogGamma(b) +
                a * Math.Log(x) + b * Math.Log(1 - x));
            if (x < (a + 1) / (a + b + 2))
                return bt * BetaCf(a, b, x) / a;
            return 1 - bt * BetaCf(b, a, 1 - x) / b;
        }

        private double BetaCf(double a, double b, double x)
        {
            var qab = a + b; var qap = a + 1; var qam = a - 1;
            var c = 1.0; var d = 1.0 - qab * x / qap;
            if (Math.Abs(d) < 1e-30) d = 1e-30;
            d = 1.0 / d; var h = d;
            for (int m = 1; m <= 100; m++)
            {
                var m2 = 2 * m;
                var aa = m * (b - m) * x / ((qam + m2) * (a + m2));
                d = 1.0 + aa * d;
                if (Math.Abs(d) < 1e-30) d = 1e-30;
                c = 1.0 + aa / c;
                if (Math.Abs(c) < 1e-30) c = 1e-30;
                d = 1.0 / d; h *= d * c;
                aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2));
                d = 1.0 + aa * d;
                if (Math.Abs(d) < 1e-30) d = 1e-30;
                c = 1.0 + aa / c;
                if (Math.Abs(c) < 1e-30) c = 1e-30;
                d = 1.0 / d;
                var del = d * c;
                h *= del;
                if (Math.Abs(del - 1.0) < 3e-7) break;
            }
            return h;
        }

        private double LogGamma(double x)
        {
            double[] c = { 76.18009172947146, -86.50532032941677,
                24.01409824083091, -1.231739572450155,
                0.001208650973866179, -0.000005395239384953 };
            double y = x, tmp = x + 5.5;
            tmp -= (x + 0.5) * Math.Log(tmp);
            double sum = 1.000000000190015;
            for (int j = 0; j < 6; j++)
                sum += c[j] / ++y;
            return -tmp + Math.Log(2.5066282746310005 * sum / x);
        }
    }

    public class VariantMetrics
    {
        public double Value { get; set; }
        public string VariationKey { get; set; }
        public DateTime Timestamp { get; set; }
    }

    public class ExperimentResult
    {
        public double ControlMean { get; set; }
        public double TreatmentMean { get; set; }
        public double Lift { get; set; }
        public double TStatistic { get; set; }
        public double PValue { get; set; }
        public int DegreesOfFreedom { get; set; }
        public bool IsSignificant { get; set; }
        public double ConfidenceLevel { get; set; }
        public int SampleSizeControl { get; set; }
        public int SampleSizeTreatment { get; set; }
    }

    // ═══════════════════════════════════════════════════════════
    // SECTION 8: Feature Flag Client (Main Entry Point)
    // ═══════════════════════════════════════════════════════════

    public class FeatureFlagClient : IDisposable
    {
        private readonly EvaluationEngine _engine;
        private readonly SseStreamingClient _streaming;
        private readonly EventTracker _tracker;
        private readonly PersistentFlagCache _cache;
        private UserContext _currentContext;
        private bool _initialized;

        public event EventHandler<FlagChangeEvent> FlagChanged;
        public bool IsInitialized => _initialized;

        public FeatureFlagClient(string sdkKey, string baseUrl,
            string cacheDir = null)
        {
            _engine = new EvaluationEngine();
            _streaming = new SseStreamingClient(baseUrl, sdkKey);
            _tracker = new EventTracker(baseUrl, sdkKey);
            _cache = new PersistentFlagCache(cacheDir ?? Path.Combine(
                Path.GetTempPath(), "ff-cache"));

            _streaming.FlagChanged += (s, flag) =>
            {
                _engine.UpdateFlag(flag);
                FlagChanged?.Invoke(this, new FlagChangeEvent
                {
                    FlagKey = flag.Key, NewVersion = flag.Version
                });
            };
        }

        public async Task InitializeAsync(string environmentKey,
            UserContext context = null)
        {
            _currentContext = context;

            // Load from persistent cache
            var cached = await _cache.LoadFlagsAsync(environmentKey);
            if (cached.Count > 0)
                _engine.LoadFlags(cached.Values);

            // Start streaming
            _ = _streaming.StartAsync();

            // Wait for initial connection
            await _streaming.WaitForInitialization(TimeSpan.FromSeconds(10));
            _initialized = true;

            // Save to cache
            var allFlags = _engine.GetAllFlags()
                .ToDictionary(f => f.Key, f => f);
            await _cache.SaveFlagsAsync(environmentKey, allFlags);
        }

        public bool BoolVariation(string flagKey, bool defaultValue,
            UserContext context = null)
        {
            var ctx = context ?? _currentContext;
            var result = _engine.Evaluate(flagKey, ctx);
            _tracker.TrackEvaluation(result, ctx);
            if (bool.TryParse(result.VariationValue, out var val))
                return val;
            return defaultValue;
        }

        public string StringVariation(string flagKey, string defaultValue,
            UserContext context = null)
        {
            var ctx = context ?? _currentContext;
            var result = _engine.Evaluate(flagKey, ctx);
            _tracker.TrackEvaluation(result, ctx);
            return result.VariationValue ?? defaultValue;
        }

        public int IntVariation(string flagKey, int defaultValue,
            UserContext context = null)
        {
            var ctx = context ?? _currentContext;
            var result = _engine.Evaluate(flagKey, ctx);
            _tracker.TrackEvaluation(result, ctx);
            if (int.TryParse(result.VariationValue, out var val))
                return val;
            return defaultValue;
        }

        public T JsonVariation<T>(string flagKey, T defaultValue,
            UserContext context = null)
        {
            var ctx = context ?? _currentContext;
            var result = _engine.Evaluate(flagKey, ctx);
            _tracker.TrackEvaluation(result, ctx);
            try
            {
                return JsonSerializer.Deserialize<T>(result.VariationValue);
            }
            catch { return defaultValue; }
        }

        public void Track(string eventName,
            Dictionary<string, object> data = null,
            UserContext context = null)
        {
            _tracker.Track(eventName, context ?? _currentContext, data);
        }

        public void Identify(UserContext context)
            => _currentContext = context;

        public EvaluationResult Evaluate(string flagKey,
            UserContext context = null)
        {
            var ctx = context ?? _currentContext;
            var result = _engine.Evaluate(flagKey, ctx);
            _tracker.TrackEvaluation(result, ctx);
            return result;
        }

        public void Dispose()
        {
            _streaming?.Dispose();
            _tracker?.Dispose();
        }
    }
}

Deep Dive: Why Deterministic Bucketing Matters for Experimentation

Deterministic bucketing is the single most important technical requirement for a valid A/B testing platform. Without it, the fundamental assumptions of statistical hypothesis testing break down. When a user enters an experiment, they must be consistently assigned to the same variation for the entire duration of the experiment. If a user could flip between control and treatment, the measured effect would be diluted and the experiment would never reach statistical significance, wasting weeks of engineering time and traffic.

The bucketing algorithm we described earlier uses a cryptographic-strength hash function (MurmurHash3) that produces a uniformly distributed value from 0 to 9999. This 0.01% granularity means the platform can precisely control what percentage of users sees each variation. When the product manager asks to roll out a new checkout flow to exactly 25% of US-based premium users, the system can target that precisely without any drift or rounding errors that accumulate over millions of users.

A critical subtlety is the use of a per-flag salt in the hash computation. Without a salt, the same user would always fall into the same percentage bucket across all flags, creating unwanted correlations between experiments. By using a unique salt per flag, each experiment creates an independent random assignment, ensuring that statistical independence assumptions hold between concurrent experiments.

Deep Dive: Graceful Degradation and Offline Mode

The availability requirements for a feature flag platform are unusually strict because the platform sits on the critical path of every feature in the customer's application. If the flag platform goes down and features start behaving unexpectedly, the customer's entire product is affected. This is fundamentally different from most third-party services where downtime means a lost feature, not a broken application.

The multi-layered degradation strategy ensures this never happens. During normal operation, the SDK receives flag updates via SSE with sub-2-second latency. If the SSE connection drops, the SDK continues operating with the last known flag state from its in-memory cache. If the application process restarts and the platform is unreachable, the SDK loads from its persistent disk cache, which contains the last successfully fetched flag state. This means even a complete platform outage lasting hours would not cause any features to change state in the customer's application.

This design principle is called "last known good" and it is the cornerstone of feature flag platform reliability. The only time a flag value changes is when the SDK receives an explicit update from the platform. The platform can never accidentally set a flag to a default state during an outage because the SDK simply stops accepting updates when it cannot reach the server. This is a crucial safety guarantee that distinguishes enterprise-grade platforms from homegrown feature flag solutions.

Deep Dive: Event Sourcing for Flag State Changes

Event sourcing is a powerful pattern for feature flag platforms because it provides a complete audit trail, enables point-in-time recovery, and makes the flag state derivable from first principles. Instead of storing only the current state of a flag, we store every state transition as an immutable event in an append-only log.

When a developer changes a flag from "off" to "on with 10% rollout to US users," that change is recorded as a flag event with the complete before/after state. This event is then replayed through the SSE streaming layer to update all connected SDKs. The event also flows into the audit log, the analytics pipeline, and any connected integrations (Jira, Slack, Datadog). Because events are immutable and ordered, we can reconstruct the state of any flag at any point in time by replaying events up to that timestamp.

This is particularly valuable for debugging production incidents. When a customer reports that a feature started misbehaving at 3:42 PM, the engineering team can query the flag event log to see exactly what changed around that time, which developer made the change, and what the previous state was. Combined with the rollback feature, this enables rapid incident response with full traceability.

Deep Dive: Statistical Power and Experiment Duration

One of the most common mistakes in A/B testing is stopping an experiment too early because the results "look significant." This is known as peeking, and it dramatically increases the false positive rate. A result that appears significant at 1,000 users may completely disappear at 10,000 users as the variance decreases and the true effect size becomes apparent.

The platform addresses this in two ways. For Frequentist experiments, it enforces a fixed-horizon stopping rule: the experiment must run until the pre-calculated minimum sample size is reached before results can be declared significant. For Bayesian experiments, the platform uses expected loss as the decision criterion rather than p-values, which naturally handles the peeking problem and allows continuous monitoring.

Both approaches also require capturing at least one full weekly cycle (7 days) to account for day-of-week effects. An experiment that runs only on weekdays might show a 15% lift simply because the treatment group had more weekday traffic. Running for a minimum of 14 days ensures that both control and treatment groups experience the same distribution of weekly traffic patterns, making the comparison valid.

25. Conclusion

Building a feature flag and experimentation platform at scale is a complex engineering challenge that touches virtually every aspect of distributed systems design. From deterministic bucketing algorithms that ensure consistent user experiences across millions of SDK instances, to real-time streaming architectures that propagate flag changes globally within 2 seconds, to statistical analysis engines that power data-driven product decisions, the platform requires careful consideration of performance, reliability, accuracy, and security.

The key architectural decisions that define a world-class feature flag platform are:

  • SDK-side evaluation eliminates network latency from the critical path, enabling sub-millisecond evaluations that never become a bottleneck
  • Control plane / data plane separation ensures SDK evaluations continue even during platform outages, maintaining the highest availability standards
  • Push-based streaming with replay provides near-instant flag propagation while ensuring no updates are lost during network interruptions
  • Deterministic hash-based bucketing guarantees consistent experiment assignment across all SDK instances without coordination
  • Guardrail metrics with automated rollback protects businesses from the impact of poorly performing features during progressive rollouts
  • Multi-tier caching (in-memory, Redis, persistent disk) balances performance with freshness across all failure modes

The financial opportunity is substantial: with operating costs around $70K/month and a revenue model that can generate $3.6M+ ARR from 1,000 customers, the unit economics are favorable for a feature flag platform that competes with LaunchDarkly's $1.3B+ market position.

Key Takeaways for System Design Interviews:
  • Always start with the distinction between control plane and data plane
  • Emphasize SDK-side evaluation for latency-critical paths
  • Explain deterministic bucketing for experiment consistency
  • Discuss graceful degradation and offline mode for availability
  • Cover the full lifecycle: creation, targeting, experimentation, progressive rollout, and cleanup
  • Address the data pipeline challenge: trillions of events require streaming aggregation, not raw storage
  • Mention multi-region design for global availability and data residency compliance

Whether you are preparing for a senior+ system design interview at a FAANG company or building a feature flag platform for your organization, the patterns and principles covered in this guide provide a comprehensive foundation. The feature flag and experimentation space continues to evolve with advances in streaming infrastructure, machine learning for automated experiment analysis, and edge computing for even lower latency evaluations. Master these fundamentals, and you will be well-prepared to design and build platforms that serve trillions of evaluations while enabling teams to ship features with confidence.

© 2026 Ayodhyya. All rights reserved.

Feature Flag & Experimentation Platform System Design Guide

ayodhyya.com