How to Design Feature Flag & Experimentation Platform like LaunchDarkly
Building feature flags, A/B testing, and progressive rollouts for engineering teams at scale
Table of Contents
- Introduction - Why Feature Flags Matter
- Functional & Non-Functional Requirements
- Capacity Estimation
- Data Model
- API Design
- High-Level Architecture
- Flag Evaluation Engine
- SDK Architecture
- Real-Time Flag Streaming
- A/B Testing & Experimentation Framework
- Statistical Analysis
- Progressive Rollout & Canary Deployments
- Flag Lifecycle Management
- Audit Logging & Change Tracking
- Multi-Variate & Complex Targeting
- Integration Ecosystem
- Performance & Latency Optimization
- Data Pipeline for Metrics & Insights
- Database Design
- Caching Strategy
- Multi-Region Design
- Cost Estimation
- Interview Q&A
- Full C# Implementation
- Conclusion
1. Introduction - Why Feature Flags Matter
Feature flags (also called feature toggles, feature switches, or feature gates) are one of the most powerful techniques in modern software engineering. They allow teams to decouple code deployment from feature release, enabling safer, faster, and more controlled rollouts. When combined with experimentation and A/B testing, feature flags become the backbone of a data-driven product development culture.
LaunchDarkly, founded in 2014, pioneered the commercial feature management space and now serves over 2,500 customers including IBM, Atlassian, NBC, and Microsoft. The platform processes an astounding 10+ trillion flag evaluations per day, with sub-millisecond latency for SDK evaluations. Split.io (acquired by Harness) built a feature delivery platform tightly integrated with experimentation, while Flagsmith and Unleash offer open-source alternatives.
The global feature flag management market was valued at $1.3 billion in 2025 and is projected to reach $4.8 billion by 2030, growing at a CAGR of 29.5%. This explosive growth is driven by the adoption of DevOps practices, continuous delivery, and the need for data-driven feature releases.
In this comprehensive system design guide, we will architect a feature flag and experimentation platform from scratch, covering flag management, real-time streaming, SDK design, A/B testing, statistical analysis, progressive rollouts, and the data pipeline. This guide is aimed at senior+ engineers preparing for system design interviews or building similar platforms in production.
- Scale: Trillions of evaluations per day with sub-millisecond latency
- Consistency: All SDK instances must see the same flag state within seconds of a change
- Accuracy: Experimentation requires deterministic bucketing and statistical rigor
- Availability: SDKs must continue working even if the platform is down
- Security: Customer data never leaves their infrastructure; HIPAA/SOC2 compliance
2. Functional & Non-Functional Requirements
Functional Requirements
| # | Requirement | Description |
|---|---|---|
| FR1 | Flag CRUD Operations | Create, read, update, delete feature flags with metadata, tags, and descriptions |
| FR2 | Targeting Rules | Define rules to target users by attributes (country, plan, age, custom attributes) |
| FR3 | Percentage Rollouts | Gradually roll out features to a percentage of users with consistent bucketing |
| FR4 | Multi-Variate Flags | Support boolean, string, number, JSON variants with multiple values |
| FR5 | A/B Testing | Create experiments with control/treatment groups and track metrics |
| FR6 | Progressive Delivery | Canary deployments, ring-based rollouts, and scheduled releases |
| FR7 | Audit Logging | Track all flag changes with who, when, what, and diff |
| FR8 | SDK Distribution | Client-side, server-side, and edge SDKs for multiple platforms |
| FR9 | Real-Time Updates | Push flag changes to SDKs within 2 seconds via SSE/WebSocket |
| FR10 | Integrations | Jira, GitHub, Slack, Datadog, PagerDuty integrations |
| FR11 | Flag Scheduling | Schedule flag changes for future dates and time windows |
| FR12 | Approval Workflows | Require approvals before flag changes in production environments |
Non-Functional Requirements
| # | Requirement | Target |
|---|---|---|
| NFR1 | Latency (SDK evaluation) | <1ms at p99, <0.5ms at p50 |
| NFR2 | Availability | 99.99% for SDK evaluation path; 99.9% for management API |
| NFR3 | Consistency | Flag propagation to all SDKs within 2 seconds globally |
| NFR4 | Scalability | Support 10+ trillion evaluations/day, 1M+ concurrent SDK connections |
| NFR5 | Durability | No flag state lost on platform failure; SDKs cache last-known-good state |
| NFR6 | Security | TLS everywhere, SDK keys scoped by environment, SOC2 Type II compliant |
| NFR7 | Offline Mode | SDKs function with cached flag state when platform is unreachable |
| NFR8 | Multi-Tenancy | Complete isolation between customer organizations and projects |
3. Capacity Estimation
Let us estimate the system scale based on industry benchmarks and LaunchDarkly published metrics:
Flag & Customer Scale
| Entity | Count | Calculation |
|---|---|---|
| Organizations | 5,000 | Enterprise + SMB customers |
| Projects per org | 10 avg | 5,000 x 10 = 50,000 projects |
| Environments per project | 4 avg | Dev, Staging, Beta, Production |
| Flags per project | 200 avg | 50,000 x 200 = 10M flags |
| Active flags | 5M | ~50% of all flags are active |
| SDK instances (server) | 500K | Microservices across all customers |
| SDK instances (client) | 50M | Mobile + web browser clients |
Evaluation Volume
Calculations:
- Total evaluations per day: 12 trillion (12 x 10^12)
- Evaluations per second: ~140M (peak: 500M/s)
- Average flags evaluated per context: 50 (batch evaluation)
- Unique evaluation requests per second: ~2.8M
- Data per evaluation response: ~200 bytes
- Bandwidth for evaluation responses: ~560 MB/s (peak: 2 GB/s)
Storage Estimation
| Data Type | Size per Record | Annual Volume | Total Storage |
|---|---|---|---|
| Flag configurations | 5 KB | 10M flags (snapshot) | ~50 GB |
| Evaluation events | 200 bytes | 12T/day x 365 | ~876 TB/year |
| Audit logs | 1 KB | 10M changes/day | ~3.6 TB/year |
| Experiment metrics | 500 bytes | 1B events/day | ~182 TB/year |
| User contexts | 500 bytes | 100M users | ~50 GB |
4. Data Model
The data model must support multi-tenancy, hierarchical organization, and complex targeting rules:
Flag States per Environment
| Field | Type | Description |
|---|---|---|
flag_id | UUID | Reference to the flag definition |
environment_id | UUID | Reference to the environment |
enabled | Boolean | Whether the flag is enabled in this environment |
default_variation | String | Variation returned when no rules match |
targets | JSON | Individual user targeting (on/off variations) |
rules | JSON Array | Ordered targeting rules with conditions |
fallthrough | JSON | Default rule when no targeting matches |
prerequisites | JSON Array | Prerequisite flags that must be on |
scheduled_changes | JSON Array | Pending future state changes |
5. API Design
Management API (Dashboard & Admin)
POST /v1/orgs // Create organization
GET /v1/orgs/{orgId} // Get organization details
PATCH /v1/orgs/{orgId} // Update organization
POST /v1/orgs/{orgId}/projects // Create project
GET /v1/orgs/{orgId}/projects // List projects
POST /v1/projects/{projectKey}/envs // Create environment
GET /v1/projects/{projectKey}/envs // List environments
POST /v1/envs/{envKey}/clone // Clone environment
POST /v1/projects/{projectKey}/flags // Create flag
GET /v1/projects/{projectKey}/flags // List flags (paginated)
GET /v1/flags/{flagKey} // Get flag with all env states
PATCH /v1/flags/{flagKey} // Update flag metadata
DELETE /v1/flags/{flagKey} // Archive flag
GET /v1/flags/{flagKey}/envs/{envKey} // Get flag state
PATCH /v1/flags/{flagKey}/envs/{envKey} // Update flag state
POST /v1/flags/{flagKey}/envs/{envKey}/rules // Add targeting rule
DELETE /v1/flags/{flagKey}/envs/{envKey}/rules/{id} // Remove rule
POST /v1/flags/{flagKey}/envs/{envKey}/experiments // Create experiment
GET /v1/experiments/{experimentId}/results // Get experiment results
GET /v1/audit-log?projectKey={pk}&since={date} // Get audit entries
Client-Side SDK API
public interface IFeatureFlagClientSDK
{
bool BoolVariation(string flagKey, bool defaultValue, UserContext context);
string StringVariation(string flagKey, string defaultValue, UserContext context);
int IntVariation(string flagKey, int defaultValue, UserContext context);
double DoubleVariation(string flagKey, double defaultValue, UserContext context);
T JsonVariation<T>(string flagKey, T defaultValue, UserContext context);
Dictionary<string, FlagValue> AllFlags(UserContext context);
void Identify(UserContext context);
void Clear();
void Track(string eventName, UserContext context, Dictionary<string, object> data);
Task<bool> WaitForInitialization(TimeSpan timeout);
void Close();
}
Server-Side SDK API
public interface IFeatureFlagServerSDK
{
EvaluationResult Evaluate(string flagKey, UserContext context);
Dictionary<string, EvaluationResult> EvaluateAll(UserContext context);
Flag GetFlag(string flagKey);
IReadOnlyList<Flag> GetAllFlags();
void Track(Event event);
ClientStatus Status { get; }
event EventHandler<FlagChangedEventArgs> FlagChanged;
}
6. High-Level Architecture
The platform follows a control plane + data plane separation. The control plane manages flag configurations and experiments, while the data plane handles flag evaluations and event collection at the edge.
Architecture Principles
- Control Plane / Data Plane Separation: The control plane operates independently from the data plane. SDK evaluations continue even if the control plane is down.
- SDK-Side Evaluation: All flag evaluation happens locally within the SDK. No network calls needed.
- Push-Based Distribution: Flag changes are pushed to SDKs via SSE/WebSocket, ensuring near-instant propagation.
- Eventual Consistency with Bounded Staleness: SDKs may lag behind by at most 2 seconds in normal operation.
7. Flag Evaluation Engine
The flag evaluation engine is the heart of the system. It takes a flag key, user context, and configuration, then returns the correct variation. The engine must be deterministic, fast, and consistent across all SDK instances.
Evaluation Flow
Deterministic Bucketing Algorithm
For percentage rollouts, we use a deterministic hash function that consistently assigns users to buckets:
The bucketing algorithm uses a composite key of user_key + flag_key + salt to compute a deterministic hash. The hash maps to a 0-9999 range for 0.01% granularity.
Evaluation Example
var context = new UserContext
{
Key = "user-12345",
Custom = new Dictionary<string, object>
{
["country"] = "US",
["plan"] = "enterprise"
}
};
var result = sdk.Evaluate("new-checkout-flow", context);
// Trace:
// 1. Flag enabled? Yes
// 2. Individual targeting? No match
// 3. Rule 1: country == "US" AND plan == "enterprise" -> "treatment-v2" (100%)
// 4. Result: "treatment-v2", Reason: RULE_MATCH
8. SDK Architecture
The SDK is the primary interface between customer applications and the feature flag platform.
SDK Initialization Sequence
SDK Types Comparison
| Feature | Client-Side | Server-Side | Edge |
|---|---|---|---|
| Evaluation Location | In-browser | In-process | Edge worker |
| Authentication | Client key | Server key | Edge key |
| Flag Distribution | SSE streaming | SSE + polling | Edge KV |
| Persistent Cache | localStorage | File system | Edge KV |
| Evaluation Latency | <1ms | <1ms | <5ms |
| Event Sending | Batched 30s | Batched 60s | Aggregated |
| Security | Rules exposed | Rules hidden | Trusted edge |
9. Real-Time Flag Streaming
Real-time flag propagation is critical. When a developer toggles a flag, all connected SDKs must receive the update within 2 seconds. We use Server-Sent Events (SSE) as the primary protocol.
SSE Message Protocol
// SSE stream format for flag updates
// Connection: GET /v1/stream?sdk-key=xxx
// Initial connection
event: connected
data: {"version":42,"flags":1250,"interval":30000}
// Individual flag change
event: patch
data: {"flag":"new-checkout","version":43,"data":{"enabled":true,"variation":"treatment"}}
// Flag deletion
event: delete
data: {"flag":"old-feature","version":44}
// Full sync (reconnect)
event: sync
data: {"flags":{"new-checkout":{...}},"version":45}
// Keep-alive
: keepalive
Reconnection Strategy
Exponential backoff with jitter:
- Attempt 1: 1 second
- Attempt 2: 2 seconds + random(0-1s) jitter
- Attempt 3: 4 seconds + random(0-2s) jitter
- Attempt 4: 8 seconds + random(0-4s) jitter
- Max delay: 30 seconds
- On reconnect: send
Last-Event-IDheader for replay
The server maintains a flag change log (last 1000 changes) per stream for replay during reconnection.
10. A/B Testing & Experimentation Framework
An experimentation framework transforms feature flags from a deployment tool into a decision-making engine.
Experiment Configuration
| Field | Description | Example |
|---|---|---|
experiment_key | Unique identifier | "checkout-redesign-v2" |
hypothesis | Expected outcome | "Redesign increases conversion by 10%" |
flag_key | Controlling flag | "new-checkout-flow" |
variations | Control + treatment | ["control", "v2-minimal", "v2-steps"] |
traffic_allocation | % of users in experiment | 50% |
primary_metric | Main success metric | "checkout_completed" |
guardrail_metrics | Must not regress | ["error_rate", "page_load_time"] |
minimum_sample_size | Required per variation | 10,000 |
significance_level | Statistical threshold | 0.05 (95% confidence) |
Event Tracking Pipeline
11. Statistical Analysis
The experimentation engine supports both Frequentist and Bayesian analysis approaches.
Frequentist vs Bayesian
| Aspect | Frequentist | Bayesian |
|---|---|---|
| Core Concept | p-value, confidence intervals | Posterior probability distributions |
| Sample Size | Pre-computed requirement | Can peek at any time |
| Interpretation | 95% chance result not due to chance | 94% probability treatment is better |
| Guardrails | Requires Bonferroni correction | Naturally handles multiple comparisons |
| Best For | Large samples, fixed hypotheses | Early-stage, quick decisions |
| Stopping Rule | Fixed-horizon only | Continuous monitoring allowed |
Sample Size Formula: n = (Z_{1-a/2} + Z_{1-b})^2 x 2s^2 / d^2
- Z_{1-a/2} = 1.96 for 95% confidence
- Z_{1-b} = 0.84 for 80% power
- s = Standard deviation of metric
- d = Minimum detectable effect (e.g., 5% lift)
For s=1.0 and 5% lift: n ~ 15,700 per variation.
Streaming Statistics
For real-time dashboards, we use streaming statistical algorithms:
- Count-Min Sketch: Approximate event counting with bounded error
- T-Digest: Streaming quantile estimation for metrics like revenue
- HyperLogLog: Unique user counting with 0.81% standard error
- Streaming Variance: Welford's algorithm for running mean and variance
12. Progressive Rollout & Canary Deployments
Progressive rollout gradually exposes a feature to increasing percentages of users while monitoring for issues.
Rollout Configuration
| Stage | Traffic | Duration | Auto-Advance | Rollback Trigger |
|---|---|---|---|---|
| Canary | 1% | 24 hours | Yes | Error rate > 0.1% increase |
| Early Adopters | 5% | 48 hours | Yes | Error rate > 0.05% increase |
| Partial | 10% | 48 hours | Manual | Guardrail metric regression |
| Growth | 25-50% | 72 hours each | Manual | Error rate, latency, conversion |
| Full | 100% | 7 days | No | Any critical alert |
Automatic Rollback
// Rollout monitor - runs as background service
public class RolloutMonitor : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
var activeRollouts = await _rolloutService.GetActiveRollouts();
foreach (var rollout in activeRollouts)
{
var metrics = await _metricsService
.GetGuardrailMetrics(rollout.FlagKey, rollout.Environment);
foreach (var guardrail in rollout.Guardrails)
{
var current = metrics[guardrail.MetricName];
var baseline = await _baselineService
.GetBaseline(guardrail.MetricName, rollout.Environment);
if (current > baseline * (1 + guardrail.Threshold))
{
await _flagService.Rollback(rollout.FlagKey,
rollout.Environment, rollout.PreviousVariation);
await _alertService.Send(new RollbackAlert
{
FlagKey = rollout.FlagKey,
Metric = guardrail.MetricName,
Current = current,
Baseline = baseline
});
}
}
}
await Task.Delay(TimeSpan.FromSeconds(30), stoppingToken);
}
}
}
13. Flag Lifecycle Management
Feature flags have a lifecycle that must be managed to prevent flag debt.
Flag Cleanup Automation
- Day 60: Warning notification to the flag owner
- Day 90: Flag marked as cleanup_suggested; creates Jira ticket
- Day 120: Escalates to engineering manager; flag becomes read-only
- Day 180: Flag auto-archived; cleanup ticket priority escalated
14. Audit Logging & Change Tracking
Every action on the platform must be logged for compliance (SOC2, HIPAA, FedRAMP).
| Field | Type | Description |
|---|---|---|
audit_id | UUID | Unique identifier |
timestamp | ISO 8601 | When the action occurred |
actor | Object | { id, email, name, ip_address, user_agent } |
action | Enum | CREATE, UPDATE, DELETE, ENABLE, DISABLE, APPROVE, ROLLBACK |
resource_type | String | FLAG, ENVIRONMENT, PROJECT, EXPERIMENT |
resource_id | UUID | ID of the affected resource |
environment | String | Which environment was affected |
changes | JSON | Diff of before/after state (JSON Patch) |
metadata | JSON | Additional context (Jira ticket, PR link) |
Audit Log Storage Tiers
- Hot (Elasticsearch): Last 90 days - fast search, real-time queries
- Warm (S3 Parquet): 90 days to 2 years - compressed, queryable via Athena
- Cold (Glacier): 2+ years - compliance archive
15. Multi-Variate & Complex Targeting
Beyond simple boolean flags, multi-variate flags allow testing multiple variations simultaneously.
Complex Targeting Rules Example
// Example: Multi-variate flag with complex targeting
public static class TargetingRuleExample
{
public static FlagConfiguration ConfigureHomepageLayout()
{
return new FlagConfiguration
{
Key = "homepage-layout",
Type = FlagType.String,
DefaultVariation = "classic",
Variations = new[]
{
new Variation { Key = "classic", Value = "classic" },
new Variation { Key = "modern", Value = "modern" },
new Variation { Key = "minimal", Value = "minimal" },
new Variation { Key = "beta", Value = "beta" }
},
Rules = new[]
{
// Rule 1: Internal employees always get beta
new TargetingRule
{
Priority = 1,
Conditions = new[]
{
new Condition { Attribute = "email",
Op = "endsWith", Values = ["@company.com"] }
},
Variation = "beta"
},
// Rule 2: Premium users in US get modern
new TargetingRule
{
Priority = 2,
Conditions = new[]
{
new Condition { Attribute = "plan",
Op = "in", Values = ["premium", "enterprise"] },
new Condition { Attribute = "country",
Op = "in", Values = ["US", "CA"] }
},
Variation = "modern"
},
// Rule 3: 30% rollout of minimal to everyone else
new TargetingRule
{
Priority = 3,
PercentageRollout = new[]
{
new PercentageVariation { Variation = "minimal", Weight = 30 },
new PercentageVariation { Variation = "classic", Weight = 70 }
}
}
}
};
}
}
Supported Operators
| Operator | Description | Example |
|---|---|---|
in | Value is in set | country in ["US", "CA", "UK"] |
notIn | Value not in set | plan notIn ["free", "trial"] |
endsWith | String ends with | email endsWith ["@company.com"] |
startsWith | String starts with | version startsWith ["2."] |
contains | String contains | browser contains ["Chrome"] |
greaterThan | Numeric greater | age greaterThan ["18"] |
lessThan | Numeric less | score lessThan ["50"] |
between | Numeric between | version between ["1.0", "2.0"] |
exists | Attribute exists | beta_tester exists |
before | Date before | signupDate before ["2025-01-01"] |
after | Date after | signupDate after ["2024-06-01"] |
16. Integration Ecosystem
Feature flag platforms thrive when connected to the developer's existing tools.
Webhook Events
| Event | Description | Key Fields |
|---|---|---|
flag.created | New flag created | flag_key, project, created_by |
flag.updated | Config changed | flag_key, environment, changes |
flag.enabled | Turned on | flag_key, environment, variation |
flag.disabled | Turned off | flag_key, environment |
experiment.started | Experiment launched | experiment_key, variations |
experiment.concluded | Results ready | experiment_key, winner, confidence |
rollout.alert | Guardrail breach | flag_key, metric, threshold |
approval.requested | Awaiting approval | flag_key, environment |
17. Performance & Latency Optimization
Sub-millisecond evaluation latency is a core requirement.
Optimization Layers
| Layer | Optimization | Impact |
|---|---|---|
| In-Memory | Flags stored as flat byte array; zero-copy | <0.1ms |
| Compression | Brotli compression; delta updates | 90% less bandwidth |
| Hashing | MurmurHash3 (hardware-optimized) | <0.01ms per hash |
| Batching | Multiple evaluations in single call | Amortized overhead |
| Edge Caching | Flag snapshots at CDN edge | <5ms TTFB |
| Persistent Cache | Disk cache avoids cold start | 0ms startup |
| SSE Reconnect | Last-Event-ID replay | Instant recovery |
Benchmark Comparison
| Platform | p50 Latency | p99 Latency | Propagation |
|---|---|---|---|
| LaunchDarkly | <0.5ms | <1ms | <2 seconds |
| Split.io | <1ms | <3ms | <3 seconds |
| Flagsmith | <2ms | <10ms | <5 seconds |
| Unleash | <3ms | <15ms | <10 seconds |
| Our Platform | <0.5ms | <1ms | <2 seconds |
18. Data Pipeline for Metrics & Insights
Data Retention Policy
| Data Type | Hot (Queryable) | Warm (Compressed) | Cold (Archive) |
|---|---|---|---|
| Evaluation Events | 7 days (ClickHouse) | 90 days (S3 Parquet) | 1 year (Glacier) |
| Custom Events | 30 days (ClickHouse) | 1 year (S3 Parquet) | 3 years (Glacier) |
| Audit Logs | 90 days (Elasticsearch) | 2 years (S3 Parquet) | 7 years (Glacier) |
| Aggregated Metrics | 30 days (Redis) | 2 years (ClickHouse) | Indefinite (S3) |
| Experiment Results | 90 days (ClickHouse) | Indefinite (S3) | N/A |
19. Database Design
PostgreSQL Schema (Core Tables)
public class DatabaseSchema
{
// Organizations
// CREATE TABLE organizations (
// id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
// name VARCHAR(255) NOT NULL,
// key VARCHAR(100) UNIQUE NOT NULL,
// plan VARCHAR(50) NOT NULL DEFAULT 'free',
// settings JSONB DEFAULT '{}',
// created_at TIMESTAMPTZ DEFAULT NOW(),
// version INTEGER DEFAULT 1
// );
// Projects
// CREATE TABLE projects (
// id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
// org_id UUID REFERENCES organizations(id),
// name VARCHAR(255) NOT NULL,
// key VARCHAR(100) UNIQUE NOT NULL,
// created_at TIMESTAMPTZ DEFAULT NOW(),
// version INTEGER DEFAULT 1
// );
// Feature Flags
// CREATE TABLE flags (
// id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
// project_id UUID REFERENCES projects(id),
// key VARCHAR(255) NOT NULL,
// name VARCHAR(255) NOT NULL,
// flag_type VARCHAR(20) DEFAULT 'boolean',
// tags TEXT[] DEFAULT '{}',
// archived BOOLEAN DEFAULT false,
// created_at TIMESTAMPTZ DEFAULT NOW(),
// version INTEGER DEFAULT 1,
// UNIQUE(project_id, key)
// );
// Flag States (per environment)
// CREATE TABLE flag_states (
// id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
// flag_id UUID REFERENCES flags(id),
// environment_id UUID REFERENCES environments(id),
// enabled BOOLEAN DEFAULT false,
// default_variation VARCHAR(255),
// targets JSONB DEFAULT '[]',
// rules JSONB DEFAULT '[]',
// fallthrough JSONB DEFAULT '{}',
// version INTEGER DEFAULT 1,
// UNIQUE(flag_id, environment_id)
// );
// Variations
// CREATE TABLE variations (
// id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
// flag_id UUID REFERENCES flags(id),
// key VARCHAR(255) NOT NULL,
// value TEXT NOT NULL,
// name VARCHAR(255),
// UNIQUE(flag_id, key)
// );
// Experiments
// CREATE TABLE experiments (
// id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
// flag_id UUID REFERENCES flags(id),
// environment_id UUID REFERENCES environments(id),
// key VARCHAR(255) NOT NULL,
// hypothesis TEXT,
// traffic_percentage DECIMAL(5,2) DEFAULT 100.00,
// status VARCHAR(20) DEFAULT 'draft',
// start_time TIMESTAMPTZ,
// end_time TIMESTAMPTZ,
// version INTEGER DEFAULT 1
// );
// Audit Log
// CREATE TABLE audit_logs (
// id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
// timestamp TIMESTAMPTZ DEFAULT NOW(),
// actor_id UUID,
// actor_email VARCHAR(255),
// action VARCHAR(50) NOT NULL,
// resource_type VARCHAR(50) NOT NULL,
// resource_id UUID,
// environment VARCHAR(100),
// changes JSONB,
// metadata JSONB
// );
// Event Sourcing Table
// CREATE TABLE flag_events (
// id BIGSERIAL PRIMARY KEY,
// flag_id UUID NOT NULL,
// environment_id UUID NOT NULL,
// event_type VARCHAR(50) NOT NULL,
// payload JSONB NOT NULL,
// version INTEGER NOT NULL,
// created_at TIMESTAMPTZ DEFAULT NOW(),
// UNIQUE(flag_id, environment_id, version)
// );
}
Optimistic Concurrency Control
public class FlagStateService
{
public async Task<FlagState> UpdateFlagState(
string flagKey, string envKey, FlagStateUpdate update, int expectedVersion)
{
using var connection = await _connectionFactory.CreateConnection();
await connection.OpenAsync();
using var transaction = await connection.BeginTransactionAsync();
try
{
var rowsAffected = await connection.ExecuteAsync(@"
UPDATE flag_states
SET enabled = @Enabled,
default_variation = @DefaultVariation,
targets = @Targets::jsonb,
rules = @Rules::jsonb,
version = version + 1,
updated_at = NOW()
WHERE flag_id = (SELECT id FROM flags WHERE key = @FlagKey)
AND environment_id = (SELECT id FROM environments WHERE key = @EnvKey)
AND version = @ExpectedVersion",
new {
update.Enabled, update.DefaultVariation,
Targets = JsonSerializer.Serialize(update.Targets),
Rules = JsonSerializer.Serialize(update.Rules),
FlagKey = flagKey, EnvKey = envKey,
ExpectedVersion = expectedVersion
}, transaction);
if (rowsAffected == 0)
{
throw new ConcurrencyConflictException(
$"Flag '{flagKey}' was modified by another user. " +
$"Expected version {expectedVersion}. Please refresh and retry.");
}
// Record event for event sourcing
await connection.ExecuteAsync(@"
INSERT INTO flag_events
(flag_id, environment_id, event_type, payload, version)
VALUES ((SELECT id FROM flags WHERE key = @FlagKey),
(SELECT id FROM environments WHERE key = @EnvKey),
'state_updated', @Payload::jsonb, @Version)",
new {
FlagKey = flagKey, EnvKey = envKey,
Payload = JsonSerializer.Serialize(update),
Version = expectedVersion + 1
}, transaction);
await transaction.CommitAsync();
// Publish change to streaming layer
await _eventBus.PublishAsync(new FlagStateChangedEvent
{
FlagKey = flagKey,
EnvironmentKey = envKey,
Version = expectedVersion + 1,
Timestamp = DateTime.UtcNow
});
return await GetFlagState(flagKey, envKey);
}
catch
{
await transaction.RollbackAsync();
throw;
}
}
}
20. Caching Strategy
Cache Invalidation Strategy
| Trigger | Method | Propagation |
|---|---|---|
| Flag state updated | Redis invalidation + SSE push | <2 seconds |
| Flag created/archived | Project-level invalidation | <5 seconds |
| SDK reconnects | Full snapshot from API | Immediate |
| SDK cold start | Disk cache then API fallback | Disk 0ms, API <100ms |
| Redis failure | Direct DB query | Immediate (slower) |
21. Multi-Region Design
Regional Data Strategy
- Flag Configurations: Written to primary (US-EAST), async replicated to EU/APAC. Lag <500ms.
- Evaluation Events: Collected locally, streamed to regional Kafka, aggregated regionally.
- SDK Streaming: Each region has its own SSE gateway. SDKs connect via GeoDNS.
- Failover: Global load balancer routes to nearest healthy region on outage.
22. Cost Estimation
| Component | Spec | Monthly Cost |
|---|---|---|
| PostgreSQL (Primary + 3 Replicas) | r6g.2xlarge, 1TB gp3 | $6,000 |
| Redis Cluster (6 nodes) | r6g.xlarge, 256GB | $4,800 |
| Kafka (6 brokers) | m5.2xlarge, 2TB EBS | $5,400 |
| ClickHouse Cluster (3 nodes) | r6g.2xlarge, 2TB | $4,500 |
| SSE/WebSocket Servers (30) | c5.large | $3,200 |
| API Servers (20) | c5.xlarge | $4,200 |
| Event Ingestion (15) | c5.xlarge | $3,150 |
| Elasticsearch (3 nodes) | r5.xlarge, 1TB | $2,400 |
| S3 Storage (500TB) | Standard + Glacier | $12,000 |
| CloudFront / Cloudflare | 50TB/month transfer | $4,000 |
| Monitoring | Datadog/Grafana | $5,000 |
| Multi-Region Replication | EU + APAC | $15,000 |
| Total | ~$69,650/month |
Revenue Model:
- Free tier: 1,000 flags, 50K evaluations/month, 2 users
- Pro ($99/month): 10,000 flags, 5M evaluations/month, 10 users
- Enterprise ($499/month): Unlimited flags, 100M evaluations/month
- At 1,000 customers averaging $300/month: $300K MRR = $3.6M ARR
23. Interview Q&A
Q1: How do you ensure all SDK instances see the same flag value for a given user?
Answer: We use deterministic bucketing with a composite hash of user_key + flag_key + flag_salt using MurmurHash3. Since all SDKs have the same flag configuration (pushed via SSE) and use the same hash function with the same seed, they independently compute the same result. The hash maps to a 0-9999 range for 0.01% rollout precision. This guarantees a user sees the same variation across all SDK instances without coordination.
Q2: What happens if the feature flag platform goes down?
Answer: SDKs operate in "last known good" mode. Flags are persisted to disk during initialization. If the streaming connection drops, the SDK uses the in-memory cache. If the SDK restarts and cannot reach the platform, it loads from the persistent cache. We implement graceful degradation — a flag that was on before the outage stays on. The platform being down never causes features to accidentally turn off.
Q3: How do you propagate flag changes to millions of SDK instances within 2 seconds?
Answer: We use a multi-tier streaming architecture: (1) Flag change is written to the database, (2) Published to Redis Pub/Sub, (3) Propagated to regional SSE gateways, (4) SSE gateways push to all connected SDKs. This fan-out reaches millions of SDKs within 2 seconds. On reconnect, SDKs send Last-Event-ID for replay from the server's buffer (last 1000 changes).
Q4: How does percentage-based rollout work consistently across millions of users?
Answer: We use deterministic hash-based bucketing. For each user-flag pair: MurmurHash3(user_key + flag_key + salt) % 10000. If the result is less than rollout_percentage * 100, the user gets treatment. This ensures: (1) Each user always gets the same variation, (2) Distribution matches target within +/-0.1%, (3) Changing percentage only affects boundary users.
Q5: How do you prevent a poorly designed experiment from hurting the business?
Answer: We implement guardrail metrics and automated rollback. Every experiment specifies guardrail metrics (error rate, page load time, revenue per session) with thresholds. If any guardrail breaches (e.g., error rate +0.1%), the system: (1) Disables the experiment flag, (2) Reverts all users to control, (3) Alerts the owner. We also enforce minimum duration (7+ days) and sample size.
Q6: Explain the trade-offs between client-side and server-side SDK evaluation.
Answer: Client-side evaluates in the browser/device — all targeting rules are visible to the client. Good for non-sensitive flags, uses SSE streaming, stores in localStorage. Server-side evaluates in your backend — all rules hidden. Has access to server-side attributes (IP, internal data), uses SSE + polling, caches in Redis. Key trade-off: security vs latency.
Q7: How would you handle 500 million evaluations per second at peak?
Answer: Evaluation must be entirely local. The SDK loads all configurations into an in-memory hash map. Optimizations: (1) Pre-compiled byte arrays for zero-copy deserialization, (2) Decision tree compilation at load time, (3) Hardware-optimized MurmurHash3 (AES-NI), (4) Bulk evaluation in single SDK call, (5) Each SDK instance is isolated with no shared state. 500M/s distributed across millions of instances means each handles only a few hundred per second.
Q8: How do you handle targeting rules that reference missing attributes?
Answer: When a rule references an attribute not in the user context, the rule is treated as not matching and evaluation proceeds to the next rule. The SDK reports ATTRIBUTE_MISSING in the evaluation event for analytics. For server-side SDKs, the application can enrich context before evaluation. For client-side, we recommend using only reliably available attributes.
Q9: What is the difference between a feature flag and an experiment?
Answer: A feature flag toggles features for deployment control. An experiment adds metric tracking, traffic splitting, and statistical analysis. Differences: (1) Experiments require metric events, (2) Defined duration and sample size, (3) Deterministic bucketing for consistency, (4) Statistical results (p-values, confidence intervals). A flag can exist without an experiment, but an experiment always needs a flag.
Q10: How do you handle flag dependencies (prerequisite flags)?
Answer: Prerequisite flags allow one flag to depend on another being enabled. The evaluation engine checks prerequisites first: if the prerequisite is off, the dependent flag returns a prerequisite-failed default. We validate the dependency graph is acyclic at creation time. The SDK resolves prerequisites locally during evaluation.
Q11: How would you migrate a customer from their existing config flags?
Answer: Our migration toolkit: (1) Scans codebase for existing patterns (@FeatureFlag, env vars, config files), (2) Generates mapping to our data model, (3) Creates flags via Management API, (4) Provides replacement code snippets. We support a hybrid mode where the SDK references both platform flags and legacy sources, enabling incremental migration.
Q12: How do you prevent evaluation from becoming a bottleneck in high-throughput services?
Answer: Strategies: (1) Batch evaluation — all flags in one call, (2) Lazy evaluation — only used code paths, (3) Pre-computation — evaluate at identify-time, (4) Local-only — zero I/O, zero network, (5) Async events — queued, never blocking evaluation, (6) Tiny SDK — <50KB compiled. Target: <0.1ms overhead per service.
24. Full C# Implementation
Below is a complete, production-quality C# implementation of a feature flag SDK with evaluation engine, streaming client, and experiment tracking.
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net.Http;
using System.Text;
using System.Text.Json;
using System.Text.Json.Serialization;
using System.Threading;
using System.Threading.Tasks;
using System.Timers;
namespace FeatureFlagPlatform.SDK
{
// ═══════════════════════════════════════════════════════════
// SECTION 1: Core Models
// ═══════════════════════════════════════════════════════════
public enum FlagType { Boolean, String, Number, Json }
public enum EvaluationReason
{
FlagDisabled, NotFound, InvalidContext, IndividualTarget,
RuleMatch, PercentageRollout, Fallthrough, PrerequisiteFailed, Error
}
public class UserContext
{
public string Key { get; set; }
public string Name { get; set; }
public string Email { get; set; }
public string Ip { get; set; }
public string Country { get; set; }
public Dictionary<string, object> Custom { get; set; } = new();
public object GetAttribute(string attribute)
{
return attribute?.ToLowerInvariant() switch
{
"key" or "userkey" or "user_key" => Key,
"name" => Name,
"email" => Email,
"ip" or "ipaddress" => Ip,
"country" or "countrycode" => Country,
_ => Custom.TryGetValue(attribute, out var val) ? val : null
};
}
}
public class Variation
{
public string Key { get; set; }
public string Value { get; set; }
public string Name { get; set; }
}
public class TargetingCondition
{
public string Attribute { get; set; }
public string Operator { get; set; }
public List<string> Values { get; set; } = new();
}
public class PercentageVariation
{
public string VariationKey { get; set; }
public int Weight { get; set; }
}
public class TargetingRule
{
public int Priority { get; set; }
public List<TargetingCondition> Conditions { get; set; } = new();
public string VariationKey { get; set; }
public List<PercentageVariation> PercentageRollout { get; set; }
}
public class PrerequisiteFlag
{
public string FlagKey { get; set; }
public string VariationValue { get; set; }
}
public class FlagConfiguration
{
public string Key { get; set; }
public string Name { get; set; }
public FlagType Type { get; set; }
public bool Enabled { get; set; }
public List<Variation> Variations { get; set; } = new();
public string DefaultVariation { get; set; }
public string OffVariation { get; set; }
public List<TargetingRule> Rules { get; set; } = new();
public TargetingRule Fallthrough { get; set; }
public List<PrerequisiteFlag> Prerequisites { get; set; } = new();
public Dictionary<string, string> IndividualTargets { get; set; } = new();
public string Salt { get; set; }
public int Version { get; set; }
public DateTime LastUpdated { get; set; }
}
public class EvaluationResult
{
public string FlagKey { get; set; }
public string VariationKey { get; set; }
public string VariationValue { get; set; }
public bool Enabled { get; set; }
public EvaluationReason Reason { get; set; }
public double LatencyMs { get; set; }
public DateTime Timestamp { get; set; }
}
public class FlagChangeEvent : EventArgs
{
public string FlagKey { get; set; }
public int NewVersion { get; set; }
}
// ═══════════════════════════════════════════════════════════
// SECTION 2: Hashing and Bucketing
// ═══════════════════════════════════════════════════════════
public static class MurmurHash3
{
public static uint Compute(string key, uint seed = 0)
{
byte[] data = Encoding.UTF8.GetBytes(key);
uint hash = seed;
int length = data.Length;
int index = 0;
while (length >= 4)
{
uint k = BitConverter.ToUInt32(data, index);
k *= 0xcc9e2d51;
k = RotateLeft(k, 15);
k *= 0x1b873593;
hash ^= k;
hash = RotateLeft(hash, 13);
hash = hash * 5 + 0xe6546b64;
index += 4;
length -= 4;
}
uint tail = 0;
switch (length)
{
case 3: tail ^= (uint)data[index + 2] << 16; goto case 2;
case 2: tail ^= (uint)data[index + 1] << 8; goto case 1;
case 1:
tail ^= data[index];
tail *= 0xcc9e2d51;
tail = RotateLeft(tail, 15);
tail *= 0x1b873593;
hash ^= tail;
break;
}
hash ^= (uint)data.Length;
hash ^= hash >> 16;
hash *= 0x85ebca6b;
hash ^= hash >> 13;
hash *= 0xc2b2ae35;
hash ^= hash >> 16;
return hash;
}
private static uint RotateLeft(uint value, int count)
=> (value << count) | (value >> (32 - count));
public static int Bucket(string userKey, string flagKey, string salt)
{
string composite = $"{userKey}:{flagKey}:{salt}";
uint hash = Compute(composite);
return (int)(hash % 10000);
}
}
// ═══════════════════════════════════════════════════════════
// SECTION 3: Flag Evaluation Engine
// ═══════════════════════════════════════════════════════════
public class EvaluationEngine
{
private readonly ConcurrentDictionary<string, FlagConfiguration> _flags = new();
public void LoadFlags(IEnumerable<FlagConfiguration> flags)
{
foreach (var flag in flags)
_flags[flag.Key] = flag;
}
public void UpdateFlag(FlagConfiguration flag)
=> _flags[flag.Key] = flag;
public void RemoveFlag(string flagKey)
=> _flags.TryRemove(flagKey, out _);
public EvaluationResult Evaluate(string flagKey, UserContext context)
{
var sw = System.Diagnostics.Stopwatch.StartNew();
try
{
if (context == null || string.IsNullOrEmpty(context.Key))
return CreateResult(flagKey, EvaluationReason.InvalidContext, sw);
if (!_flags.TryGetValue(flagKey, out var flag))
return CreateResult(flagKey, EvaluationReason.NotFound, sw);
if (!flag.Enabled)
return CreateResult(flagKey, EvaluationReason.FlagDisabled, sw,
variation: flag.OffVariation);
// Check prerequisites
if (flag.Prerequisites?.Count > 0)
{
foreach (var prereq in flag.Prerequisites)
{
if (_flags.TryGetValue(prereq.FlagKey, out var prereqFlag))
{
var prereqResult = Evaluate(prereq.FlagKey, context);
if (prereqResult.VariationValue != prereq.VariationValue)
return CreateResult(flagKey,
EvaluationReason.PrerequisiteFailed, sw);
}
else
return CreateResult(flagKey,
EvaluationReason.PrerequisiteFailed, sw);
}
}
// Check individual targets
if (flag.IndividualTargets?.TryGetValue(context.Key, out var targetVar) == true)
return CreateResult(flagKey, EvaluationReason.IndividualTarget, sw,
variation: targetVar);
// Evaluate targeting rules
if (flag.Rules != null)
{
foreach (var rule in flag.Rules.OrderBy(r => r.Priority))
{
if (EvaluateConditions(rule.Conditions, context))
{
if (rule.PercentageRollout?.Count > 0)
{
var bucket = MurmurHash3.Bucket(
context.Key, flagKey, flag.Salt);
int cumulative = 0;
foreach (var pv in rule.PercentageRollout)
{
cumulative += pv.Weight;
if (bucket < cumulative)
return CreateResult(flagKey,
EvaluationReason.RuleMatch, sw,
variation: pv.VariationKey);
}
return CreateResult(flagKey, EvaluationReason.RuleMatch,
sw, variation: rule.PercentageRollout.Last().VariationKey);
}
return CreateResult(flagKey, EvaluationReason.RuleMatch, sw,
variation: rule.VariationKey);
}
}
}
// Fallthrough
if (flag.Fallthrough != null)
{
if (flag.Fallthrough.PercentageRollout?.Count > 0)
{
var bucket = MurmurHash3.Bucket(context.Key, flagKey, flag.Salt);
int cumulative = 0;
foreach (var pv in flag.Fallthrough.PercentageRollout)
{
cumulative += pv.Weight;
if (bucket < cumulative)
return CreateResult(flagKey,
EvaluationReason.Fallthrough, sw,
variation: pv.VariationKey);
}
return CreateResult(flagKey, EvaluationReason.Fallthrough, sw,
variation: flag.Fallthrough.PercentageRollout.Last().VariationKey);
}
return CreateResult(flagKey, EvaluationReason.Fallthrough, sw,
variation: flag.Fallthrough.VariationKey);
}
return CreateResult(flagKey, EvaluationReason.Fallthrough, sw,
variation: flag.DefaultVariation);
}
catch (Exception)
{
sw.Stop();
return new EvaluationResult
{
FlagKey = flagKey, VariationKey = "error",
VariationValue = "error", Reason = EvaluationReason.Error,
LatencyMs = sw.Elapsed.TotalMilliseconds,
Timestamp = DateTime.UtcNow
};
}
}
private bool EvaluateConditions(
List<TargetingCondition> conditions, UserContext context)
{
if (conditions == null || conditions.Count == 0) return true;
foreach (var condition in conditions)
{
var attrValue = context.GetAttribute(condition.Attribute);
if (!EvaluateCondition(condition, attrValue)) return false;
}
return true;
}
private bool EvaluateCondition(TargetingCondition condition, object attrValue)
{
var strValue = attrValue?.ToString() ?? "";
var opValues = condition.Values ?? new List<string>();
return condition.Operator?.ToLowerInvariant() switch
{
"in" => opValues.Any(v =>
string.Equals(v, strValue, StringComparison.OrdinalIgnoreCase)),
"notin" or "not_in" => !opValues.Any(v =>
string.Equals(v, strValue, StringComparison.OrdinalIgnoreCase)),
"startswith" or "starts_with" => opValues.Any(v =>
strValue.StartsWith(v, StringComparison.OrdinalIgnoreCase)),
"endswith" or "ends_with" => opValues.Any(v =>
strValue.EndsWith(v, StringComparison.OrdinalIgnoreCase)),
"contains" => opValues.Any(v =>
strValue.Contains(v, StringComparison.OrdinalIgnoreCase)),
"greaterthan" or "greater_than" =>
double.TryParse(strValue, out var num) &&
double.TryParse(opValues.FirstOrDefault(), out var thr) && num > thr,
"lessthan" or "less_than" =>
double.TryParse(strValue, out var num) &&
double.TryParse(opValues.FirstOrDefault(), out var thr) && num < thr,
"exists" => attrValue != null,
"notexists" or "not_exists" => attrValue == null,
_ => false
};
}
private EvaluationResult CreateResult(
string flagKey, EvaluationReason reason,
System.Diagnostics.Stopwatch sw, string variation = null)
{
sw.Stop();
string varKey = variation ?? "off";
string varValue = varKey;
if (_flags.TryGetValue(flagKey, out var flag))
{
var match = flag.Variations?.FirstOrDefault(v => v.Key == varKey);
if (match != null) varValue = match.Value;
}
return new EvaluationResult
{
FlagKey = flagKey, VariationKey = varKey,
VariationValue = varValue, Enabled = flag?.Enabled ?? false,
Reason = reason, LatencyMs = sw.Elapsed.TotalMilliseconds,
Timestamp = DateTime.UtcNow
};
}
}
}
// ═══════════════════════════════════════════════════════════
// SECTION 4: SSE Streaming Client
// ═══════════════════════════════════════════════════════════
public class SseStreamingClient : IDisposable
{
private readonly HttpClient _httpClient;
private readonly string _streamUrl;
private readonly string _sdkKey;
private CancellationTokenSource _cts;
private Task _streamTask;
private int _reconnectDelay = 1000;
private const int MaxReconnectDelay = 30000;
private string _lastEventId;
public event EventHandler<FlagConfiguration> FlagChanged;
public event EventHandler<FlagConfiguration> FlagDeleted;
public event EventHandler Connected;
public event EventHandler<Exception> Disconnected;
public bool IsConnected { get; private set; }
public SseStreamingClient(string baseUrl, string sdkKey)
{
_httpClient = new HttpClient { Timeout = TimeSpan.FromMilliseconds(-1) };
_streamUrl = $"{baseUrl}/v1/stream";
_sdkKey = sdkKey;
}
public Task StartAsync(CancellationToken cancellationToken = default)
{
_cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
_streamTask = StreamLoopAsync(_cts.Token);
return Task.CompletedTask;
}
private async Task StreamLoopAsync(CancellationToken ct)
{
while (!ct.IsCancellationRequested)
{
try
{
var request = new HttpRequestMessage(HttpMethod.Get, _streamUrl);
request.Headers.Add("Authorization", _sdkKey);
request.Headers.Add("Accept", "text/event-stream");
request.Headers.Add("Cache-Control", "no-cache");
if (!string.IsNullOrEmpty(_lastEventId))
request.Headers.Add("Last-Event-ID", _lastEventId);
var response = await _httpClient.SendAsync(
request, HttpCompletionOption.ResponseHeadersRead, ct);
response.EnsureSuccessStatusCode();
IsConnected = true;
_reconnectDelay = 1000;
Connected?.Invoke(this, EventArgs.Empty);
using var stream = await response.Content.ReadAsStreamAsync(ct);
using var reader = new StreamReader(stream);
string eventType = null, eventData = null, eventId = null;
while (!ct.IsCancellationRequested && !reader.EndOfStream)
{
var line = await reader.ReadLineAsync();
if (string.IsNullOrEmpty(line))
{
if (!string.IsNullOrEmpty(eventData))
{
ProcessEvent(eventType, eventData, eventId);
eventId = null;
eventType = null;
eventData = null;
}
}
else if (line.StartsWith("event:"))
eventType = line.Substring(6).Trim();
else if (line.StartsWith("data:"))
eventData = line.Substring(5).Trim();
else if (line.StartsWith("id:"))
eventId = line.Substring(3).Trim();
}
}
catch (Exception ex) when (!ct.IsCancellationRequested)
{
IsConnected = false;
Disconnected?.Invoke(this, ex);
await Task.Delay(_reconnectDelay, ct);
_reconnectDelay = Math.Min(_reconnectDelay * 2, MaxReconnectDelay);
_reconnectDelay += Random.Shared.Next(0, _reconnectDelay / 2);
}
}
}
private void ProcessEvent(string eventType, string data, string eventId)
{
if (!string.IsNullOrEmpty(eventId)) _lastEventId = eventId;
switch (eventType)
{
case "patch":
var patch = JsonSerializer.Deserialize<FlagPatch>(data);
if (patch != null)
FlagChanged?.Invoke(this, new FlagConfiguration
{
Key = patch.FlagKey,
Enabled = patch.Data?.Enabled ?? false,
Version = patch.Version,
LastUpdated = DateTime.UtcNow
});
break;
case "delete":
var del = JsonSerializer.Deserialize<FlagDelete>(data);
if (del?.FlagKey != null)
FlagDeleted?.Invoke(this,
new FlagConfiguration { Key = del.FlagKey });
break;
}
}
public void Dispose()
{
_cts?.Cancel();
_streamTask?.Wait(TimeSpan.FromSeconds(5));
_httpClient?.Dispose();
}
private class FlagPatch
{
[JsonPropertyName("flag")] public string FlagKey { get; set; }
[JsonPropertyName("version")] public int Version { get; set; }
[JsonPropertyName("data")] public PatchData Data { get; set; }
}
private class PatchData
{
[JsonPropertyName("enabled")] public bool? Enabled { get; set; }
[JsonPropertyName("variation")] public string Variation { get; set; }
}
private class FlagDelete
{
[JsonPropertyName("flag")] public string FlagKey { get; set; }
}
}
// ═══════════════════════════════════════════════════════════
// SECTION 5: Event Tracker for experimentation
// ═══════════════════════════════════════════════════════════
public class EventTracker : IDisposable
{
private readonly ConcurrentQueue<TrackingEvent> _eventQueue = new();
private readonly System.Timers.Timer _flushTimer;
private readonly HttpClient _httpClient;
private readonly string _ingestUrl;
private readonly string _sdkKey;
public EventTracker(string baseUrl, string sdkKey,
TimeSpan? flushInterval = null)
{
_httpClient = new HttpClient();
_ingestUrl = $"{baseUrl}/v1/events";
_sdkKey = sdkKey;
_flushTimer = new System.Timers.Timer(
(flushInterval ?? TimeSpan.FromSeconds(30)).TotalMilliseconds);
_flushTimer.Elapsed += async (s, e) => await FlushEventsAsync();
_flushTimer.AutoReset = true;
_flushTimer.Start();
}
public void Track(string eventName, UserContext context,
Dictionary<string, object> data = null)
{
_eventQueue.Enqueue(new TrackingEvent
{
EventType = eventName, UserKey = context.Key,
Timestamp = DateTime.UtcNow,
Data = data ?? new Dictionary<string, object>()
});
}
public void TrackEvaluation(EvaluationResult result, UserContext context)
{
_eventQueue.Enqueue(new TrackingEvent
{
EventType = "$flag_evaluation", UserKey = context.Key,
Timestamp = result.Timestamp,
Data = new Dictionary<string, object>
{
["flag"] = result.FlagKey,
["variation"] = result.VariationKey,
["enabled"] = result.Enabled,
["reason"] = result.Reason.ToString(),
["latencyMs"] = result.LatencyMs
}
});
}
public async Task FlushEventsAsync()
{
var events = new List<TrackingEvent>();
while (_eventQueue.TryDequeue(out var evt) && events.Count < 500)
events.Add(evt);
if (events.Count == 0) return;
try
{
var payload = JsonSerializer.Serialize(new { events });
var content = new StringContent(payload,
Encoding.UTF8, "application/json");
content.Headers.Add("Authorization", _sdkKey);
await _httpClient.PostAsync(_ingestUrl, content);
}
catch
{
foreach (var evt in events)
_eventQueue.Enqueue(evt);
}
}
public void Dispose()
{
_flushTimer?.Stop();
_flushTimer?.Dispose();
FlushEventsAsync().GetAwaiter().GetResult();
}
private class TrackingEvent
{
public string EventType { get; set; }
public string UserKey { get; set; }
public DateTime Timestamp { get; set; }
public Dictionary<string, object> Data { get; set; }
}
}
// ═══════════════════════════════════════════════════════════
// SECTION 6: Persistent Cache
// ═══════════════════════════════════════════════════════════
public class PersistentFlagCache
{
private readonly string _cacheDirectory;
public PersistentFlagCache(string cacheDirectory)
{
_cacheDirectory = cacheDirectory;
Directory.CreateDirectory(_cacheDirectory);
}
public async Task SaveFlagsAsync(string environmentKey,
Dictionary<string, FlagConfiguration> flags)
{
var filePath = GetFilePath(environmentKey);
var data = JsonSerializer.Serialize(flags,
new JsonSerializerOptions { WriteIndented = false });
await File.WriteAllTextAsync(filePath, data);
}
public async Task<Dictionary<string, FlagConfiguration>>
LoadFlagsAsync(string environmentKey)
{
var filePath = GetFilePath(environmentKey);
if (!File.Exists(filePath))
return new Dictionary<string, FlagConfiguration>();
var data = await File.ReadAllTextAsync(filePath);
return JsonSerializer.Deserialize<
Dictionary<string, FlagConfiguration>>(data)
?? new Dictionary<string, FlagConfiguration>();
}
private string GetFilePath(string environmentKey)
{
var safeName = Convert.ToBase64String(
Encoding.UTF8.GetBytes(environmentKey)).Replace('/', '_');
return Path.Combine(_cacheDirectory, $"flags_{safeName}.json");
}
}
// ═══════════════════════════════════════════════════════════
// SECTION 7: Experiment Analyzer
// ═══════════════════════════════════════════════════════════
public class ExperimentAnalyzer
{
public ExperimentResult Analyze(
List<VariantMetrics> control,
List<VariantMetrics> treatment,
double significanceLevel = 0.05)
{
var controlMean = control.Average(m => m.Value);
var treatmentMean = treatment.Average(m => m.Value);
var controlVar = CalculateVariance(control, controlMean);
var treatmentVar = CalculateVariance(treatment, treatmentMean);
var n1 = control.Count;
var n2 = treatment.Count;
var se = Math.Sqrt(controlVar / n1 + treatmentVar / n2);
if (se == 0) return new ExperimentResult { IsSignificant = false };
var tStat = (treatmentMean - controlMean) / se;
var df = Math.Pow(controlVar / n1 + treatmentVar / n2, 2) /
(Math.Pow(controlVar / n1, 2) / (n1 - 1) +
Math.Pow(treatmentVar / n2, 2) / (n2 - 1));
var pValue = CalculatePValue(Math.Abs(tStat), (int)df);
var lift = controlMean != 0
? ((treatmentMean - controlMean) / controlMean) * 100 : 0;
return new ExperimentResult
{
ControlMean = controlMean, TreatmentMean = treatmentMean,
Lift = lift, TStatistic = tStat, PValue = pValue,
DegreesOfFreedom = (int)df,
IsSignificant = pValue < significanceLevel,
ConfidenceLevel = (1 - pValue) * 100,
SampleSizeControl = n1, SampleSizeTreatment = n2
};
}
private double CalculateVariance(
List<VariantMetrics> metrics, double mean)
=> metrics.Sum(m => Math.Pow(m.Value - mean, 2)) / (metrics.Count - 1);
private double CalculatePValue(double tStat, int df)
{
var x = df / (df + tStat * tStat);
return IncompleteBeta(df / 2.0, 0.5, x);
}
private double IncompleteBeta(double a, double b, double x)
{
if (x == 0) return 0;
if (x == 1) return 1;
var bt = Math.Exp(LogGamma(a + b) - LogGamma(a) - LogGamma(b) +
a * Math.Log(x) + b * Math.Log(1 - x));
if (x < (a + 1) / (a + b + 2))
return bt * BetaCf(a, b, x) / a;
return 1 - bt * BetaCf(b, a, 1 - x) / b;
}
private double BetaCf(double a, double b, double x)
{
var qab = a + b; var qap = a + 1; var qam = a - 1;
var c = 1.0; var d = 1.0 - qab * x / qap;
if (Math.Abs(d) < 1e-30) d = 1e-30;
d = 1.0 / d; var h = d;
for (int m = 1; m <= 100; m++)
{
var m2 = 2 * m;
var aa = m * (b - m) * x / ((qam + m2) * (a + m2));
d = 1.0 + aa * d;
if (Math.Abs(d) < 1e-30) d = 1e-30;
c = 1.0 + aa / c;
if (Math.Abs(c) < 1e-30) c = 1e-30;
d = 1.0 / d; h *= d * c;
aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2));
d = 1.0 + aa * d;
if (Math.Abs(d) < 1e-30) d = 1e-30;
c = 1.0 + aa / c;
if (Math.Abs(c) < 1e-30) c = 1e-30;
d = 1.0 / d;
var del = d * c;
h *= del;
if (Math.Abs(del - 1.0) < 3e-7) break;
}
return h;
}
private double LogGamma(double x)
{
double[] c = { 76.18009172947146, -86.50532032941677,
24.01409824083091, -1.231739572450155,
0.001208650973866179, -0.000005395239384953 };
double y = x, tmp = x + 5.5;
tmp -= (x + 0.5) * Math.Log(tmp);
double sum = 1.000000000190015;
for (int j = 0; j < 6; j++)
sum += c[j] / ++y;
return -tmp + Math.Log(2.5066282746310005 * sum / x);
}
}
public class VariantMetrics
{
public double Value { get; set; }
public string VariationKey { get; set; }
public DateTime Timestamp { get; set; }
}
public class ExperimentResult
{
public double ControlMean { get; set; }
public double TreatmentMean { get; set; }
public double Lift { get; set; }
public double TStatistic { get; set; }
public double PValue { get; set; }
public int DegreesOfFreedom { get; set; }
public bool IsSignificant { get; set; }
public double ConfidenceLevel { get; set; }
public int SampleSizeControl { get; set; }
public int SampleSizeTreatment { get; set; }
}
// ═══════════════════════════════════════════════════════════
// SECTION 8: Feature Flag Client (Main Entry Point)
// ═══════════════════════════════════════════════════════════
public class FeatureFlagClient : IDisposable
{
private readonly EvaluationEngine _engine;
private readonly SseStreamingClient _streaming;
private readonly EventTracker _tracker;
private readonly PersistentFlagCache _cache;
private UserContext _currentContext;
private bool _initialized;
public event EventHandler<FlagChangeEvent> FlagChanged;
public bool IsInitialized => _initialized;
public FeatureFlagClient(string sdkKey, string baseUrl,
string cacheDir = null)
{
_engine = new EvaluationEngine();
_streaming = new SseStreamingClient(baseUrl, sdkKey);
_tracker = new EventTracker(baseUrl, sdkKey);
_cache = new PersistentFlagCache(cacheDir ?? Path.Combine(
Path.GetTempPath(), "ff-cache"));
_streaming.FlagChanged += (s, flag) =>
{
_engine.UpdateFlag(flag);
FlagChanged?.Invoke(this, new FlagChangeEvent
{
FlagKey = flag.Key, NewVersion = flag.Version
});
};
}
public async Task InitializeAsync(string environmentKey,
UserContext context = null)
{
_currentContext = context;
// Load from persistent cache
var cached = await _cache.LoadFlagsAsync(environmentKey);
if (cached.Count > 0)
_engine.LoadFlags(cached.Values);
// Start streaming
_ = _streaming.StartAsync();
// Wait for initial connection
await _streaming.WaitForInitialization(TimeSpan.FromSeconds(10));
_initialized = true;
// Save to cache
var allFlags = _engine.GetAllFlags()
.ToDictionary(f => f.Key, f => f);
await _cache.SaveFlagsAsync(environmentKey, allFlags);
}
public bool BoolVariation(string flagKey, bool defaultValue,
UserContext context = null)
{
var ctx = context ?? _currentContext;
var result = _engine.Evaluate(flagKey, ctx);
_tracker.TrackEvaluation(result, ctx);
if (bool.TryParse(result.VariationValue, out var val))
return val;
return defaultValue;
}
public string StringVariation(string flagKey, string defaultValue,
UserContext context = null)
{
var ctx = context ?? _currentContext;
var result = _engine.Evaluate(flagKey, ctx);
_tracker.TrackEvaluation(result, ctx);
return result.VariationValue ?? defaultValue;
}
public int IntVariation(string flagKey, int defaultValue,
UserContext context = null)
{
var ctx = context ?? _currentContext;
var result = _engine.Evaluate(flagKey, ctx);
_tracker.TrackEvaluation(result, ctx);
if (int.TryParse(result.VariationValue, out var val))
return val;
return defaultValue;
}
public T JsonVariation<T>(string flagKey, T defaultValue,
UserContext context = null)
{
var ctx = context ?? _currentContext;
var result = _engine.Evaluate(flagKey, ctx);
_tracker.TrackEvaluation(result, ctx);
try
{
return JsonSerializer.Deserialize<T>(result.VariationValue);
}
catch { return defaultValue; }
}
public void Track(string eventName,
Dictionary<string, object> data = null,
UserContext context = null)
{
_tracker.Track(eventName, context ?? _currentContext, data);
}
public void Identify(UserContext context)
=> _currentContext = context;
public EvaluationResult Evaluate(string flagKey,
UserContext context = null)
{
var ctx = context ?? _currentContext;
var result = _engine.Evaluate(flagKey, ctx);
_tracker.TrackEvaluation(result, ctx);
return result;
}
public void Dispose()
{
_streaming?.Dispose();
_tracker?.Dispose();
}
}
}
Deep Dive: Why Deterministic Bucketing Matters for Experimentation
Deterministic bucketing is the single most important technical requirement for a valid A/B testing platform. Without it, the fundamental assumptions of statistical hypothesis testing break down. When a user enters an experiment, they must be consistently assigned to the same variation for the entire duration of the experiment. If a user could flip between control and treatment, the measured effect would be diluted and the experiment would never reach statistical significance, wasting weeks of engineering time and traffic.
The bucketing algorithm we described earlier uses a cryptographic-strength hash function (MurmurHash3) that produces a uniformly distributed value from 0 to 9999. This 0.01% granularity means the platform can precisely control what percentage of users sees each variation. When the product manager asks to roll out a new checkout flow to exactly 25% of US-based premium users, the system can target that precisely without any drift or rounding errors that accumulate over millions of users.
A critical subtlety is the use of a per-flag salt in the hash computation. Without a salt, the same user would always fall into the same percentage bucket across all flags, creating unwanted correlations between experiments. By using a unique salt per flag, each experiment creates an independent random assignment, ensuring that statistical independence assumptions hold between concurrent experiments.
Deep Dive: Graceful Degradation and Offline Mode
The availability requirements for a feature flag platform are unusually strict because the platform sits on the critical path of every feature in the customer's application. If the flag platform goes down and features start behaving unexpectedly, the customer's entire product is affected. This is fundamentally different from most third-party services where downtime means a lost feature, not a broken application.
The multi-layered degradation strategy ensures this never happens. During normal operation, the SDK receives flag updates via SSE with sub-2-second latency. If the SSE connection drops, the SDK continues operating with the last known flag state from its in-memory cache. If the application process restarts and the platform is unreachable, the SDK loads from its persistent disk cache, which contains the last successfully fetched flag state. This means even a complete platform outage lasting hours would not cause any features to change state in the customer's application.
This design principle is called "last known good" and it is the cornerstone of feature flag platform reliability. The only time a flag value changes is when the SDK receives an explicit update from the platform. The platform can never accidentally set a flag to a default state during an outage because the SDK simply stops accepting updates when it cannot reach the server. This is a crucial safety guarantee that distinguishes enterprise-grade platforms from homegrown feature flag solutions.
Deep Dive: Event Sourcing for Flag State Changes
Event sourcing is a powerful pattern for feature flag platforms because it provides a complete audit trail, enables point-in-time recovery, and makes the flag state derivable from first principles. Instead of storing only the current state of a flag, we store every state transition as an immutable event in an append-only log.
When a developer changes a flag from "off" to "on with 10% rollout to US users," that change is recorded as a flag event with the complete before/after state. This event is then replayed through the SSE streaming layer to update all connected SDKs. The event also flows into the audit log, the analytics pipeline, and any connected integrations (Jira, Slack, Datadog). Because events are immutable and ordered, we can reconstruct the state of any flag at any point in time by replaying events up to that timestamp.
This is particularly valuable for debugging production incidents. When a customer reports that a feature started misbehaving at 3:42 PM, the engineering team can query the flag event log to see exactly what changed around that time, which developer made the change, and what the previous state was. Combined with the rollback feature, this enables rapid incident response with full traceability.
Deep Dive: Statistical Power and Experiment Duration
One of the most common mistakes in A/B testing is stopping an experiment too early because the results "look significant." This is known as peeking, and it dramatically increases the false positive rate. A result that appears significant at 1,000 users may completely disappear at 10,000 users as the variance decreases and the true effect size becomes apparent.
The platform addresses this in two ways. For Frequentist experiments, it enforces a fixed-horizon stopping rule: the experiment must run until the pre-calculated minimum sample size is reached before results can be declared significant. For Bayesian experiments, the platform uses expected loss as the decision criterion rather than p-values, which naturally handles the peeking problem and allows continuous monitoring.
Both approaches also require capturing at least one full weekly cycle (7 days) to account for day-of-week effects. An experiment that runs only on weekdays might show a 15% lift simply because the treatment group had more weekday traffic. Running for a minimum of 14 days ensures that both control and treatment groups experience the same distribution of weekly traffic patterns, making the comparison valid.
25. Conclusion
Building a feature flag and experimentation platform at scale is a complex engineering challenge that touches virtually every aspect of distributed systems design. From deterministic bucketing algorithms that ensure consistent user experiences across millions of SDK instances, to real-time streaming architectures that propagate flag changes globally within 2 seconds, to statistical analysis engines that power data-driven product decisions, the platform requires careful consideration of performance, reliability, accuracy, and security.
The key architectural decisions that define a world-class feature flag platform are:
- SDK-side evaluation eliminates network latency from the critical path, enabling sub-millisecond evaluations that never become a bottleneck
- Control plane / data plane separation ensures SDK evaluations continue even during platform outages, maintaining the highest availability standards
- Push-based streaming with replay provides near-instant flag propagation while ensuring no updates are lost during network interruptions
- Deterministic hash-based bucketing guarantees consistent experiment assignment across all SDK instances without coordination
- Guardrail metrics with automated rollback protects businesses from the impact of poorly performing features during progressive rollouts
- Multi-tier caching (in-memory, Redis, persistent disk) balances performance with freshness across all failure modes
The financial opportunity is substantial: with operating costs around $70K/month and a revenue model that can generate $3.6M+ ARR from 1,000 customers, the unit economics are favorable for a feature flag platform that competes with LaunchDarkly's $1.3B+ market position.
- Always start with the distinction between control plane and data plane
- Emphasize SDK-side evaluation for latency-critical paths
- Explain deterministic bucketing for experiment consistency
- Discuss graceful degradation and offline mode for availability
- Cover the full lifecycle: creation, targeting, experimentation, progressive rollout, and cleanup
- Address the data pipeline challenge: trillions of events require streaming aggregation, not raw storage
- Mention multi-region design for global availability and data residency compliance
Whether you are preparing for a senior+ system design interview at a FAANG company or building a feature flag platform for your organization, the patterns and principles covered in this guide provide a comprehensive foundation. The feature flag and experimentation space continues to evolve with advances in streaming infrastructure, machine learning for automated experiment analysis, and edge computing for even lower latency evaluations. Master these fundamentals, and you will be well-prepared to design and build platforms that serve trillions of evaluations while enabling teams to ship features with confidence.