1. Introduction & Why Event-Driven Architecture

Modern distributed systems face a fundamental tension: components must coordinate to deliver business value, yet direct coupling between services creates fragile, hard-to-evolve architectures. Event-Driven Architecture (EDA) resolves this tension by replacing direct service-to-service calls with a publish-subscribe model where producers emit events and consumers react to them independently. Rather than Service A calling Service B synchronously and waiting for a response, Service A publishes an event ("OrderPlaced") and any number of interested services react to that event on their own schedule. This decoupling transforms the system from a tightly coupled web of dependencies into a loosely connected constellation of autonomous components.

The business case for EDA is compelling. In a synchronous request-response architecture, a single service outage cascades across the entire system. If the notification service is down when an order is placed, the order service either fails or must implement complex retry logic. In an event-driven system, the notification service's outage is absorbed by the message broker: events accumulate in the queue and are processed when the service recovers. The order service never knows the difference. This resilience is not a nice-to-have — it is a survival requirement at scale. When Netflix handles 250 million streaming sessions per day, synchronous cross-service calls would create a fragile dependency chain where any single failure brings down the entire platform.

Event-driven architecture is not a silver bullet. It introduces its own complexities: eventual consistency (consumers see data changes with a delay), ordering guarantees (events from different producers may arrive out of order), debugging difficulty (tracing a request across asynchronous event chains is harder than following a synchronous call stack), and operational overhead (managing message brokers, monitoring consumer lag, handling dead letter queues). The decision to adopt EDA should be driven by specific requirements — high throughput, fault tolerance, independent deployability, and team autonomy — not by architectural fashion. This guide provides the depth needed to make informed decisions and implement EDA correctly in production.

The evolution from monolithic to event-driven architectures mirrors the evolution of software itself. In the 1990s, enterprise application integration (EAI) used message-oriented middleware (MOM) like IBM MQ and TIBCO. The 2000s brought enterprise service buses (ESBs) that centralized message routing. The 2010s saw the rise of log-based platforms like Apache Kafka that treated events as an immutable, replayable log. Today, event-driven architectures are the backbone of cloud-native systems, with every major cloud provider offering managed event streaming services (Amazon Kinesis, Azure Event Hubs, Google Pub/Sub). Understanding EDA is no longer optional for senior engineers — it is foundational knowledge for building systems that operate at internet scale.

When to Choose Event-Driven Architecture

2. Core Concepts: Events, Commands, Queries

The vocabulary of event-driven architecture is precise, and conflating terms leads to design errors. An Event is a fact — something that happened in the past. It is immutable, named in past tense ("OrderPlaced", "PaymentProcessed", "InventoryReserved"), and carries the data necessary for consumers to react. Events are broadcast: every interested consumer receives every relevant event. No single producer knows or cares who consumes its events. This broadcast nature is what creates loose coupling.

A Command is an instruction — something that should happen. Commands are named in imperative tense ("PlaceOrder", "ProcessPayment", "ReserveInventory"), are directed at a specific handler, and may be rejected (validation failure, business rule violation). Unlike events, commands imply a contract: the sender expects the handler to do something specific. This makes commands inherently more coupled than events. In a well-designed EDA system, commands are used within a single service boundary (between components of the same bounded context) while events cross service boundaries.

A Query is a request for data — it never changes state. In event-driven systems, queries often hit read models that are optimized for specific query patterns (CQRS read side). The distinction matters because queries can be served from eventually consistent read replicas without the consistency concerns that affect commands. A query for an order's status might hit a read model that is 500ms behind the write model — this is acceptable because the query is not changing state.

C#
// Event — immutable fact from the past
public record OrderPlacedEvent(
    Guid OrderId,
    Guid CustomerId,
    IReadOnlyList<OrderItem> Items,
    decimal TotalAmount,
    DateTime PlacedAtUtc);

// Command — instruction to perform an action
public record PlaceOrderCommand(
    Guid CustomerId,
    IReadOnlyList<OrderItem> Items,
    string ShippingAddress);

// Query — request for data, never changes state
public record GetOrderQuery(Guid OrderId);
public record GetOrderHistoryQuery(
    Guid CustomerId,
    int PageNumber,
    int PageSize);

Event Taxonomy

Not all events are created equal. Understanding the taxonomy prevents design errors:

The directionality of data flow also matters. Commands flow inward — from the API layer toward the domain model. Events flow outward — from the domain model toward the outside world. This unidirectional flow makes systems easier to reason about. When you see a consumer processing an event, you know it is reacting to a fact, not making a decision. When you see a handler processing a command, you know it is making a decision that may produce new events. This clarity is essential for debugging distributed systems where tracing the chain of causality across services is already challenging.

In practice, many systems use a combination of both patterns. An API gateway receives HTTP requests, translates them into commands, and dispatches them to the appropriate service. The service validates the command, executes business logic, and publishes events. Other services subscribe to those events and react accordingly. The command flow is synchronous within the service boundary (for immediate validation and response), while the event flow is asynchronous across service boundaries (for loose coupling and resilience).

C#
public class OrderService
{
    private readonly IEventPublisher _eventPublisher;
    private readonly IOrderRepository _orderRepo;

    public async Task<OrderResult> HandleAsync(
        PlaceOrderCommand command, CancellationToken ct)
    {
        // Validate the command
        var customer = await _customerRepo.GetByIdAsync(
            command.CustomerId, ct);
        if (customer == null)
            return OrderResult.Failure("Customer not found");

        // Execute business logic
        var order = Order.Create(
            command.CustomerId,
            command.Items,
            command.ShippingAddress);

        // Persist the state change
        await _orderRepo.AddAsync(order, ct);

        // Publish the event (fire and forget — resilience via Outbox)
        await _eventPublisher.PublishAsync(
            new OrderPlacedEvent(
                order.Id,
                order.CustomerId,
                order.Items,
                order.TotalAmount,
                DateTime.UtcNow),
            ct);

        return OrderResult.Success(order.Id);
    }
}

3. Event Sourcing Pattern

Event sourcing is a persistence pattern where the state of an entity is derived from a sequence of events rather than stored as a mutable row in a database. Instead of updating an orders table to set status = "shipped", you append an OrderShippedEvent to an event store. To reconstruct the current state of an order, you replay all events for that order from the beginning. This approach provides a complete, immutable audit trail of every state change — invaluable for debugging, compliance, and temporal queries.

The event store is the system of record. It is append-only — events are never updated or deleted. This immutability provides several guarantees: the event log is a perfect audit trail, events can be replayed to rebuild read models or fix bugs in event handlers, and temporal queries ("what was the order's status at 3 PM yesterday?") are trivial — just replay events up to that point. The tradeoff is that read operations require either maintaining projections (materialized views updated by event handlers) or replaying events at query time (expensive for entities with long event histories).

C#
// Event Store — append-only log of domain events
public class EventStore : IEventStore
{
    private readonly IDbConnection _db;

    public async Task AppendAsync<TEvent>(
        Guid aggregateId,
        TEvent @event,
        int expectedVersion,
        CancellationToken ct = default)
    {
        var sql = @"
            INSERT INTO event_store
                (aggregate_id, event_type, event_data,
                 version, occurred_at)
            VALUES
                (@aggregateId, @eventType, @eventData,
                 @version, @occurredAt)";

        // Optimistic concurrency: version check prevents
        // two concurrent appends from both succeeding
        var rows = await _db.ExecuteAsync(sql, new
        {
            aggregateId,
            eventType = typeof(TEvent).Name,
            eventData = JsonSerializer.Serialize(@event),
            version = expectedVersion,
            occurredAt = DateTime.UtcNow
        });

        if (rows == 0)
            throw new ConcurrencyException(
                $"Expected version {expectedVersion} " +
                $"conflict for aggregate {aggregateId}");
    }

    public async Task<IReadOnlyList<StoredEvent>>
        GetEventsAsync(Guid aggregateId,
            int fromVersion = 0,
            CancellationToken ct = default)
    {
        var sql = @"
            SELECT * FROM event_store
            WHERE aggregate_id = @aggregateId
              AND version > @fromVersion
            ORDER BY version ASC";
        return (await _db.QueryAsync<StoredEvent>(
            sql, new { aggregateId, fromVersion })).ToList();
    }
}

// Aggregate root rebuilt from events
public class OrderAggregate
{
    public Guid Id { get; private set; }
    public OrderStatus Status { get; private set; }
    public decimal TotalAmount { get; private set; }
    public int Version { get; private set; }
    private readonly List<object> _uncommittedEvents = new();

    public static OrderAggregate Load(
        Guid id, IEnumerable<StoredEvent> history)
    {
        var aggregate = new OrderAggregate { Id = id };
        foreach (var storedEvent in history)
        {
            var @event = JsonSerializer.Deserialize(
                storedEvent.EventType, storedEvent.EventData);
            aggregate.Apply(@event);
        }
        return aggregate;
    }

    private void Apply(object @event)
    {
        switch (@event)
        {
            case OrderPlacedEvent e:
                Status = OrderStatus.Placed;
                TotalAmount = e.TotalAmount;
                break;
            case OrderPaidEvent e:
                Status = OrderStatus.Paid;
                break;
            case OrderShippedEvent e:
                Status = OrderStatus.Shipped;
                break;
            case OrderCancelledEvent e:
                Status = OrderStatus.Cancelled;
                break;
        }
        Version++;
    }
}

Event Store Schema

SQL
CREATE TABLE event_store (
    event_id        UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_id    UUID NOT NULL,
    event_type      VARCHAR(255) NOT NULL,
    event_data      JSONB NOT NULL,
    metadata        JSONB,
    version         INTEGER NOT NULL,
    occurred_at     TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (aggregate_id, version)
);

CREATE INDEX idx_events_aggregate
    ON event_store (aggregate_id, version ASC);

CREATE INDEX idx_events_type_time
    ON event_store (event_type, occurred_at DESC);

-- For catching up projections efficiently
CREATE INDEX idx_events_unprocessed
    ON event_store (occurred_at)
    WHERE processed = FALSE;

Snapshots for Performance

Replaying thousands of events to reconstruct an aggregate is expensive. Snapshots capture the aggregate state at a point in time, allowing the system to load the snapshot and replay only events that occurred after it. A common strategy is to snapshot every 100 events. When loading an aggregate, the system checks for the latest snapshot and replays events from that version forward. For an order with 500 events, this reduces replay from 500 events to the last 100 — an 80% reduction in reconstruction time.

C#
public class SnapshotStore
{
    public async Task<OrderAggregate> LoadAsync(Guid orderId)
    {
        // Try loading from snapshot first
        var snapshot = await _db.QuerySingleOrDefaultAsync<
            OrderSnapshot>(
            "SELECT * FROM order_snapshots " +
            "WHERE order_id = @orderId " +
            "ORDER BY version DESC LIMIT 1",
            new { orderId });

        int fromVersion = 0;
        OrderAggregate aggregate;

        if (snapshot != null)
        {
            aggregate = OrderAggregate.LoadFromSnapshot(
                snapshot.State);
            fromVersion = snapshot.Version;
        }
        else
        {
            aggregate = new OrderAggregate { Id = orderId };
        }

        // Replay events after the snapshot
        var events = await _eventStore.GetEventsAsync(
            orderId, fromVersion);
        foreach (var storedEvent in events)
        {
            var @event = Deserialize(storedEvent);
            aggregate.Apply(@event);
        }

        return aggregate;
    }

    public async Task SaveSnapshotAsync(OrderAggregate order)
    {
        await _db.ExecuteAsync(
            "INSERT INTO order_snapshots " +
            "(order_id, version, state) " +
            "VALUES (@orderId, @version, @state)",
            new
            {
                orderId = order.Id,
                version = order.Version,
                state = JsonSerializer.Serialize(
                    order.ToSnapshotState())
            });
    }
}

4. CQRS (Command Query Responsibility Segregation)

CQRS separates the read model (optimized for queries) from the write model (optimized for business operations). In a traditional CRUD system, the same database table serves both reads and writes, leading to compromises: indexes that speed up reads slow down writes, normalization that prevents anomalies complicates queries, and the same schema must accommodate both operational and analytical workloads. CQRS eliminates these compromises by allowing each side to use the storage technology and schema design that best fits its purpose.

The write side receives commands, validates them against business rules, and produces events. It uses a domain model rich with behavior (entities, value objects, aggregates) and persists to an event store or normalized database optimized for transactional writes. The read side receives events from the write side and updates denormalized read models optimized for specific query patterns. An "OrderSummary" read model might combine data from orders, customers, and products into a single flat document that can be queried without joins.

C#
// Write side: Command handler produces events
public class PlaceOrderHandler : ICommandHandler<PlaceOrderCommand>
{
    private readonly IEventStore _eventStore;

    public async Task HandleAsync(
        PlaceOrderCommand command, CancellationToken ct)
    {
        var order = Order.Create(
            command.CustomerId,
            command.Items,
            command.ShippingAddress);

        // Business rules validation happens here
        order.ValidateBusinessRules();

        // Persist as events
        foreach (var @event in order.GetUncommittedEvents())
        {
            await _eventStore.AppendAsync(
                order.Id, @event, order.Version, ct);
        }
    }
}

// Read side: Event handler updates read model
public class OrderPlacedProjection :
    IEventHandler<OrderPlacedEvent>
{
    private readonly IDbConnection _db;

    public async Task HandleAsync(
        OrderPlacedEvent @event, CancellationToken ct)
    {
        // Denormalized read model — optimized for list queries
        var sql = @"
            INSERT INTO order_summaries
                (order_id, customer_id, customer_name,
                 total_amount, item_count, status,
                 placed_at, shipping_address)
            SELECT
                @orderId,
                c.id, c.name,
                @totalAmount,
                @itemCount,
                'Placed',
                @placedAt,
                @shippingAddress
            FROM customers c WHERE c.id = @customerId";

        await _db.ExecuteAsync(sql, new
        {
            @event.OrderId,
            @event.CustomerId,
            @event.TotalAmount,
            ItemCount = @event.Items.Count,
            @event.PlacedAt,
            ShippingAddress = "" // from event
        });
    }
}

CQRS with Event Sourcing

When CQRS Adds Value vs. Overhead

CQRS is valuable when read and write workloads have fundamentally different characteristics: when read patterns are complex (search, analytics, dashboards) while writes are simple domain operations, when read and write scaling requirements differ significantly (100K reads/sec vs. 1K writes/sec), or when different storage technologies serve each side better (Elasticsearch for search, PostgreSQL for transactions). CQRS adds overhead when the system is simple enough that a single database handles both adequately, when eventual consistency creates unacceptable user experience (a user places an order and immediately sees a "not found" page), or when the team lacks the operational maturity to manage multiple data stores and projection synchronization.

5. Message Brokers: Kafka, RabbitMQ, Azure Service Bus

The message broker is the backbone of any event-driven system. It decouples producers from consumers, absorbs traffic spikes, and guarantees message delivery even when consumers are temporarily unavailable. The choice of broker fundamentally shapes the architecture: Kafka provides a distributed commit log optimized for high throughput and replay, RabbitMQ offers flexible routing with traditional message queue semantics, and Azure Service Bus provides enterprise features like sessions and deduplication. Understanding the tradeoffs between these brokers is essential for making the right choice.

Broker Comparison

Apache Kafka Architecture

Kafka stores events in an immutable, append-only log partitioned across a cluster of brokers. Each partition is an ordered, immutable sequence of events. Producers append events to the end of a partition, and consumers read from any position in the log. This log-based architecture provides several critical properties: messages are never deleted after consumption (allowing replay), ordering is guaranteed within a partition, and multiple consumer groups can independently read the same log without affecting each other. The tradeoff is that Kafka requires careful partition management — choosing the wrong partition key can create hot partitions that bottleneck throughput.

C#
// Kafka producer with idempotent writes
public class KafkaEventPublisher : IEventPublisher
{
    private readonly IProducer<string, string> _producer;

    public async Task PublishAsync<T>(
        T @event, CancellationToken ct = default)
    {
        var topic = typeof(T).Name switch
        {
            nameof(OrderPlacedEvent) => "orders.events",
            nameof(PaymentProcessedEvent) => "payments.events",
            nameof(InventoryReservedEvent) => "inventory.events",
            _ => "generic.events"
        };

        // Partition key ensures all events for the same
        // aggregate go to the same partition (ordered)
        var partitionKey = ExtractPartitionKey(@event);
        var value = JsonSerializer.Serialize(@event);

        var message = new Message<string, string>
        {
            Key = partitionKey,
            Value = value,
            Headers = new Headers
            {
                { "event-type", Encoding.UTF8.GetBytes(
                    typeof(T).Name) },
                { "correlation-id", Encoding.UTF8.GetBytes(
                    Guid.NewGuid().ToString()) },
                { "timestamp", Encoding.UTF8.GetBytes(
                    DateTime.UtcNow.ToString("o")) }
            }
        };

        // Idempotent delivery — Kafka retries with
        // the same producer ID and sequence number
        var result = await _producer.ProduceAsync(
            topic, message, ct);

        if (result.Status != PersistenceStatus.Persisted)
            throw new EventPublishException(
                $"Event not persisted to partition " +
                $"{result.Partition}");
    }
}

// Kafka consumer with consumer group
public class KafkaEventConsumer : BackgroundService
{
    private readonly IConsumer<string, string> _consumer;
    private readonly IEventHandlerFactory _handlerFactory;

    protected override async Task ExecuteAsync(
        CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            var result = _consumer.Consume(ct);
            var eventType = result.Message.Headers
                .GetString("event-type");
            var handler = _handlerFactory.GetHandler(eventType);

            try
            {
                await handler.HandleAsync(
                    result.Message.Value, ct);
                _consumer.Commit(result);
            }
            catch (Exception ex) when (IsTransient(ex))
            {
                // Don't commit — will be retried
                _consumer.Assign(result.TopicPartitionOffset);
            }
            catch (Exception ex)
            {
                // Send to DLQ after max retries
                await SendToDlqAsync(result, ex, ct);
                _consumer.Commit(result);
            }
        }
    }
}

When to Choose Which Broker

Choose Kafka when you need high throughput (millions of events/sec), event replay capability, stream processing (Kafka Streams, ksqlDB), or when multiple consumer groups must independently process the same events. Kafka excels in data pipeline architectures, event sourcing backends, and real-time analytics. Choose RabbitMQ when you need complex routing logic (topic exchanges, header-based routing), traditional work queue semantics, or when message ordering per queue is sufficient. RabbitMQ is simpler to operate for small-to-medium workloads and excels in task distribution scenarios. Choose Azure Service Bus when you need enterprise features like sessions (ordered processing per entity), duplicate detection, auto-forwarding, or deep integration with Azure services. Service Bus is the pragmatic choice for .NET shops on Azure that need reliable messaging without managing broker infrastructure.

6. Event Schema Evolution & Versioning

Event schemas inevitably change as business requirements evolve. A field is added, a type changes, a field is renamed. In a synchronous API, you can version the API endpoint (/v1/, /v2/) and migrate clients incrementally. In an event-driven system, old events persist in the log forever, and consumers must be able to read events from any version. Schema evolution — the ability to change event schemas while maintaining backward and forward compatibility — is one of the most critical and underappreciated aspects of event-driven architecture.

Compatibility Matrix

The recommended approach for event schema management is Apache Avro with a schema registry. Avro schemas define the exact structure of each event version, and the schema registry enforces compatibility rules. When a producer registers a new schema version, the registry checks that it is backward compatible with the previous version. Consumers always deserialize using the latest schema they know about, and the schema registry handles translating between versions.

JSON
// Avro schema for OrderPlacedEvent v1
{
    "type": "record",
    "name": "OrderPlacedEvent",
    "namespace": "com.company.events",
    "fields": [
        { "name": "orderId", "type": "string" },
        { "name": "customerId", "type": "string" },
        { "name": "totalAmount", "type": "double" },
        { "name": "placedAtUtc", "type": "long",
          "logicalType": "timestamp-millis" }
    ]
}

// Avro schema for OrderPlacedEvent v2
// (backward compatible — new optional field with default)
{
    "type": "record",
    "name": "OrderPlacedEvent",
    "namespace": "com.company.events",
    "fields": [
        { "name": "orderId", "type": "string" },
        { "name": "customerId", "type": "string" },
        { "name": "totalAmount", "type": "double" },
        { "name": "placedAtUtc", "type": "long",
          "logicalType": "timestamp-millis" },
        { "name": "currency",
          "type": "string",
          "default": "USD" },
        { "name": "discountCode",
          "type": ["null", "string"],
          "default": null }
    ]
}

C#
// Schema registry client for versioned serialization
public class SchemaRegistryEventSerializer
{
    private readonly ISchemaRegistryClient _registry;
    private readonly ISerializer<byte[]> _avroSerializer;

    public async Task<byte[]> SerializeAsync<T>(
        T @event, CancellationToken ct = default)
    {
        var schemaId = await _registry.GetSchemaIdAsync(
            typeof(T).Name, ct);

        // Serialize with Avro and prepend schema ID
        var payload = _avroSerializer.Serialize(
            @event, schemaId);

        // Wire format: [magic byte][schema ID (4 bytes)][payload]
        var result = new byte[1 + 4 + payload.Length];
        result[0] = 0x0; // magic byte
        BitConverter.GetBytes(schemaId).CopyTo(result, 1);
        payload.CopyTo(result, 5);

        return result;
    }

    public async Task<T> DeserializeAsync<T>(
        byte[] data, CancellationToken ct = default)
    {
        var schemaId = BitConverter.ToInt32(data, 1);
        var payload = data.Skip(5).ToArray();

        // Registry resolves schema version and deserializes
        var schema = await _registry.GetSchemaAsync(
            schemaId, ct);
        return _avroSerializer.Deserialize<T>(
            payload, schema);
    }
}

// Event upcaster for legacy events without schema registry
public class OrderPlacedEventUpcaster : IEventUpcaster
{
    public bool CanUpcast(string eventType, int version)
        => eventType == "OrderPlacedEvent" && version == 1;

    public object Upcast(object oldEvent, int version)
    {
        if (version == 1 && oldEvent is OrderPlacedV1 v1)
        {
            return new OrderPlacedV2(
                v1.OrderId,
                v1.CustomerId,
                v1.TotalAmount,
                v1.PlacedAtUtc,
                Currency: "USD",        // default
                DiscountCode: null);     // default
        }
        throw new UpcastException($"Cannot upcast v{version}");
    }
}

7. Saga Pattern for Distributed Transactions

In a microservices architecture, a single business operation often spans multiple services. "Place an order" requires reserving inventory, processing payment, and creating a shipping record — each in a different service with its own database. Distributed transactions (2PC/XA) are technically possible but practically unusable at scale: they require all participants to be available simultaneously, create lock contention, and perform poorly across network boundaries. The Saga pattern provides an alternative: a sequence of local transactions, each updating its own service, with compensating transactions to undo changes if any step fails.

Choreography vs. Orchestration

Sagas can be implemented in two styles. Choreography is decentralized — each service publishes events and listens for events from other services, implementing its own reaction logic. Service A completes its work, publishes an event, and Service B picks up the event and does its part. There is no central coordinator. This approach is simple for small sagas with few steps but becomes difficult to manage as the number of steps grows — understanding the overall flow requires tracing events across multiple services.

Orchestration uses a central saga orchestrator that tells each service what to do. The orchestrator maintains the saga state and coordinates the sequence of steps. When a step fails, the orchestrator executes compensating transactions in reverse order. This approach is easier to understand, test, and debug because the entire flow is defined in one place. The tradeoff is that the orchestrator is a single point of complexity (though not a single point of failure if designed correctly).

C#
// Saga orchestrator with compensating transactions
public class PlaceOrderSaga : ISagaOrchestrator
{
    private readonly IEventPublisher _events;
    private readonly ISagaStateStore _stateStore;

    public async Task ExecuteAsync(
        PlaceOrderCommand command, CancellationToken ct)
    {
        var sagaId = Guid.NewGuid();
        var state = new PlaceOrderSagaState
        {
            SagaId = sagaId,
            OrderId = Guid.NewGuid(),
            CustomerId = command.CustomerId,
            Items = command.Items,
            Steps = new List<SagaStep>(),
            Status = SagaStatus.Running
        };

        try
        {
            // Step 1: Create order (compensate: cancel order)
            await ExecuteStepAsync(state, "CreateOrder",
                async () =>
                {
                    await _orderService.CreateAsync(
                        new CreateOrderCommand(
                            state.OrderId, state.CustomerId,
                            state.Items), ct);
                },
                async () =>
                {
                    await _orderService.CancelAsync(
                        state.OrderId, ct);
                }, ct);

            // Step 2: Reserve inventory
            // (compensate: release inventory)
            await ExecuteStepAsync(state, "ReserveInventory",
                async () =>
                {
                    await _inventoryService.ReserveAsync(
                        new ReserveInventoryCommand(
                            state.OrderId, state.Items), ct);
                },
                async () =>
                {
                    await _inventoryService.ReleaseAsync(
                        state.OrderId, ct);
                }, ct);

            // Step 3: Process payment
            // (compensate: refund payment)
            await ExecuteStepAsync(state, "ProcessPayment",
                async () =>
                {
                    await _paymentService.ChargeAsync(
                        new ChargePaymentCommand(
                            state.OrderId, state.CustomerId,
                            state.TotalAmount), ct);
                },
                async () =>
                {
                    await _paymentService.RefundAsync(
                        state.OrderId, ct);
                }, ct);

            // Step 4: Create shipment (no compensation needed)
            await ExecuteStepAsync(state, "CreateShipment",
                async () =>
                {
                    await _shippingService.CreateAsync(
                        new CreateShipmentCommand(
                            state.OrderId,
                            command.ShippingAddress), ct);
                },
                compensate: null, ct);

            state.Status = SagaStatus.Completed;
        }
        catch (Exception ex)
        {
            state.Status = SagaStatus.Failed;
            state.Error = ex.Message;

            // Execute compensating transactions
            // in reverse order
            await CompensateAsync(state, ct);
        }

        await _stateStore.SaveAsync(state, ct);
    }

    private async Task ExecuteStepAsync(
        PlaceOrderSagaState state,
        string stepName,
        Func<Task> action,
        Func<Task>? compensate,
        CancellationToken ct)
    {
        state.Steps.Add(new SagaStep
        {
            Name = stepName,
            Status = StepStatus.Running,
            StartedAt = DateTime.UtcNow
        });

        await action();

        state.Steps.Last().Status = StepStatus.Completed;
        state.Steps.Last().CompletedAt = DateTime.UtcNow;
    }

    private async Task CompensateAsync(
        PlaceOrderSagaState state, CancellationToken ct)
    {
        var completedSteps = state.Steps
            .Where(s =>
                s.Status == StepStatus.Completed &&
                s.Compensate != null)
            .Reverse();

        foreach (var step in completedSteps)
        {
            try
            {
                await step.Compensate!();
                step.Status = StepStatus.Compensated;
            }
            catch (Exception ex)
            {
                // Compensation failure — manual intervention needed
                step.Status = StepStatus.CompensationFailed;
                step.Error = ex.Message;
                await _alertService.AlertAsync(
                    $"Saga {state.SagaId}: " +
                    $"Compensation failed for {step.Name}",
                    ct);
            }
        }
    }
}

Saga Failure Scenarios

8. Outbox Pattern for Reliable Publishing

The outbox pattern solves a fundamental problem in event-driven systems: how to atomically update the database and publish an event. Without the outbox pattern, you face the dual-write problem: if the database write succeeds but the event publish fails, the database state and event log diverge. If the event publish succeeds but the database write fails, consumers receive events for state changes that never happened. The Transactional Outbox pattern stores events in an outbox table within the same database transaction as the business state change, then a separate process polls the outbox and publishes events to the broker.

C#
// Outbox table schema
// CREATE TABLE outbox (
//     id UUID PRIMARY KEY,
//     event_type VARCHAR(255) NOT NULL,
//     payload JSONB NOT NULL,
//     created_at TIMESTAMPTZ DEFAULT NOW(),
//     published BOOLEAN DEFAULT FALSE,
//     published_at TIMESTAMPTZ
// );

// Repository with atomic outbox write
public class OrderRepositoryWithOutbox : IOrderRepository
{
    private readonly IDbConnection _db;

    public async Task CreateOrderAsync(
        Order order,
        IReadOnlyList<object> events,
        CancellationToken ct = default)
    {
        // Both the order AND the events are written
        // in a single database transaction
        using var transaction = _db.BeginTransaction();

        try
        {
            // Write the business state
            await _db.ExecuteAsync(
                "INSERT INTO orders (id, customer_id, status) " +
                "VALUES (@id, @customerId, @status)",
                new { order.Id, order.CustomerId,
                      Status = order.Status.ToString() },
                transaction);

            // Write events to the outbox table
            foreach (var @event in events)
            {
                await _db.ExecuteAsync(
                    "INSERT INTO outbox " +
                    "(id, event_type, payload) " +
                    "VALUES (@id, @eventType, @payload)",
                    new
                    {
                        id = Guid.NewGuid(),
                        eventType = @event.GetType().Name,
                        payload = JsonSerializer.Serialize(@event)
                    },
                    transaction);
            }

            transaction.Commit();
            // Both succeeded or both failed — atomic
        }
        catch
        {
            transaction.Rollback();
            throw;
        }
    }
}

// Background poller that publishes outbox events
public class OutboxPoller : BackgroundService
{
    private readonly IDbConnection _db;
    private readonly IEventPublisher _publisher;
    private readonly TimeSpan _pollInterval =
        TimeSpan.FromSeconds(1);

    protected override async Task ExecuteAsync(
        CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            var unpublished = await _db.QueryAsync<
                OutboxEntry>(
                "SELECT * FROM outbox " +
                "WHERE published = FALSE " +
                "ORDER BY created_at ASC " +
                "LIMIT 100 " +
                "FOR UPDATE SKIP LOCKED");

            foreach (var entry in unpublished)
            {
                try
                {
                    await _publisher.PublishRawAsync(
                        entry.EventType, entry.Payload, ct);

                    await _db.ExecuteAsync(
                        "UPDATE outbox " +
                        "SET published = TRUE, " +
                        "    published_at = @now " +
                        "WHERE id = @id",
                        new { entry.Id,
                              Now = DateTime.UtcNow });
                }
                catch (Exception ex)
                {
                    // Log but don't throw — will retry on next poll
                    _logger.LogError(ex,
                        "Failed to publish outbox entry {Id}",
                        entry.Id);
                }
            }

            await Task.Delay(_pollInterval, ct);
        }
    }
}

Outbox Polling vs. CDC

9. Dead Letter Queues & Error Handling

Every event-driven system will encounter events that cannot be processed: malformed messages, transient failures that exhaust retries, schema mismatches, and business rule violations. Without a dead letter queue (DLQ), these poison messages block the consumer and potentially the entire processing pipeline. A DLQ is a separate queue where unprocessable events are sent after exhausting retry attempts, allowing the main queue to continue processing while operators investigate and resolve the problematic events.

Error Handling Pipeline

C#
public class ResilientEventHandler<T> : IEventHandler<T>
{
    private readonly IEventHandler<T> _inner;
    private readonly IDlqPublisher _dlq;
    private readonly IRetryPolicy _retryPolicy;
    private readonly int _maxRetries = 5;

    public async Task HandleAsync(
        T @event, EventContext context,
        CancellationToken ct)
    {
        var attempt = 0;
        Exception? lastException = null;

        while (attempt <= _maxRetries)
        {
            try
            {
                await _inner.HandleAsync(
                    @event, context, ct);
                return; // Success
            }
            catch (DeserializationException ex)
            {
                // Permanent — bad message format
                await _dlq.SendAsync(@event, context,
                    DlqReason.BadFormat, ex.Message, ct);
                return;
            }
            catch (BusinessRuleException ex)
            {
                // Permanent — business validation failed
                await _dlq.SendAsync(@event, context,
                    DlqReason.BusinessRule,
                    ex.Message, ct);
                return;
            }
            catch (Exception ex) when (
                IsTransient(ex) &
                attempt < _maxRetries)
            {
                lastException = ex;
                attempt++;
                var delay = _retryPolicy.GetDelay(attempt);
                await Task.Delay(delay, ct);
            }
            catch (Exception ex)
            {
                lastException = ex;
                break;
            }
        }

        // Exhausted all retries
        await _dlq.SendAsync(@event, context,
            DlqReason.MaxRetriesExceeded,
            lastException?.Message ?? "Unknown", ct);
    }

    private bool IsTransient(Exception ex)
        => ex is TimeoutException
            or HttpRequestException
            or ConnectionException
            or IOException;
}

public enum DlqReason
{
    BadFormat,
    BusinessRule,
    MaxRetriesExceeded,
    SchemaMismatch,
    Unknown
}

DLQ Operations

10. Event-Driven Microservices

Event-driven architecture is the natural companion to microservices. In a monolith, components communicate through method calls — fast, reliable, and strongly typed. When you decompose the monolith into microservices, those method calls become network calls — slow, unreliable, and loosely typed. Event-driven communication mitigates these problems: instead of Service A calling Service B synchronously, Service A publishes an event and Service B processes it asynchronously. The message broker absorbs network failures, handles retry logic, and decouples the lifecycle of producer and consumer.

Event-Driven Communication Patterns

C#
// Event-driven microservice registration
public static class ServiceCollectionExtensions
{
    public static IServiceCollection AddEventDrivenServices(
        this IServiceCollection services,
        IConfiguration config)
    {
        // Register the message broker
        services.AddSingleton<IMessageBroker>(sp =>
        {
            var logger = sp.GetRequiredService<
                ILogger<KafkaBroker>();
            return new KafkaBroker(
                config["Kafka:BootstrapServers"]!,
                logger);
        });

        // Register event handlers
        services.AddScoped<
            IEventHandler<OrderPlacedEvent>,
            SendOrderConfirmationHandler>();
        services.AddScoped<
            IEventHandler<OrderPlacedEvent>,
            UpdateInventoryHandler>();
        services.AddScoped<
            IEventHandler<PaymentProcessedEvent>,
            ShipOrderHandler>();

        // Register the consumer background service
        services.AddHostedService<EventConsumerService>();

        // Register the outbox poller
        services.AddHostedService<OutboxPoller>();

        return services;
    }
}

// Consumer service that routes events to handlers
public class EventConsumerService : BackgroundService
{
    private readonly IMessageBroker _broker;
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger<EventConsumerService> _logger;

    protected override async Task ExecuteAsync(
        CancellationToken ct)
    {
        // Subscribe to relevant topics
        await _broker.SubscribeAsync("order-events",
            async (message, ct) =>
        {
            using var scope = _scopeFactory.CreateScope();
            var eventType = message.Headers["event-type"];
            var handlers = scope.ServiceProvider
                .GetServices(typeof(IEventHandler<>)
                    .MakeGenericType(
                        Type.GetType(eventType)!));

            foreach (var handler in handlers)
            {
                var handleMethod = handler.GetType()
                    .GetMethod("HandleAsync")!;
                await (Task)handleMethod.Invoke(
                    handler,
                    new object[] { message.Payload, ct })!;
            }
        }, ct);
    }
}

Data Ownership in Event-Driven Microservices

Each microservice owns its data. No service directly accesses another service's database. Instead, services share data through events. When the Order Service needs customer information, it doesn't query the Customer Database — it subscribes to CustomerUpdatedEvents and maintains its own local copy (read model) of the customer data it needs. This data ownership principle has profound implications: each service can evolve its schema independently, scale its database independently, and choose the storage technology that best fits its needs. The tradeoff is eventual consistency — the Order Service's copy of customer data may be seconds behind the Customer Service's authoritative copy.

11. Real-World Case Studies: Netflix, Uber, Shopify

Understanding how major technology companies use event-driven architecture provides practical insights into the patterns, tradeoffs, and operational realities of EDA at scale. Each company faces unique challenges — Netflix handles streaming for 260 million subscribers, Uber processes millions of ride events per minute, and Shopify manages flash sales where traffic spikes 100x in seconds. Their solutions reveal the evolution of EDA from theoretical patterns to battle-tested production systems.

Netflix: Keystone — The Media Pipeline Platform

Netflix processes over 2 trillion events per day through their Keystone platform, which serves as the backbone for data processing, monitoring, and personalization. Every action a user takes — play, pause, search, browse, rate — generates an event that flows through Keystone. The platform is built on Apache Kafka and processes events in real-time to power recommendations, adjust encoding quality based on network conditions, and trigger content licensing analytics.

Netflix's key innovation is the "event mesh" — a layer that routes events between hundreds of microservices without requiring each service to know about every other service. The event mesh handles schema evolution, dead letter queues, and cross-region replication automatically. When a new service needs to consume existing events, it registers with the mesh and starts receiving events without requiring changes to the producers. This self-service model enables Netflix's 100+ engineering teams to independently develop and deploy services that participate in the event ecosystem.

Uber: Cadence — Durable Execution Platform

Uber built Cadence (now open-source, with the successor being Temporal) to handle the complex, long-running workflows inherent in ride-sharing: matching riders with drivers, managing surge pricing calculations, processing payments across multiple currencies, and coordinating driver onboarding. Cadence uses event sourcing internally — every workflow execution is a sequence of events that can be replayed to reconstruct the workflow's state at any point. This enables developers to write sequential code that internally executes as a durable, fault-tolerant event-driven workflow.

Uber's innovation is treating workflows as code rather than configuration. Instead of defining a DAG in YAML, you write regular C# or Go code with function calls. The Cadence/Temporal framework intercepts these function calls, records them as events, and replays them on recovery. This eliminates the impedance mismatch between "what the developer writes" and "what the workflow engine executes." Uber runs over 100 million workflow executions per day across hundreds of microservices, with workflow durations ranging from seconds to months (some compliance workflows run for 90+ days).

Shopify: Event-Driven Commerce at Flash-Sale Scale

Shopify's architecture must handle extreme traffic spikes during flash sales (Black Friday, product launches) where merchants process thousands of orders per second. Their event-driven system processes order events, inventory updates, payment confirmations, and shipping notifications across millions of merchants. The key challenge is tenant isolation — a flash sale on one merchant's store must not impact the performance of other merchants' stores.

Shopify uses a combination of Kafka for event streaming and Redis for real-time state management. When a customer places an order, the event flows through a pipeline: inventory reservation → payment processing → order confirmation → shipping label generation → notification dispatch. Each step is an independent consumer that processes events from its own Kafka topic, enabling independent scaling. During flash sales, Shopify auto-scales their Kafka consumer groups to handle 10x normal throughput while maintaining per-merchant rate limiting to prevent any single merchant from overwhelming the system.

C#
// Pattern inspired by Netflix's event mesh
public class EventMeshRouter
{
    private readonly IEventSubscriptionRegistry _registry;
    private readonly IMessageBroker _broker;

    public async Task RouteAsync(
        IntegrationEvent @event, CancellationToken ct)
    {
        // Find all subscribers for this event type
        var subscribers = await _registry
            .GetSubscribersAsync(@event.GetType().Name);

        foreach (var subscriber in subscribers)
        {
            // Filter: does this subscriber want this
            // specific event instance?
            if (subscriber.Filter != null &&
                !subscriber.Filter.Matches(@event))
                continue;

            var message = new Message
            {
                Topic = subscriber.Topic,
                Key = @event.AggregateId.ToString(),
                Payload = Serialize(@event),
                Headers = new Dictionary<string, string>
                {
                    ["event-type"] = @event.GetType().Name,
                    ["event-version"] = @event.SchemaVersion,
                    ["correlation-id"] = @event.CorrelationId,
                    ["source-service"] = @event.SourceService
                }
            };

            await _broker.PublishAsync(message, ct);
        }
    }
}

public interface IEventFilter
{
    bool Matches(IntegrationEvent @event);
}

// Filter: only events from US merchants
public class RegionFilter : IEventFilter
{
    private readonly string _region;
    public bool Matches(IntegrationEvent @event)
        => @event.Metadata.TryGetValue("region", out var r)
           && r == _region;
}

12. Testing Event-Driven Systems

Testing event-driven systems is fundamentally harder than testing synchronous systems. In a synchronous call, you send a request and observe the response — the test boundary is clear. In an event-driven system, the action (publishing an event) and the reaction (processing by consumers) are separated by time, message brokers, and potentially other services. Tests must account for eventual consistency, message ordering, duplicate delivery, and the asynchronous nature of event processing.

Testing Pyramid for EDA

C#
// Unit test for event handler — no broker dependency
[TestClass]
public class OrderPlacedHandlerTests
{
    [TestMethod]
    public async Task Should_Send_Confirmation_Email()
    {
        // Arrange
        var emailService = new Mock<IEmailService>();
        var handler = new SendOrderConfirmationHandler(
            emailService.Object);

        var @event = new OrderPlacedEvent(
            OrderId: Guid.NewGuid(),
            CustomerId: Guid.NewGuid(),
            Items: new[] { new OrderItem("Widget", 2, 9.99m) },
            TotalAmount: 19.98m,
            PlacedAtUtc: DateTime.UtcNow);

        // Act
        await handler.HandleAsync(@event,
            CancellationToken.None);

        // Assert
        emailService.Verify(x =>
            x.SendAsync(
                It.IsAny<string>(),
                It.Is<string>(body =>
                    body.Contains(@event.OrderId.ToString())),
                It.IsAny<CancellationToken>()),
            Times.Once);
    }
}

// Integration test with Testcontainers
[TestClass]
public class KafkaEventFlowTests : IAsyncLifetime
{
    private GenericContainer _kafka;
    private IProducer<string, string> _producer;
    private IConsumer<string, string> _consumer;

    public async Task InitializeAsync()
    {
        _kafka = new GenericBuilder<GenericContainer>()
            .WithImage("confluentinc/cp-kafka:7.5.0")
            .WithPortBinding(9092)
            .WithWaitStrategy(
                Wait.ForUnixContainer()
                    .UntilPortIsAvailable(9092))
            .Build();
        await _kafka.StartAsync();

        // Setup producer and consumer
        _producer = new ProducerBuilder<string, string>(
            new ProducerConfig
            {
                BootstrapServers =
                    $"localhost:{_kafka.GetMappedPublicPort(9092)}"
            }).Build();
    }

    [TestMethod]
    public async Task Should_Deliver_Event_To_Consumer()
    {
        // Arrange
        var @event = new OrderPlacedEvent(
            Guid.NewGuid(), Guid.NewGuid(),
            Array.Empty<OrderItem>(),
            29.99m, DateTime.UtcNow);

        // Act — publish
        await _producer.ProduceAsync("orders.events",
            new Message<string, string>
            {
                Key = @event.OrderId.ToString(),
                Value = JsonSerializer.Serialize(@event)
            });

        // Assert — consumer receives it
        var result = _consumer.Consume(
            TimeSpan.FromSeconds(10));
        var received = JsonSerializer.Deserialize<
            OrderPlacedEvent>(result.Message.Value);
        Assert.AreEqual(@event.OrderId, received.OrderId);
    }

    public async Task DisposeAsync()
    {
        await _kafka.StopAsync();
    }
}

// Contract test for event schema compatibility
[TestClass]
public class EventSchemaContractTests
{
    [TestMethod]
    public void OrderPlacedEvent_V2_Should_Be_Backward_Compatible()
    {
        var registry = new SchemaRegistry();
        var v1Schema = registry.GetSchema(
            "OrderPlacedEvent", version: 1);
        var v2Schema = registry.GetSchema(
            "OrderPlacedEvent", version: 2);

        var compatibility = registry
            .CheckCompatibility(v1Schema, v2Schema);

        Assert.AreEqual(
            CompatibilityStatus.Compatible,
            compatibility.Status);
    }

    [TestMethod]
    public void V1_Event_Should_Deserialize_With_V2_Schema()
    {
        var v1Event = new OrderPlacedV1(
            "order-123", "customer-456", 99.99,
            DateTime.UtcNow);

        var serialized = AvroSerializer.Serialize(
            v1Event, schemaVersion: 1);

        var deserialized = AvroDeserializer.Deserialize<
            OrderPlacedV2>(serialized, schemaVersion: 2);

        Assert.AreEqual("order-123", deserialized.OrderId);
        Assert.AreEqual("USD", deserialized.Currency);
    }
}

Chaos Testing Event-Driven Systems

C#
// Chaos test: verify event delivery survives broker restart
[TestMethod]
public async Task Should_Deliver_All_Events_After_Broker_Restart()
{
    // Arrange
    var events = Enumerable.Range(0, 100)
        .Select(i => new TestEvent($"event-{i}"))
        .ToList();

    // Publish first batch
    foreach (var e in events.Take(50))
        await PublishEventAsync(e);

    // Chaos: restart the broker
    await _kafkaContainer.RestartAsync();

    // Publish second batch
    foreach (var e in events.Skip(50))
        await PublishEventAsync(e);

    // Assert: consumer eventually receives all 100 events
    var received = await ConsumeAllAsync(
        timeout: TimeSpan.FromSeconds(30));
    CollectionAssert.AreEquivalent(
        events.Select(e => e.Id).ToList(),
        received.Select(e => e.Id).ToList());
}

13. Monitoring & Observability

Observability in event-driven systems requires tracking events across service boundaries. A single user action might generate events processed by 5 different services, flowing through 3 Kafka topics, and persisting in 4 databases. Without proper observability, debugging "why did this order not ship?" requires manually searching through logs across multiple services — a process that can take hours. The three pillars of observability — metrics, logs, and traces — must be augmented with event-specific monitoring: consumer lag, event throughput, schema evolution tracking, and dead letter queue growth.

Essential EDA Metrics

C#
// Structured logging for event processing
public class ObservableEventHandler<T> : IEventHandler<T>
{
    private readonly IEventHandler<T> _inner;
    private readonly ILogger<ObservableEventHandler<T>> _logger;
    private readonly Meter _meter;
    private readonly Counter<long> _eventsProcessed;
    private readonly Histogram<double> _processingDuration;

    public ObservableEventHandler(
        IEventHandler<T> inner,
        ILogger<ObservableEventHandler<T>> logger)
    {
        _inner = inner;
        _logger = logger;
        _meter = new Meter($"events.{typeof(T).Name}");
        _eventsProcessed = _meter.CreateCounter<long>(
            "events.processed");
        _processingDuration = _meter.CreateHistogram<double>(
            "events.processing_duration_ms");
    }

    public async Task HandleAsync(
        T @event, EventContext context,
        CancellationToken ct)
    {
        var sw = Stopwatch.StartNew();

        _logger.LogInformation(
            "Processing event {EventType} " +
            "with ID {EventId} from topic {Topic}",
            typeof(T).Name, context.EventId,
            context.Topic);

        try
        {
            await _inner.HandleAsync(@event, context, ct);
            sw.Stop();

            _eventsProcessed.Add(1,
                new KeyValuePair<string, object?>(
                    "status", "success"));
            _processingDuration.Record(
                sw.ElapsedMilliseconds,
                new KeyValuePair<string, object?>(
                    "status", "success"));

            _logger.LogInformation(
                "Event {EventType} processed successfully " +
                "in {Duration}ms",
                typeof(T).Name, sw.ElapsedMilliseconds);
        }
        catch (Exception ex)
        {
            sw.Stop();

            _eventsProcessed.Add(1,
                new KeyValuePair<string, object?>(
                    "status", "error"));
            _processingDuration.Record(
                sw.ElapsedMilliseconds,
                new KeyValuePair<string, object?>(
                    "status", "error"));

            _logger.LogError(ex,
                "Event {EventType} processing failed " +
                "after {Duration}ms: {Error}",
                typeof(T).Name, sw.ElapsedMilliseconds,
                ex.Message);
            throw;
        }
    }
}

// Distributed tracing across event chains
public class EventTracingMiddleware
{
    private readonly RequestDelegate _next;

    public async Task InvokeAsync(HttpContext context)
    {
        // Extract trace context from event headers
        var traceId = context.Request.Headers["trace-id"]
            .FirstOrDefault();
        var spanId = context.Request.Headers["span-id"]
            .FirstOrDefault();

        using var activity = ActivitySource
            .StartActivity("ProcessEvent");
        activity?.SetTag("trace.id", traceId);
        activity?.SetTag("parent.span.id", spanId);

        await _next(context);
    }
}

Distributed Tracing for Events

In synchronous systems, distributed tracing (OpenTelemetry, Jaeger) works by propagating trace context in HTTP headers. In event-driven systems, the trace context must be propagated through event metadata. When a producer publishes an event, it includes the current trace ID and span ID in the event headers. When a consumer processes the event, it extracts the trace context and creates a child span. This maintains the causal chain across service boundaries, enabling end-to-end tracing of a request that spans synchronous API calls and asynchronous event processing.

14. Performance & Scalability

Event-driven systems must handle massive throughput while maintaining low latency. Performance bottlenecks in EDA typically occur at three points: the broker (partition throughput limits), the consumer (processing speed limits), and the database (write throughput limits). Understanding these bottlenecks and designing for them is essential for building systems that scale.

Kafka Performance Tuning

C#
// High-performance batch consumer
public class BatchEventConsumer : BackgroundService
{
    private readonly IConsumer<byte[], byte[]> _consumer;
    private readonly IEventHandlerFactory _handlerFactory;
    private readonly int _batchSize = 500;
    private readonly TimeSpan _batchTimeout =
        TimeSpan.FromSeconds(5);

    protected override async Task ExecuteAsync(
        CancellationToken ct)
    {
        while (!ct.IsCancellationRequested)
        {
            // Fetch a batch of messages
            var batch = _consumer.ConsumeBatch(
                _batchTimeout, _batchSize);

            if (!batch.Any()) continue;

            // Group by partition for ordered processing
            var byPartition = batch
                .GroupBy(x => x.Partition);

            // Process partitions in parallel
            var tasks = byPartition.Select(partitionGroup =>
                ProcessPartitionAsync(
                    partitionGroup.Key,
                    partitionGroup.ToList(),
                    ct));

            await Task.WhenAll(tasks);

            // Commit the entire batch
            var lastOffset = batch.Last()
                .Offset.Value;
            _consumer.Commit(new TopicPartitionOffset(
                batch.First().Topic,
                batch.First().Partition,
                lastOffset + 1));
        }
    }

    private async Task ProcessPartitionAsync(
        int partition,
        List<ConsumeResult<byte[], byte[]>> messages,
        CancellationToken ct)
    {
        foreach (var msg in messages)
        {
            var eventType = Encoding.UTF8.GetString(
                msg.Message.Headers
                    .GetLastBytes("event-type"));
            var handler = _handlerFactory
                .GetHandler(eventType);
            var payload = JsonSerializer.Deserialize(
                msg.Message.Value, handler.EventType);

            await handler.HandleAsync(payload, ct);
        }
    }
}

Consumer Scaling Strategy

Consumer parallelism in Kafka is bounded by the number of partitions. A consumer group with 12 partitions can have at most 12 consumers — the 13th consumer sits idle. To scale consumers, you must first scale partitions. This means partition count is a capacity planning decision that must be made early. A common mistake is starting with too few partitions (4-6) and discovering that you need 50 consumers to handle the load but can only run 6. Start with 12-64 partitions per topic and plan for partition count to increase over time.

Database Optimization for Event Systems

Event-driven systems create heavy write workloads on databases: event store appends, outbox table writes, read model updates, and projection rebuilds. Key optimization strategies include: connection pooling (use a pool of 50-100 connections rather than opening a new connection per query), batch inserts (buffer multiple writes and insert them in a single INSERT statement), write-behind caching (buffer writes in Redis and flush to the database periodically), and table partitioning (partition large tables by time to keep index sizes manageable and enable fast archival).

C#
// Batch writer for high-throughput event store
public class BatchedEventStoreWriter
{
    private readonly IDbConnection _db;
    private readonly Channel<EventEntry> _buffer;
    private readonly int _batchSize = 500;
    private readonly TimeSpan _flushInterval =
        TimeSpan.FromMilliseconds(100);

    public async Task<ValueTask> WriteAsync(
        EventEntry entry, CancellationToken ct)
    {
        await _buffer.Writer.WriteAsync(entry, ct);
        return default;
    }

    protected override async Task ExecuteAsync(
        CancellationToken ct)
    {
        var batch = new List<EventEntry>();

        while (!ct.IsCancellationRequested)
        {
            // Collect batch (either batch size or timeout)
            var timeout = Task.Delay(_flushInterval, ct);
            while (batch.Count < _batchSize)
            {
                if (await Task.WhenAny(
                    _buffer.Reader.WaitToReadAsync(ct),
                    timeout) == timeout) break;

                if (_buffer.Reader.TryRead(out var entry))
                    batch.Add(entry);
            }

            if (batch.Count == 0) continue;

            // Single batch INSERT — much faster than
            // individual inserts
            var sql = "INSERT INTO event_store " +
                "(aggregate_id, event_type, payload, " +
                " version, occurred_at) VALUES " +
                string.Join(",",
                    batch.Select((_, i) =
                        $"(@agg{i}, @type{i}, @payload{i}, " +
                        $"@ver{i}, @time{i})"));

            var parameters = new DynamicParameters();
            for (int i = 0; i < batch.Count; i++)
            {
                parameters.Add($"agg{i}",
                    batch[i].AggregateId);
                parameters.Add($"type{i}",
                    batch[i].EventType);
                parameters.Add($"payload{i}",
                    batch[i].Payload);
                parameters.Add($"ver{i}",
                    batch[i].Version);
                parameters.Add($"time{i}",
                    batch[i].OccurredAt);
            }

            await _db.ExecuteAsync(sql, parameters);

            // Mark entries as written
            foreach (var entry in batch)
                entry.CompletionSource?.TrySetResult(true);

            batch.Clear();
        }
    }
}

15. Cost Estimation

Understanding the cost of event-driven infrastructure is essential for making architectural decisions and building business cases. The primary cost drivers are the message broker (Kafka is the most common choice), compute resources for producers and consumers, storage for the event log and projections, and network traffic between services and the broker.

Monthly Cost Breakdown (Production Deployment)

Cost Optimization Strategies

Cost vs. Architecture Decision

16. Interview Q&A Deep Dive

Event-driven architecture is a frequent topic in senior and staff-level system design interviews. Interviewers test not just your knowledge of patterns but your ability to reason about tradeoffs, handle edge cases, and make pragmatic decisions. The following questions cover the most common interview topics with production-grade answers that demonstrate senior-level thinking.

Q1: How do you handle duplicate events in an event-driven system?

Answer: Duplicate events are inevitable in distributed systems — Kafka provides at-least-once delivery by default, network retries can produce duplicates, and producer retries may send the same event twice. The defense is two-fold: idempotent consumers and deduplication at the consumer level. An idempotent consumer produces the same result regardless of how many times it processes the same event. For example, "send order confirmation email" should check if a confirmation was already sent for this order ID before sending again. For non-idempotent operations, maintain a deduplication table keyed by event ID: before processing, check if the event ID already exists in the deduplication table. If yes, skip processing. If no, insert the event ID and process the event. The deduplication table uses a TTL-based expiration (e.g., 7 days) to prevent unbounded growth. Kafka transactions provide exactly-once semantics across partitions, but they add complexity — idempotent consumers are simpler and more reliable.

Q2: How do you ensure event ordering guarantees?

Answer: Kafka guarantees ordering within a single partition. The key design decision is the partition key: all events for the same aggregate (e.g., same order ID) must go to the same partition, ensuring they are processed in order. However, ordering across partitions or across different aggregate IDs is not guaranteed. For most business use cases, per-aggregate ordering is sufficient: all events for Order #123 arrive in order, but Order #123 and Order #456 may be interleaved. For scenarios requiring global ordering (rare), use a single partition — but this limits throughput to what one partition can handle (typically 10MB/s). The practical approach is to design consumers that don't need global ordering: use idempotent processing, handle out-of-order events gracefully, and use timestamps or version numbers in events to detect and handle stale events.

C#
// Idempotent consumer with deduplication
public class IdempotentEventHandler : IEventHandler<OrderPlacedEvent>
{
    private readonly IDbConnection _db;

    public async Task HandleAsync(
        OrderPlacedEvent @event,
        CancellationToken ct)
    {
        // Check deduplication table
        var exists = await _db.ExecuteScalarAsync<bool>(
            "SELECT EXISTS(SELECT 1 FROM processed_events " +
            "WHERE event_id = @eventId)",
            new { eventId = @event.EventId });

        if (exists) return; // Already processed — skip

        // Process the event
        await ProcessOrderAsync(@event, ct);

        // Mark as processed (within the same transaction)
        await _db.ExecuteAsync(
            "INSERT INTO processed_events (event_id, processed_at) " +
            "VALUES (@eventId, @now)",
            new { @event.EventId, Now = DateTime.UtcNow });
    }
}

Q3: How do you design an event-driven system that can be debugged easily?

Answer: Debugging event-driven systems requires three things: correlation IDs, structured logging, and event store access. Every event should carry a correlation ID that traces the entire request chain — from the initial HTTP request through all event processing steps. This correlation ID is propagated in event metadata and included in all log entries. Structured logging (JSON format) with consistent fields (event_id, correlation_id, service_name, timestamp) enables centralized log search across all services. An event store or event history view allows operators to see every event that occurred for a specific aggregate within a time window. Tools like Jaeger or Tempo provide distributed tracing that shows the causal chain of events across services. Finally, maintain a "replay capability" — the ability to replay events for a specific aggregate from the event store, which helps reproduce bugs without re-executing the original trigger.

Q4: How do you handle event schema changes in a production system?

Answer: Schema changes must be backward compatible — consumers that understand the old schema must still work with new events. The process: (1) Add new fields with default values to the schema. (2) Register the new schema version with the schema registry, ensuring backward compatibility is enforced. (3) Deploy consumers that understand both old and new schemas (they use the schema registry to resolve the version). (4) Deploy producers that emit the new schema version. (5) After all consumers are updated, optionally deprecate old fields (but never remove them). For breaking changes (field type change, field removal), create a new event type (OrderPlacedV2Event) rather than modifying the existing schema. Event upcasters handle transforming old event versions into the current schema during replay. The schema registry is the single source of truth for event contracts — any schema change must go through the registry's compatibility checks.

Q5: How do you handle backpressure in an event-driven system?

Answer: Backpressure occurs when consumers cannot process events as fast as they are produced. Without backpressure handling, the broker fills up, producers block or fail, and the system degrades. Strategies: (1) Kafka consumer groups — add more consumers (up to the partition count) to increase processing throughput. (2) Buffering — Kafka's log-based architecture naturally buffers events. Consumers lag behind but events are not lost. Monitor consumer lag and alert when it exceeds thresholds. (3) Rate limiting on producers — if the system cannot keep up, slow down event production. This is a last resort but prevents cascading failures. (4) Priority queues — route high-priority events to dedicated topics with dedicated consumer groups, ensuring critical events are processed even during backpressure. (5) Batch processing — consumers can process events in batches (100-500 at a time) rather than one-by-one, amortizing overhead and increasing throughput. The key insight is that backpressure in Kafka is visible (consumer lag metric) and recoverable (events are retained until consumed), unlike TCP backpressure which drops packets.

Q6: Compare event-driven architecture with synchronous REST APIs.

Answer: The fundamental difference is coupling. REST APIs create temporal coupling (the caller must wait for the response) and availability coupling (if the callee is down, the caller fails). Event-driven architecture eliminates both: producers emit events without waiting for consumers, and the broker absorbs consumer unavailability. REST is simpler to develop, debug, and test — you call an endpoint and get a response. EDA is more resilient and scalable but harder to debug, test, and reason about consistency. REST works well for CRUD operations and request-response patterns. EDA works well for notifications, data replication, workflow orchestration, and scenarios where multiple services need to react to the same data change. In practice, most systems use both: REST for synchronous user-facing operations (place order, check status) and EDA for asynchronous backend operations (update inventory, send notifications, update analytics). The key is choosing the right boundary: what needs an immediate response (REST) versus what can happen asynchronously (EDA).

Q7: How do you test an event-driven system end-to-end?

Answer: End-to-end testing of event-driven systems requires a test infrastructure that includes a real message broker (use Testcontainers for Kafka/RabbitMQ in integration tests). The test flow: (1) Set up a test Kafka cluster using Docker/Testcontainers. (2) Publish events to test topics. (3) Verify that consumers process events correctly by checking the read model or side effects. (4) Test failure scenarios: kill a consumer, restart the broker, send malformed events. The challenge is handling eventual consistency in assertions — use polling with timeouts instead of immediate assertions. For contract testing, use Pact or a schema registry to verify that producers and consumers agree on event schemas. For chaos testing, use tools like Chaos Mesh to inject failures (network partitions, process kills) and verify the system recovers correctly. The most important tests are the ones that verify "the system eventually reaches the correct state" even when individual components fail.

Key Concepts to Master

Pre-Interview Checklist

• Know the difference between events, commands, and queries
• Explain event sourcing and when it adds value (audit trail, temporal queries)
• Discuss CQRS with and without event sourcing
• Compare Kafka, RabbitMQ, and Azure Service Bus tradeoffs
• Design schema evolution with Avro and a schema registry
• Implement the Saga pattern (orchestration vs. choreography)
• Explain the Outbox pattern and CDC alternatives
• Handle dead letter queues and operational hygiene
• Discuss consumer scaling with Kafka partitions
• Design idempotent consumers with deduplication
• Explain distributed tracing across event chains
• Handle backpressure and consumer lag scenarios
• Test event-driven systems (integration, contract, chaos)
• Estimate Kafka cluster sizing for given throughput requirements