1. Introduction & Why Microservices Exist

Microservices architecture has become the dominant paradigm for building large-scale, complex software systems. But understanding microservices requires understanding the problem they solve. As software systems grow, they face mounting pressure on multiple fronts: the codebase becomes too large for a single team to reason about, deployments slow down because unrelated changes must be coordinated, different subsystems have conflicting scaling requirements (one part needs 10x CPU, another needs 10x memory), and organizational boundaries in the engineering team do not align with the monolithic code structure. These pressures are what drive the decomposition of a monolith into microservices.

The core idea of microservices is deceptively simple: decompose a large application into smaller, independently deployable services that each own a specific business capability. Each service runs in its own process, owns its own data store, and communicates with other services over the network through well-defined APIs. But simplicity of the concept masks enormous complexity in practice. Distributed systems introduce network unreliability (services become unreachable at any moment), data consistency challenges (you can no longer use a single database transaction across multiple services), operational overhead (you now need to deploy, monitor, and debug dozens or hundreds of services instead of one), and debugging difficulty (a single user request may traverse five services, and a failure in any one of them can cause the entire request to fail).

The economic and organizational arguments for microservices are compelling at scale. Netflix decomposed its monolith into over 700 microservices to support thousands of engineers working independently. Amazon moved from a monolithic architecture to a service-oriented architecture in the early 2000s, famously mandating that all teams communicate only through service interfaces — the "API mandate." Uber evolved from a monolithic Python application ("Uber 1.0") to a microservices architecture with over 4,000 services to support its global expansion. Each of these companies made the transition when the monolithic architecture became a bottleneck for engineering velocity, not because microservices were fashionable.

Real-World Case Studies

Understanding how major companies adopted microservices provides practical insights into the tradeoffs involved:

Shopify's approach is particularly noteworthy. Rather than jumping directly to microservices, they built a "module packager" that enforces module boundaries within their monolith. Modules have defined interfaces and cannot access each other's internals. This gives many of the organizational benefits of microservices (independent team ownership, clear boundaries) without the operational overhead of distributed systems. Only when a module genuinely needs independent scaling or deployment is it extracted into a separate service. This "modular monolith first" strategy is increasingly popular among mature engineering organizations.

2. The Monolith-First Strategy

Before diving into microservices architecture, it is critical to understand why the industry consensus has shifted toward starting with a monolith. Martin Fowler's famous advice — "Almost all the successful microservice stories have started with a monolith that got too big and was broken up" — reflects hard-won experience across hundreds of organizations. The reason is straightforward: microservices introduce distributed systems complexity (network failures, eventual consistency, distributed debugging) that a monolith avoids entirely. If your team is small (under 10 engineers) and your codebase is manageable (under 500K lines), a monolith gives you faster development velocity, simpler debugging, and zero network overhead between components.

A well-structured monolith should have clear internal module boundaries that map to potential future service boundaries. These modules communicate through in-process function calls (nanoseconds) instead of network calls (milliseconds). Transactions span the entire database atomically. Deployment is a single artifact. Debugging is straightforward — you can step through the entire request lifecycle in a single debugger session. The monolith's internal structure should follow domain-driven design principles, with each module owning a specific business capability and exposing a clear interface to other modules.

The Right Time to Extract Services

Signs that your monolith is ready for decomposition include: (1) Deployment conflicts: Multiple teams cannot deploy independently because their changes are intertwined in the same codebase. (2) Scaling asymmetry: One part of the system needs 10x more resources than another, but you can only scale the entire monolith. (3) Reliability requirements: One subsystem requires five-nines availability while others can tolerate occasional downtime. (4) Technology lock-in: A subsystem would benefit from a different technology stack (e.g., a machine learning pipeline needs Python/TensorFlow while the rest is C#/.NET). (5) Team autonomy: Teams need to own their services end-to-end, from development to production, without depending on other teams.

C#
// Monolith with clear module boundaries — ready for extraction
public class OrderModule
{
    // Internal interface — only accessible within the module
    internal interface IOrderRepository
    {
        Task<Order> GetByIdAsync(Guid orderId);
        Task<Order> SaveAsync(Order order);
    }

    // Public interface — used by other modules (future service API)
    public interface IOrderService
    {
        Task<OrderDto> GetOrderAsync(Guid orderId);
        Task<Guid> CreateOrderAsync(CreateOrderCommand command);
        Task CancelOrderAsync(Guid orderId, string reason);
    }

    // This module owns the Orders table — no other module touches it
    // When extracted, this becomes the Order Service's database
}

3. Service Decomposition & Domain-Driven Design

Decomposing a monolith into microservices is the most consequential architectural decision you will make. Done well, it produces a system where teams can work independently, services can scale independently, and failures are contained. Done poorly, it produces a distributed monolith — all the costs of microservices (network latency, operational complexity, eventual consistency) with none of the benefits (independent deployment, fault isolation, team autonomy). The key to good decomposition is aligning service boundaries with business domain boundaries, not with technical layers.

Domain-Driven Design (DDD) provides the intellectual framework for identifying the right service boundaries. The central concept is the bounded context — a section of the domain where a particular model applies and where all team members speak the same language. Each bounded context becomes a candidate microservice. The "Ubiquitous Language" within a bounded context ensures that developers, domain experts, and product managers use the same terms to describe the same concepts. When two teams use the word "customer" to mean different things (one means "a person who buys" and the other means "an account that is billed"), they are in different bounded contexts and should have separate services.

Bounded Context Identification Process

Identifying bounded contexts requires understanding the business domain deeply. The process involves: (1) Event Storming — gather domain experts and developers to map all business events on a timeline, (2) Identify aggregates — clusters of events that relate to the same business entity, (3) Define boundaries — draw lines around related aggregates that share a consistent model, (4) Verify independence — ensure each bounded context can function independently without synchronous calls to other contexts.

Decomposition Patterns

C# Example: Bounded Context Models

C#
// Order Context — this is a complete, self-contained model
namespace OrderService.Domain
{
    public class Order
    {
        public Guid OrderId { get; private set; }
        public Guid CustomerId { get; private set; } // Reference ID only — no navigation property
        public List<OrderItem> Items { get; private set; }
        public OrderStatus Status { get; private set; }
        public Money TotalAmount { get; private set; }

        private readonly List<IDomainEvent> _domainEvents = new();
        public IReadOnlyList<IDomainEvent> DomainEvents => _domainEvents.AsReadOnly();

        public static Order Create(Guid customerId, List<OrderItemRequest> items)
        {
            var order = new Order
            {
                OrderId = Guid.NewGuid(),
                CustomerId = customerId,
                Items = items.Select(i => OrderItem.Create(i.ProductId, i.Quantity, i.Price)).ToList(),
                Status = OrderStatus.Created
            };
            order.TotalAmount = order.Items.Aggregate(
                Money.Zero("USD"),
                (sum, item) => sum + item.LineTotal);

            // Domain event — other services react to this
            order._domainEvents.Add(new OrderCreatedEvent(
                order.OrderId, order.CustomerId, order.TotalAmount));

            return order;
        }

        public void Confirm()
        {
            if (Status != OrderStatus.Created)
                throw new InvalidOperationException($"Cannot confirm order in {Status} status");
            Status = OrderStatus.Confirmed;
            _domainEvents.Add(new OrderConfirmedEvent(OrderId));
        }
    }

    // Payment Context — different model, different service, different database
    namespace PaymentService.Domain
    {
        public class Payment
        {
            public Guid PaymentId { get; private set; }
            public Guid OrderId { get; private set; } // Reference ID — not a foreign key
            public decimal Amount { get; private set; }
            public PaymentStatus Status { get; private set; }

            public static Payment Initiate(Guid orderId, decimal amount)
            {
                return new Payment
                {
                    PaymentId = Guid.NewGuid(),
                    OrderId = orderId,
                    Amount = amount,
                    Status = PaymentStatus.Initiated
                };
            }
        }
    }
}

4. High-Level Architecture Overview

A production microservices architecture consists of several infrastructure layers that work together: the client layer (web apps, mobile apps, third-party consumers), the edge layer (API gateway, load balancer), the service layer (the actual microservices), the communication layer (service mesh, message brokers), and the data layer (databases, caches, object stores). Understanding how these layers interact is essential before designing individual services.

Request Lifecycle: Place Order

Understanding a complete request lifecycle through the microservices architecture clarifies how all the pieces work together. When a user places an order, the request flows through the following path:

1. Client → API Gateway: The web app sends POST /api/orders to the API gateway. The gateway authenticates the JWT token, rate-limits the request, and routes it to the Order Service.
2. Order Service → Create Order: The Order Service validates the request, creates an Order aggregate with status "Created," publishes an OrderCreatedEvent to Kafka, and returns 202 Accepted with the order ID. The response does not wait for payment or inventory.
3. Payment Service ← OrderCreatedEvent: The Payment Service consumes the event, initiates a payment with the payment processor, and publishes a PaymentCompletedEvent or PaymentFailedEvent.
4. Inventory Service ← OrderCreatedEvent: Simultaneously, the Inventory Service consumes the event and reserves the requested items. It publishes an InventoryReservedEvent.
5. Order Service ← Events: The Order Service consumes PaymentCompletedEvent and InventoryReservedEvent. When both succeed, it transitions the order to "Confirmed" and publishes OrderConfirmedEvent.
6. Shipping Service ← OrderConfirmedEvent: The Shipping Service creates a shipment and begins the fulfillment process.
7. Notification Service ← OrderConfirmedEvent: The Notification Service sends a confirmation email to the customer.

This event-driven flow decouples all the services. The Order Service does not need to know about the Payment Service or Inventory Service directly. It simply publishes events and reacts to events published by others. If the Payment Service is temporarily down, the order waits in a pending state until the service recovers. If the Shipping Service is slow, it does not block the order confirmation. This temporal decoupling is one of the primary benefits of event-driven microservices.

Architecture Decision Record

5. Inter-Service Communication Patterns

Communication between microservices is fundamentally a tradeoff between simplicity and resilience. Synchronous communication (HTTP REST, gRPC) is simpler to reason about — a service sends a request and waits for a response — but creates temporal coupling: if the downstream service is slow or unavailable, the caller is blocked. Asynchronous communication (message queues, event streams) decouples services in time — the producer does not wait for the consumer — but introduces complexity: eventual consistency, message ordering challenges, duplicate delivery handling, and harder debugging.

Synchronous Communication: gRPC vs REST

For synchronous service-to-service communication, gRPC is generally preferred over REST within a microservices cluster. gRPC uses Protocol Buffers for serialization (3-10x faster than JSON), supports streaming (bidirectional, server-side, client-side), and generates strongly-typed client code from .proto definitions. HTTP/2 multiplexing allows multiple requests over a single connection, reducing connection overhead. REST remains the better choice for public-facing APIs where broad client compatibility and human readability matter.

C#
// gRPC service definition — shared between Order Service and Payment Service
// order.proto
syntax = "proto3";
package orderservice;

service OrderService {
    rpc GetOrder (GetOrderRequest) returns (OrderResponse);
    rpc CreateOrder (CreateOrderRequest) returns (OrderResponse);
    rpc StreamOrderUpdates (StreamRequest) returns (stream OrderUpdate);
}

message OrderResponse {
    string order_id = 1;
    string customer_id = 2;
    string status = 3;
    repeated OrderItem items = 4;
    double total_amount = 5;
    google.protobuf.Timestamp created_at = 6;
}

// Server implementation in Order Service
public class OrderGrpcService : OrderService.OrderServiceBase
{
    private readonly IOrderRepository _repo;
    private readonly ILogger<OrderGrpcService> _logger;

    public override async Task<OrderResponse> GetOrder(
        GetOrderRequest request, ServerCallContext context)
    {
        var orderId = Guid.Parse(request.OrderId);
        var order = await _repo.GetByIdAsync(orderId);

        if (order == null)
            throw new RpcException(new Status(
                StatusCode.NotFound, $"Order {orderId} not found"));

        return new OrderResponse
        {
            OrderId = order.OrderId.ToString(),
            CustomerId = order.CustomerId.ToString(),
            Status = order.Status.ToString(),
            TotalAmount = (double)order.TotalAmount.Amount
        };
    }

    // Streaming — Payment Service subscribes to order status changes
    public override async Task StreamOrderUpdates(
        StreamRequest request,
        IServerStreamWriter<OrderUpdate> responseStream,
        ServerCallContext context)
    {
        await foreach (var update in _orderUpdateChannel
            .Reader.ReadAllAsync(context.CancellationToken))
        {
            await responseStream.WriteAsync(new OrderUpdate
            {
                OrderId = update.OrderId.ToString(),
                NewStatus = update.Status.ToString(),
                Timestamp = Timestamp.FromDateTime(update.At.ToUniversalTime())
            });
        }
    }
}

Asynchronous Communication: Event-Driven Architecture

For asynchronous communication, Apache Kafka is the industry standard for high-throughput event streaming. Each service publishes domain events to Kafka topics, and other services subscribe to the topics they care about. Kafka provides durability (events are persisted to disk), ordering (within a partition), replay (consumers can re-read historical events), and horizontal scaling (partitioning by key distributes load across consumers).

C#
// Domain event publisher — Order Service publishes events
public class DomainEventPublisher : IDomainEventPublisher
{
    private readonly IProducer<string, DomainEvent> _producer;

    public async Task PublishAsync<T>(T domainEvent, CancellationToken ct)
        where T : IDomainEvent
    {
        var message = new Message<string, DomainEvent>
        {
            Key = domainEvent.AggregateId.ToString(), // Ensures ordering per aggregate
            Value = domainEvent,
            Headers = new Headers
            {
                { "event-type", Encoding.UTF8.GetBytes(typeof(T).Name) },
                { "correlation-id", Encoding.UTF8.GetBytes(domainEvent.CorrelationId) },
                { "timestamp", Encoding.UTF8.GetBytes(DateTime.UtcNow.ToString("O")) }
            }
        };

        // Kafka topic naming: {domain}.{entity}.{event-name}
        var topic = $"orders.order.{domainEvent.GetType().Name}";
        await _producer.ProduceAsync(topic, message, ct);
    }
}

// Event consumer — Payment Service consumes OrderCreatedEvent
public class OrderCreatedConsumer : BackgroundService
{
    private readonly IConsumer<string, OrderCreatedEvent> _consumer;
    private readonly IPaymentService _paymentService;

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        _consumer.Subscribe("orders.order.ordercreatedevent");

        while (!ct.IsCancellationRequested)
        {
            var result = _consumer.Consume(ct);
            var orderEvent = result.Message.Value;

            try
            {
                await _paymentService.InitiatePaymentAsync(
                    orderEvent.OrderId,
                    orderEvent.TotalAmount,
                    orderEvent.CustomerId);

                _consumer.Commit(result); // Manual commit — only after successful processing
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Failed to process OrderCreated for {OrderId}",
                    orderEvent.OrderId);
                // Nack → message returns to topic for retry
                _consumer.StoreOffset(result);
            }
        }
    }
}

Communication Pattern Comparison

The Request-Reply Pattern with Correlation IDs

When a service needs data from another service in an event-driven architecture, it can use the request-reply pattern: publish a request event and listen for a reply event with the same correlation ID. This maintains decoupling while enabling queries across service boundaries. The correlation ID is a GUID that travels through all events and logs, enabling end-to-end tracing of a single business operation across multiple services.

C#
// Request-reply pattern with correlation IDs and in-memory channel
public class RequestReplyClient
{
    private readonly IEventBus _eventBus;
    private readonly ConcurrentDictionary<Guid, TaskCompletionSource<object>>
        _pendingRequests = new();

    public async Task<TResponse> SendRequestAsync<TRequest, TResponse>(
        TRequest request, TimeSpan timeout)
    {
        var correlationId = Guid.NewGuid();
        var tcs = new TaskCompletionSource<object>();
        _pendingRequests[correlationId] = tcs;

        // Publish request event
        await _eventBus.PublishAsync(new RequestEnvelope<TRequest>
        {
            CorrelationId = correlationId,
            Payload = request,
            ReplyTo = "payment-service.replies"
        });

        // Wait for reply or timeout
        using var cts = new CancellationTokenSource(timeout);
        cts.Token.Register(() => tcs.TrySetCanceled());

        var reply = (ReplyEnvelope<TResponse>)await tcs.Task;
        return reply.Payload;
    }

    // Called by event consumer when reply arrives
    public void HandleReply(Guid correlationId, object reply)
    {
        if (_pendingRequests.TryRemove(correlationId, out var tcs))
            tcs.TrySetResult(reply);
    }
}

6. API Gateway & Edge Services

The API gateway is the single entry point for all external clients. It handles cross-cutting concerns that would otherwise be duplicated across every service: authentication and authorization, rate limiting, SSL termination, request routing, response aggregation, request/response transformation, and API versioning. Without a gateway, every client must know about every service, handle authentication independently, and manage different API versions — a maintenance nightmare at scale.

The gateway pattern has evolved. The traditional "fat gateway" handles all logic in a single component, which becomes a bottleneck and single point of failure. The modern approach uses a thin gateway that handles only routing and authentication, pushing business logic to the services themselves. For more complex edge concerns (aggregation, protocol translation), a dedicated Backend-for-Frontend (BFF) layer sits between the gateway and the services.

API Gateway Responsibilities

C# API Gateway Implementation

C#
// Minimal API Gateway with routing, auth, and rate limiting
public class ApiGateway
{
    private readonly IRouteConfig _routes;
    private readonly ITokenValidator _tokenValidator;
    private readonly IRateLimiter _rateLimiter;

    public async Task<HttpResponseMessage> HandleRequestAsync(
        HttpRequestMessage request)
    {
        // 1. Extract route
        var path = request.RequestUri.AbsolutePath;
        var route = _routes.Match(path, request.Method);
        if (route == null)
            return new HttpResponseMessage(HttpStatusCode.NotFound);

        // 2. Authenticate
        if (route.RequiresAuth)
        {
            var token = request.Headers.Authorization?.Parameter;
            var principal = await _tokenValidator.ValidateAsync(token);
            if (principal == null)
                return new HttpResponseMessage(HttpStatusCode.Unauthorized);
            request.Headers.Add("X-User-Id", principal.FindFirst("sub")?.Value);
        }

        // 3. Rate limit
        var clientId = request.Headers.GetValues("X-Client-Id").FirstOrDefault() ?? "anonymous";
        if (!await _rateLimiter.AllowAsync(clientId, route.RateLimit))
            return new HttpResponseMessage(HttpStatusCode.TooManyRequests);

        // 4. Forward to service
        var serviceUrl = route.ServiceUrl + path;
        var handler = new HttpClientHandler { UseCookies = false };
        using var client = new HttpClient(handler);
        var response = await client.SendAsync(request);

        // 5. Add gateway headers
        response.Headers.Add("X-Gateway-Timestamp", DateTime.UtcNow.ToString("O"));

        return response;
    }
}

// Route configuration
public record RouteConfig(
    string Path,
    string ServiceUrl,
    string Method,
    bool RequiresAuth,
    int RateLimit = 100);

Backend-for-Frontend (BFF) Pattern

The BFF pattern creates dedicated gateway layers for different client types. A mobile BFF returns compact responses with only the fields mobile apps need. A web BFF returns full responses. An internal BFF aggregates data from multiple services for administrative dashboards. Each BFF is owned by the team that builds the corresponding client, giving them full control over the API shape without affecting other clients or services.

C#
// BFF for mobile — returns compact response
[ApiController]
[Route("api/mobile")]
public class MobileBffController : ControllerBase
{
    private readonly IOrderServiceClient _orders;
    private readonly IPaymentServiceClient _payments;

    [HttpGet("orders/{orderId}")]
    public async Task<MobileOrderDto> GetOrder(string orderId)
    {
        // Parallel fan-out to multiple services
        var orderTask = _orders.GetOrderAsync(orderId);
        var paymentTask = _payments.GetPaymentForOrderAsync(orderId);

        await Task.WhenAll(orderTask, paymentTask);

        // Compact DTO — only fields mobile needs
        return new MobileOrderDto
        {
            Id = orderTask.Result.Id,
            Status = orderTask.Result.Status,
            Total = orderTask.Result.TotalAmount,
            PaymentStatus = paymentTask.Result.Status,
            // No unnecessary fields — reduces mobile data usage
        };
    }
}

7. Service Discovery & Load Balancing

In a microservices architecture, service instances are dynamic — they come and go as the system scales, deploys, and recovers from failures. A service cannot hardcode the IP addresses of its dependencies because those addresses change constantly. Service discovery solves this problem by maintaining a registry of available service instances and providing mechanisms for services to find each other dynamically. In Kubernetes environments, the platform provides built-in DNS-based service discovery, but understanding the underlying patterns is essential for debugging and for non-Kubernetes deployments.

Discovery Patterns

Load Balancing Algorithms

Once a service instance is discovered, the load balancer must choose which instance to route the request to. Different algorithms optimize for different characteristics:

C#
// Client-side load balancer with multiple strategies
public class ServiceLoadBalancer
{
    private readonly ConcurrentDictionary<string, List<ServiceInstance>>
        _instances = new();
    private int _roundRobinIndex = 0;

    // Round Robin — simplest, works well when instances are homogeneous
    public ServiceInstance GetRoundRobin(string serviceName)
    {
        var instances = _instances[serviceName];
        var index = Interlocked.Increment(ref _roundRobinIndex) % instances.Count;
        return instances[index];
    }

    // Least Connections — sends to instance with fewest active requests
    // Better when request durations vary significantly
    public ServiceInstance GetLeastConnections(string serviceName)
    {
        return _instances[serviceName]
            .OrderBy(i => i.ActiveConnections)
            .First();
    }

    // Weighted Round Robin — for heterogeneous instances
    // Instance with 4 CPU gets weight 4, instance with 2 CPU gets weight 2
    public ServiceInstance GetWeighted(string serviceName)
    {
        var instances = _instances[serviceName];
        var totalWeight = instances.Sum(i => i.Weight);
        var random = Random.Shared.Next(totalWeight);

        var cumulative = 0;
        foreach (var instance in instances)
        {
            cumulative += instance.Weight;
            if (random < cumulative) return instance;
        }
        return instances.Last();
    }

    // Consistent Hashing — same client always hits same instance
    // Best for caching — maximizes cache hits
    public ServiceInstance GetConsistentHash(string serviceName, string key)
    {
        var instances = _instances[serviceName];
        var hash = MurmurHash3.Hash(Encoding.UTF8.GetBytes(key));
        var index = hash % instances.Count;
        return instances[index];
    }
}

Health Checks & Instance Lifecycle

Service discovery is only useful if the registry reflects reality — unhealthy instances must be removed quickly and healthy instances added promptly. Kubernetes provides two types of health checks: liveness probes (is the process alive? restart if not) and readiness probes (can the service handle traffic? remove from Service endpoints if not). In non-Kubernetes environments, services must implement their own health check mechanisms.

C#
// Health check implementation for a microservice
public class ServiceHealthCheck : IHealthCheck
{
    private readonly IDatabase _db;
    private readonly IMessageBroker _broker;
    private readonly IClock _clock;

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken ct = default)
    {
        var checks = new Dictionary<string, HealthCheckResult>();

        // Database connectivity
        try
        {
            await _db.ExecuteAsync("SELECT 1", ct);
            checks["database"] = HealthCheckResult.Healthy();
        }
        catch (Exception ex)
        {
            checks["database"] = HealthCheckResult.Unhealthy($"DB: {ex.Message}");
        }

        // Message broker connectivity
        try
        {
            await _broker.IsConnectedAsync(ct);
            checks["kafka"] = HealthCheckResult.Healthy();
        }
        catch (Exception ex)
        {
            checks["kafka"] = HealthCheckResult.Degraded($"Kafka: {ex.Message}");
        }

        // Determine overall status
        var worstStatus = checks.Values
            .Select(c => c.Status)
            .Max();

        return worstStatus switch
        {
            HealthStatus.Healthy => HealthCheckResult.Healthy("All systems operational"),
            HealthStatus.Degraded => HealthCheckResult.Degraded("Partial degradation"),
            _ => HealthCheckResult.Unhealthy("Critical dependency failure")
        };
    }
}

8. Database-per-Service & Data Management

The database-per-service pattern is one of the most important and most challenging aspects of microservices architecture. Each service owns its data exclusively — no other service can directly access or modify its database. This rule enforces loose coupling: services interact only through published APIs and events, never through shared database tables. The benefit is clear: each service can evolve its data schema independently, scale its database independently, and use the database technology best suited to its data model (PostgreSQL for relational data, MongoDB for documents, Redis for caches, Elasticsearch for search).

The challenge is equally clear: how do services share data when they cannot share databases? The answer is data replication through events. When a service's data changes, it publishes a domain event. Other services that need that data subscribe to the events and maintain their own local copies. These local copies may be slightly stale (eventual consistency), but they are always eventually consistent if the event stream is working correctly.

Data Access Patterns

CQRS: Command Query Responsibility Segregation

CQRS separates the read model from the write model. The write side handles commands (create order, update inventory) and stores data in a normalized, transactional database optimized for writes. The read side handles queries (get order details, list orders) and reads from a denormalized, query-optimized database that is kept in sync via events. This separation allows each side to be optimized independently — writes can use a normalized PostgreSQL schema with strong consistency, while reads can use an Elasticsearch index optimized for search or a Redis cache optimized for fast reads.

C#
// CQRS — Write side: Order Command Handler
public class CreateOrderHandler : IRequestHandler<CreateOrderCommand, OrderDto>
{
    private readonly OrderWriteDbContext _writeDb;
    private readonly IDomainEventPublisher _eventPublisher;

    public async Task<OrderDto> Handle(
        CreateOrderCommand command, CancellationToken ct)
    {
        // Write to normalized database
        var order = Order.Create(command.CustomerId, command.Items);
        _writeDb.Orders.Add(order);
        await _writeDb.SaveChangesAsync(ct);

        // Publish event — read side updates its denormalized view
        await _eventPublisher.PublishAsync(new OrderCreatedEvent
        {
            OrderId = order.OrderId,
            CustomerId = order.CustomerId,
            Items = order.Items.Select(i => new OrderItemDto
            {
                ProductId = i.ProductId,
                Quantity = i.Quantity,
                Price = i.Price
            }).ToList(),
            TotalAmount = order.TotalAmount.Amount,
            CreatedAt = order.CreatedAt
        }, ct);

        return order.ToDto();
    }
}

// CQRS — Read side: denormalized view optimized for queries
public class OrderReadModel
{
    public Guid OrderId { get; set; }
    public Guid CustomerId { get; set; }
    public string CustomerName { get; set; } // Denormalized — joined from User Service events
    public string Status { get; set; }
    public decimal TotalAmount { get; set; }
    public List<OrderItemReadModel> Items { get; set; }
    public string ShippingAddress { get; set; } // Denormalized — from Address Service events
    public DateTime CreatedAt { get; set; }
}

// Read model projector — updates denormalized view from events
public class OrderProjectedHandler : INotificationHandler<OrderCreatedEvent>
{
    private readonly OrderReadDbContext _readDb;
    private readonly IUserServiceClient _userService;

    public async Task Handle(OrderCreatedEvent notification, CancellationToken ct)
    {
        // Fetch denormalized data from other services
        var customer = await _userService.GetByIdAsync(notification.CustomerId);

        var readModel = new OrderReadModel
        {
            OrderId = notification.OrderId,
            CustomerId = notification.CustomerId,
            CustomerName = customer.FullName, // Denormalized from User Service
            Status = "Created",
            TotalAmount = notification.TotalAmount,
            CreatedAt = notification.CreatedAt
        };

        _readDb.Orders.Add(readModel);
        await _readDb.SaveChangesAsync(ct);
    }
}

9. Saga Pattern & Distributed Transactions

The Saga pattern is the primary mechanism for maintaining data consistency across microservices without using distributed transactions (2PC). A saga is a sequence of local transactions where each transaction publishes a domain event that triggers the next transaction. If any step fails, compensating transactions undo the previous steps. Sagas can be orchestrated (a central coordinator directs the flow) or choreographed (each service listens for events and decides what to do next). Both approaches have tradeoffs that must be carefully evaluated.

Orchestration vs Choreography

Orchestrated Saga Implementation

C#
// Order Saga Orchestrator — manages the complete order flow
public class OrderSagaOrchestrator
{
    private readonly IOrderServiceClient _orders;
    private readonly IPaymentServiceClient _payments;
    private readonly IInventoryServiceClient _inventory;
    private readonly IShippingServiceClient _shipping;
    private readonly ISagaStateStore _stateStore;

    public async Task<SagaResult> ExecuteAsync(CreateOrderCommand command)
    {
        var sagaId = Guid.NewGuid();
        var state = new OrderSagaState(sagaId, command);

        try
        {
            // Step 1: Create order
            state.Status = SagaStatus.CreatingOrder;
            await _stateStore.SaveAsync(state);
            var orderId = await _orders.CreateAsync(command);
            state.OrderId = orderId;

            // Step 2: Reserve inventory
            state.Status = SagaStatus.ReservingInventory;
            await _stateStore.SaveAsync(state);
            await _inventory.ReserveAsync(orderId, command.Items);
            state.InventoryReserved = true;

            // Step 3: Process payment
            state.Status = SagaStatus.ProcessingPayment;
            await _stateStore.SaveAsync(state);
            await _payments.ChargeAsync(orderId, command.TotalAmount);
            state.PaymentCompleted = true;

            // Step 4: Confirm order
            state.Status = SagaStatus.ConfirmingOrder;
            await _stateStore.SaveAsync(state);
            await _orders.ConfirmAsync(orderId);

            state.Status = SagaStatus.Completed;
            await _stateStore.SaveAsync(state);
            return SagaResult.Success(orderId);
        }
        catch (Exception ex)
        {
            // Compensate in reverse order
            await CompensateAsync(state);
            state.Status = SagaStatus.Failed;
            state.Error = ex.Message;
            await _stateStore.SaveAsync(state);
            return SagaResult.Failure(ex.Message);
        }
    }

    private async Task CompensateAsync(OrderSagaState state)
    {
        // Reverse each completed step
        if (state.PaymentCompleted)
            await _payments.RefundAsync(state.OrderId);

        if (state.InventoryReserved)
            await _inventory.ReleaseAsync(state.OrderId);

        if (state.OrderId.HasValue)
            await _orders.CancelAsync(state.OrderId.Value, "Saga compensation");
    }
}

Choreographed Saga Implementation

C#
// Each service listens for events and acts independently — no central orchestrator
// Order Service — listens for payment and inventory events
public class OrderEventHandlers :
    INotificationHandler<PaymentCompletedEvent>,
    INotificationHandler<InventoryReservedEvent>,
    INotificationHandler<PaymentFailedEvent>
{
    private readonly IOrderRepository _repo;

    public async Task Handle(PaymentCompletedEvent notification, CancellationToken ct)
    {
        var order = await _repo.GetByIdAsync(notification.OrderId);
        order.MarkPaymentReceived();
        await _repo.SaveAsync(order);

        if (order.AllDependenciesSatisfied)
        {
            order.Confirm();
            await _repo.SaveAsync(order);
            // OrderConfirmedEvent is published automatically via domain events
        }
    }

    public async Task Handle(InventoryReservedEvent notification, CancellationToken ct)
    {
        var order = await _repo.GetByIdAsync(notification.OrderId);
        order.MarkInventoryReserved();
        await _repo.SaveAsync(order);

        if (order.AllDependenciesSatisfied)
        {
            order.Confirm();
            await _repo.SaveAsync(order);
        }
    }

    public async Task Handle(PaymentFailedEvent notification, CancellationToken ct)
    {
        // Compensation — cancel order, release inventory
        var order = await _repo.GetByIdAsync(notification.OrderId);
        order.Cancel("Payment failed");
        await _repo.SaveAsync(order);
        // OrderCancelledEvent triggers inventory release
    }
}

// Saga state tracking — Order aggregates dependency status
public class Order
{
    public bool InventoryReserved { get; private set; }
    public bool PaymentCompleted { get; private set; }
    public bool AllDependenciesSatisfied => InventoryReserved && PaymentCompleted;

    public void MarkPaymentReceived() => PaymentCompleted = true;
    public void MarkInventoryReserved() => InventoryReserved = true;
}

Saga Failure Handling

Saga compensations must themselves be idempotent. If a compensation fails (e.g., the refund API is temporarily down), the saga must retry the compensation. This creates a potential deadlock: a saga stuck in compensation cannot free its resources (reserved inventory, held payments) until the compensation succeeds. The solution is a "saga monitor" that periodically checks for stuck sagas and escalates them to human operators after a configurable timeout (typically 30 minutes). The monitor also implements retry with exponential backoff for failed compensations.

10. Resilience Patterns & Fault Tolerance

In a microservices architecture, network failures are not exceptional — they are expected. A service must be designed to handle the failure of any dependency gracefully. Resilience patterns ensure that a single service failure does not cascade into a system-wide outage. The four essential patterns are: circuit breaker (stop calling a failing service), retry with backoff (retry transient failures gracefully), bulkhead (isolate failures to prevent resource exhaustion), and timeout (prevent waiting indefinitely for a slow service).

Circuit Breaker Pattern

The circuit breaker monitors failures to a downstream service. When failures exceed a threshold, the circuit "opens" and subsequent calls fail immediately without making a network call. After a configurable timeout, the circuit enters "half-open" state and allows a single probe request. If the probe succeeds, the circuit closes; if it fails, the circuit opens again. This pattern prevents cascading failures and gives failing services time to recover.

C#
// Circuit breaker with three states: Closed, Open, Half-Open
public class CircuitBreaker
{
    private CircuitBreakerState _state = CircuitBreakerState.Closed;
    private int _failureCount = 0;
    private DateTime _lastFailureTime;
    private readonly int _failureThreshold = 5;
    private readonly TimeSpan _openDuration = TimeSpan.FromSeconds(30);

    public async Task<T> ExecuteAsync<T>(Func<Task<T>> action)
    {
        if (_state == CircuitBreakerState.Open)
        {
            if (DateTime.UtcNow - _lastFailureTime > _openDuration)
                _state = CircuitBreakerState.HalfOpen;
            else
                throw new CircuitBreakerOpenException(
                    "Circuit is open — failing fast");
        }

        try
        {
            var result = await action();

            // Success — reset failure count
            if (_state == CircuitBreakerState.HalfOpen)
            {
                _state = CircuitBreakerState.Closed;
                _failureCount = 0;
            }

            return result;
        }
        catch (Exception ex)
        {
            _failureCount++;
            _lastFailureTime = DateTime.UtcNow;

            if (_failureCount >= _failureThreshold)
                _state = CircuitBreakerState.Open;

            if (_state == CircuitBreakerState.HalfOpen)
                _state = CircuitBreakerState.Open; // Probe failed, reopen

            throw;
        }
    }
}

// Usage — wrap every inter-service call
public class ResilientPaymentClient : IPaymentServiceClient
{
    private readonly CircuitBreaker _breaker = new();
    private readonly HttpClient _client;

    public async Task<PaymentResult> ChargeAsync(
        Guid orderId, decimal amount)
    {
        return await _breaker.ExecuteAsync(async () =>
        {
            var response = await _client.PostAsJsonAsync("/api/payments/charge", new
            {
                OrderId = orderId,
                Amount = amount
            });
            response.EnsureSuccessStatusCode();
            return await response.Content.ReadFromJsonAsync<PaymentResult>();
        });
    }
}

Retry with Exponential Backoff & Jitter

C#
// Polly-based retry with exponential backoff and jitter
public static ResiliencePipeline<T> CreateRetryPipeline<T>()
{
    return new ResiliencePipelineBuilder<T>()
        .AddRetry(new RetryStrategyOptions<T>
        {
            MaxRetryAttempts = 3,
            Delay = TimeSpan.FromSeconds(1),
            BackoffType = DelayBackoffType.Exponential,
            UseJitter = true, // Prevents thundering herd
            ShouldHandle = new PredicateBuilder<T>()
                .Handle<HttpRequestException>()
                .Handle<TimeoutRejectedException>(),
            OnRetry = args =>
            {
                // Log each retry for observability
                Console.WriteLine(
                    $"Retry {args.AttemptNumber} after {args.RetryDelay}");
                return ValueTask.CompletedTask;
            }
        })
        .AddTimeout(TimeSpan.FromSeconds(10)) // Global timeout per call
        .Build();
}

Resilience Pattern Comparison

11. Caching Strategies Across Services

Caching in a microservices architecture is more complex than in a monolith because the data is spread across multiple services and databases. Each service can cache its own data, but cross-service data (like an order page that needs data from Order, Payment, User, and Shipping services) requires coordination. The API gateway can implement response caching for frequently accessed endpoints, individual services can cache their database queries, and a shared Redis cluster can serve as a distributed cache for data that is accessed across services.

Cache Patterns

C# Implementation with Event-Driven Invalidation

C#
// Cache-aside with event-driven invalidation
public class CachedProductRepository : IProductRepository
{
    private readonly IProductRepository _inner;
    private readonly IDistributedCache _cache;
    private readonly ILogger<CachedProductRepository> _logger;

    private static readonly TimeSpan CacheDuration = TimeSpan.FromMinutes(5);

    public async Task<Product> GetByIdAsync(Guid productId)
    {
        var cacheKey = $"product:{productId}";

        // Check cache first
        var cached = await _cache.GetStringAsync(cacheKey);
        if (cached != null)
        {
            _logger.LogDebug("Cache hit for product {ProductId}", productId);
            return JsonSerializer.Deserialize<Product>(cached);
        }

        // Cache miss — fetch from database
        _logger.LogDebug("Cache miss for product {ProductId}", productId);
        var product = await _inner.GetByIdAsync(productId);

        if (product != null)
        {
            await _cache.SetStringAsync(cacheKey,
                JsonSerializer.Serialize(product),
                new DistributedCacheEntryOptions
                {
                    AbsoluteExpirationRelativeToNow = CacheDuration
                });
        }

        return product;
    }

    public async Task UpdateAsync(Product product)
    {
        await _inner.UpdateAsync(product);

        // Invalidate cache — next read will fetch fresh data
        await _cache.RemoveAsync($"product:{product.Id}");
    }
}

// Event-driven cache invalidation consumer
public class ProductCacheInvalidator : INotificationHandler<ProductUpdatedEvent>
{
    private readonly IDistributedCache _cache;

    public async Task Handle(ProductUpdatedEvent notification, CancellationToken ct)
    {
        // When any service updates a product, invalidate the cache
        await _cache.RemoveAsync($"product:{notification.ProductId}");
    }
}

12. Observability: Logging, Metrics & Tracing

Debugging a distributed system is exponentially harder than debugging a monolith. A single user request in a microservices architecture may traverse five to ten services. When that request fails, you need to know which service failed, why, what the input was, what the intermediate state was, and whether the failure was due to the service itself or a downstream dependency. Observability — the ability to understand the internal state of a system from its external outputs — is not optional in microservices. It is a survival requirement.

Observability rests on three pillars: logging (structured, correlated events), metrics (quantitative measurements over time), and distributed tracing (following a request across service boundaries). OpenTelemetry has emerged as the industry standard for instrumenting all three pillars, providing vendor-neutral APIs and SDKs that export to any backend (Jaeger, Zipkin, Prometheus, Grafana, Datadog).

The Three Pillars

OpenTelemetry Integration in C#

C#
// Program.cs — OpenTelemetry setup for a microservice
using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;

var builder = WebApplication.CreateBuilder(args);

// Configure OpenTelemetry tracing
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService("order-service", serviceVersion: "1.0.0"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("OrderService.Domain") // Custom activity sources
        .AddOtlpExporter(opts => opts.Endpoint =
            new Uri("http://jaeger:4317")))
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddPrometheusExporter());

var app = builder.Build();

// Order Service endpoint with tracing
app.MapGet("/api/orders/{id}", async (
    Guid id,
    IOrderRepository repo,
    ActivitySource activitySource) =>
{
    // Create a custom span for this operation
    using var activity = activitySource.StartActivity("GetOrder");
    activity?.SetTag("order.id", id.ToString());

    var order = await repo.GetByIdAsync(id);
    if (order == null) return Results.NotFound();

    activity?.SetTag("order.status", order.Status.ToString());
    activity?.SetTag("order.item_count", order.Items.Count);

    return Results.Ok(order.ToDto());
});

// Structured logging with correlation context
public class OrderService
{
    private readonly ILogger<OrderService> _logger;

    public async Task<Order> CreateOrderAsync(CreateOrderCommand command)
    {
        // Log with structured properties — searchable in Elasticsearch
        _logger.LogInformation(
            "Creating order for customer {CustomerId} with {ItemCount} items, total {TotalAmount}",
            command.CustomerId,
            command.Items.Count,
            command.Items.Sum(i => i.Price * i.Quantity));

        var order = Order.Create(command.CustomerId, command.Items);

        _logger.LogInformation(
            "Order {OrderId} created with status {Status} for customer {CustomerId}",
            order.OrderId,
            order.Status,
            order.CustomerId);

        return order;
    }
}

RED Metrics Dashboard

The RED method provides a standard framework for monitoring microservices: Rate (requests per second), Errors (error rate as percentage), and Duration (latency percentiles — P50, P95, P99). Every service must expose these three metrics. The dashboard should also include infrastructure metrics (CPU, memory, network) and business metrics (orders per minute, revenue per hour).

Distributed Tracing: Following a Request Across Services

A distributed trace consists of spans — individual units of work within a service. Each span records the service name, operation name, start time, duration, and status. Spans are connected through parent-child relationships, forming a tree that represents the entire request lifecycle. When a request crosses service boundaries, the trace context (trace ID, span ID) is propagated through HTTP headers (traceparent header per W3C standard) or Kafka message headers.

C#
// W3C Trace Context propagation — automatically handled by OpenTelemetry
// But here's how it works under the hood:

// Sending service (Order Service) adds trace context to HTTP headers
public class TracedPaymentClient
{
    private readonly HttpClient _client;

    public async Task<PaymentResult> ChargeAsync(Guid orderId, decimal amount)
    {
        // OpenTelemetry automatically adds traceparent header
        var response = await _client.PostAsJsonAsync("/api/payments/charge", new
        {
            OrderId = orderId,
            Amount = amount
        });

        // Response also contains trace headers — correlation is automatic
        return await response.Content.ReadFromJsonAsync<PaymentResult>();
    }
}

// The trace appears in Jaeger as a tree:
// [Order Service: CreateOrder] ──12ms──▶ [Payment Service: Charge] ──200ms──▶ [Stripe: ProcessPayment]
//                                       └──3ms──▶ [Inventory Service: Reserve]

13. Security & Zero-Trust Networking

Security in microservices is fundamentally different from monolith security. In a monolith, you secure one application endpoint — the monolith handles all internal logic. In microservices, every service-to-service call is a potential attack vector. The principle of zero-trust networking means: no service trusts any other service by default, all communication is authenticated and encrypted, and every request is authorized based on the caller's identity and the requested resource.

Security Architecture Layers

JWT-Based Authentication Flow

C#
// JWT validation middleware — API Gateway validates once, passes identity downstream
public class JwtAuthenticationMiddleware
{
    private readonly RequestDelegate _next;
    private readonly TokenValidationParameters _validationParams;

    public async Task InvokeAsync(HttpContext context)
    {
        var token = context.Request.Headers.Authorization
            .FirstOrDefault()?.Replace("Bearer ", "");

        if (string.IsNullOrEmpty(token))
        {
            context.Response.StatusCode = 401;
            return;
        }

        var handler = new JwtSecurityTokenHandler();
        var principal = handler.ValidateToken(token, _validationParams, out _);

        // Set user identity on context — services read this
        context.User = principal;

        // Propagate identity to downstream services via header
        context.Request.Headers["X-User-Id"] =
            principal.FindFirst(ClaimTypes.NameIdentifier)?.Value;
        context.Request.Headers["X-User-Roles"] =
            string.Join(",", principal.FindAll(ClaimTypes.Role).Select(c => c.Value));

        await _next(context);
    }
}

// Service mesh mTLS — handled by Envoy sidecar, transparent to application
// Istio configuration for zero-trust networking
// AuthorizationPolicy: deny all by default, allow specific service-to-service calls
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: order-service-policy
spec:
  selector:
    matchLabels:
      app: order-service
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/default/sa/api-gateway"]
    to:
    - operation:
        methods: ["GET", "POST"]
        paths: ["/api/orders/*"]

Secret Management

Secrets (database passwords, API keys, encryption keys) must never be hardcoded in configuration files, environment variables in plain text, or source code. Use a secrets manager (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) with short-lived credentials, automatic rotation, and audit logging. Each service should request only the secrets it needs — follow the principle of least privilege.

C#
// Vault-backed secret provider with automatic rotation
public class VaultSecretProvider : ISecretProvider
{
    private readonly IVaultClient _vault;
    private readonly IMemoryCache _cache;
    private readonly TimeSpan _cacheDuration = TimeSpan.FromMinutes(5);

    public async Task<string> GetSecretAsync(string secretPath, string key)
    {
        var cacheKey = $"{secretPath}/{key}";
        return await _cache.GetOrCreateAsync(cacheKey, async entry =>
        {
            entry.AbsoluteExpirationRelativeToNow = _cacheDuration;

            var secret = await _vault.ReadSecretAsync(secretPath);
            return secret.Data[key].ToString();
        });
    }
}

// Usage in service configuration
public class DatabaseConfig
{
    private readonly ISecretProvider _secrets;

    public async Task<string> GetConnectionStringAsync()
    {
        var password = await _secrets.GetSecretAsync("secret/data/orders-db", "password");
        return $"Host=orders-db.internal;Database=orders;Username=svc_orders;Password={password}";
    }
}

14. Deployment, CI/CD & Container Orchestration

Microservices deployment requires infrastructure that can manage dozens or hundreds of independent services, each with its own deployment pipeline, scaling requirements, and resource needs. Kubernetes has become the de facto standard for microservices deployment because it provides service discovery, load balancing, auto-scaling, self-healing, rolling updates, and resource management out of the box. But Kubernetes is not free — it introduces operational complexity that must be managed through platform engineering.

Deployment Strategies

Kubernetes Deployment Manifest

YAML
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  labels:
    app: order-service
    version: v2.3.1
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # Allow 1 extra pod during update
      maxUnavailable: 0   # Never reduce below desired count
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
        version: v2.3.1
    spec:
      serviceAccountName: order-service-sa
      containers:
      - name: order-service
        image: registry.internal/order-service:v2.3.1
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "2000m"
            memory: "2Gi"
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        env:
        - name: DATABASE_CONNECTION_STRING
          valueFrom:
            secretKeyRef:
              name: order-service-secrets
              key: db-connection-string
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-service
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

CI/CD Pipeline

YAML
# GitHub Actions CI/CD pipeline for a microservice
name: Order Service CI/CD
on:
  push:
    branches: [main]
    paths:
      - 'services/order-service/**'

jobs:
  build-test-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build
        run: dotnet build --configuration Release

      - name: Unit Tests
        run: dotnet test --filter "Category=Unit" --logger trx

      - name: Integration Tests
        run: dotnet test --filter "Category=Integration" --logger trx
        env:
          POSTGRES_HOST: localhost
          KAFKA_BOOTSTRAP: localhost:9092

      - name: Build Docker Image
        run: docker build -t order-service:${{ github.sha }} .

      - name: Push to Registry
        run: docker push registry.internal/order-service:${{ github.sha }}

      - name: Deploy to Kubernetes (Canary)
        run: |
          kubectl set image deployment/order-service \
            order-service=registry.internal/order-service:${{ github.sha }}
          kubectl rollout status deployment/order-service --timeout=300s

15. Testing Strategies for Distributed Systems

Testing microservices is harder than testing a monolith because the system's behavior depends on the interaction between services, the reliability of the network, and the consistency of event-driven data flows. The testing strategy must cover four levels: unit tests (individual service logic), integration tests (service with its dependencies), contract tests (API compatibility between services), and end-to-end tests (full system behavior). Contract testing is especially important — it ensures that when the Order Service changes its API, the Payment Service (which depends on that API) is not broken.

Testing Pyramid for Microservices

Contract Testing with Pact

C#
// Consumer-driven contract test — Payment Service defines what it expects from Order Service
[TestClass]
public class OrderServiceContractTests
{
    [TestMethod]
    public async Task Order_Service_Should_Provide_Order_Details()
    {
        // Pact consumer test — defines the contract
        var pact = new Pact()
            .UponReceiving("a request for order details")
            .Given("order 123 exists")
            .WithRequest(HttpMethod.Get, "/api/orders/123")
            .WillRespondWith(200, builder =>
            {
                builder.WithHeader("Content-Type", "application/json");
                builder.WithBody(new
                {
                    orderId = Match.Type(Guid.NewGuid().ToString()),
                    status = Match.Regex("Created|Confirmed|Shipped",
                        "^(Created|Confirmed|Shipped)$"),
                    totalAmount = Match.Decimal(0.01m),
                    items = Match.MinType(new[]
                    {
                        new { productId = Match.AnyString(), quantity = Match.Type(1) }
                    }, 1)
                });
            });

        await pact.VerifyAsync(async ctx =>
        {
            var client = new HttpClient { BaseAddress = new Uri(ctx.MockServerUri) };
            var response = await client.GetAsync("/api/orders/123");
            response.EnsureSuccessStatusCode();
        });
    }
}

// Provider verification — Order Service verifies it meets the contract
[TestClass]
public class OrderServiceProviderTests
{
    [TestMethod]
    public async Task Should_Meet_Contract_With_Payment_Service()
    {
        var verifier = new PactVerifier();
        await verifier
            .ServiceProvider("OrderService")
            .HonoursPactWith("PaymentService")
            .FromPactFile(new FileInfo("pacts/payment-service-order-service.json"))
            .Verify();
    }
}

Chaos Engineering Tests

Chaos engineering systematically injects failures into the system to verify that resilience patterns work correctly. These tests are run in staging environments that mirror production. Key chaos tests for microservices include: killing a service instance (verify load balancer routes around it), introducing network latency between services (verify circuit breakers trigger), partitioning a service from its database (verify graceful degradation), and killing a Kafka broker (verify message delivery survives broker failure).

16. Common Anti-Patterns & How to Avoid Them

The microservices landscape is littered with cautionary tales of teams who adopted microservices and ended up with a distributed monolith — all the complexity of distributed systems with none of the benefits. Understanding common anti-patterns helps avoid these pitfalls.

Anti-Pattern Catalog

The Distributed Monolith: The Worst of Both Worlds

A distributed monolith is the most common and most painful anti-pattern. It occurs when services are deployed independently but are tightly coupled through shared databases, synchronous call chains requiring coordinated deployments, or implicit assumptions about each other's internal state. In a distributed monolith, you have all the costs of microservices (network latency, operational complexity, debugging difficulty, deployment overhead) with none of the benefits (independent deployment, fault isolation, team autonomy). If you change the Order Service API, you must also update and redeploy the Payment Service, the Shipping Service, and the Notification Service — a coordinated deployment, which is exactly what microservices were supposed to avoid.

C#
// ANTI-PATTERN: Shared database — Order Service and Payment Service access same table
// BAD: Both services directly query the Orders table
public class BadOrderService
{
    public async Task<Order> GetOrder(Guid orderId)
    {
        // Order Service queries Orders table
        return await _db.Orders.FindAsync(orderId);
    }
}

public class BadPaymentService
{
    public async Task<MarkOrderAsPaid(Guid orderId)
    {
        // Payment Service also queries Orders table — COUPLING!
        var order = await _db.Orders.FindAsync(orderId);
        order.PaymentStatus = "Paid";
        await _db.SaveChangesAsync();
    }
}

// CORRECT: Payment Service uses Order Service API or consumes events
public class GoodPaymentService
{
    private readonly IOrderServiceClient _orderClient; // Uses API, not direct DB access

    public async Task MarkOrderAsPaid(Guid orderId)
    {
        // Uses Order Service API — proper service boundary
        await _orderClient.UpdatePaymentStatusAsync(orderId, "Paid");
    }
}

17. Migrating from Monolith to Microservices

Migrating from a monolith to microservices is a multi-year journey for most organizations. The Strangler Fig pattern is the proven approach: you gradually replace parts of the monolith with new services while the monolith continues to serve production traffic. The name comes from the strangler fig tree that grows around a host tree, eventually replacing it. The key principles are: (1) never stop the monolith from serving traffic, (2) extract one service at a time, (3) each extraction must be independently valuable, and (4) the monolith shrinks as services are extracted.

Strangler Fig Migration Steps

1. Identify the first extraction target: Choose a service that is relatively self-contained, has a clear API boundary, and provides immediate value (e.g., Notification Service — it receives events and sends emails, with minimal dependencies).
2. Build the new service alongside the monolith: Create the new service with its own database, API, and deployment pipeline. Run it in parallel with the monolith's equivalent functionality.
3. Route traffic to the new service: Use a feature flag or API gateway routing to send a percentage of traffic to the new service. Monitor both old and new paths.
4. Verify and cut over: When the new service handles 100% of traffic correctly, remove the old code from the monolith. This is the "strangler" — the old code is now dead.
5. Repeat: Choose the next extraction target and repeat the process.

C#
// Strangler Fig: Feature flag to route between monolith and new service
public class OrderRoutingMiddleware
{
    private readonly IFeatureFlagProvider _featureFlags;
    private readonly IOrderServiceClient _newOrderService;
    private readonly MonolithOrderRepository _monolithRepo;

    public async Task<IActionResult> GetOrder(Guid orderId)
    {
        // Check if this order should use the new service
        var useNewService = await _featureFlags.IsEnabledAsync(
            "use-new-order-service",
            new Context { UserId = GetCurrentUserId() });

        if (useNewService)
        {
            // Route to new microservice
            return await _newOrderService.GetOrderAsync(orderId);
        }
        else
        {
            // Route to monolith
            var order = await _monolithRepo.GetByIdAsync(orderId);
            return Ok(order);
        }
    }
}

// Traffic shifting: gradually increase percentage
// Week 1: 5% of orders → new service (canary)
// Week 2: 25% of orders → new service
// Week 3: 50% of orders → new service
// Week 4: 100% of orders → new service (remove old code)

Migration Checklist

18. Interview Q&A Deep Dive

Q1: When would you NOT choose microservices?

Answer: When the team is small (under 10 engineers), the codebase is manageable (under 500K lines), deployment frequency is acceptable, and there are no conflicting scaling requirements between subsystems. Microservices introduce distributed systems complexity (network failures, eventual consistency, debugging difficulty, operational overhead) that is only justified when the organizational and technical benefits clearly outweigh the costs. A well-structured monolith with clear module boundaries provides most of the organizational benefits (team ownership, clear interfaces) without the distributed systems overhead. Start with a monolith, extract services when pain is measured, not imagined.

Q2: How do you handle data consistency across services?

Answer: Distributed transactions (2PC) are avoided because they create tight coupling and single points of failure. Instead, use the Saga pattern: a sequence of local transactions where each step publishes an event that triggers the next step. If any step fails, compensating transactions undo previous steps. For read consistency, use the CQRS pattern with event-driven data replication — each service maintains a denormalized local copy of data it needs, updated asynchronously via events. Accept eventual consistency as the price of decoupling. The consistency window is typically seconds — acceptable for most business operations. Design the UI to handle this: show "processing" states, use optimistic updates.

Q3: How do you prevent cascading failures across services?

Answer: Four defense-in-depth mechanisms: (1) Circuit breakers on every inter-service call — stop calling a failing service after N failures, retry after a timeout. (2) Timeouts on every synchronous call — never wait indefinitely. (3) Bulkheads — isolate thread pools per dependency so a slow Payment Service does not starve calls to the Inventory Service. (4) Retry budgets — limit total retries to 20% of requests per minute to prevent retry storms. Additionally, design services to be "fail-fast" — if a critical dependency is unavailable, return a degraded response immediately rather than queuing requests. Implement fallback behavior for non-critical features (show cached recommendations instead of fresh ones).

Q4: Explain the difference between orchestration and choreography in sagas.

Answer: In orchestration, a central coordinator (saga orchestrator) directs the flow — it calls each service in sequence and handles compensations if a step fails. The orchestrator has full visibility into the saga state, making debugging straightforward. In choreography, each service listens for events and decides independently what to do — the Order Service publishes OrderCreated, the Payment Service reacts by initiating payment, the Inventory Service reacts by reserving stock. There is no central coordinator, making the system more decoupled but harder to debug. Use orchestration for complex sagas (5+ steps) where you need visibility. Use choreography for simple 2-3 step flows where decoupling is more important than visibility.

Q5: How do you debug a request that fails across multiple services?

Answer: Distributed tracing is essential. Every request gets a trace ID (W3C traceparent header) that propagates across all services. When a request fails, search for the trace ID in Jaeger/Zipkin to see the full request timeline — which services were called, how long each took, where the error occurred, and what the error message was. Pair this with structured logging: every log entry includes the trace ID, service name, user ID, and operation. In Elasticsearch, you can search for all logs associated with a single trace ID to reconstruct the full context. Without distributed tracing and structured logging, debugging cross-service failures is nearly impossible — you end up grepping logs across dozens of services hoping to find the relevant entries.

Q6: How do you handle shared libraries across microservices?

Answer: Shared libraries are appropriate for utility code that rarely changes and has no business logic: logging formats, tracing instrumentation, common DTOs (like HealthCheckResponse), and serialization helpers. Never share business logic through libraries — this creates hidden coupling where changing the library requires coordinated deployment of all consuming services. For business models that need to be shared (like an OrderDto used by Order and Payment services), define the contract in a shared proto file (for gRPC) or a shared NuGet package with strict versioning. The rule: shared libraries should be "leaf" dependencies with no transitive dependencies on business logic.

Key Numbers to Remember

Pre-Interview Checklist

• Know when to choose monolith vs microservices (team size, codebase size, scaling needs)
• Understand DDD bounded contexts and how they map to service boundaries
• Explain sync (REST/gRPC) vs async (Kafka/RabbitMQ) communication tradeoffs
• Design a Saga pattern for a multi-service transaction (order → payment → inventory → shipping)
• Know the circuit breaker, retry with backoff, and bulkhead patterns
• Explain CQRS and eventual consistency — when and why to use it
• Describe the Strangler Fig migration strategy from monolith to microservices
• Know how to debug cross-service failures with distributed tracing (OpenTelemetry, Jaeger)
• Explain the difference between orchestration and choreography sagas
• Discuss security: mTLS, JWT, zero-trust networking, secrets management
• Know deployment strategies: rolling update, blue-green, canary, feature flags
• Understand the anti-patterns: distributed monolith, shared database, synchronous chains, nano services