1. Why Distributed Transactions Are Hard

In a single-service monolith with one database, a transaction is trivial. You open a connection, begin a transaction, execute your SQL statements, and either commit or rollback. The database engine handles atomicity, consistency, isolation, and durability (ACID) natively. Every row-level lock, every WAL write, every redo log entry is coordinated by a single process — the database engine — and if anything goes wrong, the database itself rolls everything back.

The moment you split your system into multiple services, each owning its own database, the game changes fundamentally. There is no single engine that can coordinate across PostgreSQL in the order service, MongoDB in the catalog service, and Redis in the cache layer. Each database has its own transaction manager, its own log, and its own notion of commit. A commit in one database has no awareness of what happens in another. This is the distributed transaction problem.

Consider a real-world e-commerce scenario. A customer places an order. The system must: (1) create the order record in the OrderService database, (2) charge the customer's credit card via the PaymentService, (3) reserve inventory in the InventoryService, and (4) schedule shipping in the ShippingService. If step 3 fails — say the item is out of stock — you have already charged the customer's card and created an order that cannot be fulfilled. Without a distributed transaction mechanism, you have an inconsistent state that will require manual intervention.

The fundamental challenge is that no single participant can see the entire transaction. Each service sees only its own operations. Network messages between services can be lost, delayed, or duplicated. A service can crash at any point. The coordinator — if one exists — can also crash. This is a manifestation of the FLP impossibility result: in an asynchronous system with even one faulty process, no deterministic consensus protocol can guarantee termination.

Distributed transactions are hard because they require coordination across failure domains. Every protocol in this guide is a different trade-off between consistency (how correct the data is), availability (how often the system can serve requests), performance (latency and throughput), and complexity (how much code you must write and maintain).

The Dual-Write Problem

One of the most common sources of data inconsistency is the dual-write problem. A service writes to its database and then sends a message to a message broker. If the database write succeeds but the service crashes before sending the message, the systems are out of sync. If the message is sent but the database write fails, the message references data that does not exist.

This is not a theoretical concern. It happens in production systems every day. The only reliable solutions are the Outbox pattern (write the message to the same database as the business data in a single transaction) or transactional messaging (the message broker participates in the same transaction as the database). We will cover both in depth later in this guide.

2. CAP Theorem and Its Implications

Before diving into specific protocols, you must understand the CAP theorem. Formally proved by Eric Brewer in 2000 and later proven rigorously by Gilbert and Lynch, the CAP theorem states that a distributed data store can provide at most two of the following three guarantees simultaneously:

• Consistency (C): Every read receives the most recent write or an error. All nodes see the same data at the same time.
• Availability (A): Every request receives a non-error response (not necessarily the most recent write). The system is always operational.
• Partition Tolerance (P): The system continues to operate despite network partitions between nodes. Messages between nodes may be lost or delayed.

In a distributed system, network partitions are inevitable. Routers fail, cables get cut, cloud availability zones lose connectivity. This means partition tolerance is not optional — it is a fact of distributed systems. The real choice is between consistency and availability during a partition.

2PC is a CP protocol — it guarantees atomicity by blocking participants during the prepare phase, but if the coordinator or a participant is unreachable, the system may be unavailable. Sagas are an AP protocol — they guarantee eventual consistency by using compensating transactions, allowing the system to remain available even during partial failures.

Beyond CAP, modern systems also consider the PACELC theorem, which extends CAP: during a partition, choose between Availability and Consistency; otherwise (in normal operation), choose between Latency and Consistency. This framework better explains the design choices of databases like Cassandra (PA/EL — prefer availability and low latency) and Spanner (PC/EC — prefer consistency even at the cost of latency).

3. Two-Phase Commit (2PC) — Deep Dive

The two-phase commit protocol is the gold standard for atomic distributed transactions. It was first described by Jim Gray in 1978 and later formalized by Dale Skeen. 2PC guarantees that either all participants commit or all participants abort — there is no partial commit. It is used internally by most relational databases (PostgreSQL, Oracle, SQL Server) for distributed queries and by X/Open XA-compliant transaction managers.

Protocol Mechanics

2PC involves a coordinator (also called the transaction manager) and two or more participants (resource managers). The protocol has two phases:

Phase 1 — Prepare (Voting): The coordinator sends a prepare message to every participant. Each participant executes the transaction up to the point of commit, writes all changes to a durable write-ahead log (WAL), acquires the necessary locks, and responds with either a "yes" (ready to commit) or "no" (cannot commit, abort). A "yes" vote is a binding promise — the participant guarantees it can commit if instructed to do so, even if it crashes and recovers.

Phase 2 — Commit/Abort (Decision): The coordinator collects all votes. If every participant voted "yes," the coordinator writes the commit decision to its own log and sends a commit message to all participants. If any participant voted "no" (or failed to respond within a timeout), the coordinator sends an abort message. Participants execute the decision and acknowledge.

Logging and Recovery

The durability of 2PC relies entirely on logging. Both the coordinator and participants write protocol messages to stable storage before sending them over the network. This is the "write-ahead log" requirement. If a participant crashes after voting "yes" but before receiving the decision, it can recover by reading its log and asking the coordinator for the decision. If the coordinator crashes after writing the commit decision to its log but before sending all commit messages, it can recover and re-send the decision. This is why 2PC requires durable storage at every node.

C#
public class TwoPhaseCommitCoordinator
{
    private readonly ITransactionLog _log;
    private readonly IEnumerable<IParticipant> _participants;

    public TwoPhaseCommitCoordinator(
        ITransactionLog log,
        IEnumerable<IParticipant> participants)
    {
        _log = log;
        _participants = participants;
    }

    public async Task<CommitResult> ExecuteAsync(
        DistributedTransaction transaction)
    {
        var transactionId = Guid.NewGuid();

        // Phase 1: Prepare — collect votes from all participants
        var votes = new Dictionary<Guid, Vote>();
        foreach (var participant in _participants)
        {
            try
            {
                var vote = await participant.PrepareAsync(transactionId, transaction);
                votes[participant.Id] = vote;

                await _log.WriteAsync(new LogEntry
                {
                    TransactionId = transactionId,
                    Phase = Phase.Prepare,
                    ParticipantId = participant.Id,
                    Vote = vote,
                    Timestamp = DateTimeOffset.UtcNow
                });
            }
            catch (Exception ex)
            {
                votes[participant.Id] = Vote.Abort;
                await _log.WriteAsync(new LogEntry
                {
                    TransactionId = transactionId,
                    Phase = Phase.Prepare,
                    ParticipantId = participant.Id,
                    Vote = Vote.Abort,
                    Error = ex.Message,
                    Timestamp = DateTimeOffset.UtcNow
                });
            }
        }

        // Phase 2: Decide
        var allReady = votes.Values.All(v => v == Vote.Ready);
        var decision = allReady ? Decision.Commit : Decision.Abort;

        await _log.WriteAsync(new LogEntry
        {
            TransactionId = transactionId,
            Phase = Phase.Decide,
            Decision = decision,
            Timestamp = DateTimeOffset.UtcNow
        });

        // Send decision to all participants
        foreach (var participant in _participants)
        {
            try
            {
                if (decision == Decision.Commit)
                    await participant.CommitAsync(transactionId);
                else
                    await participant.AbortAsync(transactionId);
            }
            catch
            {
                // Participant unreachable — will recover via log replay
            }
        }

        return new CommitResult(transactionId, decision);
    }
}

public enum Vote { Ready, Abort }
public enum Decision { Commit, Abort }
public enum Phase { Prepare, Decide }

public record LogEntry
{
    public Guid TransactionId { get; init; }
    public Phase Phase { get; init; }
    public Guid? ParticipantId { get; init; }
    public Vote? Vote { get; init; }
    public Decision? Decision { get; init; }
    public string? Error { get; init; }
    public DateTimeOffset Timestamp { get; init; }
}

The Blocking Problem

The critical weakness of 2PC is that it is a blocking protocol. After a participant votes "yes" in Phase 1, it holds locks on the transaction's resources and cannot proceed with other transactions involving those resources until it receives the coordinator's decision. If the coordinator crashes after Phase 1 but before Phase 2, participants are stuck. They cannot commit (because they do not know if everyone voted yes) and they cannot abort (because they promised to commit if instructed). They must wait for the coordinator to recover.

This blocking behavior reduces system availability. In a system where the coordinator has a 99.9% uptime, every minute of coordinator downtime means every in-flight 2PC transaction is blocked, holding locks and consuming resources. This is why 2PC is typically reserved for short-lived transactions within a single trust boundary — you do not want a 30-second e-commerce transaction holding database locks for minutes while the coordinator recovers.

2PC Failure Scenarios

4. Three-Phase Commit (3PC)

Three-phase commit was proposed by Dale Skeen as an improvement over 2PC to address the blocking problem. 3PC adds a third phase — pre-commit — between the prepare and commit phases. The idea is that if a participant knows the coordinator has collected all "yes" votes (which is what the pre-commit phase signals), it can make an independent commit decision if the coordinator becomes unavailable.

Protocol Phases

Phase 1 — CanCommit: The coordinator asks all participants whether they can commit. This is a preliminary check — participants validate preconditions (e.g., sufficient funds, available inventory) but do not acquire locks or write to the WAL. Participants respond with yes or no.

Phase 2 — PreCommit: If all participants responded yes, the coordinator sends a pre-commit message. Participants now execute the transaction, acquire locks, and write to the WAL. They respond with "pre-committed." This phase signals that a unanimous "yes" vote has been reached — if a participant receives this message, it knows every other participant also voted yes.

Phase 3 — DoCommit: If all participants pre-committed, the coordinator sends the final commit message. If the coordinator crashes after Phase 2, participants that received the pre-commit message can timeout and commit independently — because they know all participants pre-committed.

C#
public class ThreePhaseCommitCoordinator
{
    private readonly ITransactionLog _log;
    private readonly IEnumerable<IParticipant> _participants;
    private readonly TimeSpan _timeout = TimeSpan.FromSeconds(30);

    public async Task<CommitResult> ExecuteAsync(
        DistributedTransaction transaction)
    {
        var txId = Guid.NewGuid();

        // Phase 1: CanCommit — lightweight check
        var canCommitVotes = new Dictionary<Guid, bool>();
        foreach (var p in _participants)
        {
            canCommitVotes[p.Id] = await p.CanCommitAsync(txId, transaction);
        }

        if (canCommitVotes.Values.Any(v => !v))
            return new CommitResult(txId, Decision.Abort);

        // Phase 2: PreCommit — execute and prepare WAL
        var preCommitted = new Dictionary<Guid, bool>();
        foreach (var p in _participants)
        {
            try
            {
                preCommitted[p.Id] = await p.PreCommitAsync(txId);
            }
            catch
            {
                preCommitted[p.Id] = false;
            }
        }

        if (preCommitted.Values.Any(v => !v))
        {
            foreach (var p in _participants)
                await p.AbortAsync(txId);
            return new CommitResult(txId, Decision.Abort);
        }

        await _log.WriteAsync(new LogEntry
        {
            TransactionId = txId,
            Phase = Phase.Decide,
            Decision = Decision.Commit,
            Timestamp = DateTimeOffset.UtcNow
        });

        // Phase 3: DoCommit
        foreach (var p in _participants)
        {
            await p.DoCommitAsync(txId);
        }

        return new CommitResult(txId, Decision.Commit);
    }
}

2PC vs 3PC Comparison

5. Saga Pattern — Compensating Transactions

The Saga pattern was first described by Hector Garcia-Molina and Kenneth Salem in 1987 as a way to handle long-lived transactions in a database. The modern incarnation, adapted for microservices, replaces distributed locks with compensating transactions. A Saga is a sequence of local transactions where each step commits immediately. If any step fails, the Saga executes compensating transactions for all previously completed steps in reverse order.

The key insight is that compensating transactions are not the same as rollbacks. A rollback erases the effects of a transaction as if it never happened. A compensating transaction creates a new transaction that logically undoes the effects. For example, a payment charge is compensated by a refund — the original charge record still exists, but a new refund record cancels it out. This distinction matters for auditing, compliance, and understanding the system's history.

Saga Properties

• Eventual Consistency: Sagas do not provide atomic visibility. Between steps, different parts of the system may see different states. For example, after payment but before inventory reservation, the payment is visible but the inventory is not yet reserved.
• Compensability: Every forward operation must have a corresponding compensation. If an operation cannot be compensated (e.g., sending an email), the Saga must handle it differently — typically by making it the last step.
• Retriability: Each step (and its compensation) must be idempotent. Network failures may cause retries, and the system must handle duplicate executions safely.
• Isolation: Unlike database transactions, Sagas do not provide isolation. Other transactions can see intermediate states. This is the price of avoiding distributed locks.

Saga Failure Semantics

When a step in a Saga fails, there are three possible scenarios. First, the step itself failed before committing — in this case, the Saga simply runs compensations for previously completed steps. Second, the step committed successfully but the response was lost — the Saga coordinator does not know if the step succeeded, so it retries the step (which must be idempotent) or retries the compensation. Third, the step committed and the coordinator knows it failed — the coordinator runs compensations.

The second scenario is the most dangerous. If a step committed but the coordinator did not receive the acknowledgment, the coordinator may attempt to compensate without knowing whether the step actually ran. The compensation must be idempotent to handle this case safely. This is why idempotency is not optional in a Saga implementation — it is a fundamental requirement.

Comparison: 2PC vs Saga

6. Orchestration vs Choreography Sagas

Sagas can be implemented in two fundamental ways: orchestration and choreography. The choice between them affects system coupling, observability, and maintainability.

Orchestration Sagas

In an orchestration saga, a central coordinator (the orchestrator) manages the entire saga flow. The orchestrator contains the saga logic as a state machine. It sends commands to each participant, waits for responses, and handles failures by triggering compensations. Participants are simple — they receive a command, execute it, and return a result. They do not need to know about other participants or the overall saga flow.

The advantages of orchestration are clear. The entire saga logic is in one place — you can read the orchestrator and understand the complete flow, including error handling. Adding a new step or modifying the compensation logic requires changing only the orchestrator. Monitoring is straightforward — you can instrument the orchestrator to emit metrics and traces for every step. Debugging is easier because the control flow is centralized.

The disadvantages are that the orchestrator is a single point of failure and a potential bottleneck. If the orchestrator crashes mid-saga, the saga is incomplete until it recovers. The orchestrator also knows about every participant, creating coupling between the coordinator and all services involved in the saga.

C#
public class OrderSagaOrchestrator
{
    private readonly IPaymentService _paymentService;
    private readonly IInventoryService _inventoryService;
    private readonly IShippingService _shippingService;
    private readonly ISagaLog _sagaLog;

    public async Task<SagaResult> ExecuteAsync(Order order)
    {
        var sagaId = Guid.NewGuid();
        var completedSteps = new List<SagaStep>();

        try
        {
            // Step 1: Create Order
            var orderId = await _orderService.CreateAsync(order);
            completedSteps.Add(new SagaStep("CreateOrder", orderId));

            // Step 2: Charge Payment
            var paymentId = await _paymentService.ChargeAsync(
                sagaId, order.Total, order.PaymentMethod);
            completedSteps.Add(new SagaStep("ChargePayment", paymentId));

            // Step 3: Reserve Inventory
            var reservationId = await _inventoryService.ReserveAsync(
                sagaId, order.Items);
            completedSteps.Add(new SagaStep("ReserveInventory", reservationId));

            // Step 4: Create Shipment
            var shipmentId = await _shippingService.CreateShipmentAsync(
                sagaId, order.Address, order.Items);
            completedSteps.Add(new SagaStep("CreateShipment", shipmentId));

            await _sagaLog.LogCompletion(sagaId, SagaStatus.Completed);
            return SagaResult.Success(sagaId);
        }
        catch (Exception ex)
        {
            // Compensate in reverse order
            await CompensateAsync(sagaId, completedSteps, ex);
            return SagaResult.Failure(sagaId, ex.Message);
        }
    }

    private async Task CompensateAsync(
        Guid sagaId,
        List<SagaStep> completedSteps,
        Exception originalError)
    {
        foreach (var step in completedSteps.AsEnumerable().Reverse())
        {
            try
            {
                switch (step.Name)
                {
                    case "ReserveInventory":
                        await _inventoryService.ReleaseAsync(
                            sagaId, step.ResourceId);
                        break;
                    case "ChargePayment":
                        await _paymentService.RefundAsync(
                            sagaId, step.ResourceId);
                        break;
                    case "CreateOrder":
                        await _orderService.CancelAsync(step.ResourceId);
                        break;
                }
                await _sagaLog.LogCompensation(
                    sagaId, step.Name, CompensationStatus.Success);
            }
            catch (Exception compEx)
            {
                // Log compensation failure — requires manual intervention
                await _sagaLog.LogCompensation(
                    sagaId, step.Name, CompensationStatus.Failed);
            }
        }
    }
}

Choreography Sagas

In a choreography saga, there is no central coordinator. Instead, each service publishes events when it completes its step, and other services subscribe to those events. The order service publishes "OrderCreated," the payment service consumes it, processes payment, and publishes "PaymentProcessed." The inventory service consumes "PaymentProcessed" and reserves inventory. The flow emerges from the event subscriptions.

The advantage of choreography is loose coupling. No service needs to know about the overall saga flow. Adding a new step means adding a new event subscription — no changes to existing services. This makes choreography sagas easy to extend and deploy independently.

The disadvantage is that the control flow is distributed and implicit. To understand the complete flow, you must trace the chain of events across multiple services. Debugging a failed saga requires correlating events from different services. Compensation logic is also distributed — each service must handle its own compensation when it receives a failure event.

C#
// Order Service — publishes OrderCreated
public class OrderCreatedHandler
{
    private readonly IMessageBus _messageBus;

    public async Task HandleAsync(Order order)
    {
        await _orderRepository.CreateAsync(order);
        await _messageBus.PublishAsync(new OrderCreated
        {
            OrderId = order.Id,
            Total = order.Total,
            Items = order.Items,
            CorrelationId = Guid.NewGuid()
        });
    }
}

// Payment Service — consumes OrderCreated, publishes PaymentProcessed
public class OrderCreatedConsumer
{
    private readonly IPaymentRepository _payments;
    private readonly IMessageBus _messageBus;

    public async Task ConsumeAsync(OrderCreated evt)
    {
        try
        {
            var payment = await _payments.ChargeAsync(
                evt.OrderId, evt.Total);
            await _messageBus.PublishAsync(new PaymentProcessed
            {
                OrderId = evt.OrderId,
                PaymentId = payment.Id,
                CorrelationId = evt.CorrelationId
            });
        }
        catch (Exception)
        {
            await _messageBus.PublishAsync(new PaymentFailed
            {
                OrderId = evt.OrderId,
                CorrelationId = evt.CorrelationId
            });
        }
    }
}

// Inventory Service — consumes PaymentProcessed
public class PaymentProcessedConsumer
{
    private readonly IInventoryRepository _inventory;
    private readonly IMessageBus _messageBus;

    public async Task ConsumeAsync(PaymentProcessed evt)
    {
        try
        {
            var reservation = await _inventory.ReserveAsync(
                evt.OrderId);
            await _messageBus.PublishAsync(new InventoryReserved
            {
                OrderId = evt.OrderId,
                ReservationId = reservation.Id,
                CorrelationId = evt.CorrelationId
            });
        }
        catch (Exception)
        {
            await _messageBus.PublishAsync(new InventoryReservationFailed
            {
                OrderId = evt.OrderId,
                CorrelationId = evt.CorrelationId
            });
        }
    }
}

Decision Framework

7. Try-Confirm-Cancel (TCC) Pattern

Try-Confirm-Cancel (TCC) is a distributed transaction pattern proposed by Pat Helland. It provides stronger consistency guarantees than Sagas while avoiding the blocking behavior of 2PC. TCC requires each participant to implement three operations: Try, Confirm, and Cancel.

Try: Reserves resources and validates preconditions. This is not a commit — it is a reservation. For example, a payment service would authorize a hold on the credit card. An inventory service would reserve stock. A seat booking service would hold seats temporarily. The Try phase acquires "soft locks" that are scoped to the TCC transaction.

Confirm: Commits the reservation. If all participants' Try operations succeed, the coordinator sends Confirm to each participant. The Confirm operation must be idempotent — if a participant receives Confirm twice (due to a retry), it should return the same result.

Cancel: Releases the reservation. If any participant's Try fails, or if the coordinator decides to abort, the coordinator sends Cancel to all participants. Cancel releases the reserved resources. Cancel must also be idempotent.

C#
public interface ITccParticipant
{
    Task<TryResult> TryAsync(Guid transactionId, TransactionContext context);
    Task ConfirmAsync(Guid transactionId);
    Task CancelAsync(Guid transactionId);
}

public class PaymentTccParticipant : ITccParticipant
{
    private readonly IPaymentGateway _gateway;
    private readonly IPaymentRepository _repository;

    public async Task<TryResult> TryAsync(
        Guid transactionId, TransactionContext context)
    {
        // Authorize (hold) the amount — no charge yet
        var authorization = await _gateway.AuthorizeAsync(
            context.PaymentMethod, context.Amount);

        await _repository.SaveAuthorizationAsync(new Authorization
        {
            TransactionId = transactionId,
            AuthorizationCode = authorization.Code,
            Amount = context.Amount,
            Status = AuthorizationStatus.Pending
        });

        return TryResult.Success(transactionId);
    }

    public async Task ConfirmAsync(Guid transactionId)
    {
        // Idempotent: check if already confirmed
        var auth = await _repository.GetAuthorizationAsync(transactionId);
        if (auth.Status == AuthorizationStatus.Confirmed)
            return;

        // Capture the held funds
        await _gateway.CaptureAsync(auth.AuthorizationCode, auth.Amount);
        auth.Status = AuthorizationStatus.Confirmed;
        await _repository.UpdateAuthorizationAsync(auth);
    }

    public async Task CancelAsync(Guid transactionId)
    {
        // Idempotent: check if already cancelled
        var auth = await _repository.GetAuthorizationAsync(transactionId);
        if (auth == null || auth.Status == AuthorizationStatus.Cancelled)
            return;

        // Release the hold
        await _gateway.VoidAsync(auth.AuthorizationCode);
        auth.Status = AuthorizationStatus.Cancelled;
        await _repository.UpdateAuthorizationAsync(auth);
    }
}

TCC Coordinator

C#
public class TccCoordinator
{
    private readonly IEnumerable<ITccParticipant> _participants;
    private readonly ITccLog _log;

    public async Task<CommitResult> ExecuteAsync(
        TransactionContext context)
    {
        var txId = Guid.NewGuid();

        // Phase 1: Try all participants
        var tryResults = new Dictionary<Guid, TryResult>();
        foreach (var p in _participants)
        {
            try
            {
                var result = await p.TryAsync(txId, context);
                tryResults[p.Id] = result;
            }
            catch
            {
                tryResults[p.Id] = TryResult.Failed;
            }
        }

        // Phase 2: Confirm or Cancel
        var allSucceeded = tryResults.Values.All(r => r.IsSuccess);

        await _log.WriteAsync(txId,
            allSucceeded ? TxDecision.Confirm : TxDecision.Cancel);

        var operation = allSucceeded
            ? p => p.ConfirmAsync(txId)
            : p => p.CancelAsync(txId);

        foreach (var p in _participants)
        {
            try { await operation(p); }
            catch { /* Retry with idempotency */ }
        }

        return new CommitResult(txId, allSucceeded);
    }
}

TCC vs 2PC vs Saga

8. The Outbox Pattern — Solving the Dual-Write Problem

The Outbox pattern is not a distributed transaction protocol itself — it is a building block that makes other patterns (especially Sagas) reliable. It solves the fundamental dual-write problem: writing to a database and sending a message to a message broker atomically.

The Problem

Consider a service that needs to update its database and notify other services via a message broker. The naive approach is:

1. Write to the database
2. Send a message to the broker

If step 1 succeeds but step 2 fails (service crashes, network issue, broker down), the database is updated but no notification is sent. Other services remain unaware of the change. If step 2 succeeds but step 1 fails, a message references data that does not exist. There is no way to make these two operations atomic without a shared transaction manager.

The Outbox Solution

The Outbox pattern eliminates the dual-write problem by writing the business data and the integration event to the same database in a single local transaction:

1. Write the business data (e.g., INSERT INTO orders) and the event (e.g., INSERT INTO outbox) in the same database transaction.
2. A separate relay process (the "outbox relay" or "polling publisher") reads unpublished events from the outbox table.
3. The relay publishes each event to the message broker and marks it as published in the outbox table.

Since the business data and the event are in the same database transaction, they are atomically committed. If the transaction rolls back, neither the business data nor the event is persisted. The relay process ensures that events are eventually published — even if it crashes, it will pick up unpublished events on restart.

Outbox Implementation Considerations

• Relay Process: Can be a background worker, a separate deployment, or built into the database (e.g., Debezium for CDC). Polling-based relays are simpler but introduce latency. CDC-based relays (using database change streams) are more efficient but require database-specific configuration.
• Event Ordering: The outbox table should have a monotonically increasing sequence number. The relay publishes events in order to maintain causality.
• At-Least-Once Delivery: The relay may publish an event and then crash before marking it as published. On restart, it will re-publish the event. Consumers must be idempotent.
• Cleanup: Published events should be archived or deleted periodically to prevent the outbox table from growing indefinitely.

9. Transactional Outbox with C# and Entity Framework

This section provides a production-ready implementation of the Outbox pattern using C#, Entity Framework Core, and a generic message bus. The implementation includes the outbox entity, the relay worker, and integration with the business service.

Outbox Entity and DbContext

C#
public class OutboxMessage
{
    public Guid Id { get; set; }
    public string EventType { get; set; } = string.Empty;
    public string Payload { get; set; } = string.Empty;
    public long SequenceNumber { get; set; }
    public DateTimeOffset CreatedAt { get; set; }
    public DateTimeOffset? PublishedAt { get; set; }
    public int RetryCount { get; set; }
}

public class AppDbContext : DbContext
{
    public DbSet<OutboxMessage> OutboxMessages { get; set; } = null!;
    public DbSet<Order> Orders { get; set; } = null!;

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        modelBuilder.Entity<OutboxMessage>(e =>
        {
            e.HasKey(x => x.Id);
            e.Property(x => x.SequenceNumber)
                .UseIdentityAlwaysColumn();
            e.HasIndex(x => new { x.PublishedAt, x.SequenceNumber })
                .HasFilter("\"PublishedAt\" IS NULL");
        });
    }
}

Business Service with Outbox

C#
public class OrderService
{
    private readonly AppDbContext _db;
    private readonly IJsonSerializer _serializer;

    public OrderService(AppDbContext db, IJsonSerializer serializer)
    {
        _db = db;
        _serializer = serializer;
    }

    public async Task<Order> PlaceOrderAsync(PlaceOrderCommand command)
    {
        // Both the order and the outbox event are written
        // in a SINGLE database transaction
        using var transaction = await _db.Database.BeginTransactionAsync();

        try
        {
            var order = new Order
            {
                Id = Guid.NewGuid(),
                CustomerId = command.CustomerId,
                Items = command.Items,
                Total = command.Items.Sum(i => i.Price * i.Quantity),
                Status = OrderStatus.Created,
                CreatedAt = DateTimeOffset.UtcNow
            };

            _db.Orders.Add(order);

            // Write the integration event to the outbox
            var outboxMessage = new OutboxMessage
            {
                Id = Guid.NewGuid(),
                EventType = nameof(OrderCreated),
                Payload = _serializer.Serialize(new OrderCreated
                {
                    OrderId = order.Id,
                    CustomerId = order.CustomerId,
                    Total = order.Total,
                    Items = order.Items
                }),
                CreatedAt = DateTimeOffset.UtcNow
            };

            _db.OutboxMessages.Add(outboxMessage);

            await _db.SaveChangesAsync();
            await transaction.CommitAsync();

            return order;
        }
        catch
        {
            await transaction.RollbackAsync();
            throw;
        }
    }
}

Outbox Relay Worker

C#
public class OutboxRelayWorker : BackgroundService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly ILogger<OutboxRelayWorker> _logger;
    private readonly TimeSpan _pollingInterval = TimeSpan.FromMilliseconds(500);

    public OutboxRelayWorker(
        IServiceProvider serviceProvider,
        ILogger<OutboxRelayWorker> logger)
    {
        _serviceProvider = serviceProvider;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Outbox relay worker started");

        while (!stoppingToken.IsCancellationRequested)
        {
            try
            {
                await ProcessPendingMessagesAsync(stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error processing outbox messages");
            }

            await Task.Delay(_pollingInterval, stoppingToken);
        }
    }

    private async Task ProcessPendingMessagesAsync(CancellationToken ct)
    {
        using var scope = _serviceProvider.CreateScope();
        var db = scope.ServiceProvider
            .GetRequiredService<AppDbContext>();
        var messageBus = scope.ServiceProvider
            .GetRequiredService<IMessageBus>();

        var pendingMessages = await db.OutboxMessages
            .Where(m => m.PublishedAt == null && m.RetryCount < 5)
            .OrderBy(m => m.SequenceNumber)
            .Take(50)
            .ToListAsync(ct);

        foreach (var message in pendingMessages)
        {
            try
            {
                await messageBus.PublishAsync(
                    message.EventType, message.Payload);

                message.PublishedAt = DateTimeOffset.UtcNow;
                _logger.LogDebug(
                    "Published outbox message {Id} ({Type})",
                    message.Id, message.EventType);
            }
            catch (Exception ex)
            {
                message.RetryCount++;
                _logger.LogWarning(ex,
                    "Failed to publish outbox message {Id}, retry {Count}",
                    message.Id, message.RetryCount);
            }
        }

        await db.SaveChangesAsync(ct);
    }
}

10. Idempotency — The Foundation of Reliable Distributed Systems

Idempotency is the property of an operation where executing it multiple times produces the same result as executing it once. In distributed systems, idempotency is not a nice-to-have — it is a survival mechanism. Networks are unreliable. Messages are duplicated. Retries are inevitable. Without idempotency, every retry risks duplicating a payment, double-reserving inventory, or creating duplicate orders.

Why Idempotency Matters

Consider a payment service that receives a charge request. The request succeeds, but the response is lost due to a network timeout. The client retries. Without idempotency, the customer is charged twice. With idempotency, the payment service detects that the same transaction ID was already processed and returns the original result without re-charging.

This is not just about network retries. In a Saga, when a step fails and the coordinator retries it, the step may have already committed locally. In 2PC, when a participant recovers from a crash, it must re-execute decisions from its log. In the Outbox pattern, the relay may re-publish an event if it crashes after publishing but before marking the event as published. In every case, the system relies on idempotency to prevent duplicate effects.

Idempotency Implementation

C#
public class IdempotencyService
{
    private readonly AppDbContext _db;

    public IdempotencyService(AppDbContext db)
    {
        _db = db;
    }

    public async Task<T?> ExecuteOnceAsync<T>(
        string idempotencyKey,
        Func<Task<T>> operation)
    {
        // Check if this operation was already executed
        var existing = await _db.IdempotencyKeys
            .FirstOrDefaultAsync(k => k.Key == idempotencyKey);

        if (existing != null)
        {
            return JsonSerializer.Deserialize<T>(existing.Result);
        }

        // Execute the operation
        var result = await operation();

        // Store the result with a unique constraint to prevent races
        var key = new IdempotencyKeyEntity
        {
            Key = idempotencyKey,
            Result = JsonSerializer.Serialize(result),
            CreatedAt = DateTimeOffset.UtcNow,
            ExpiresAt = DateTimeOffset.UtcNow.AddHours(24)
        };

        try
        {
            _db.IdempotencyKeys.Add(key);
            await _db.SaveChangesAsync();
        }
        catch (DbUpdateException)
        {
            // Another request already stored this key — race condition
            // Return the existing result
            existing = await _db.IdempotencyKeys
                .FirstOrDefaultAsync(k => k.Key == idempotencyKey);
            return JsonSerializer.Deserialize<T>(existing!.Result);
        }

        return result;
    }
}

// Usage in a payment controller
[ApiController]
[Route("api/[controller]")]
public class PaymentsController : ControllerBase
{
    private readonly IdempotencyService _idempotency;
    private readonly IPaymentService _paymentService;

    public PaymentsController(
        IdempotencyService idempotency,
        IPaymentService paymentService)
    {
        _idempotency = idempotency;
        _paymentService = paymentService;
    }

    [HttpPost]
    public async Task<ActionResult<PaymentResult>> Charge(
        [FromBody] ChargeRequest request,
        [FromHeader(Name = "X-Idempotency-Key")] string idempotencyKey)
    {
        var result = await _idempotency.ExecuteOnceAsync(
            idempotencyKey,
            () => _paymentService.ChargeAsync(request));

        return Ok(result);
    }
}

Idempotency Key Generation

11. Distributed Locking and Lease-Based Coordination

Distributed locks provide mutual exclusion across multiple nodes. They are used to prevent concurrent Saga executions from conflicting, to coordinate leaders in high-availability systems, and to implement singletons (e.g., only one node should process a given partition). While not a distributed transaction protocol, distributed locks are a building block used by several transaction patterns.

Redlock Algorithm

The Redlock algorithm, proposed by Antirez (Salvatore Sanfilippo), provides a distributed lock using N independent Redis instances. The client attempts to acquire the lock from each instance. If the lock is acquired from a majority (N/2 + 1) of instances within the lock's TTL, the lock is held. The lock's validity time is the minimum TTL minus the time taken to acquire the lock.

C#
public class DistributedLock : IAsyncDisposable
{
    private readonly IDatabase[] _redisInstances;
    private readonly string _resource;
    private readonly string _lockValue;
    private readonly TimeSpan _ttl;
    private bool _isAcquired;

    public DistributedLock(
        IDatabase[] redisInstances,
        string resource,
        TimeSpan ttl)
    {
        _redisInstances = redisInstances;
        _resource = resource;
        _lockValue = Guid.NewGuid().ToString();
        _ttl = ttl;
    }

    public async Task<bool> AcquireAsync()
    {
        var acquiredCount = 0;
        var startTime = DateTimeOffset.UtcNow;

        // Try to acquire lock from each Redis instance
        foreach (var instance in _redisInstances)
        {
            try
            {
                var acquired = await instance.StringSetAsync(
                    $"lock:{_resource}",
                    _lockValue,
                    _ttl,
                    When.NotExists);

                if (acquired)
                    acquiredCount++;
            }
            catch
            {
                // Instance unreachable — don't count it
            }
        }

        // Check if majority acquired and lock is still valid
        var elapsed = DateTimeOffset.UtcNow - startTime;
        var lockValidity = _ttl - elapsed;
        var majority = _redisInstances.Length / 2 + 1;

        _isAcquired = acquiredCount >= majority && lockValidity > TimeSpan.Zero;
        return _isAcquired;
    }

    public async Task<bool> ReleaseAsync()
    {
        if (!_isAcquired) return false;

        // Only release if the value matches (not stolen by another client)
        var released = true;
        foreach (var instance in _redisInstances)
        {
            try
            {
                var script = @"
                    if redis.call('get', KEYS[1]) == ARGV[1] then
                        return redis.call('del', KEYS[1])
                    else
                        return 0
                    end";
                var result = await instance.ScriptEvaluateAsync(
                    script,
                    new RedisKey[] { $"lock:{_resource}" },
                    new RedisValue[] { _lockValue });
                released &= (long)result == 1;
            }
            catch
            {
                released = false;
            }
        }

        _isAcquired = false;
        return released;
    }

    public async ValueTask DisposeAsync()
    {
        await ReleaseAsync();
    }
}

Lease-Based Coordination

For distributed transaction coordinators, lease-based coordination is more robust than simple locking. A lease gives a node exclusive rights to a role (e.g., "saga coordinator") for a fixed period. If the lease expires, another node can take over. This prevents the split-brain problem where two nodes believe they are the coordinator.

In practice, systems like ZooKeeper, etcd, and Consul provide lease primitives with strong consistency guarantees. The saga coordinator acquires a lease before executing a saga. If the coordinator crashes, the lease expires, and another node can detect the failure and take over. The new coordinator reads the saga log and resumes or compensates incomplete sagas.

12. Event Sourcing and CQRS for Transactional Consistency

Event Sourcing stores the state of an entity as a sequence of events rather than as a single snapshot. Instead of updating a row in a database, you append an event to an event log. The current state is derived by replaying all events. This provides a complete audit trail, enables temporal queries, and naturally supports the Outbox pattern — the event log IS the outbox.

Event Sourcing with Aggregate Roots

C#
public abstract class AggregateRoot
{
    public Guid Id { get; protected set; }
    public int Version { get; protected set; }
    private readonly List<DomainEvent> _uncommittedEvents = new();

    public IReadOnlyList<DomainEvent> UncommittedEvents =>
        _uncommittedEvents.AsReadOnly();

    protected void Apply(DomainEvent @event)
    {
        @event.Version = Version + 1;
        When(@event);
        Version = @event.Version;
        _uncommittedEvents.Add(@event);
    }

    protected abstract void When(DomainEvent @event);

    public void ClearUncommittedEvents()
    {
        _uncommittedEvents.Clear();
    }
}

public class Order : AggregateRoot
{
    public OrderStatus Status { get; private set; }
    public decimal Total { get; private set; }
    public List<OrderItem> Items { get; private set; } = new();

    // Factory method
    public static Order Create(Guid orderId, List<OrderItem> items)
    {
        var order = new Order();
        order.Apply(new OrderCreated
        {
            AggregateId = orderId,
            Items = items,
            Total = items.Sum(i => i.Price * i.Quantity)
        });
        return order;
    }

    public void ConfirmPayment(Guid paymentId)
    {
        if (Status != OrderStatus.Created)
            throw new InvalidOperationException(
                $"Cannot confirm payment in {Status} state");

        Apply(new OrderPaymentConfirmed
        {
            AggregateId = Id,
            PaymentId = paymentId
        });
    }

    public void Cancel(string reason)
    {
        if (Status == OrderStatus.Shipped)
            throw new InvalidOperationException(
                "Cannot cancel shipped order");

        Apply(new OrderCancelled
        {
            AggregateId = Id,
            Reason = reason
        });
    }

    protected override void When(DomainEvent @event)
    {
        switch (@event)
        {
            case OrderCreated e:
                Id = e.AggregateId;
                Items = e.Items;
                Total = e.Total;
                Status = OrderStatus.Created;
                break;
            case OrderPaymentConfirmed:
                Status = OrderStatus.PaymentConfirmed;
                break;
            case OrderCancelled:
                Status = OrderStatus.Cancelled;
                break;
        }
    }
}

CQRS — Command Query Responsibility Segregation

CQRS separates the write model (commands) from the read model (queries). The write model uses Event Sourcing to ensure every state change is recorded as an immutable event. The read model is a denormalized projection optimized for specific query patterns. Changes to the write model are published as events, and a projector updates the read model.

In a distributed transaction context, CQRS provides a significant advantage: the write side can use Event Sourcing with the Outbox pattern to guarantee reliable event publishing, while the read side can use eventual consistency without affecting the correctness of writes. The read model can be rebuilt from the event log at any time, making it resilient to bugs in the projection logic.

Event Sourcing and Saga Integration

Event Sourcing integrates naturally with Sagas. A Saga can publish a "start" event, and each service consumes the event, performs its step, and publishes a completion event. The Saga coordinator (or the event chain in choreography) listens for all completion events. If a step fails, the failure event triggers compensations. Because every event is stored durably, you can replay the event log to reconstruct the Saga's history and verify that all compensations executed correctly.

Event Sourcing also simplifies debugging distributed transaction issues. When a Saga fails in production, you can query the event store for the specific Saga's correlation ID and see every event that occurred, in order, with timestamps. This is far more valuable than scattered logs across multiple services.

13. Transaction SAGA in .NET — Full Implementation

This section provides a complete, production-ready Saga implementation in C# with a state machine, persistent saga log, compensations, and idempotency. This is not a toy example — it is the pattern used in real financial and e-commerce systems.

Saga State Machine

C#
public enum SagaStatus
{
    Pending,
    Step1_Completed,
    Step2_Completed,
    Step3_Completed,
    Compensating,
    Completed,
    Failed
}

public class SagaInstance
{
    public Guid SagaId { get; set; }
    public Guid CorrelationId { get; set; }
    public SagaStatus Status { get; set; }
    public string Payload { get; set; } = string.Empty;
    public int CurrentStep { get; set; }
    public DateTimeOffset CreatedAt { get; set; }
    public DateTimeOffset? CompletedAt { get; set; }
}

public class OrderSagaState
{
    public Guid OrderId { get; set; }
    public Guid? PaymentId { get; set; }
    public Guid? ReservationId { get; set; }
    public Guid? ShipmentId { get; set; }
    public string CustomerId { get; set; } = string.Empty;
    public decimal Total { get; set; }
    public List<OrderItem> Items { get; set; } = new();
}

Saga Executor with Persistence

C#
public class PersistentSagaExecutor
{
    private readonly ISagaRepository _repository;
    private readonly IPaymentService _paymentService;
    private readonly IInventoryService _inventoryService;
    private readonly IShippingService _shippingService;
    private readonly ILogger<PersistentSagaExecutor> _logger;

    public PersistentSagaExecutor(
        ISagaRepository repository,
        IPaymentService paymentService,
        IInventoryService inventoryService,
        IShippingService shippingService,
        ILogger<PersistentSagaExecutor> logger)
    {
        _repository = repository;
        _paymentService = paymentService;
        _inventoryService = inventoryService;
        _shippingService = shippingService;
        _logger = logger;
    }

    public async Task ExecuteAsync(OrderSagaState state)
    {
        var saga = new SagaInstance
        {
            SagaId = Guid.NewGuid(),
            CorrelationId = Guid.NewGuid(),
            Status = SagaStatus.Pending,
            Payload = JsonSerializer.Serialize(state),
            CurrentStep = 0,
            CreatedAt = DateTimeOffset.UtcNow
        };

        await _repository.SaveAsync(saga);

        try
        {
            // Step 1: Charge Payment
            state.PaymentId = await WithIdempotency(
                saga.SagaId, "ChargePayment",
                () => _paymentService.ChargeAsync(
                    saga.CorrelationId, state.CustomerId, state.Total));
            saga.Status = SagaStatus.Step1_Completed;
            saga.CurrentStep = 1;
            await _repository.UpdateAsync(saga);

            // Step 2: Reserve Inventory
            state.ReservationId = await WithIdempotency(
                saga.SagaId, "ReserveInventory",
                () => _inventoryService.ReserveAsync(
                    saga.CorrelationId, state.Items));
            saga.Status = SagaStatus.Step2_Completed;
            saga.CurrentStep = 2;
            await _repository.UpdateAsync(saga);

            // Step 3: Create Shipment
            state.ShipmentId = await WithIdempotency(
                saga.SagaId, "CreateShipment",
                () => _shippingService.CreateShipmentAsync(
                    saga.CorrelationId, state.Items));
            saga.Status = SagaStatus.Completed;
            saga.CompletedAt = DateTimeOffset.UtcNow;
            await _repository.UpdateAsync(saga);

            _logger.LogInformation(
                "Saga {SagaId} completed successfully", saga.SagaId);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex,
                "Saga {SagaId} failed at step {Step}",
                saga.SagaId, saga.CurrentStep);

            saga.Status = SagaStatus.Compensating;
            await _repository.UpdateAsync(saga);

            await CompensateAsync(saga, state);
        }
    }

    private async Task CompensateAsync(SagaInstance saga, OrderSagaState state)
    {
        // Compensate in reverse order based on current step
        if (state.ShipmentId.HasValue)
        {
            await WithIdempotency(saga.SagaId, "CancelShipment",
                () => _shippingService.CancelShipmentAsync(
                    saga.CorrelationId, state.ShipmentId.Value));
        }

        if (state.ReservationId.HasValue)
        {
            await WithIdempotency(saga.SagaId, "ReleaseInventory",
                () => _inventoryService.ReleaseAsync(
                    saga.CorrelationId, state.ReservationId.Value));
        }

        if (state.PaymentId.HasValue)
        {
            await WithIdempotency(saga.SagaId, "RefundPayment",
                () => _paymentService.RefundAsync(
                    saga.CorrelationId, state.PaymentId.Value));
        }

        saga.Status = SagaStatus.Failed;
        saga.CompletedAt = DateTimeOffset.UtcNow;
        await _repository.UpdateAsync(saga);
    }

    private async Task<T> WithIdempotency<T>(
        Guid sagaId,
        string stepName,
        Func<Task<T>> operation)
    {
        var key = $"{sagaId}:{stepName}";
        var existing = await _repository.GetStepResultAsync<T>(key);
        if (existing != null)
            return existing;

        var result = await operation();
        await _repository.SaveStepResultAsync(key, result);
        return result;
    }
}

Saga Recovery on Startup

C#
public class SagaRecoveryService : IHostedService
{
    private readonly ISagaRepository _repository;
    private readonly PersistentSagaExecutor _executor;
    private readonly ILogger<SagaRecoveryService> _logger;

    public SagaRecoveryService(
        ISagaRepository repository,
        PersistentSagaExecutor executor,
        ILogger<SagaRecoveryService> logger)
    {
        _repository = repository;
        _executor = executor;
        _logger = logger;
    }

    public async Task StartAsync(CancellationToken cancellationToken)
    {
        // Find all incomplete sagas and resume them
        var incompleteSagas = await _repository
            .GetIncompleteSagasAsync();

        foreach (var saga in incompleteSagas)
        {
            _logger.LogInformation(
                "Recovering saga {SagaId} (status: {Status})",
                saga.SagaId, saga.Status);

            try
            {
                var state = JsonSerializer.Deserialize<OrderSagaState>(
                    saga.Payload)!;

                // Reset to the last completed step and re-execute
                // The WithIdempotency guard prevents duplicate operations
                await _executor.ExecuteAsync(state);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex,
                    "Failed to recover saga {SagaId}",
                    saga.SagaId);
            }
        }
    }

    public Task StopAsync(CancellationToken cancellationToken)
        => Task.CompletedTask;
}

14. Testing Distributed Transactions

Testing distributed transactions is significantly harder than testing local transactions because failures are non-deterministic, timing-dependent, and involve multiple systems. A comprehensive testing strategy must cover normal flows, failure scenarios, retries, idempotency, and recovery.

Testing Levels

Saga Unit Test Example

C#
public class OrderSagaTests
{
    private readonly Mock<IPaymentService> _paymentService = new();
    private readonly Mock<IInventoryService> _inventoryService = new();
    private readonly Mock<IShippingService> _shippingService = new();
    private readonly Mock<ISagaRepository> _repository = new();

    [Fact]
    public async Task Execute_WhenInventoryFails_RefundsPaymentAndCancelsOrder()
    {
        // Arrange
        _paymentService.Setup(s => s.ChargeAsync(
                It.IsAny<Guid>(), It.IsAny<string>(), It.IsAny<decimal>()))
            .ReturnsAsync(Guid.NewGuid());

        _inventoryService.Setup(s => s.ReserveAsync(
                It.IsAny<Guid>(), It.IsAny<List<OrderItem>>()))
            .ThrowsAsync(new InsufficientInventoryException());

        var executor = new PersistentSagaExecutor(
            _repository.Object,
            _paymentService.Object,
            _inventoryService.Object,
            _shippingService.Object,
            Substitute.For<ILogger<PersistentSagaExecutor>>());

        var state = new OrderSagaState
        {
            OrderId = Guid.NewGuid(),
            CustomerId = "CUST-001",
            Total = 99.99m,
            Items = new List<OrderItem>
            {
                new() { ProductId = "PROD-1", Quantity = 1, Price = 99.99m }
            }
        };

        // Act
        await executor.ExecuteAsync(state);

        // Assert — payment was charged and then refunded
        _paymentService.Verify(s => s.ChargeAsync(
            It.IsAny<Guid>(), It.IsAny<string>(), 99.99m),
            Times.Once);
        _paymentService.Verify(s => s.RefundAsync(
            It.IsAny<Guid>(), It.IsAny<Guid>()),
            Times.Once);

        // Inventory was never reserved (it failed)
        _inventoryService.Verify(s => s.ReserveAsync(
            It.IsAny<Guid>(), It.IsAny<List<OrderItem>>()),
            Times.Once);

        // Shipping was never called
        _shippingService.Verify(s => s.CreateShipmentAsync(
            It.IsAny<Guid>(), It.IsAny<List<OrderItem>>()),
            Times.Never);
    }

    [Fact]
    public async Task Execute_WhenPaymentAlreadyCharged_Idempotent()
    {
        // Arrange — simulate idempotency key already processed
        var sagaId = Guid.NewGuid();
        var paymentId = Guid.NewGuid();

        _repository.Setup(r => r.GetStepResultAsync<Guid>(
            $"{sagaId}:ChargePayment"))
            .ReturnsAsync(paymentId);

        _inventoryService.Setup(s => s.ReserveAsync(
                It.IsAny<Guid>(), It.IsAny<List<OrderItem>>()))
            .ReturnsAsync(Guid.NewGuid());

        _shippingService.Setup(s => s.CreateShipmentAsync(
                It.IsAny<Guid>(), It.IsAny<List<OrderItem>>()))
            .ReturnsAsync(Guid.NewGuid());

        // Act & Assert — payment should NOT be charged again
        // Idempotency guard returns cached paymentId
        // Verify that ChargeAsync is never called
    }
}

Chaos Engineering for Distributed Transactions

Chaos testing injects real failures into your system to verify that your distributed transaction logic handles them correctly. Key scenarios to test include:

• Network partition between coordinator and participant: The participant should timeout and the coordinator should handle the retry or compensation.
• Service crash after commit but before acknowledgment: The coordinator should retry the operation (idempotent) or the compensation.
• Duplicate message delivery: The consumer should detect and discard duplicates using idempotency keys.
• Outbox relay crash during event publishing: On restart, the relay should pick up unpublished events and re-publish them.
• Database failure during saga step: The saga should compensate all completed steps.
• Concurrent saga executions with shared resources: Locking or reservation logic should prevent double-spending or double-reservation.

Tools like Toxiproxy can simulate network conditions (latency, packet loss, connection resets) between services. Testcontainers can spin up real databases and message brokers in Docker for integration testing. Custom middleware can inject faults at specific points in the saga flow.

15. Performance Trade-offs and Latency Analysis

Every distributed transaction protocol has a performance cost. Understanding these costs is essential for making informed architecture decisions and for meeting latency SLAs.

Latency Breakdown by Protocol

Optimizing Saga Performance

Although Sagas execute steps sequentially (because step N+1 depends on the result of step N), there are several optimizations:

• Parallel steps: If steps do not depend on each other, execute them in parallel. For example, charging payment and reserving inventory can run concurrently if they do not share data.
• Pipeline prefetch: The Outbox relay can batch-publish events, reducing the per-event overhead. Instead of publishing one event at a time, the relay fetches 50-100 events and publishes them in a single batch.
• Local event publishing: If the downstream service is in the same process (e.g., an in-process event bus), skip the message broker entirely for lower latency.
• Async processing: The Saga coordinator should be non-blocking. Use async/await in C# to avoid thread pool starvation during long waits.
• Connection pooling: Reuse database and message broker connections across Saga executions. Connection establishment is expensive.

C#
// Parallel Saga execution for independent steps
public async Task ExecuteParallelStepsAsync(OrderSagaState state)
{
    // These steps are independent — they can run concurrently
    var paymentTask = WithIdempotency(
        state.OrderId, "ChargePayment",
        () => _paymentService.ChargeAsync(
            state.OrderId, state.CustomerId, state.Total));

    var inventoryTask = WithIdempotency(
        state.OrderId, "ReserveInventory",
        () => _inventoryService.ReserveAsync(
            state.OrderId, state.Items));

    // Wait for both to complete
    await Task.WhenAll(paymentTask, inventoryTask);

    state.PaymentId = paymentTask.Result;
    state.ReservationId = inventoryTask.Result;

    // This step depends on both previous steps
    state.ShipmentId = await WithIdempotency(
        state.OrderId, "CreateShipment",
        () => _shippingService.CreateShipmentAsync(
            state.OrderId, state.Items));
}

16. When to Use Which Protocol — Decision Matrix

Choosing the right distributed transaction protocol depends on your specific requirements. This decision matrix provides guidance based on common scenarios.

Decision Flow

Protocol Selection by Domain

17. Real-World Architecture Patterns

Pattern 1: E-Commerce Order Processing

An e-commerce platform uses an orchestration Saga with the Outbox pattern. The OrderService is the orchestrator. When a customer places an order:

1. OrderService creates the order and writes an OrderCreated event to the outbox in a single database transaction.
2. The Outbox relay publishes OrderCreated to Kafka.
3. PaymentService consumes OrderCreated, charges the card, and publishes PaymentProcessed to its outbox.
4. InventoryService consumes PaymentProcessed, reserves stock, and publishes InventoryReserved to its outbox.
5. ShippingService consumes InventoryReserved and creates a shipping label.

If InventoryService fails (out of stock), it publishes InventoryReservationFailed. OrderService consumes this event and publishes a compensating command: RefundPayment and CancelOrder. Each service processes the compensation idempotently.

Pattern 2: Financial Transfer

A banking system uses 2PC for intra-bank transfers (between accounts in the same database cluster) and TCC for cross-bank transfers. The 2PC provides strong consistency for local transfers where latency must be under 50ms. TCC provides soft-lock guarantees for cross-bank transfers that may take seconds to complete.

Pattern 3: Travel Booking

A travel platform uses a choreography Saga for booking flights, hotels, and car rentals from different providers. Each provider publishes events independently: FlightBooked, HotelBooked, CarBooked. A coordination service listens for all events and sends a confirmation. If any booking fails, a BookingFailed event triggers cancellations of already-completed bookings. The booking holds (TCC-style) are managed by each provider independently — the airline holds the seat, the hotel holds the room — until the entire booking is confirmed.

Pattern 4: Event-Driven Microservices with CQRS

A SaaS platform uses Event Sourcing with CQRS for its core domain. Every state change is an event stored in EventStoreDB. The write side appends events; the read side projects them into optimized views. Sagas orchestrate cross-aggregate workflows. The event store doubles as the Outbox — downstream services consume events from the event store's subscription mechanism. This eliminates the separate Outbox table entirely.

18. Interview Q&A — Senior+ Level

Q1: What is the difference between 2PC and Saga?

2PC provides true atomicity by holding locks across all participants during a prepare-commit cycle. No participant sees intermediate states. Sagas provide logical atomicity via compensating transactions — each step commits immediately, and compensations undo completed steps on failure. The key trade-off is consistency vs availability: 2PC blocks during coordinator failure (lower availability), while Sagas never hold distributed locks (higher availability) but expose intermediate states to other transactions.

Q2: When should I use 2PC vs Saga?

Use 2PC for short-lived transactions (under 100ms) within a single trust boundary — same organization, same data center, reliable participants. 2PC is used internally by most databases for distributed joins and by XA-compliant transaction managers. Use Sagas for cross-service transactions, long-running workflows (seconds to minutes), and when high availability is required. Sagas are the default choice for most microservices architectures.

Q3: How do you handle a Saga step that cannot be compensated?

Some operations cannot be undone — sending an email, publishing a notification, calling a third-party API. The standard approach is to make these operations the LAST step in the Saga. By the time the Saga reaches the non-compensatable step, all preceding steps have been committed and confirmed. If the non-compensatable step fails, you compensate the preceding steps. If it succeeds, the Saga is complete. If you cannot control the ordering, use the "Picky Commit" pattern — defer the non-compensatable step until a confirmation step (e.g., a scheduled job that confirms the order after a short delay).

Q4: What is the Outbox pattern and why is it essential?

The Outbox pattern solves the dual-write problem: writing to a database and publishing a message atomically. You write both the business data and an integration event to the same database in a single local transaction. A separate relay process reads unpublished events and publishes them to the message broker. This guarantees at-least-once delivery without distributed transactions. The Outbox pattern is essential for reliable event-driven microservices because without it, database writes and message sends are not atomic — a crash between them leaves the system inconsistent.

Q5: How do you test distributed transaction failure scenarios?

Test at multiple levels. Unit tests verify saga state machine logic and compensation ordering with mocked services. Integration tests use Testcontainers to run real databases and message brokers, verifying the Outbox relay and message delivery. Chaos tests inject real failures using Toxiproxy (network partitions, latency, packet loss) and custom fault injection middleware. Key scenarios to test: coordinator crash mid-saga, participant crash after commit, duplicate message delivery, outbox relay crash, and concurrent saga executions with shared resources. Use correlation IDs to trace a single saga execution across all services in your test assertions.

Q6: What is idempotency and why is it critical for distributed transactions?

Idempotency means an operation produces the same result whether executed once or multiple times. It is critical because in distributed systems, retries are inevitable — network timeouts, service crashes, and message broker redelivery all cause operations to be executed more than once. Without idempotency, retries would create duplicate charges, duplicate orders, or inconsistent state. Every operation in a Saga, every message consumer in an event-driven system, and every API endpoint exposed to external clients must be idempotent. Use idempotency keys (UUIDs) stored with unique database constraints to detect and prevent duplicates.

Q7: Explain the CAP theorem and how it relates to distributed transactions.

The CAP theorem states that a distributed system can provide at most two of three guarantees: Consistency (all nodes see the same data), Availability (every request gets a response), and Partition tolerance (the system works despite network failures). Since network partitions are inevitable, the real choice is between CP (consistency over availability) and AP (availability over consistency). 2PC is a CP protocol — it guarantees atomicity but reduces availability during failures. Sagas are an AP protocol — they guarantee eventual consistency while maintaining availability. The PACELC theorem further extends this: even when the network is healthy, you must choose between Latency and Consistency.

Q8: How do you monitor distributed transactions in production?

Monitor at three levels. First, track saga completion rates, compensation rates, and step-level latency using distributed tracing (Jaeger, Zipkin, Application Insights). Each saga execution should have a correlation ID that appears in all traces across services. Second, monitor the Outbox relay lag — the time between event creation and publication. High lag indicates relay performance issues. Third, track idempotency key collision rates — a high rate indicates excessive retries, which may signal upstream problems. Set up alerts for: compensation rate exceeding threshold, Outbox relay lag exceeding 30 seconds, and saga execution time exceeding SLA.

Q9: What are the alternatives to 2PC and Saga?

Alternatives include: TCC (Try-Confirm-Cancel) for stronger consistency without blocking; Event Sourcing with CQRS for audit-trail-friendly consistency; the Outbox pattern for reliable event publishing; transactional messaging (e.g., Azure Service Bus transactions, RabbitMQ publisher confirms with local transactions); NewSQL databases (CockroachDB, Google Spanner) that provide ACID transactions across nodes using consensus protocols; and the Transactional Inbox pattern for ensuring exactly-once processing of messages. For most systems, the combination of Saga + Outbox + Idempotency covers the vast majority of use cases.

Q10: Design a distributed transaction system for an e-commerce platform.

Use an orchestration Saga with the Outbox pattern. The OrderService is the orchestrator and maintains the saga state machine. Each service (Order, Payment, Inventory, Shipping) writes business data and integration events to its local database via the Outbox pattern. A Kafka-backed Outbox relay publishes events with at-least-once delivery. Each consumer uses idempotency keys to handle duplicate messages. Compensations are triggered by failure events — if inventory reservation fails, a PaymentRefund event is published and the order is cancelled. Run saga recovery on service startup to handle incomplete sagas from crashes. Monitor with distributed tracing using a shared correlation ID. The architecture provides eventual consistency with high availability, suitable for an e-commerce platform that needs to handle thousands of orders per minute.

Frequently Asked Questions

What is the CAP theorem and how does it relate to distributed transactions?

The CAP theorem states that a distributed system can provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Since network partitions are inevitable, the real choice is between CP (consistency over availability, e.g., 2PC) and AP (availability over consistency, e.g., Saga). Distributed transaction protocols operate on this spectrum.

What is the difference between 2PC and 3PC?

2PC has two phases (prepare and commit) and is blocking — participants hold locks during coordinator failure. 3PC adds a pre-commit phase that allows participants to timeout and decide independently, making it non-blocking. However, 3PC requires synchronous messaging and is rarely used in practice due to its sensitivity to network partitions.

When should I use the Saga pattern over 2PC?

Use Sagas when transactions span multiple services, are long-running, or require high availability. Sagas avoid distributed locks and use compensating transactions instead. Use 2PC when transactions are short-lived, participants are reliable, and strong consistency is required within a single trust boundary.

What is the Outbox pattern and why is it important?

The Outbox pattern solves the dual-write problem by writing business data and integration events to the same database in a single local transaction. A separate relay process reads the outbox and publishes events to the message broker. This guarantees at-least-once delivery without distributed transactions, and is a building block for Saga implementations.

How do you handle idempotency in distributed transactions?

Assign a unique idempotency key (UUID) to every operation. Store processed keys in a database table with a unique constraint. Before executing an operation, check if the key was already processed. If yes, return the cached result. For high-throughput systems, use Redis with TTL. Every operation in a saga or retry scenario must be idempotent.

What is the difference between orchestration and choreography sagas?

Orchestration uses a central coordinator that manages the saga flow and handles compensations. It is easier to monitor, debug, and modify. Choreography uses events — each service publishes and subscribes to events independently. It is more loosely coupled but harder to trace and manage failure flows. Most teams start with orchestration.

What are compensating transactions and how do they work?

Compensating transactions are the logical undo of a previously committed step. If charging a payment fails after the payment was already processed, the compensation is a refund. Compensations must be idempotent and must handle cases where the original operation partially completed. They do not erase history — they create a new correcting transaction.