1. Introduction — Why Webhooks and Event-Driven Architecture

In modern software systems, services need to communicate with each other in real-time. Traditionally, this was achieved through polling — a client repeatedly queries a server to check if new data is available. While simple, polling is wasteful. It consumes network bandwidth, wastes CPU cycles, and introduces latency proportional to the polling interval. Webhooks solve this fundamental problem by flipping the communication model from pull to push.

A webhook is essentially a user-defined HTTP callback. When a specific event occurs in a source system, the system makes an HTTP POST request to a pre-configured URL provided by the consumer. This push-based model ensures that the receiving system is notified instantly without ever needing to poll for changes. The concept is elegant in its simplicity, yet building a reliable, secure, and scalable webhook delivery system is one of the most challenging problems in distributed systems engineering.

Consider how widespread webhooks have become across the industry. GitHub sends webhooks when code is pushed, pull requests are opened, or issues are updated. Stripe notifies merchant backends when payments succeed or fail. Shopify triggers webhooks when orders are placed or inventory changes. Slack uses webhooks to deliver messages to channels. Discord, Twilio, SendGrid, PayPal, Square, and virtually every major SaaS platform exposes webhooks as a first-class integration primitive.

Polling vs. Webhooks: A Fundamental Shift

To understand the value proposition, let us compare the two approaches in detail. With polling, a client sends a request every N seconds asking "Is there anything new?" The server must process this request, query its database, and respond — even if nothing has changed. For a system with 10,000 clients polling every 10 seconds, that is 1,000 requests per second of pure overhead. With webhooks, the server sends a single HTTP request to each registered client only when an event actually occurs. If there are no events, there are zero requests. This is not just an optimization; it is an architectural paradigm shift that reduces waste and improves responsiveness simultaneously.

However, webhooks introduce their own complexity. What happens when the receiving server is down? What if the webhook payload is too large? How do you guarantee that a webhook was delivered exactly once? How do you secure webhooks against spoofing and replay attacks? How do you debug failures when your webhook is being delivered to a third-party server you do not control? These are the questions that make webhook system design a genuinely challenging interview topic and a critical production concern.

The Event-Driven Architecture Context

Webhooks are one manifestation of the broader event-driven architecture (EDA) pattern. In EDA, components communicate by producing and consuming events — immutable records of something that happened. Within a single service, events might flow through an internal message bus like RabbitMQ or Kafka. Webhooks extend this pattern to cross-boundary communication, delivering events to external systems over the public internet via HTTP. Understanding this context is important because many of the design patterns you would use internally — event sourcing, idempotency keys, dead letter queues — apply directly to webhook delivery systems.

In this article, we will design a complete webhook and event delivery system from the ground up. We will cover requirements analysis, capacity estimation, data modeling, API design, high-level architecture, delivery mechanics, security, retry strategies, monitoring, and scaling. By the end, you will have a thorough understanding of how to design and build a production-grade webhook system that can handle millions of events per day with reliability, security, and operational visibility.

2. Requirements Gathering

Before diving into architecture, we must clearly define what the system needs to do. A well-defined set of requirements guides every downstream design decision. For a webhook and event delivery system, the requirements fall into several categories.

Functional Requirements

1. Event Ingestion: The system must accept events from internal services via a publish API. Events carry a type, payload, and metadata (timestamp, source, correlation ID).
2. Endpoint Registration: Users must be able to register HTTP endpoints (URLs) and subscribe to specific event types. Each endpoint can have its own filtering rules and delivery configuration.
3. Delivery: When an event matches a subscription, the system must deliver the event payload as an HTTP POST request to the registered endpoint. The request must include a signature header for verification.
4. Retry on Failure: If the target endpoint returns a non-2xx status code or times out, the system must retry delivery with exponential backoff and jitter.
5. Delivery Status Tracking: Every delivery attempt must be logged with request/response details, timestamps, and status codes. Users must be able to query delivery history for debugging.
6. Manual Replay: Users must be able to manually replay failed or historical deliveries for a given endpoint.
7. Event Filtering: Users should be able to filter which events are delivered based on event type, payload fields, or custom expressions.

Non-Functional Requirements

Ordering Requirements

Ordering is a nuanced requirement in webhook systems. Many use cases, such as payment processing and order management, require that events for a given entity (e.g., a specific order) are delivered in the order they were produced. However, guaranteeing global ordering across all events is prohibitively expensive. The standard approach is to provide per-entity ordering within a partitioned dispatch model. This means two events for the same order will arrive in order, but events for different orders may arrive out of order.

Security Requirements

1. Every webhook delivery must include a cryptographic signature that the receiver can use to verify authenticity.
2. Delivery endpoints must use HTTPS with valid TLS certificates.
3. The system must support secret rotation so that consumers can periodically update their verification keys without downtime.
4. Replay protection via timestamps and nonces must be built in.
5. Optionally, support mTLS (mutual TLS) for high-security use cases.

These requirements define the boundaries of our design space. Every architectural decision we make from this point forward should be traceable back to one of these requirements.

3. Capacity Estimation

Capacity estimation grounds the design in reality. Without concrete numbers, it is easy to either over-engineer or under-provision. Let us establish reasonable assumptions for a mid-to-large scale webhook delivery system.

Assumptions

Storage Estimation

Let us estimate the storage needed for delivery logs over a 90-day retention period. At 500,000 peak deliveries per second, with an average of 150,000 per second sustained, the daily delivery volume is approximately 13 billion deliveries. Each delivery log entry includes request headers, payload, response headers, response body, and metadata — roughly 5 KB per entry.

Daily storage: 13 billion multiplied by 5 KB equals 65 TB per day. Over 90 days, that is approximately 5.85 petabytes. This is clearly impractical for a traditional database. We need a tiered storage strategy: recent delivery logs (last 7 days) in a fast database like Cassandra or ScyllaDB, and older logs in object storage like S3 with a compression ratio of approximately 10:1, bringing the 90-day total to roughly 585 TB in compressed form.

Network Bandwidth

At 500,000 deliveries per second with an average 2 KB payload, the outbound bandwidth is approximately 1 GB/s or 8 Gbps. This is well within the capacity of a single data center, but at scale across multiple regions, network planning becomes critical. The return traffic (responses) is negligible at approximately 250 MB/s.

Compute Estimation

HTTP delivery is I/O-bound, not CPU-bound. A single delivery worker thread can maintain approximately 50 to 100 concurrent HTTP connections. At 500,000 QPS, we need roughly 5,000 to 10,000 delivery worker threads. Across a fleet of 64-core machines, this translates to approximately 100 to 200 delivery worker instances, assuming each instance handles 2,500 to 5,000 QPS.

Additionally, we need dispatch workers to filter and fan-out events, retry scheduler workers to manage the retry queue, and API servers to handle registration and status queries. A reasonable estimate is 50 dispatch workers, 20 retry scheduler instances, and 10 API servers.

4. Data Model Design

The data model is the backbone of the system. It defines what information we store and how different entities relate to each other. Let us design the core entities for our webhook delivery system.

Entity Relationship Diagram

Endpoint

An endpoint represents a registered HTTP URL that will receive webhook deliveries. Each endpoint belongs to a customer and has its own URL, status (active, paused, disabled), and configuration. The configuration includes timeout settings, retry policy overrides, and content type preferences. Endpoints are identified by a unique UUID and are the top-level entity in the delivery hierarchy.

Subscription

A subscription links an endpoint to one or more event types with optional filter expressions. For example, an endpoint might subscribe to payment.completed events but only for payments in the EUR currency. Filter expressions are stored as JSON and evaluated by the dispatch pipeline before delivery. Each subscription is scoped to exactly one endpoint and one event type, but an endpoint can have many subscriptions across different event types.

Event

An event represents something that happened in the source system. Events have a type (e.g., order.created), a payload (the event data), a source identifier, and a timestamp. Events are immutable once created. They are stored in an append-only table or log. The event store is the system of record for all events and supports replay from any point in time.

Delivery

A delivery represents a single attempt to deliver an event to a specific endpoint. A delivery is created when the dispatch pipeline matches an event to a subscription. It tracks the current status (pending, delivered, failed, dead_lettered), the number of attempts made, and the next retry time. Deliveries are the central entity in the retry system and represent the unit of work for delivery workers.

Delivery Attempt

Each delivery attempt is logged with the HTTP status code, response time, response headers, response body (truncated), and any error messages. This data is invaluable for debugging and is retained according to the configured retention policy. Delivery attempts are append-only — new attempts are always inserted, never updated.

Delivery Secret

Delivery secrets store the HMAC signing key for each endpoint. Secrets are stored as salted hashes (not plaintext) and support rotation. An endpoint can have multiple active secrets during a rotation window. The most recent active secret is used for signing, while all active secrets are accepted for verification by consumers.

5. API Design

The API layer provides the interface for customers to register endpoints, manage subscriptions, and query delivery status. We follow RESTful conventions with JSON request and response bodies.

Core API Endpoints

Register Endpoint — Request and Response

JSON
{
  "url": "https://api.merchant.com/webhooks",
  "description": "Payment notification handler",
  "config": {
    "timeout_ms": 10000,
    "content_type": "application/json",
    "retry_policy": {
      "max_attempts": 8,
      "initial_delay_ms": 1000,
      "max_delay_ms": 3600000
    }
  },
  "verify_url": true
}

// Response — 201 Created
{
  "id": "ep_7f3a8b2c",
  "url": "https://api.merchant.com/webhooks",
  "status": "pending_verification",
  "secret": "whsec_a1b2c3d4e5f6g7h8i9j0",
  "created_at": "2026-07-01T10:00:00Z"
}

The verify_url flag triggers the challenge-response verification flow described in Section 8. The secret is returned once at creation time and never again — the customer must store it securely.

Publish Event — Request and Response

JSON
{
  "type": "payment.completed",
  "source": "billing-service",
  "data": {
    "payment_id": "pay_9x8y7z",
    "amount": 4999,
    "currency": "USD",
    "customer_id": "cust_abc123"
  },
  "idempotency_key": "idem_pay_9x8y7z_20260701"
}

// Response — 202 Accepted
{
  "event_id": "evt_f4e3d2c1",
  "status": "accepted",
  "matched_subscriptions": 12,
  "created_at": "2026-07-01T10:00:00Z"
}

The idempotency_key ensures that duplicate event submissions are deduplicated. The response immediately indicates how many subscriptions matched, giving the publisher confidence that the event will be delivered.

Get Delivery Status — Response

JSON
{
  "id": "dlv_1a2b3c",
  "endpoint_id": "ep_7f3a8b2c",
  "event_id": "evt_f4e3d2c1",
  "status": "delivered",
  "attempt_count": 2,
  "created_at": "2026-07-01T10:00:01Z",
  "delivered_at": "2026-07-01T10:00:03Z",
  "attempts": [
    {
      "attempt_number": 1,
      "http_status": 503,
      "duration_ms": 5023,
      "error": "Service Unavailable",
      "attempted_at": "2026-07-01T10:00:01Z"
    },
    {
      "attempt_number": 2,
      "http_status": 200,
      "duration_ms": 234,
      "response_body": "{\"ok\":true}",
      "attempted_at": "2026-07-01T10:00:03Z"
    }
  ]
}

6. High-Level Architecture

The architecture of a webhook delivery system consists of several interconnected components. Let us explore the high-level design and then drill into each component in subsequent sections.

Component Responsibilities

Event API Gateway: The entry point for all incoming events. It handles authentication, rate limiting, schema validation, and idempotency checking. Events that pass validation are published to the Event Bus. The gateway is a stateless HTTP server that can be horizontally scaled behind a load balancer.

Event Bus (Kafka): The central nervous system of the architecture. Kafka provides durable, ordered, partitioned event storage. Events are partitioned by entity ID to ensure per-entity ordering. The Event Bus decouples ingestion from delivery and provides natural backpressure when delivery workers fall behind.

Dispatch Processor: Consumes events from the Event Bus, looks up matching subscriptions, and creates delivery tasks. This is where fan-out occurs — a single event may produce multiple delivery tasks, one per matching subscription. The dispatch processor is the most compute-intensive component due to filter evaluation and subscription matching.

Event Filter: Evaluates subscription filter expressions against event payloads. Only events that match a subscription filters are delivered to that subscription endpoint. Filters are compiled into efficient evaluation functions at subscription creation time.

Delivery Workers: The workhorses of the system. Delivery workers consume delivery tasks and make HTTP POST requests to target endpoints. They handle timeouts, response validation, signature generation, and retry scheduling. Workers are stateless and horizontally scalable, making them the easiest component to scale.

Retry Queue: A priority queue (backed by Kafka or a dedicated message broker) that holds delivery tasks that failed and need to be retried. Tasks are ordered by next retry time and dequeued when the retry time arrives.

Dead Letter Queue (DLQ): Holds deliveries that have exhausted all retry attempts. DLQ entries can be inspected, manually replayed, or archived to cold storage. The DLQ is the last resort for deliveries that cannot be completed through automatic retries.

Data Flow Summary

The end-to-end flow is: (1) A service publishes an event to the Event API Gateway. (2) The event is validated and written to the Event Bus. (3) The Dispatch Processor consumes the event, queries subscriptions, applies filters, and creates delivery tasks. (4) Delivery Workers consume delivery tasks, generate signatures, and make HTTP POST requests. (5) On success, the delivery is marked as delivered. On failure, it enters the Retry Queue. (6) After exhausting retries, deliveries move to the DLQ.

7. Event Schema Design

A well-designed event schema is critical for the long-term maintainability of a webhook system. Events must be self-describing, versioned, and backward-compatible as the system evolves.

Event Structure

JSON
{
  "id": "evt_f4e3d2c1b0a9",
  "type": "payment.completed",
  "version": "2026-07-01",
  "source": "billing-service",
  "data": {
    "payment_id": "pay_9x8y7z",
    "amount": 4999,
    "currency": "USD",
    "customer_id": "cust_abc123"
  },
  "metadata": {
    "correlation_id": "corr_xyz789",
    "trace_id": "abc123def456"
  },
  "timestamp": "2026-07-01T10:00:00.000Z",
  "idempotency_key": "idem_pay_9x8y7z_20260701"
}

Schema Fields

Event Type Naming Convention

Event types follow a consistent dot-separated naming convention: {domain}.{entity}.{action}. For example:

• payment.completed — A payment was successfully processed
• payment.failed — A payment attempt failed
• order.created — A new order was placed
• order.shipped — An order was shipped
• user.registered — A new user account was created
• invoice.overdue — An invoice passed its due date

Schema Versioning Strategy

Schema evolution is inevitable. The key principle is backward compatibility — new fields are additive, and existing fields are never removed or renamed. When breaking changes are necessary, a new event type is introduced (e.g., payment.completed.v2) and consumers migrate gradually.

The version field allows consumers to handle multiple schema versions simultaneously. The dispatch processor includes the schema version in the delivery request headers, so consumers know which version they are receiving.

Payload Size Guidelines

Webhook payloads should be compact but informative. Aim for payloads under 10 KB. If the full entity state is larger, include a summary in the event payload and a link to a fetch API where consumers can retrieve the full data. This pattern — known as thin event, thick API — keeps webhook delivery fast while still providing access to complete data when needed.

8. Webhook Registration and Subscription

Before a webhook endpoint can receive deliveries, it must be validated. Endpoint verification ensures that the registered URL actually belongs to the customer and is capable of receiving and processing webhook events.

URL Validation Flow

Challenge-Response Verification

The challenge-response protocol works as follows. When a new endpoint is registered, the system sends a special verification request to the URL. This request includes a unique challenge token in the query string or as a header. The customer server must echo back this token in the response body within a time limit (typically 30 seconds). This proves two things: (1) the customer controls the server at the specified URL, and (2) the server is capable of handling incoming HTTP requests.

HTTP
GET /webhooks?challenge=tok_abc123def456 HTTP/1.1
Host: api.merchant.com
User-Agent: Ayodhyya-WebhookVerifier/1.0

---

HTTP/1.1 200 OK
Content-Type: application/json

{"challenge": "tok_abc123def456", "status": "ok"}

Subscription Management

Once an endpoint is verified, customers create subscriptions to specify which event types they want to receive. Each subscription is scoped to a single endpoint and can optionally include filter expressions.

JSON
{
  "event_type": "payment.completed",
  "filter": {
    "data.currency": {"eq": "USD"},
    "data.amount": {"gte": 1000}
  }
}

This subscription would only deliver payment.completed events where the currency is USD and the amount is at least 10.00 dollars. Filter expressions are evaluated by the dispatch processor before delivery tasks are created.

Endpoint Configuration Options

9. Event Dispatch Pipeline

The dispatch pipeline is where raw events are transformed into delivery tasks. This is the most complex component in the system, as it must efficiently match events to subscriptions, apply filters, and fan out to potentially thousands of endpoints.

Pipeline Stages

Stage 1: Event Consumption

The dispatch processor consumes events from the Event Bus (Kafka). Each partition is consumed by exactly one consumer instance, ensuring ordering within a partition. Events are partitioned by entity ID (e.g., order ID, payment ID), so all events for the same entity are processed by the same consumer in order.

Stage 2: Subscription Lookup

For each event, the processor looks up all subscriptions that match the event type. This is a hot-path operation that must be fast. We use a two-tier lookup strategy: first check the in-memory cache (backed by Redis), and on cache miss, query the database. The subscription cache is updated asynchronously via a change data capture (CDC) stream from the subscription table.

Stage 3: Filter Evaluation

For each matching subscription, the processor evaluates the subscription filter expression against the event payload. Filters use a JSONPath-like syntax for field access and support operators like eq, ne, gt, gte, lt, lte, in, nin, and regex. Subscriptions without filters pass all events of the matching type.

Stage 4: Delivery Task Creation

For each subscription that passes the filter, the processor creates a delivery task. A delivery task contains the endpoint URL, the serialized event payload, the signing secret, and any endpoint-specific configuration (timeout, custom headers, retry policy). Delivery tasks are written to the delivery queue (another Kafka topic or a dedicated task queue).

C#
public class DispatchProcessor : IEventConsumer
{
    private readonly ISubscriptionCache _subscriptionCache;
    private readonly IFilterEvaluator _filterEvaluator;
    private readonly IDeliveryTaskProducer _taskProducer;

    public async Task ProcessEventAsync(Event incomingEvent)
    {
        var subscriptions = await _subscriptionCache
            .GetSubscriptionsForEventTypeAsync(incomingEvent.Type);

        foreach (var subscription in subscriptions)
        {
            if (_filterEvaluator.Matches(incomingEvent, subscription.Filter))
            {
                var task = new DeliveryTask
                {
                    Id = Guid.NewGuid(),
                    EventId = incomingEvent.Id,
                    Event = incomingEvent,
                    Endpoint = subscription.Endpoint,
                    SubscriptionId = subscription.Id,
                    CreatedAt = DateTime.UtcNow,
                    AttemptNumber = 0
                };

                await _taskProducer.ProduceAsync(task);
            }
        }
    }
}

Fan-Out Considerations

Fan-out is the process of creating multiple delivery tasks from a single event. At scale, a popular event type might match thousands of subscriptions. To avoid creating a thundering herd, we batch delivery tasks and write them in bulk. We also apply per-endpoint rate limiting at this stage to avoid overwhelming slow endpoints.

Parallelism and Partitioning

The dispatch processor scales horizontally by adding more consumer instances. Kafka partitions ensure that each instance processes a disjoint set of event partitions. For a system processing 100,000 events per second with 5x fan-out, we need the dispatch processor to generate 500,000 delivery tasks per second. With 50 consumer instances, each instance handles 2,000 events per second and generates 10,000 delivery tasks per second.

10. HTTP Delivery Engine

The delivery engine is responsible for making HTTP POST requests to target endpoints and handling the responses. This component must be efficient, resilient, and observant. It operates in a highly I/O-bound environment where network latency dominates.

Delivery Flow

HTTP Client Configuration

The delivery engine uses a custom HTTP client configured for webhook delivery specifically. Key configuration points include:

• Connection pooling: Reuse TCP connections to the same host to reduce handshake overhead. Configure a per-host connection limit of 10 to 50 connections.
• DNS resolution caching: Cache DNS results for 60 seconds to avoid repeated lookups while still respecting DNS changes.
• Timeout configuration: Default 10-second timeout per request. Configurable per endpoint. Use separate connect timeout (3s) and read timeout (7s).
• Redirect handling: Follow up to 3 redirects. Record each hop for debugging purposes.
• Response size limits: Read up to 1 MB of response body. Discard the rest to prevent memory exhaustion from malicious endpoints.
• User-Agent header: Identify the webhook system (e.g., Ayodhyya-Webhook/1.0) for endpoint operators to identify traffic.

Response Validation

Not all 2xx responses are equal. The delivery engine considers a delivery successful if the response status code is in the 200 to 299 range. Some systems additionally validate that the response body contains a specific acknowledgment field, but this adds complexity and is generally not recommended.

For non-2xx responses, the classification is:

TLS Requirements

All webhook deliveries must use TLS 1.2 or higher. The delivery engine should reject endpoints with expired, self-signed, or revoked TLS certificates. Certificate validation includes checking the certificate chain, verifying the hostname matches the certificate Subject Alternative Name (SAN), and confirming the certificate is not expired or revoked.

11. Signature Verification

Signature verification is the cornerstone of webhook security. It allows receivers to verify that a webhook was genuinely sent by the trusted source and was not tampered with in transit. We use HMAC-SHA256 as the primary signing algorithm.

Signing Process

The delivery engine generates a signature for each webhook delivery. The signature is computed over the concatenation of a timestamp and the raw request body, using the endpoint signing secret. This signature is included in the X-Webhook-Signature header.

C#
public class SignatureVerifier
{
    private const string SignatureHeader = "X-Webhook-Signature";
    private const string TimestampHeader = "X-Webhook-Timestamp";
    private const string Scheme = "sha256=";

    public string GenerateSignature(byte[] payload, byte[] secret, long timestamp)
    {
        var timestampBytes = Encoding.UTF8.GetBytes(timestamp.ToString());
        var payloadWithTimestamp = new byte[timestampBytes.Length + payload.Length];
        Buffer.BlockCopy(timestampBytes, 0, payloadWithTimestamp, 0, timestampBytes.Length);
        Buffer.BlockCopy(payload, 0, payloadWithTimestamp, timestampBytes.Length, payload.Length);

        using var hmac = new HMACSHA256(secret);
        var hash = hmac.ComputeHash(payloadWithTimestamp);
        return Scheme + Convert.ToHexString(hash).ToLowerInvariant();
    }

    public bool VerifySignature(byte[] payload, byte[] secret, long timestamp, string providedSignature)
    {
        var expectedSignature = GenerateSignature(payload, secret, timestamp);

        // Constant-time comparison to prevent timing attacks
        return CryptographicOperations.FixedTimeEquals(
            Encoding.UTF8.GetBytes(expectedSignature),
            Encoding.UTF8.GetBytes(providedSignature));
    }
}

Replay Protection

HMAC signatures alone prevent tampering but not replay attacks. An attacker who captures a valid webhook request could replay it later. To prevent this, we include a timestamp in the signed payload and reject requests where the timestamp is more than 5 minutes old.

C#
public class ReplayProtection
{
    private const int MaxTimestampAgeSeconds = 300;
    private readonly IDistributedCache _nonceCache;

    public async Task<bool> ValidateRequestAsync(
        long timestamp, string nonce, string signature)
    {
        var currentTime = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
        if (Math.Abs(currentTime - timestamp) > MaxTimestampAgeSeconds)
            return false;

        var nonceKey = $"webhook:nonce:{nonce}";
        var exists = await _nonceCache.GetAsync(nonceKey);
        if (exists != null)
            return false;

        await _nonceCache.SetAsync(nonceKey, new byte[] { 1 },
            new DistributedCacheEntryOptions
            {
                AbsoluteExpirationRelativeToNow =
                    TimeSpan.FromSeconds(MaxTimestampAgeSeconds + 60)
            });

        return true;
    }
}

Signature Headers Sent with Each Delivery

12. Retry Strategy

Retries are essential for reliable delivery. Target endpoints will inevitably experience downtime, network issues, and temporary overloads. A well-designed retry strategy maximizes delivery success while avoiding excessive load on struggling endpoints.

Exponential Backoff with Jitter

The standard approach is exponential backoff: each retry waits exponentially longer than the previous one. Combined with jitter (random variation), this prevents retry storms where many deliveries retry at the same time.

C#
public class RetryScheduler
{
    private const int BaseDelayMs = 1000;
    private const int MaxDelayMs = 3600000;
    private const double JitterFactor = 0.3;
    private static readonly Random _random = new();

    public TimeSpan CalculateRetryDelay(int attemptNumber)
    {
        var exponentialDelay = BaseDelayMs * Math.Pow(2, attemptNumber - 1);
        var cappedDelay = Math.Min(exponentialDelay, MaxDelayMs);
        var jitterRange = cappedDelay * JitterFactor;
        var jitter = (_random.NextDouble() * 2 - 1) * jitterRange;
        var finalDelay = cappedDelay + jitter;

        return TimeSpan.FromMilliseconds(Math.Max(0, finalDelay));
    }
}

Default Retry Schedule

This schedule spreads retries over approximately 2 hours. After 10 attempts, the delivery moves to the dead letter queue. The exact schedule is configurable per endpoint.

Retry Queue Implementation

The retry queue is implemented as a Kafka topic with a delay-based ordering. Each message includes a next_retry_at timestamp. Consumers read messages in timestamp order using a priority queue adapter. Alternatively, some systems use a dedicated delayed message broker like RabbitMQ with the dead-letter exchange pattern, or a time-based polling approach with a sorted set in Redis.

Failure Classification

Not all failures should trigger retries. Permanent failures — such as a 400 Bad Request due to malformed payload, or a 401 Unauthorized due to invalid credentials — should immediately mark the delivery as failed without further retries. Retrying these wastes resources and generates unnecessary load. Only transient failures — 408, 429, 5xx, timeouts, and network errors — should trigger the retry mechanism.

13. Delivery Log and Debugging

Visibility into delivery behavior is critical for both the system operators and the webhook consumers. When a customer reports that they are not receiving expected webhooks, the delivery log is the first place to look.

What to Log

• Request: Full URL, method, headers (including custom headers and generated signature headers), request body size, and timestamp.
• Response: Status code, response headers, response body (truncated to 4 KB), and response time in milliseconds.
• Errors: If the request failed, capture the error type (timeout, DNS resolution, connection refused), the error message, and the stack trace from the delivery worker.
• Context: Delivery ID, event ID, endpoint ID, attempt number, and the delivery worker instance that processed it.

Log Retention Tiers

Debugging Dashboard

A well-designed debugging dashboard provides both high-level metrics and drill-down capability. The top level shows delivery success rate, average latency, and error distribution over time. Clicking into a specific endpoint shows all deliveries for that endpoint, with the ability to filter by event type, status, and time range. Each delivery entry expands to show the full request/response details, including headers, body, and timing breakdown.

Structured Logging Format

All delivery logs should be structured (JSON format) and include consistent fields for easy querying. Use a schema registry to evolve the log format without breaking existing queries. Example log entry:

JSON
{
  "delivery_id": "dlv_1a2b3c",
  "event_id": "evt_f4e3d2c1",
  "endpoint_id": "ep_7f3a8b2c",
  "attempt_number": 2,
  "status": "success",
  "http_status": 200,
  "request_url": "https://api.merchant.com/webhooks",
  "request_body_size": 2048,
  "response_status": 200,
  "response_time_ms": 234,
  "worker_id": "worker-3",
  "timestamp": "2026-07-01T10:00:03Z"
}

14. Ordering Guarantees

Ordering is one of the most nuanced aspects of webhook delivery. Let us explore the different ordering guarantees and their trade-offs.

Ordering Models

Global Ordering

Global ordering means that all events are delivered in the exact order they were produced, across all endpoints. This is the strongest guarantee but is prohibitively expensive. It requires a single partition per endpoint (eliminating parallelism) and introduces head-of-line blocking — a slow endpoint delays all subsequent deliveries.

Per-Entity Ordering (Recommended)

Per-entity ordering guarantees that events for the same entity (e.g., the same order) are delivered in order, but events for different entities may be interleaved. This is achieved by partitioning events by entity ID and processing each partition sequentially. Within a partition, delivery tasks are created in order and consumed by a single worker, preserving ordering.

C#
public class OrderingManager
{
    private readonly ConcurrentDictionary<string, long> _lastSequenceNumbers = new();

    public bool CheckOrdering(string entityId, long sequenceNumber)
    {
        return _lastSequenceNumbers.AddOrUpdate(
            entityId,
            sequenceNumber,
            (key, lastSeq) =>
            {
                if (sequenceNumber <= lastSeq)
                    return false;
                return sequenceNumber;
            }) == sequenceNumber;
    }
}

Sequence Numbers

Each event carries a monotonically increasing sequence number within its entity. The dispatch processor checks whether the sequence number is the expected next value for that entity. If not, the event is held in a pending buffer until the missing sequence arrives. This handles the case where events for the same entity are published to different Kafka partitions and arrive out of order.

Delivery Attempt Ordering

For a given delivery, retry attempts are always ordered — attempt 2 always happens after attempt 1. This is trivially guaranteed by the retry scheduler, which enforces the delay between attempts.

15. Rate Limiting

Rate limiting protects both the webhook system and its consumers. Without rate limiting, a burst of events could overwhelm a slow endpoint, causing cascading failures. Rate limiting also prevents abusive patterns where a misconfigured endpoint generates excessive load.

Rate Limiting Layers

Token Bucket Algorithm

We use the token bucket algorithm for rate limiting. Each endpoint has a bucket that fills at a configured rate (tokens per second) and has a maximum capacity. Each delivery attempt consumes one token. If the bucket is empty, the delivery is delayed until a token becomes available or the delay exceeds a maximum threshold.

C#
public class TokenBucketRateLimiter
{
    private readonly string _endpointId;
    private readonly int _maxTokens;
    private readonly double _refillRatePerSecond;
    private double _currentTokens;
    private DateTime _lastRefill;

    public TokenBucketRateLimiter(string endpointId, int maxTokens, double refillRatePerSecond)
    {
        _endpointId = endpointId;
        _maxTokens = maxTokens;
        _refillRatePerSecond = refillRatePerSecond;
        _currentTokens = maxTokens;
        _lastRefill = DateTime.UtcNow;
    }

    public bool TryConsume()
    {
        Refill();
        if (_currentTokens >= 1)
        {
            _currentTokens -= 1;
            return true;
        }
        return false;
    }

    private void Refill()
    {
        var now = DateTime.UtcNow;
        var elapsed = (now - _lastRefill).TotalSeconds;
        _currentTokens = Math.Min(_maxTokens, _currentTokens + elapsed * _refillRatePerSecond);
        _lastRefill = now;
    }
}

Backpressure Mechanism

When an endpoint consistently hits its rate limit, the system applies backpressure. This means reducing the dispatch rate for that endpoint deliveries. Instead of immediately creating delivery tasks, the dispatch processor defers them to a delay queue with a short delay (e.g., 1 to 5 seconds). This smooths out bursts and prevents delivery workers from being overwhelmed.

Default Rate Limits

16. Event Filtering

Event filtering allows consumers to receive only the events they care about, reducing noise and processing overhead. A robust filtering system supports multiple filter dimensions.

Filter Types

1. Event Type Filtering: The most basic filter. An endpoint subscribes to specific event types (e.g., payment.completed) and only receives those events.
2. Payload-Based Filtering: Subscriptions can include expressions that match against fields in the event payload. For example, only deliver events where data.amount is greater than 1000.
3. Header-Based Filtering: Some systems support filtering on event metadata headers, such as the source service or environment.
4. Regex Filtering: Advanced filter expressions support regular expressions for complex matching patterns.

Filter Expression Syntax

JSON
{
  "all": [
    {"field": "data.currency", "operator": "eq", "value": "USD"},
    {"field": "data.amount", "operator": "gte", "value": 1000},
    {
      "any": [
        {"field": "data.type", "operator": "eq", "value": "subscription"},
        {"field": "data.type", "operator": "eq", "value": "one_time"}
      ]
    }
  ]
}

This expression uses all (AND) and any (OR) combinators to build complex filter logic. The filter evaluator walks the expression tree and evaluates each leaf node against the event payload.

Filter Evaluation Performance

Filter evaluation happens on the hot path of the dispatch processor. A naive implementation that parses and evaluates filter expressions for every event would be too slow at scale. Instead, we compile filter expressions into an optimized evaluation function at subscription creation time. This compilation step parses the expression once and produces a delegate that can be invoked repeatedly with minimal overhead.

Default Filtering Behavior

If a subscription has no filter expression, it receives all events of the subscribed type. If an event does not match any subscription filter, it is not delivered to that subscription. Events that match no subscriptions at all are still retained in the event store for potential future subscriptions or replay.

17. Webhook Security

Webhook security is a multi-layered concern. The system must protect against spoofing, tampering, replay attacks, and unauthorized access. Let us explore the complete security model.

Security Layers

IP Allowlisting

For high-security consumers, we publish the IP addresses from which webhook deliveries originate. Consumers can configure their firewalls to only accept traffic from these IPs. The IP list is published at a well-known endpoint and updated automatically when IPs change. We provide a 48-hour advance notice before any IP change.

Mutual TLS (mTLS)

mTLS provides the strongest transport-layer authentication. Both the delivery system and the target endpoint present TLS certificates to each other. The delivery system verifies the target certificate against a trusted CA, and the target verifies the delivery system certificate. This prevents man-in-the-middle attacks even if an attacker has compromised DNS.

Secret Rotation

Secrets should be rotated periodically (e.g., every 90 days) to limit the damage from potential secret compromise. The rotation process is:

1. The system generates a new secret and marks it as pending.
2. Both the old and new secrets are used for verification during the transition window. The new secret is used for signing, while both old and new are accepted for verification.
3. After a grace period (e.g., 7 days), the old secret is retired and only the new secret is active.

C#
public class SecretRotationManager
{
    public async Task<string> RotateSecretAsync(string endpointId)
    {
        var currentSecret = await _secretStore.GetActiveSecretAsync(endpointId);
        var newSecret = GenerateSecureSecret();

        currentSecret.Status = SecretStatus.Rotating;
        currentSecret.RetiredAt = DateTime.UtcNow.AddDays(7);
        await _secretStore.UpdateAsync(currentSecret);

        var newSecretRecord = new DeliverySecret
        {
            EndpointId = endpointId,
            SecretHash = HashSecret(newSecret),
            Status = SecretStatus.Active,
            CreatedAt = DateTime.UtcNow
        };
        await _secretStore.InsertAsync(newSecretRecord);

        return newSecret;
    }

    public async Task<List<byte[]>> GetVerificationSecretsAsync(string endpointId)
    {
        var secrets = await _secretStore
            .GetActiveAndRotatingSecretsAsync(endpointId);
        return secrets.Select(s =>
            Encoding.UTF8.GetBytes(s.SecretHash)).ToList();
    }
}

18. Scale-Out Design

Scaling a webhook delivery system requires horizontal scaling of each component independently. Let us examine how each layer scales.

Partitioned Dispatch

The system scales by adding more Kafka partitions and more delivery worker instances. Each delivery worker consumes from one or more partitions and handles the HTTP delivery for events in those partitions. The number of partitions determines the maximum parallelism. For a system processing 100,000 events per second with 5x fan-out, we need at least 500 Kafka partitions to ensure sufficient parallelism.

Worker Pool Architecture

Delivery workers are organized into pools. Each pool handles a subset of endpoints, enabling affinity-based optimizations. For example, endpoints owned by the same customer are often served by the same pool, allowing connection reuse and locality benefits. Pools are assigned dynamically based on endpoint hash, and rebalancing occurs automatically when workers are added or removed.

Auto-Scaling Rules

Connection Pool Sizing

Each delivery worker maintains a connection pool to target endpoints. The pool size per host is configurable but defaults to 10 connections. For a fleet of 200 delivery workers targeting the same endpoint, that is 2,000 concurrent connections — enough for most endpoints. Endpoints that require more concurrent connections can request a pool size increase.

19. Dead Letter Queue and Manual Replay

Despite best efforts, some deliveries will fail permanently. The dead letter queue (DLQ) provides a safety net for these deliveries, enabling inspection, manual replay, and eventual resolution.

DLQ Workflow

DLQ Entry Structure

JSON
{
  "id": "dlq_abc123",
  "delivery_id": "dlv_1a2b3c",
  "event_id": "evt_f4e3d2c1",
  "endpoint_id": "ep_7f3a8b2c",
  "endpoint_url": "https://api.merchant.com/webhooks",
  "event_type": "payment.completed",
  "final_status": "failed_permanently",
  "total_attempts": 8,
  "first_attempt_at": "2026-07-01T10:00:01Z",
  "last_attempt_at": "2026-07-01T12:00:00Z",
  "failure_reason": "All attempts returned 500 status codes",
  "created_at": "2026-07-01T12:00:01Z"
}

Manual Replay

Customers can replay deliveries from the DLQ via the API or dashboard. Replay creates a new delivery with a new delivery ID but the same event payload. The original delivery is marked as replayed for audit purposes. Replay supports options for replaying the most recent attempt or starting from attempt 1 (with the full retry schedule).

The replay API:

HTTP
POST /v1/deliveries/dlv_1a2b3c/replay
{
  "from_attempt": 1,
  "note": "Customer fixed their endpoint, retrying"
}

// Response: 202 Accepted
{
  "new_delivery_id": "dlv_4d5e6f",
  "status": "pending",
  "created_at": "2026-07-02T09:00:00Z"
}

Bulk Replay

For scenarios where an endpoint was down for an extended period, customers can request a bulk replay of all failed deliveries within a time range. The bulk replay creates new deliveries for all matching DLQ entries and processes them through the standard delivery pipeline with the endpoint current configuration.

20. Analytics and Monitoring

Comprehensive monitoring and analytics are essential for operating a webhook delivery system at scale. Let us define the key metrics and monitoring strategy.

Key Metrics

Monitoring Stack

We recommend a three-layer monitoring stack:

1. Real-time dashboards (Grafana plus Prometheus): Live metrics with 15-second granularity. Used for operational monitoring and incident response.
2. Alerting (PagerDuty or OpsGenie): Automated alerts based on the thresholds above. Alerts are routed to the appropriate on-call team based on severity.
3. Analytics (ClickHouse or BigQuery): Historical analysis and reporting. Used for capacity planning, SLA reporting, and customer-facing analytics.

Customer-Facing Analytics

Customers should have access to a dashboard showing their delivery metrics over time. This includes success rate trends, latency percentiles, error breakdowns, and delivery volume. These insights help customers proactively identify and resolve issues with their endpoints.

21. Database Design

The database layer must support high-throughput writes (delivery logs) and low-latency reads (endpoint configuration, delivery status). Let us design the schema for the primary database tables.

Core Tables

SQL
CREATE TABLE endpoints (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    customer_id UUID NOT NULL,
    url TEXT NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'pending_verification',
    config JSONB NOT NULL DEFAULT '{}',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    verified_at TIMESTAMPTZ,
    disabled_at TIMESTAMPTZ
);

CREATE INDEX idx_endpoints_customer ON endpoints(customer_id);
CREATE INDEX idx_endpoints_status ON endpoints(status);

CREATE TABLE subscriptions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    endpoint_id UUID NOT NULL REFERENCES endpoints(id),
    event_type VARCHAR(100) NOT NULL,
    filter_expression JSONB,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE(endpoint_id, event_type)
);

CREATE INDEX idx_subscriptions_event_type ON subscriptions(event_type);

CREATE TABLE events (
    id UUID PRIMARY KEY,
    event_type VARCHAR(100) NOT NULL,
    source VARCHAR(50) NOT NULL,
    payload JSONB NOT NULL,
    metadata JSONB,
    idempotency_key VARCHAR(255) UNIQUE,
    occurred_at TIMESTAMPTZ NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_events_type ON events(event_type);
CREATE INDEX idx_events_occurred ON events(occurred_at);

CREATE TABLE deliveries (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    endpoint_id UUID NOT NULL REFERENCES endpoints(id),
    event_id UUID NOT NULL REFERENCES events(id),
    status VARCHAR(20) NOT NULL DEFAULT 'pending',
    attempt_count INT NOT NULL DEFAULT 0,
    next_retry_at TIMESTAMPTZ,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    delivered_at TIMESTAMPTZ,
    failed_at TIMESTAMPTZ,
    dead_letter_id UUID
);

CREATE INDEX idx_deliveries_endpoint ON deliveries(endpoint_id, created_at DESC);
CREATE INDEX idx_deliveries_status ON deliveries(status, next_retry_at);
CREATE INDEX idx_deliveries_event ON deliveries(event_id);

CREATE TABLE delivery_attempts (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    delivery_id UUID NOT NULL REFERENCES deliveries(id),
    attempt_number INT NOT NULL,
    http_status INT,
    request_headers JSONB,
    request_body_size INT,
    response_headers JSONB,
    response_body TEXT,
    duration_ms INT,
    error_message TEXT,
    error_type VARCHAR(50),
    worker_id VARCHAR(50),
    attempted_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_delivery_attempts_delivery ON delivery_attempts(delivery_id);

CREATE TABLE delivery_secrets (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    endpoint_id UUID NOT NULL REFERENCES endpoints(id),
    secret_hash VARCHAR(128) NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'active',
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    retired_at TIMESTAMPTZ
);

CREATE INDEX idx_delivery_secrets_endpoint ON delivery_secrets(endpoint_id, status);

Partitioning Strategy

The delivery_attempts table is the largest and benefits from time-based partitioning. Partition by month to keep active partitions small and enable efficient data lifecycle management (dropping old partitions after retention expires).

The events table can also be partitioned by time or by event type hash. For a system ingesting 100,000 events per second, time-based partitioning with daily granularity keeps each partition manageable.

22. Caching Strategy

Caching reduces database load and improves latency for frequently accessed data. In a webhook delivery system, the most important caches are endpoint configuration and subscription lookup.

Cache Hierarchy

What to Cache

Cache Stampede Prevention

When a cache entry expires and multiple workers simultaneously request the same data, they can all hit the database simultaneously — a cache stampede. We prevent this using a probabilistic early expiration technique and a distributed lock.

C#
public class CacheStampedePrevention<T>
{
    private readonly IDistributedLock _lock;
    private readonly Func<Task<T>> _dataSource;

    public async Task<T> GetOrRefreshAsync(string cacheKey, TimeSpan ttl)
    {
        var cached = await _cache.GetAsync<T>(cacheKey);
        if (cached != null) return cached;

        var lockKey = $"lock:{cacheKey}";
        var acquired = await _lock.TryAcquireAsync(lockKey, TimeSpan.FromSeconds(10));

        if (acquired)
        {
            try
            {
                cached = await _cache.GetAsync<T>(cacheKey);
                if (cached != null) return cached;

                var data = await _dataSource();
                await _cache.SetAsync(cacheKey, data, ttl);
                return data;
            }
            finally
            {
                await _lock.ReleaseAsync(lockKey);
            }
        }

        await Task.Delay(100);
        return await GetOrRefreshAsync(cacheKey, ttl);
    }
}

23. Multi-Region Design

For global availability and low-latency delivery, a webhook system must operate across multiple regions. Multi-region design introduces challenges around data consistency, cross-region delivery, and failover.

Multi-Region Architecture

Regional Endpoint Affinity

Each endpoint is assigned a primary region based on its geographic location. Deliveries to that endpoint are preferentially routed through the primary region to minimize latency. If the primary region fails, deliveries automatically fail over to a secondary region. The region assignment is stored in the endpoint configuration and can be overridden by the customer.

Cross-Region Event Replication

Events are produced to the local Kafka cluster and then replicated to other regions via Kafka MirrorMaker 2. This ensures that all regions have access to the full event stream. The replication lag is typically under 1 second, which is acceptable for webhook delivery where sub-second global consistency is not required.

Failover Strategy

Region failover is triggered when the primary region health check fails for more than 60 seconds. The failover process:

1. DNS is updated to route traffic to the secondary region via a global load balancer or DNS failover.
2. The secondary region resumes delivery for affected endpoints using its replicated Kafka data.
3. Delivery status is reconciled after the primary region recovers, using the idempotent delivery model to avoid duplicates.

Region-Specific Considerations

24. Cost Estimation

Understanding the cost structure of a webhook delivery system is essential for business planning. Let us break down the major cost components at scale.

Infrastructure Costs (Monthly Estimates)

Cost Optimization Strategies

1. Spot Instances for Delivery Workers: Delivery workers are stateless and can be interrupted. Using spot instances (60 to 70% discount) reduces the worker fleet cost from ,600 to approximately ,000.
2. Reserved Instances: For always-on components like Kafka and PostgreSQL, 1-year reserved instances provide 30 to 40% savings.
3. Tiered Storage: Moving delivery logs to S3 after 7 days significantly reduces Cassandra cluster size and cost.
4. Payload Compression: Gzip compression of delivery payloads reduces network transfer costs by 60 to 80%.

Cost Per Delivery

At 13 billion deliveries per day (400 billion per month), the total infrastructure cost of approximately translates to roughly .0000018 per delivery — less than two thousandths of a cent. Even with significant overhead for engineering, operations, and margin, the per-delivery cost remains well under a penny, making webhooks an economically viable integration mechanism at any scale.

25. Interview Q&A

This section covers the most common interview questions about webhook and event delivery system design, along with detailed answers.

Q1: How do you achieve exactly-once delivery?

Answer: True exactly-once delivery is impossible over HTTP. However, we can achieve effectively-once delivery through idempotency. Each delivery has a unique ID (X-Webhook-Delivery-ID header), and each event has a unique ID (X-Webhook-Event-ID header). The consumer uses the event ID for idempotent processing — if they receive the same event ID twice, they return the cached result from the first processing. On the delivery side, we use deduplication at ingestion (via idempotency_key) to prevent duplicate event creation, and we track delivery attempts to avoid duplicate deliveries for the same event-endpoint pair.

Q2: How do you handle a slow endpoint that causes delivery delays?

Answer: Slow endpoints (those that respond after several seconds) consume delivery worker connections and reduce overall throughput. We handle this with several mechanisms: (1) Per-endpoint timeouts enforced at the HTTP client level. (2) Per-endpoint connection limits to prevent a single slow endpoint from monopolizing worker connections. (3) Per-endpoint rate limiting that throttles deliveries when the endpoint is consistently slow. (4) Circuit breaker pattern that temporarily stops deliveries to an endpoint that has consecutive timeouts, allowing the endpoint to recover.

Q3: How do you prevent replay attacks?

Answer: We use a three-part defense: (1) Timestamps in the signed payload ensure requests older than 5 minutes are rejected. (2) Unique nonces in each request are tracked in Redis with a TTL matching the timestamp window — duplicate nonces are rejected. (3) Consumers are encouraged to use the event ID for idempotent processing, so even if a replay makes it through, the processing is idempotent. The combination of these three mechanisms makes replay attacks practically infeasible.

Q4: How do you handle schema evolution in event payloads?

Answer: We follow the additive-only evolution principle: new fields can be added, but existing fields cannot be removed or have their types changed. The event version field allows consumers to handle multiple schema versions. When breaking changes are necessary, we introduce a new event type (e.g., v2 suffix) and deprecate the old one with a 90-day transition period. We provide a schema registry where consumers can look up the schema for each event type and version.

Q5: How do you test a webhook system?

Answer: We test at multiple levels: (1) Unit tests for filter evaluation, signature generation, and retry delay calculation. (2) Integration tests using a mock HTTP server that simulates various response scenarios (success, failure, timeout, slow response). (3) Chaos engineering that randomly terminates delivery workers, corrupts network traffic, and simulates endpoint failures. (4) Load testing that pushes the system to 2x expected peak load. (5) A canary deployment strategy that routes 5% of traffic to new code before full rollout.

Q6: Why use Kafka instead of a simpler message queue like RabbitMQ?

Answer: Kafka provides three critical capabilities for webhook delivery: (1) Durable, ordered log that allows replaying events from any point in time — essential for debugging and recovery. (2) Horizontal scalability to millions of events per second. (3) Partitioning that enables per-entity ordering. RabbitMQ is simpler but lacks durable message replay, which is essential for a reliable delivery system. That said, for smaller-scale systems (under 10,000 events/second), RabbitMQ with the delayed message plugin is a perfectly viable choice.

Q7: How do you ensure the system is resilient to cascading failures?

Answer: We apply the bulkhead pattern at multiple levels: (1) Connection pools are partitioned per-endpoint, so one slow endpoint cannot exhaust connections to other endpoints. (2) Delivery workers are isolated from dispatch workers by separate queues, so delivery backlogs do not prevent event ingestion. (3) The retry queue is separate from the primary delivery queue, so retry storms do not impact first-time deliveries. (4) Circuit breakers on delivery workers stop requests to endpoints that are consistently failing, preventing wasted resources.

Q8: How would you handle delivery ordering across multiple delivery workers?

Answer: We partition the delivery task queue by entity ID (e.g., order ID). This ensures that all deliveries for the same entity are processed by the same worker in order. Different entities can be processed by different workers in parallel. Within a single worker, we process delivery tasks sequentially per entity. If a higher-sequence-numbered delivery arrives before a lower one, we hold it in a buffer for up to 30 seconds, after which we deliver it anyway with an out-of-order flag.

Q9: How do you debug delivery failures for a specific customer?

Answer: We provide a delivery log viewer in the customer dashboard with the following capabilities: (1) Search by event ID, delivery ID, or time range. (2) Filter by delivery status (success, failed, pending). (3) View full request/response details for each delivery attempt, including headers, body, and timing. (4) View retry schedule showing when each retry occurred or will occur. (5) DLQ view showing permanently failed deliveries with the failure reason and option to replay. The underlying data is stored in Cassandra for fast query performance across billions of records.

Q10: How do you handle events that need to be delivered to thousands of endpoints?

Answer: High-fan-out events are handled through batch processing in the dispatch pipeline. Instead of creating one delivery task at a time, we batch-create delivery tasks for all matching subscriptions in a single database transaction. The delivery tasks are then published to Kafka in batches, and delivery workers consume them in bulk. We also apply per-endpoint rate limiting to prevent overwhelming slow endpoints. For extreme fan-out (10,000+ endpoints), we use a dedicated fan-out queue with multiple consumer groups to parallelize the work.

Q11: What happens when the webhook delivery system itself goes down?

Answer: Events are safely stored in Kafka, which has durable replication across multiple availability zones. When the delivery system recovers, it resumes processing from where it left off, picking up events from the Kafka offset. No events are lost. The only impact is delayed delivery during the outage window. The retry mechanism ensures that any deliveries that failed during the outage are retried. We target RTO (Recovery Time Objective) of under 30 seconds via automated health checks and failover.

26. Full C# Implementation

The following is a complete, production-grade C# implementation of the core webhook delivery system components. This code demonstrates the key patterns discussed throughout this article.

Configuration Models

C#
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;

namespace WebhookDeliverySystem
{
    public class WebhookSystemOptions
    {
        public int DefaultTimeoutMs { get; set; } = 10000;
        public int MaxRetryAttempts { get; set; } = 8;
        public int BaseRetryDelayMs { get; set; } = 1000;
        public int MaxRetryDelayMs { get; set; } = 3600000;
        public double JitterFactor { get; set; } = 0.3;
        public int MaxResponseSizeBytes { get; set; } = 1048576;
        public int ReplayProtectionWindowSeconds { get; set; } = 300;
    }

    public class EndpointConfig
    {
        public Guid Id { get; set; }
        public Guid CustomerId { get; set; }
        public string Url { get; set; } = string.Empty;
        public string Status { get; set; } = "active";
        public int TimeoutMs { get; set; } = 10000;
        public int MaxRetryAttempts { get; set; } = 8;
        public string ContentType { get; set; } = "application/json";
        public Dictionary<string, string> CustomHeaders { get; set; } = new();
    }

    public class Subscription
    {
        public Guid Id { get; set; }
        public Guid EndpointId { get; set; }
        public string EventType { get; set; } = string.Empty;
        public JsonElement? FilterExpression { get; set; }
        public EndpointConfig? Endpoint { get; set; }
    }

    public class WebhookEvent
    {
        public Guid Id { get; set; }
        public string Type { get; set; } = string.Empty;
        public string Version { get; set; } = "2026-07-01";
        public string Source { get; set; } = string.Empty;
        public JsonElement Data { get; set; }
        public string? IdempotencyKey { get; set; }
        public DateTime Timestamp { get; set; }
        public DateTime ReceivedAt { get; set; }
    }

    public class DeliveryTask
    {
        public Guid DeliveryId { get; set; }
        public Guid EventId { get; set; }
        public WebhookEvent Event { get; set; } = null!;
        public EndpointConfig Endpoint { get; set; } = null!;
        public Guid SubscriptionId { get; set; }
        public int AttemptNumber { get; set; }
        public DateTime CreatedAt { get; set; }
        public DateTime? NextRetryAt { get; set; }
    }

    public class DeliveryResult
    {
        public Guid DeliveryId { get; set; }
        public int AttemptNumber { get; set; }
        public bool Success { get; set; }
        public int? HttpStatusCode { get; set; }
        public int DurationMs { get; set; }
        public string? ResponseBody { get; set; }
        public string? ErrorMessage { get; set; }
        public string? ErrorType { get; set; }
        public DateTime AttemptedAt { get; set; }
    }
}

Signature Verifier

C#
namespace WebhookDeliverySystem
{
    public interface ISignatureVerifier
    {
        string GenerateSignature(byte[] payload, byte[] secret, long timestamp);
        bool VerifySignature(byte[] payload, byte[] secret,
            long timestamp, string providedSignature);
    }

    public class SignatureVerifier : ISignatureVerifier
    {
        private const string Scheme = "sha256=";

        public string GenerateSignature(byte[] payload, byte[] secret, long timestamp)
        {
            var timestampBytes = Encoding.UTF8.GetBytes(timestamp.ToString());
            var combined = new byte[timestampBytes.Length + payload.Length];
            Buffer.BlockCopy(timestampBytes, 0, combined, 0, timestampBytes.Length);
            Buffer.BlockCopy(payload, 0, combined, timestampBytes.Length, payload.Length);

            using var hmac = new HMACSHA256(secret);
            var hash = hmac.ComputeHash(combined);
            return Scheme + Convert.ToHexString(hash).ToLowerInvariant();
        }

        public bool VerifySignature(byte[] payload, byte[] secret,
            long timestamp, string providedSignature)
        {
            var expected = GenerateSignature(payload, secret, timestamp);
            return CryptographicOperations.FixedTimeEquals(
                Encoding.UTF8.GetBytes(expected),
                Encoding.UTF8.GetBytes(providedSignature));
        }
    }
}

Replay Protection

C#
namespace WebhookDeliverySystem
{
    public interface IReplayProtection
    {
        Task<bool> ValidateAsync(long timestamp, string nonce);
    }

    public class ReplayProtection : IReplayProtection
    {
        private readonly ICacheService _cache;
        private readonly int _windowSeconds;

        public ReplayProtection(ICacheService cache, int windowSeconds = 300)
        {
            _cache = cache;
            _windowSeconds = windowSeconds;
        }

        public async Task<bool> ValidateAsync(long timestamp, string nonce)
        {
            var now = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
            if (Math.Abs(now - timestamp) > _windowSeconds)
                return false;

            var nonceKey = $"webhook:nonce:{nonce}";
            var exists = await _cache.ExistsAsync(nonceKey);
            if (exists) return false;

            await _cache.SetAsync(nonceKey, "1",
                TimeSpan.FromSeconds(_windowSeconds + 60));
            return true;
        }
    }
}

Retry Scheduler

C#
namespace WebhookDeliverySystem
{
    public interface IRetryScheduler
    {
        TimeSpan CalculateDelay(int attemptNumber);
        DateTime GetNextRetryTime(int attemptNumber);
        bool ShouldRetry(int attemptNumber, int maxAttempts, DeliveryResult result);
    }

    public class RetryScheduler : IRetryScheduler
    {
        private readonly WebhookSystemOptions _options;
        private static readonly Random _random = new();

        public RetryScheduler(IOptions<WebhookSystemOptions> options)
        {
            _options = options.Value;
        }

        public TimeSpan CalculateDelay(int attemptNumber)
        {
            var exponential = _options.BaseRetryDelayMs
                * Math.Pow(2, attemptNumber - 1);
            var capped = Math.Min(exponential, _options.MaxRetryDelayMs);
            var jitterRange = capped * _options.JitterFactor;
            var jitter = (_random.NextDouble() * 2 - 1) * jitterRange;
            var final = Math.Max(0, capped + jitter);
            return TimeSpan.FromMilliseconds(final);
        }

        public DateTime GetNextRetryTime(int attemptNumber)
        {
            return DateTime.UtcNow.Add(CalculateDelay(attemptNumber));
        }

        public bool ShouldRetry(int attemptNumber, int maxAttempts,
            DeliveryResult result)
        {
            if (attemptNumber >= maxAttempts) return false;
            if (result.Success) return false;
            if (result.HttpStatusCode is >= 400 and < 500
                and not 408 and not 429)
                return false;
            return true;
        }
    }
}

Rate Limiter

C#
namespace WebhookDeliverySystem
{
    public interface IRateLimiter
    {
        Task<bool> TryConsumeAsync(string endpointId);
    }

    public class TokenBucketRateLimiter : IRateLimiter
    {
        private readonly ICacheService _cache;
        private readonly int _maxTokens;
        private readonly double _refillRatePerSecond;

        public TokenBucketRateLimiter(ICacheService cache,
            int maxTokens = 100, double refillRate = 100.0 / 60)
        {
            _cache = cache;
            _maxTokens = maxTokens;
            _refillRatePerSecond = refillRate;
        }

        public async Task<bool> TryConsumeAsync(string endpointId)
        {
            var key = $"ratelimit:{endpointId}";
            var bucket = await _cache.GetAsync<TokenBucket>(key);

            if (bucket == null)
            {
                bucket = new TokenBucket
                {
                    Tokens = _maxTokens - 1,
                    LastRefill = DateTime.UtcNow
                };
                await _cache.SetAsync(key, bucket, TimeSpan.FromHours(1));
                return true;
            }

            var elapsed = (DateTime.UtcNow - bucket.LastRefill).TotalSeconds;
            bucket.Tokens = Math.Min(_maxTokens,
                bucket.Tokens + elapsed * _refillRatePerSecond);
            bucket.LastRefill = DateTime.UtcNow;

            if (bucket.Tokens >= 1)
            {
                bucket.Tokens -= 1;
                await _cache.SetAsync(key, bucket, TimeSpan.FromHours(1));
                return true;
            }
            return false;
        }

        private class TokenBucket
        {
            public double Tokens { get; set; }
            public DateTime LastRefill { get; set; }
        }
    }
}

Filter Evaluator

C#
namespace WebhookDeliverySystem
{
    public interface IFilterEvaluator
    {
        bool Matches(WebhookEvent evt, JsonElement? filterExpression);
    }

    public class FilterEvaluator : IFilterEvaluator
    {
        private readonly ILogger<FilterEvaluator> _logger;

        public FilterEvaluator(ILogger<FilterEvaluator> logger)
        {
            _logger = logger;
        }

        public bool Matches(WebhookEvent evt, JsonElement? filterExpression)
        {
            if (filterExpression == null ||
                filterExpression.Value.ValueKind == JsonValueKind.Null)
                return true;

            try
            {
                return EvaluateNode(evt.Data, filterExpression.Value);
            }
            catch (Exception ex)
            {
                _logger.LogWarning(ex, "Filter evaluation failed");
                return false;
            }
        }

        private bool EvaluateNode(JsonElement data, JsonElement node)
        {
            if (node.TryGetProperty("all", out var allFilters))
                return allFilters.EnumerateArray()
                    .All(f => EvaluateNode(data, f));

            if (node.TryGetProperty("any", out var anyFilters))
                return anyFilters.EnumerateArray()
                    .Any(f => EvaluateNode(data, f));

            if (node.TryGetProperty("field", out var field)
                && node.TryGetProperty("operator", out var op)
                && node.TryGetProperty("value", out var val))
            {
                var fieldValue = ResolveField(data, field.GetString()!);
                return EvaluateCondition(fieldValue, op.GetString()!, val);
            }
            return true;
        }

        private JsonElement? ResolveField(JsonElement data, string path)
        {
            var parts = path.Split('.'); var current = data;
            foreach (var part in parts)
            {
                if (current.TryGetProperty(part, out var next))
                    current = next;
                else return null;
            }
            return current;
        }

        private bool EvaluateCondition(JsonElement? fieldValue,
            string op, JsonElement expectedValue)
        {
            if (fieldValue == null) return false;
            var fv = fieldValue.Value;
            switch (op)
            {
                case "": return fv.GetRawText() == expectedValue.GetRawText();
                case "": return fv.GetRawText() != expectedValue.GetRawText();
                case "": return fv.GetDouble() > expectedValue.GetDouble();
                case "": return fv.GetDouble() >= expectedValue.GetDouble();
                case "": return fv.GetDouble() < expectedValue.GetDouble();
                case "": return fv.GetDouble() <= expectedValue.GetDouble();
                case "": return expectedValue.EnumerateArray()
                    .Any(v => v.GetRawText() == fv.GetRawText());
                default: return false;
            }
        }
    }
}

Dispatch Processor

C#
namespace WebhookDeliverySystem
{
    public interface IDispatchProcessor
    {
        Task ProcessEventAsync(WebhookEvent evt,
            CancellationToken ct = default);
    }

    public class DispatchProcessor : IDispatchProcessor
    {
        private readonly ISubscriptionRepository _subscriptionRepo;
        private readonly IFilterEvaluator _filterEvaluator;
        private readonly IDeliveryTaskProducer _taskProducer;
        private readonly ILogger<DispatchProcessor> _logger;

        public DispatchProcessor(
            ISubscriptionRepository subscriptionRepo,
            IFilterEvaluator filterEvaluator,
            IDeliveryTaskProducer taskProducer,
            ILogger<DispatchProcessor> logger)
        {
            _subscriptionRepo = subscriptionRepo;
            _filterEvaluator = filterEvaluator;
            _taskProducer = taskProducer;
            _logger = logger;
        }

        public async Task ProcessEventAsync(WebhookEvent evt,
            CancellationToken ct = default)
        {
            _logger.LogInformation(
                "Processing event {EventId} type {EventType}",
                evt.Id, evt.Type);

            var subscriptions = await _subscriptionRepo
                .GetByEventTypeAsync(evt.Type, ct);

            var tasksCreated = 0;
            foreach (var subscription in subscriptions)
            {
                if (subscription.Endpoint == null) continue;
                if (subscription.Endpoint.Status != "active") continue;

                if (!_filterEvaluator.Matches(evt, subscription.FilterExpression))
                    continue;

                var task = new DeliveryTask
                {
                    DeliveryId = Guid.NewGuid(),
                    EventId = evt.Id,
                    Event = evt,
                    Endpoint = subscription.Endpoint,
                    SubscriptionId = subscription.Id,
                    AttemptNumber = 0,
                    CreatedAt = DateTime.UtcNow
                };

                await _taskProducer.ProduceAsync(task, ct);
                tasksCreated++;
            }

            _logger.LogInformation(
                "Event {EventId}: {Count} subscriptions, " +
                "{Tasks} delivery tasks",
                evt.Id, subscriptions.Count, tasksCreated);
        }
    }
}

HTTP Delivery Engine

C#
namespace WebhookDeliverySystem
{
    public interface IDeliveryEngine
    {
        Task<DeliveryResult> DeliverAsync(DeliveryTask task,
            CancellationToken ct = default);
    }

    public class HttpDeliveryEngine : IDeliveryEngine
    {
        private readonly IHttpClientFactory _httpClientFactory;
        private readonly ISignatureVerifier _signatureVerifier;
        private readonly ILogger<HttpDeliveryEngine> _logger;
        private readonly WebhookSystemOptions _options;

        public HttpDeliveryEngine(
            IHttpClientFactory httpClientFactory,
            ISignatureVerifier signatureVerifier,
            IOptions<WebhookSystemOptions> options,
            ILogger<HttpDeliveryEngine> logger)
        {
            _httpClientFactory = httpClientFactory;
            _signatureVerifier = signatureVerifier;
            _options = options.Value;
            _logger = logger;
        }

        public async Task<DeliveryResult> DeliverAsync(
            DeliveryTask task, CancellationToken ct = default)
        {
            var sw = System.Diagnostics.Stopwatch.StartNew();
            var attemptTime = DateTime.UtcNow;

            try
            {
                var payload = JsonSerializer.SerializeToUtf8Bytes(
                    task.Event,
                    new JsonSerializerOptions
                    {
                        PropertyNamingPolicy = JsonNamingPolicy.CamelCase
                    });

                var timestamp = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
                var secret = Encoding.UTF8.GetBytes(
                    task.Endpoint.Id.ToString("N"));
                var signature = _signatureVerifier
                    .GenerateSignature(payload, secret, timestamp);

                var client = _httpClientFactory.CreateClient();
                client.Timeout = TimeSpan.FromMilliseconds(
                    task.Endpoint.TimeoutMs > 0
                        ? task.Endpoint.TimeoutMs
                        : _options.DefaultTimeoutMs);

                var request = new HttpRequestMessage(
                    HttpMethod.Post, task.Endpoint.Url)
                {
                    Content = new ByteArrayContent(payload)
                };

                request.Content.Headers.ContentType =
                    new System.Net.Http.Headers
                        .MediaTypeHeaderValue(task.Endpoint.ContentType);

                request.Headers.Add("X-Webhook-Signature", signature);
                request.Headers.Add("X-Webhook-Timestamp",
                    timestamp.ToString());
                request.Headers.Add("X-Webhook-Event-ID",
                    task.EventId.ToString());
                request.Headers.Add("X-Webhook-Delivery-ID",
                    task.DeliveryId.ToString());
                request.Headers.Add("X-Webhook-Event-Type",
                    task.Event.Type);
                request.Headers.Add("User-Agent",
                    "Ayodhyya-Webhook/1.0");

                foreach (var header in task.Endpoint.CustomHeaders)
                    request.Headers.TryAddWithoutValidation(
                        header.Key, header.Value);

                var response = await client.SendAsync(request, ct);
                sw.Stop();

                var body = await response.Content.ReadAsStringAsync(ct);
                if (body.Length > _options.MaxResponseSizeBytes)
                    body = body[.._options.MaxResponseSizeBytes]
                        + "(truncated)";

                return new DeliveryResult
                {
                    DeliveryId = task.DeliveryId,
                    AttemptNumber = task.AttemptNumber + 1,
                    Success = response.IsSuccessStatusCode,
                    HttpStatusCode = (int)response.StatusCode,
                    DurationMs = (int)sw.ElapsedMilliseconds,
                    ResponseBody = body,
                    AttemptedAt = attemptTime
                };
            }
            catch (TaskCanceledException) when (!ct.IsCancellationRequested)
            {
                sw.Stop();
                return new DeliveryResult
                {
                    DeliveryId = task.DeliveryId,
                    AttemptNumber = task.AttemptNumber + 1,
                    Success = false,
                    DurationMs = (int)sw.ElapsedMilliseconds,
                    ErrorMessage = "Request timed out",
                    ErrorType = "timeout",
                    AttemptedAt = attemptTime
                };
            }
            catch (HttpRequestException ex)
            {
                sw.Stop();
                return new DeliveryResult
                {
                    DeliveryId = task.DeliveryId,
                    AttemptNumber = task.AttemptNumber + 1,
                    Success = false,
                    DurationMs = (int)sw.ElapsedMilliseconds,
                    ErrorMessage = ex.Message,
                    ErrorType = ex.GetType().Name,
                    AttemptedAt = attemptTime
                };
            }
        }
    }
}

Delivery Worker

C#
namespace WebhookDeliverySystem
{
    public interface IDeliveryWorker
    {
        Task StartAsync(CancellationToken ct = default);
        Task StopAsync();
    }

    public class DeliveryWorker : IDeliveryWorker
    {
        private readonly IDeliveryEngine _deliveryEngine;
        private readonly IRetryScheduler _retryScheduler;
        private readonly IRateLimiter _rateLimiter;
        private readonly IDeliveryRepository _deliveryRepo;
        private readonly IDeliveryTaskConsumer _taskConsumer;
        private readonly ILogger<DeliveryWorker> _logger;
        private readonly WebhookSystemOptions _options;
        private CancellationTokenSource? _cts;

        public DeliveryWorker(
            IDeliveryEngine deliveryEngine,
            IRetryScheduler retryScheduler,
            IRateLimiter rateLimiter,
            IDeliveryRepository deliveryRepo,
            IDeliveryTaskConsumer taskConsumer,
            IOptions<WebhookSystemOptions> options,
            ILogger<DeliveryWorker> logger)
        {
            _deliveryEngine = deliveryEngine;
            _retryScheduler = retryScheduler;
            _rateLimiter = rateLimiter;
            _deliveryRepo = deliveryRepo;
            _taskConsumer = taskConsumer;
            _options = options.Value;
            _logger = logger;
        }

        public async Task StartAsync(CancellationToken ct = default)
        {
            _cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
            _logger.LogInformation("Delivery worker starting");

            await foreach (var task in
                _taskConsumer.ConsumeAsync(_cts.Token))
            {
                try
                {
                    await ProcessDeliveryTaskAsync(task, _cts.Token);
                }
                catch (Exception ex)
                {
                    _logger.LogError(ex,
                        "Error processing delivery {DeliveryId}",
                        task.DeliveryId);
                }
            }
        }

        public Task StopAsync()
        {
            _cts?.Cancel();
            _logger.LogInformation("Delivery worker stopping");
            return Task.CompletedTask;
        }

        private async Task ProcessDeliveryTaskAsync(
            DeliveryTask task, CancellationToken ct)
        {
            if (!await _rateLimiter
                .TryConsumeAsync(task.Endpoint.Id.ToString()))
            {
                _logger.LogWarning(
                    "Rate limited endpoint {EndpointId}",
                    task.Endpoint.Id);
                task.NextRetryAt = DateTime.UtcNow.AddSeconds(5);
                await _deliveryRepo.UpdateNextRetryAsync(
                    task.DeliveryId, task.NextRetryAt, ct);
                return;
            }

            await _deliveryRepo
                .EnsureDeliveryExistsAsync(task, ct);

            var result = await _deliveryEngine
                .DeliverAsync(task, ct);

            await _deliveryRepo
                .LogAttemptAsync(result, ct);

            if (result.Success)
            {
                await _deliveryRepo
                    .MarkDeliveredAsync(task.DeliveryId, ct);

                _logger.LogInformation(
                    "Delivery {DeliveryId} succeeded " +
                    "on attempt {Attempt} ({Duration}ms)",
                    task.DeliveryId, result.AttemptNumber,
                    result.DurationMs);
                return;
            }

            var maxAttempts = task.Endpoint.MaxRetryAttempts > 0
                ? task.Endpoint.MaxRetryAttempts
                : _options.MaxRetryAttempts;

            if (_retryScheduler.ShouldRetry(
                result.AttemptNumber, maxAttempts, result))
            {
                var nextRetry = _retryScheduler
                    .GetNextRetryTime(result.AttemptNumber);

                await _deliveryRepo
                    .ScheduleRetryAsync(
                        task.DeliveryId, nextRetry, ct);

                _logger.LogWarning(
                    "Delivery {DeliveryId} failed attempt " +
                    "{Attempt}, retry at {NextRetry}",
                    task.DeliveryId, result.AttemptNumber,
                    nextRetry);
            }
            else
            {
                await _deliveryRepo
                    .MarkDeadLetteredAsync(
                        task.DeliveryId, result, ct);

                _logger.LogError(
                    "Delivery {DeliveryId} dead-lettered " +
                    "after {Attempt} attempts",
                    task.DeliveryId, result.AttemptNumber);
            }
        }
    }
}

System Composition

The following code shows how to wire all components together using dependency injection in an ASP.NET Core application:

C#
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

namespace WebhookDeliverySystem
{
    public static class ServiceCollectionExtensions
    {
        public static IServiceCollection AddWebhookDeliverySystem(
            this IServiceCollection services)
        {
            services.Configure<WebhookSystemOptions>(
                opts =>
            {
                opts.DefaultTimeoutMs = 10000;
                opts.MaxRetryAttempts = 8;
                opts.BaseRetryDelayMs = 1000;
                opts.MaxRetryDelayMs = 3600000;
                opts.JitterFactor = 0.3;
            });

            services.AddSingleton<ISignatureVerifier,
                SignatureVerifier>();
            services.AddSingleton<IReplayProtection,
                ReplayProtection>();
            services.AddSingleton<IRetryScheduler,
                RetryScheduler>();
            services.AddSingleton<IFilterEvaluator,
                FilterEvaluator>();
            services.AddSingleton<IRateLimiter,
                TokenBucketRateLimiter>();

            services.AddScoped<IDispatchProcessor,
                DispatchProcessor>();
            services.AddScoped<IDeliveryEngine,
                HttpDeliveryEngine>();

            services.AddHttpClient();
            services.AddHostedService<DeliveryWorkerHostedService>();

            return services;
        }
    }

    public class DeliveryWorkerHostedService : BackgroundService
    {
        private readonly IDeliveryWorker _worker;

        public DeliveryWorkerHostedService(IDeliveryWorker worker)
        {
            _worker = worker;
        }

        protected override async Task ExecuteAsync(
            CancellationToken stoppingToken)
        {
            await _worker.StartAsync(stoppingToken);
        }
    }
}

This composition root registers all the components we have designed: the signature verifier for HMAC-SHA256 signing, replay protection for timestamp and nonce validation, the retry scheduler with exponential backoff and jitter, the filter evaluator for subscription matching, the rate limiter using token buckets, the dispatch processor for event fan-out, and the HTTP delivery engine for making webhook POST requests. The delivery worker runs as a hosted service, consuming delivery tasks and processing them continuously.

27. Conclusion

Designing a webhook and event delivery system is a masterclass in distributed systems engineering. It touches virtually every aspect of system design: from API design and data modeling to security, reliability, scalability, and operational observability. The challenges are real — delivering HTTP requests to untrusted, unreliable endpoints across the public internet requires careful attention to retry strategies, signature verification, rate limiting, and dead letter handling.

Throughout this article, we have covered the complete lifecycle of a webhook delivery system. We started with requirements gathering, establishing clear functional and non-functional requirements. We estimated capacity for a system handling 100,000 events per second with 500,000 delivery QPS at peak. We designed the data model with endpoints, subscriptions, events, deliveries, and delivery attempts. We defined a RESTful API for registration, event publishing, and delivery status queries.

The architecture leverages Kafka as the central event bus, providing durable, ordered, partitioned event storage that decouples ingestion from delivery. The dispatch processor performs efficient subscription matching with filter evaluation, fanning out a single event into multiple delivery tasks. The HTTP delivery engine handles the actual POST requests with proper timeout management, TLS validation, and response classification. Signature verification using HMAC-SHA256 ensures authenticity and integrity, while replay protection via timestamps and nonces prevents replay attacks.

The retry strategy uses exponential backoff with jitter, carefully spreading retries over time to avoid overwhelming struggling endpoints. Deliveries that exhaust their retry budget are moved to the dead letter queue, where they can be inspected, replayed, or archived. Rate limiting at multiple levels — global, per-endpoint, and per-IP — prevents cascading failures and protects both the system and its consumers.

At scale, the system partitions work across Kafka topics and delivery worker pools, enabling linear horizontal scaling. Multi-region deployment ensures global availability and low-latency delivery, with automated failover providing sub-30-second recovery. The tiered storage strategy — hot data in Cassandra, warm data in compressed Cassandra, cold data in S3 Parquet, and archival data in Glacier — manages the enormous volume of delivery logs efficiently.

Key Takeaways

1. Webhooks are push-based event delivery across trust boundaries. Every design decision must account for the fact that the target is untrusted and unreliable.
2. HMAC signatures are non-negotiable. Every delivery must be signed, and consumers must verify signatures before processing.
3. Retry logic is where most of the complexity lives. Exponential backoff, jitter, failure classification, and the dead letter queue are all essential components.
4. Observability is not optional. Without comprehensive delivery logging and monitoring, debugging webhook issues is nearly impossible.
5. Design for horizontal scale from day one. Stateless workers, partitioned queues, and externalized state in Redis and Cassandra make scaling straightforward.
6. Storage costs dominate at scale. Tiered storage and aggressive log rotation are essential for cost management.

Whether you are building a webhook system for your own platform or designing one in a system design interview, the patterns and principles covered in this article provide a solid foundation. The key is to think through each layer of the stack — ingestion, dispatch, delivery, retry, and monitoring — and make explicit design decisions that balance reliability, performance, security, and cost.