1. Introduction: The Three Pillars of Observability

In the era of microservices, serverless functions, and globally distributed systems, understanding what is happening inside your production environment has become one of the most critical engineering challenges. A single user request at a company like Netflix or Uber may traverse dozens of services, each deployed independently, written in different programming languages, and running across multiple availability zones. When something goes wrong—a latency spike, an intermittent error, a cascading failure—the engineering team needs to answer a deceptively simple question: what exactly happened?

Observability is the discipline that answers this question. It is not merely logging or monitoring; it is the ability to understand the internal state of a system by examining its external outputs. The concept originates from control theory, where a system is observable if its internal state can be inferred from its outputs. In modern software engineering, observability is built upon three pillars: traces, metrics, and logs. Each pillar captures a different dimension of system behavior, and together they provide the holistic view required to operate complex distributed systems reliably.

Traces

A distributed trace represents the end-to-end journey of a single request as it flows through multiple services. Each unit of work within a service is represented as a span, and spans are connected in a parent-child relationship to form a trace tree. Traces answer the question: what path did this request take, and where did it spend its time? They are indispensable for diagnosing latency issues, identifying bottlenecks, and understanding cross-service dependencies. The W3C Trace Context specification standardizes how trace context is propagated across service boundaries, enabling vendor-neutral tracing across polyglot environments.

Metrics

Metrics are numerical measurements collected over time. They answer the question: how is the system performing right now, and how does that compare to yesterday? Common metric types include counters (total requests), gauges (current memory usage), histograms (request latency distribution), and summaries. Metrics are lightweight, highly aggregatable, and ideal for dashboards and alerting. Unlike traces, which capture individual request detail, metrics provide a statistical overview of system behavior. The OpenTelemetry Metrics API defines standard instruments like Counter, Histogram, and ObservableGauge that produce metrics conforming to the Prometheus or OpenMetrics format.

Logs

Logs are discrete, timestamped event records produced by applications and infrastructure. They answer the question: what exactly happened at this specific moment? Structured logs—emitted as JSON with consistent fields—enable powerful querying and filtering. When correlated with traces and metrics, logs provide the deepest level of detail for debugging. A log entry that includes a TraceId can be instantly linked to the full distributed trace, allowing engineers to jump from a high-level metric alert to the exact log lines that explain the root cause.

Why a Unified Platform?

Most organizations start by deploying separate tools for each pillar: Prometheus for metrics, ELK for logs, and Jaeger or Zipkin for traces. However, operating three separate systems creates fragmentation. Correlating data across systems is manual and error-prone. Context is lost when switching between tools. A unified observability platform—one that ingests, stores, and correlates all three pillars in a single system—dramatically reduces mean time to resolution (MTTR) and improves the developer experience. This article walks through the complete system design of such a platform, from instrumentation SDKs and collector pipelines to storage backends, visualization, alerting, and multi-tenancy.

The design challenges are substantial. A platform serving thousands of microservices must ingest billions of spans per day, store them efficiently, support sub-second query latency for recent data, and provide real-time alerting—all while maintaining strict multi-tenant isolation. The following sections address each of these challenges in detail, presenting architectural patterns that have been battle-tested at companies operating at Internet scale.

2. OpenTelemetry Standard Deep Dive

OpenTelemetry (OTel) has emerged as the de facto standard for instrumenting, generating, collecting, and exporting telemetry data. Maintained by the Cloud Native Computing Foundation (CNCF), it consolidates the former OpenTracing and OpenCensus projects into a single, comprehensive framework. As of 2026, OpenTelemetry is the second most active CNCF project after Kubernetes, with SDKs available for Java, .NET, Python, Go, JavaScript, Rust, and more.

Core Components

The OpenTelemetry specification defines several key components. The API provides vendor-neutral interfaces for creating spans, metrics, and logs. The SDK implements the API with configurable behavior—sampling, resource detection, batch processing, and export. The Collector is a vendor-agnostic proxy that receives, processes, and exports telemetry data. The Semantic Conventions define standardized naming for spans, attributes, and resources, ensuring consistency across different instrumentation libraries.

Resource and Scope

Every telemetry signal is associated with a Resource, which represents the entity producing the data. A Resource includes attributes like service.name, service.version, host.name, and k8s.namespace.name. Resources are immutable and attached to all signals emitted by a given service instance. Scope identifies the instrumentation library that created the signal, enabling consumers to understand which library produced a given span or metric.

The OTLP Protocol

OpenTelemetry Protocol (OTLP) is the native protocol for exporting telemetry data. It supports both gRPC and HTTP/protobuf transports, with bidirectional streaming for gRPC. OTLP defines separate RPCs for traces, metrics, and logs. The protocol includes built-in support for compression (gzip, zstd), retry with exponential backoff, and keepalive. OTLP has become the universal language of telemetry transport, with virtually every observability backend supporting OTLP ingestion natively or via a collector exporter.

Semantic Conventions in Practice

Semantic conventions eliminate naming inconsistency across teams. Instead of one team naming a span HTTP_GET and another naming it http-request, the conventions prescribe http.request.method and url.full as the standard attributes. For databases, spans use db.system, db.query.text, and db.namespace. For messaging systems, attributes include messaging.system, destination.name, and messaging.operation.type. Adhering to these conventions enables the platform to provide unified query experiences—a single query for all HTTP server spans with status code 500 works across every service in the organization.

OpenTelemetry and the W3C Standards

OpenTelemetry natively supports W3C Trace Context propagation, with the traceparent and tracestate headers injected and extracted automatically. This integration ensures that traces can flow across services regardless of the transport protocol—HTTP, gRPC, message queues, or even Lambda invocations. The strategic advantage of building an observability platform on OpenTelemetry is vendor lock-in avoidance. Instrumentation code is written against the OTel API, not against a vendor SDK. The Collector can export to multiple backends simultaneously, providing portability that is invaluable for organizations that anticipate changing their backend stack over time.

3. System Architecture Overview

Designing a distributed tracing and observability platform requires careful thought about data flow, processing stages, and storage trade-offs. The platform must ingest telemetry from thousands of service instances, process it in real-time, store it efficiently, and serve it through query APIs and dashboards—all while handling millions of data points per second.

Ingestion Layer

The ingestion layer receives telemetry data from instrumented services. In a Kubernetes environment, this typically takes the form of a sidecar or DaemonSet deployment pattern. In the sidecar model, each service pod gets a co-located collector instance that receives telemetry over localhost, minimizing network hops. In the DaemonSet model, one collector runs per node, serving all pods on that node. Both patterns have trade-offs: sidecars offer stronger isolation and per-service configuration, while DaemonSets are more resource-efficient. Agent collectors perform initial processing—batching, compression, basic filtering—before forwarding to gateway collectors.

Gateway Layer

Gateway collectors are deployed as horizontally scalable deployments behind a load balancer. They receive data from all agent collectors and perform heavier processing: tail-based sampling decisions, attribute enrichment from external sources, and protocol translation. Gateways are stateless and can be scaled horizontally based on throughput. They maintain in-memory buffers for tail-based sampling, which requires holding complete traces before making accept/reject decisions. The gateway layer is also where data is routed to different storage backends based on signal type—traces to ClickHouse, metrics to Prometheus, logs to Elasticsearch.

Storage Layer

Each signal type benefits from a storage backend optimized for its access pattern. Traces are best stored in columnar databases like ClickHouse, which excels at high-throughput ingestion and analytical queries over structured span data. Metrics fit naturally into time-series databases like Prometheus or Thanos, which provide efficient label-based querying and long-term storage. Logs can be stored in Elasticsearch for full-text search or in cost-effective systems like Loki or ClickHouse. A critical architectural decision is whether to use a unified storage backend (ClickHouse for everything) or specialized backends per signal.

Cross-Cutting Concerns

Several concerns span the entire architecture. Authentication and authorization ensure that only authorized services can push telemetry and only authorized users can query it. Rate limiting prevents any single tenant from overwhelming the platform. Data retention policies automatically expire old data to manage storage costs. Encryption in transit (mTLS between collectors) and at rest (encrypted storage volumes) protect sensitive telemetry data. Health monitoring of the observability platform itself requires a separate, lightweight monitoring stack often called the meta-observability layer.

4. Trace Context Propagation

Trace context propagation is the mechanism that enables distributed traces to span multiple services. Without propagation, each service would create isolated spans with no connection to one another. Propagation works by injecting context into request headers at the calling service and extracting it at the receiving service. This seemingly simple concept has significant complexity in practice, particularly in polyglot environments with multiple transport protocols and messaging systems.

W3C Trace Context

The W3C Trace Context specification defines two headers: traceparent and tracestate. The traceparent header carries the trace identifier, parent span ID, and trace flags in a compact format: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. The first field (00) is the version, the second is the 32-character hex trace ID, the third is the 16-character hex span ID, and the fourth is the trace flags (01 = sampled). The tracestate header carries vendor-specific data as key-value pairs.

B3 Propagation (Zipkin)

B3 propagation uses multiple headers: X-B3-TraceId, X-B3-SpanId, X-B3-ParentSpanId, X-B3-Sampled, and X-B3-Flags. While B3 is widely supported, the W3C Trace Context standard has been broadly adopted as the default. OpenTelemetry supports both formats via pluggable propagators, allowing services using different propagation standards to interoperate through the collector.

Injection and Extraction Flow

When Service A makes an outgoing call to Service B, the OTel SDK propagator injects the current span context into the request headers. When Service B receives the request, its OTel SDK extracts the context from the headers and creates a new child span linked to the parent. This process repeats at every service boundary. For asynchronous communication (message queues), the same pattern applies—the producer injects context into message headers, and the consumer extracts it. The W3C specification recommends that each consumer creates its own span linked to the producer context rather than continuing the same trace when messages may be forked or retried.

Propagation Configuration in C#

C#
using OpenTelemetry.Context.Propagation;
using OpenTelemetry.Trace;

var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddSource("MyApp.Services")
    .SetSampler(new ParentBasedSampler(root: new AlwaysOnSampler()))
    .AddOtlpExporter(opts =>
    {
        opts.Endpoint = new Uri("otel-collector:4317");
        opts.Protocol = OtlpExportProtocol.Grpc;
    })
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddSqlClientInstrumentation()
    .Build();

var compositePropagator = new CompositeTextMapPropagator(new TextMapPropagator[]
{
    TraceContextPropagator.Instance,
    BaggagePropagator.Instance,
});

using var activity = MyActivitySource.StartActivity("ProcessOrder");
activity?.SetTag("order.id");

var propagationContext = new PropagationContext(
    activity?.Context ?? default, Baggage.Current);

var headers = new Dictionary<string, string>();
compositePropagator.Inject(propagationContext, headers, (carrier, key, value) =>
{
    carrier[key] = value;
});

foreach (var header in headers)
{
    httpClient.DefaultRequestHeaders.TryAddWithoutValidation(header.Key, header.Value);
}

Propagation best practices include using 128-bit trace IDs to prevent collision, never exposing trace context in user-facing responses, ensuring propagation works across all transport types, testing propagation thoroughly with chaos engineering, and monitoring propagation failure rates. A single broken propagation link creates orphaned spans, making traces useless for debugging.

5. Instrumentation SDK Architecture

Instrumentation is the process of adding telemetry collection code to applications. The choice between automatic instrumentation and manual instrumentation—and how to balance the two—is one of the most important design decisions for an observability platform.

Auto-Instrumentation

Auto-instrumentation uses framework hooks, middleware, and runtime interception to capture telemetry without modifying application code. In the .NET ecosystem, this is achieved through OpenTelemetry.AutoInstrumentation, which leverages the CLR Profiler API to instrument libraries at runtime. Auto-instrumentation captures HTTP request/response details, database queries, gRPC calls, and message queue operations. The primary advantage is instant coverage—teams add a NuGet package and an environment variable, and they immediately get traces for all supported libraries. The limitation is that auto-instrumentation can only capture what the framework exposes through hooks.

Manual Instrumentation

Manual instrumentation involves developers explicitly creating spans around specific operations using the OTel API. This approach provides the richest context because developers know the business meaning of each operation. A payment processing span might include attributes like payment.amount, payment.currency, and payment.method—information that no auto-instrumentation library can infer. Manual instrumentation also enables fine-grained control over span hierarchy, allowing developers to create nested spans that mirror the logical structure of their code.

The Instrumentation Spectrum

In practice, most production systems use a combination of both. Auto-instrumentation provides the foundational layer—all HTTP, database, and messaging calls are traced automatically. Manual instrumentation is layered on top for business-critical operations. The OTel SDK allows both to coexist because manual spans created with named ActivitySource instances are independent of auto-instrumented spans. The SDK merges them into a single trace tree based on the active context.

Instrumenting a .NET Service

C#
using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

public class Program
{
    public static void Main(string[] args)
    {
        var builder = WebApplication.CreateBuilder(args);
        var resource = ResourceBuilder.CreateDefault()
            .AddService(
                serviceName: "OrderService",
                serviceVersion: typeof(Program).Assembly.GetName().Version?.ToString() ?? "1.0.0",
                serviceInstanceId: Environment.MachineName)
            .AddAttributes(new Dictionary<string, object>
            {
                { "deployment.environment", builder.Environment.EnvironmentName },
                { "cloud.provider", "azure" },
                { "cloud.region", "eastus2" }
            });

        builder.Services.AddOpenTelemetry()
            .WithTracing(tracerBuilder =>
            {
                tracerBuilder
                    .SetResourceBuilder(resource)
                    .AddAspNetCoreInstrumentation(opts =>
                    {
                        opts.RecordException = true;
                        opts.Filter = (ctx) => !ctx.Request.Path.StartsWithSegments("/health");
                        opts.EnrichWithHttpRequest = (activity, req) =>
                            activity.SetTag("tenant.id", req.Headers["X-Tenant-Id"].FirstOrDefault());
                    })
                    .AddHttpClientInstrumentation()
                    .AddSqlClientInstrumentation(opts => opts.SetDbStatementForText = true)
                    .AddSource("OrderService.BusinessLogic")
                    .SetSampler(new ParentBasedSampler(
                        root: new TraceIdRatioBasedSampler(0.1)));
            })
            .WithMetrics(meterBuilder =>
            {
                meterBuilder
                    .SetResourceBuilder(resource)
                    .AddAspNetCoreInstrumentation()
                    .AddHttpClientInstrumentation()
                    .AddRuntimeInstrumentation()
                    .AddMeter("OrderService.Metrics")
                    .AddOtlpExporter();
            });

        var app = builder.Build();
        app.MapControllers();
        app.Run();
    }
}

public static class OrderActivities
{
    private static readonly ActivitySource Source = new("OrderService.BusinessLogic");

    public static Activity? StartProcessOrder(Guid orderId, string customerId)
    {
        var activity = Source.StartActivity("ProcessOrder", ActivityKind.Internal);
        activity?.SetTag("order.id", orderId.ToString());
        activity?.SetTag("customer.id", customerId);
        return activity;
    }
}

eBPF-Based Auto-Instrumentation

eBPF represents a new frontier in auto-instrumentation. By attaching to kernel-level system calls, eBPF-based agents can trace network requests, file system operations, and process scheduling without any application-level changes. Projects like Pixie and Coroot use eBPF to automatically generate service maps and capture network flows. While eBPF provides extraordinary breadth, it lacks the depth of application-level instrumentation. The optimal architecture combines eBPF-based system-level tracing with application-level OTel instrumentation, with the collector correlating both data sources into unified traces.

6. Span Data Model and Lifecycle

A span is the fundamental unit of work in distributed tracing. Understanding the span data model in detail—its fields, relationships, lifecycle states, and how it maps to storage—is essential for designing both the instrumentation SDK and the storage backend.

Span Fields

Every span contains several mandatory fields. The TraceId (128-bit) identifies the trace and is consistent across all spans. The SpanId (64-bit) uniquely identifies this span. The ParentSpanId links to the parent span. The OperationName describes the work performed. The StartTime and EndTime define the span duration with nanosecond precision. The Status indicates success (Unset), error (Error), or cancellation (Cancelled). Optional fields include attributes (key-value contextual data), events (timestamped annotations), and links (causal relationships to other traces).

Span Lifecycle

The lifecycle of a span within the SDK follows several stages. First, the span is created by an ActivitySource.StartActivity() call. During execution, attributes, events, and status are modified. Finally, the span is ended by calling activity.Stop(), which finalizes the end time and enqueues the span for export. The SDK BatchSpanProcessor holds ended spans in a memory buffer and exports them in batches. If the buffer fills or the batch interval elapses, the processor flushes the batch to the exporter.

Span to Storage Mapping

C#
using System.Diagnostics;
using OpenTelemetry;

public class EnrichingSpanProcessor : BaseProcessor<Activity>
{
    private readonly Resource _resource;
    public EnrichingSpanProcessor(Resource resource) { _resource = resource; }

    public override void OnStart(Activity data)
    {
        data.SetTag("host.ip", GetLocalIpAddress());
        data.SetTag("process.pid", Environment.ProcessId);
        data.SetTag("runtime.version", Environment.Version.ToString());
    }

    public override void OnEnd(Activity data)
    {
        var duration = data.Duration.TotalMilliseconds;
        data.SetTag("span.duration_ms", duration);
        var latencyTier = duration switch
        {
            < 100 => "fast",
            < 500 => "normal",
            < 2000 => "slow",
            _ => "very_slow"
        };
        data.SetTag("latency.tier", latencyTier);

        if (data.Status == ActivityStatusCode.Error)
        {
            data.SetTag("error.severity", "high");
            data.SetTag("process.memory_working_set_mb",
                GC.GetTotalMemory(false) / (1024 * 1024));
        }
    }

    private static string GetLocalIpAddress()
    {
        try
        {
            var host = System.Net.Dns.GetHostName();
            var addresses = System.Net.Dns.GetHostAddresses(host);
            return addresses.FirstOrDefault()?.ToString() ?? "unknown";
        }
        catch { return "unknown"; }
    }
}

In ClickHouse, spans are typically stored in a single wide table with columns for each standard field and a Map column for dynamic attributes. This design supports both structured queries (filter by service name, operation, duration) and attribute-based queries. The trace tree structure is reconstructed at query time by joining spans on TraceId and building the parent-child hierarchy in memory.

7. Collector Pipeline

The OpenTelemetry Collector is the backbone of the telemetry data pipeline. It is a vendor-agnostic proxy that receives telemetry data, processes it through a configurable pipeline, and exports it to one or more backends. The pluggable architecture built around receivers, processors, and exporters makes it extraordinarily flexible.

Architecture Components

A Collector instance consists of four component types. Receivers accept telemetry data in various formats—OTLP (gRPC/HTTP), Jaeger, Zipkin, Prometheus, and Kafka. Processors transform, filter, and enrich data in flight—batch, sampling, attribute manipulation, and memory limiting. Exporters send processed data to backends—ClickHouse, Prometheus, Elasticsearch, Kafka, or another Collector instance. Connectors link pipelines together, allowing the output of one pipeline to feed the input of another.

Three Common Topologies

The agent topology deploys one Collector per host or pod, performing initial processing locally before forwarding to a central gateway. This is the most common pattern in Kubernetes. The gateway topology deploys a horizontally scalable cluster of Collectors behind a load balancer for centralized processing. The combined topology uses a single deployment for both functions, suitable for smaller deployments. Most production systems use the agent-gateway combination: agents for local reduction, gateways for centralized decisions like tail-based sampling.

Agent Collector Configuration

C#
public class CollectorConfigGenerator
{
    public static string GenerateAgentConfig(string serviceName, int samplingRate) =>
        $@"
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 4
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  batch:
    timeout: 5s
    send_batch_size: 8192
    send_batch_max_size: 16384
  resource:
    attributes:
      - key: service.deployment.environment
        value: production
        action: upsert
      - key: collector.tier
        value: agent
        action: insert
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: error-policy
        type: status_code
        status_code: {{ status_codes: [ERROR] }}
      - name: slow-traces
        type: latency
        latency: {{ threshold_ms: 5000 }}
      - name: probabilistic
        type: probabilistic
        probabilistic: {{ sampling_percentage: {samplingRate} }}

exporters:
  otlp/gateway:
    endpoint: otel-gateway:4317
    tls:
      cert_file: /certs/agent.pem
      key_file: /certs/agent-key.pem
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource, tail_sampling, batch]
      exporters: [otlp/gateway]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/gateway]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [otlp/gateway]
  telemetry:
    metrics:
      address: 0.0.0.0:8888
      level: detailed
";
}

Processors Deep Dive

The batch processor accumulates spans in memory and exports them in batches, reducing network overhead. The memory limiter prevents OOM by monitoring heap usage and dropping data when limits are exceeded. The tail sampling processor makes sampling decisions after seeing the complete trace. The k8sattributes processor enriches spans with Kubernetes metadata by watching the K8s API. The attributes processor adds, modifies, or deletes span attributes. The filter processor removes signals matching specified criteria.

Scaling the Collector

Scaling requires understanding bottlenecks. The receiver is typically first—each incoming OTLP request must be deserialized and queued. Processing is CPU-bound for attribute transformations and memory-bound for tail sampling. Exporters are I/O-bound for network throughput. In Kubernetes, horizontal scaling is achieved by increasing replica count. For stateful processors like tail sampling, consistent hashing ensures all spans from a given trace go to the same Collector instance. The Collector pipeline should be treated as a first-class production system—with its own monitoring, alerting, and capacity planning—because failures directly impact observability coverage.

8. Trace Storage Backend

The trace storage backend persists span data, supports efficient trace retrieval, and enables analytical queries over large trace datasets. The choice of storage technology profoundly impacts query performance, storage cost, and the types of analysis the platform can support.

ClickHouse

ClickHouse is a columnar OLAP database designed for real-time analytics. It excels at high-throughput ingestion (millions of rows per second per node) and fast analytical queries over structured data. ClickHouse stores data in columns, enabling highly compressed storage and efficient scans for queries that touch many rows but few columns. The MergeTree engine provides automatic background merging, TTL-based expiration, and materialized views for pre-aggregation.

Cassandra

Apache Cassandra is a distributed wide-column store designed for high availability and linear scaling. The TraceId serves as the partition key, ensuring all spans for a trace are co-located on the same node. This makes trace retrieval extremely fast. Cassandra write performance is exceptional. However, query flexibility is limited—secondary indexes are inefficient at scale, and analytical queries spanning many partitions are slow.

Elasticsearch

Elasticsearch provides full-text search, structured queries, and aggregations. The inverted index enables fast searches across any attribute. Aggregations support analytics like latency histograms. The downside is storage efficiency—the inverted index consumes significantly more storage than columnar formats.

ClickHouse Schema for Traces

C#
public class ClickHouseTraceSchema
{
    public static string GetCreateTableSql() =>
    "CREATE TABLE IF NOT EXISTS traces (" +
    "  trace_id       FixedString(32)," +
    "  span_id        FixedString(16)," +
    "  parent_span_id FixedString(16)," +
    "  service_name   LowCardinality(String)," +
    "  operation_name LowCardinality(String)," +
    "  span_kind      Enum8('INTERNAL'=1,'SERVER'=2,'CLIENT'=3,'PRODUCER'=4,'CONSUMER'=5)," +
    "  start_time     DateTime64(9)," +
    "  end_time       DateTime64(9)," +
    "  duration_us    UInt64," +
    "  status_code    Enum8('UNSET'=0,'OK'=1,'ERROR'=2)," +
    "  status_message String," +
    "  attributes     Map(LowCardinality(String), String)," +
    "  resource_attrs Map(LowCardinality(String), String)," +
    "  tenant_id      LowCardinality(String) DEFAULT ''" +
    ") ENGINE = MergeTree()" +
    "PARTITION BY toYYYYMMDD(start_time)" +
    "ORDER BY (tenant_id, service_name, start_time, trace_id, span_id)" +
    "TTL start_time + INTERVAL 30 DAY" +
    "SETTINGS index_granularity = 8192;";

    public static string GetTraceByIdSql(string traceId) =>
        $"SELECT * FROM traces WHERE trace_id = '{traceId}' ORDER BY start_time ASC;";

    public static string GetErrorRateSql(string serviceName) =>
        $"SELECT toDate(start_time) AS day, count() AS total, " +
        $"countIf(status_code = 'ERROR') AS errors, " +
        $"round(errors/total*100,2) AS error_rate " +
        $"FROM traces WHERE service_name = '{serviceName}' " +
        $"GROUP BY day ORDER BY day;";
}

Query Patterns and Retention

Trace queries follow several distinct patterns: trace retrieval (fetching all spans for a TraceId), trace search (finding traces matching criteria), service analytics (computing error rates and latency percentiles), and dependency analysis (mapping service-to-service call relationships). Recent traces (24-48 hours) are stored at full fidelity. Older traces may be compressed or sampled. Very old traces can be moved to object storage via tiered storage. Most organizations retain full traces for 3-7 days and sampled traces for 30-90 days.

9. Metrics Collection and Aggregation

Metrics provide the statistical backbone of observability. While traces capture individual request detail, metrics aggregate system behavior over time to reveal trends, anomalies, and capacity signals. A comprehensive metrics subsystem supports counters, gauges, histograms, and exemplars, with efficient storage and query capabilities that scale to millions of active time series.

Metric Types in OpenTelemetry

OpenTelemetry defines four primary instrument types. The Counter is monotonically increasing—total requests, total bytes sent. The Gauge represents a current value that can fluctuate—CPU usage, queue depth. The Histogram records a distribution of values—request latency, response size—and computes configurable percentiles. The Exponential Histogram is a memory-efficient variant using power-of-two bucket boundaries. Exemplars bridge metrics and traces by attaching specific trace IDs to histogram samples, enabling drill-down from P99 latency metrics to the actual trace that contributed to the measurement.

Custom Metrics Instrumentation in C#

C#
using System.Diagnostics.Metrics;
using OpenTelemetry.Metrics;

public class OrderMetrics : IDisposable
{
    public const string MeterName = "OrderService.Metrics";
    private readonly Meter _meter;
    private readonly Counter<long> _ordersCreated;
    private readonly Counter<long> _ordersFailed;
    private readonly Histogram<double> _orderProcessingDuration;
    private readonly Histogram<double> _paymentGatewayLatency;
    private readonly ObservableGauge<int> _pendingOrders;
    private long _totalPendingOrders;

    public OrderMetrics()
    {
        _meter = new Meter(MeterName, "1.0");
        _ordersCreated = _meter.CreateCounter<long>("orders.created.total",
            description: "Total orders created", unit: "{order}");
        _ordersFailed = _meter.CreateCounter<long>("orders.failed.total",
            description: "Total failed orders");
        _orderProcessingDuration = _meter.CreateHistogram<double>(
            "orders.processing.duration", unit: "ms",
            advice: new HistogramAdvice
            {
                ExplicitBucketBoundaries = new[] { 10, 25, 50, 100, 250, 500, 1000, 2500, 5000 }
            });
        _paymentGatewayLatency = _meter.CreateHistogram<double>(
            "payments.gateway.latency", unit: "ms");
        _pendingOrders = _meter.CreateObservableGauge<int>(
            "orders.pending.count",
            () => new Measurement<int>((int)Interlocked.Read(ref _totalPendingOrders)));
    }

    public void RecordOrderCreated(string region)
    {
        _ordersCreated.Add(1, new("region", region));
        Interlocked.Increment(ref _totalPendingOrders);
    }
    public void RecordOrderFailed(string region, string reason)
    {
        _ordersFailed.Add(1, new("region", region), new("reason", reason));
    }
    public void RecordProcessingDuration(double durationMs, string service)
    {
        _orderProcessingDuration.Record(durationMs, new("target_service", service));
    }
    public void RecordPaymentLatency(double latencyMs, string provider, bool success)
    {
        _paymentGatewayLatency.Record(latencyMs,
            new("provider", provider), new("success", success));
    }
    public void Dispose() { _meter?.Dispose(); }
}

The metrics subsystem must handle the cardinality challenge. Cardinality is the number of unique label combinations. A metric with 10 services, 20 endpoints, and 5 status codes has 1,000 unique time series. Add unbounded labels like user IDs, and cardinality explodes. The platform must enforce cardinality limits and detect high-cardinality metrics at ingestion time. Metrics are typically stored in Prometheus with Thanos or Grafana Mimir providing multi-tenancy and long-term storage via object stores.

10. Log Aggregation and Correlation

Logs remain the most detailed source of system information. While traces provide request-level flow and metrics provide statistical aggregation, logs capture the precise details of individual events. The challenge is not collecting logs but correlating them across services and making them searchable at scale.

Structured Logging

Structured logging emits logs as JSON objects with consistent fields—timestamp, level, message, service name, trace ID, span ID, and custom business fields. Structured logs enable precise queries like finding all ERROR logs from a specific service where a particular order ID appears, which is impossible with free-form text.

Trace-Log Correlation

The most powerful correlation technique is embedding trace context in log entries. When the OTel SDK is configured, it automatically injects TraceId and SpanId into the log context. All log entries emitted during a request automatically include these fields. From any span in the trace viewer, you can jump to the exact logs emitted during that span. Conversely, from a log entry you can navigate to the full distributed trace. This bidirectional correlation eliminates traditional friction between looking at logs and looking at traces.

Logging with Trace Correlation in C#

C#
using Microsoft.Extensions.Logging;
using OpenTelemetry.Logs;

public static class ObservabilityLoggingSetup
{
    public static ILoggingBuilder AddObservabilityLogging(
        this ILoggingBuilder builder, string otlpEndpoint)
    {
        builder
            .ClearProviders()
            .AddConsole(options =>
            {
                options.IncludeScopes = true;
                options.TimestampFormat = "yyyy-MM-dd HH:mm:ss.fff zzz";
            })
            .AddOpenTelemetry(options =>
            {
                options.IncludeScopes = true;
                options.IncludeFormattedMessage = true;
                options.ParseStateValues = true;
                options.IncludeExceptionDetails = true;
                options.SetResourceBuilder(
                    ResourceBuilder.CreateDefault()
                        .AddService("OrderService", "1.0.0"));
                options.AddOtlpExporter(o =>
                {
                    o.Endpoint = new Uri(otlpEndpoint);
                    o.Protocol = OtlpExportProtocol.Grpc;
                });
            });
        return builder;
    }
}

public class OrderProcessingService
{
    private readonly ILogger<OrderProcessingService> _logger;
    public OrderProcessingService(ILogger<OrderProcessingService> logger)
    {
        _logger = logger;
    }

    public async Task ProcessOrderAsync(Order order)
    {
        _logger.LogInformation(
            "Processing order {OrderId} for customer {CustomerId} " +
            "with {ItemCount} items totaling {TotalAmount:C}",
            order.Id, order.CustomerId, order.Items.Count, order.TotalAmount);

        try
        {
            await ValidateInventoryAsync(order);
            await ProcessPaymentAsync(order);
            await ConfirmOrderAsync(order);
        }
        catch (PaymentFailedException ex)
        {
            _logger.LogError(ex,
                "Payment failed for order {OrderId}: {FailureReason}",
                order.Id, ex.FailureReason);

            using (_logger.BeginScope(new Dictionary<string, object>
            {
                ["order.state"] = order.CurrentState,
                ["payment.attempt"] = order.PaymentAttempts,
                ["payment.provider"] = order.PaymentProvider
            }))
            {
                _logger.LogWarning(
                    "Payment retry scheduled for order {OrderId}", order.Id);
            }
            throw;
        }
    }
}

Log Storage and Lifecycle

Log storage must balance query capability against cost. Hot storage retains recent logs (1-7 days) for interactive debugging. Warm storage holds older logs (7-30 days) for compliance. Cold storage archives beyond 30 days. The platform should implement log sampling—retaining all ERROR and WARN logs but sampling INFO and DEBUG logs at 10-50%—to manage storage costs without losing critical diagnostic information.

11. Distributed Trace Visualization

Visualization is where raw trace data becomes actionable insight. The best storage backend and the most complete instrumentation are wasted if engineers cannot quickly understand the trace visualizations. This section covers waterfall charts, flame graphs, service maps, and the design considerations for building effective trace viewers.

Waterfall Charts

The waterfall chart displays spans as horizontal bars along a timeline, with parent-child relationships shown as indentation. Each bar width represents the span duration, and horizontal position represents start/end time relative to the root span. Waterfall charts excel at showing sequential request flow—which services were called, where time was spent, and where gaps exist. Key UX decisions include zoom behavior, span color coding (by service, status, or duration), and tooltip content.

Flame Graphs

Flame graphs represent the trace as a hierarchical stack where each row is a service and each block is a span. The horizontal axis represents percentage of total trace duration, and vertical axis represents depth. Flame graphs excel at showing where time was spent in aggregate—a wide block at the bottom indicates a service that consumed most of the trace duration. They are useful for comparing multiple traces and identifying patterns across many requests.

Service Map Visualization

Service maps show service dependencies as a directed graph. Nodes represent services, edges represent call relationships with metrics (request rate, error rate, latency). Maps are computed from trace data by analyzing parent-child relationships across many traces. The visualization must handle dynamic topology and graph complexity—a large organization might have hundreds of services with thousands of edges, requiring layout algorithms and progressive disclosure.

Trace Viewer API in C#

C#
[ApiController]
[Route("api/v1/traces")]
public class TraceQueryController : ControllerBase
{
    private readonly ITraceRepository _traceRepo;

    public TraceQueryController(ITraceRepository traceRepo) { _traceRepo = traceRepo; }

    [HttpGet("{traceId}")]
    public async Task<IActionResult> GetTrace(string traceId, CancellationToken ct)
    {
        var sw = Stopwatch.StartNew();
        var spans = await _traceRepo.GetSpansByTraceIdAsync(traceId, ct);
        sw.Stop();

        if (!spans.Any())
            return NotFound(new { error = "Trace not found", traceId });

        var traceStart = spans.Min(s => s.StartTime);
        var traceDuration = spans.Max(s => s.EndTime) - traceStart;

        var waterfall = spans
            .OrderBy(s => s.StartTime).ThenBy(s => s.Depth)
            .Select(s => new WaterfallEntry
            {
                SpanId = s.SpanId,
                OperationName = s.OperationName,
                ServiceName = s.ServiceName,
                StartOffset = s.StartTime - traceStart,
                Duration = s.EndTime - s.StartTime,
                PercentageOfTotal = traceDuration.TotalMilliseconds > 0
                    ? (s.EndTime - s.StartTime).TotalMilliseconds /
                      traceDuration.TotalMilliseconds * 100 : 0,
                HasError = s.Status == SpanStatus.Error,
                Depth = s.Depth
            }).ToList();

        return Ok(new
        {
            TraceId = traceId,
            TotalDuration = traceDuration,
            SpanCount = spans.Count,
            ServiceCount = spans.Select(s => s.ServiceName).Distinct().Count(),
            HasError = spans.Any(s => s.Status == SpanStatus.Error),
            Waterfall = waterfall,
            QueryTimeMs = sw.ElapsedMilliseconds
        });
    }

    [HttpGet("search")]
    public async Task<IActionResult> SearchTraces(
        [FromQuery] string? serviceName,
        [FromQuery] string? operationName,
        [FromQuery] TimeSpan? minDuration,
        [FromQuery] bool? hasError,
        [FromQuery] DateTime? startTime,
        [FromQuery] DateTime? endTime,
        [FromQuery] int limit = 20,
        CancellationToken ct = default)
    {
        var request = new TraceSearchRequest
        {
            ServiceName = serviceName,
            OperationName = operationName,
            MinDuration = minDuration,
            HasError = hasError,
            StartTime = startTime ?? DateTime.UtcNow.AddHours(-1),
            EndTime = endTime ?? DateTime.UtcNow,
            Limit = Math.Min(limit, 100)
        };
        var results = await _traceRepo.SearchTracesAsync(request, ct);
        return Ok(results);
    }
}

Trace visualization queries must return within 200ms for interactive use. The most critical query is trace retrieval—fetching all spans for a TraceId and building the tree. In ClickHouse, this is a single partition read. Client-side rendering uses virtualized lists and canvas-based drawing for smooth performance with large trace trees.

12. Alerting and Anomaly Detection

Alerting transforms observability from a reactive debugging tool into a proactive reliability system. The goal is to detect problems before users are impacted and to route alerts to the right teams with actionable context.

Alert Rule Types

Threshold alerts trigger when a metric crosses a fixed value—error rate above 5% for 5 minutes. Anomaly alerts trigger when a metric deviates significantly from its expected pattern. Predictive alerts use time-series forecasting to anticipate future threshold violations. SLO-based alerts track error budgets and alert when consumption rate threatens the SLO target. Multi-window burn rate alerting, as described in the Google SRE Workbook, is the gold standard—it combines short-window burn rate (detecting sudden incidents) with long-window burn rate (confirming sustained degradation).

Alert Routing and Escalation

Alerts must be routed to the correct team based on service ownership, severity, and on-call schedule. Alert grouping combines related alerts to reduce noise. Alert suppression prevents duplicate notifications when a single root cause triggers multiple symptoms. Every alert should include a link to the relevant trace, a runbook URL, and the service owner.

Pre-configured Alert Rules in C#

C#
public enum AlertSeverity { Critical, High, Warning, Info }
public enum AlertType { Threshold, AnomalyDetection, SLOBurnRate, Composite, LogBased }

public static class ProductionAlertRules
{
    public static List<AlertRule> GetDefaultRules() => new()
    {
        new AlertRule
        {
            Name = "HighErrorRate",
            Description = "Service error rate exceeds 5% for 5 minutes",
            Severity = AlertSeverity.Critical,
            Type = AlertType.Threshold,
            Query = "rate(traces_status_total{status='ERROR'}[5m]) / " +
                    "rate(traces_total[5m]) * 100 > 5",
            ForDuration = TimeSpan.FromMinutes(5),
            NotifyChannels = new[] { "pagerduty", "slack-incidents" },
            Tags = new[] { "sli", "reliability" }
        },
        new AlertRule
        {
            Name = "HighLatencyP99",
            Description = "P99 latency exceeds SLA threshold",
            Severity = AlertSeverity.Warning,
            Type = AlertType.Threshold,
            Query = "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 2.0",
            ForDuration = TimeSpan.FromMinutes(10),
            NotifyChannels = new[] { "slack-engineering" },
            Tags = new[] { "sli", "performance" }
        },
        new AlertRule
        {
            Name = "SLOBurnRateCritical",
            Description = "Error budget burning at 14.4x (2% remaining in 1h)",
            Severity = AlertSeverity.Critical,
            Type = AlertType.SLOBurnRate,
            Query = "slo:burn_rate:5m{window='1h'} > 14.4 and slo:burn_rate:1h{window='6h'} > 14.4",
            ForDuration = TimeSpan.FromMinutes(2),
            NotifyChannels = new[] { "pagerduty", "slack-incidents" },
            Tags = new[] { "slo", "error-budget" }
        },
        new AlertRule
        {
            Name = "CollectorBackpressure",
            Description = "Collector queue approaching capacity",
            Severity = AlertSeverity.High,
            Type = AlertType.Threshold,
            Query = "otelcol_processor_batch_batch_size / " +
                    "otelcol_processor_batch_batch_size_max > 0.8",
            ForDuration = TimeSpan.FromMinutes(3),
            NotifyChannels = new[] { "slack-platform" },
            Tags = new[] { "infrastructure", "observability" }
        }
    };
}

public class AlertRule
{
    public string Id { get; set; } = Guid.NewGuid().ToString();
    public string Name { get; set; } = string.Empty;
    public string Description { get; set; } = string.Empty;
    public AlertSeverity Severity { get; set; }
    public AlertType Type { get; set; }
    public string Query { get; set; } = string.Empty;
    public TimeSpan ForDuration { get; set; } = TimeSpan.FromMinutes(5);
    public string[] NotifyChannels { get; set; } = Array.Empty<string>();
    public string[] Tags { get; set; } = Array.Empty<string>();
}

Anomaly Detection Algorithms

Anomaly detection typically uses statistical methods rather than deep learning. The most common approaches are Exponential Weighted Moving Average (EWMA) for level shifts, Seasonal-Hybrid ESD for seasonal patterns, and Prophet for forecasting with trend and seasonality decomposition. Multi-metric anomaly detection examines correlated metrics simultaneously—if latency, error rate, and CPU all spike together, confidence is higher. The SLO-based approach replaces threshold alerting with error budget consumption tracking, automatically accounting for traffic patterns and varying severity.

13. Service Dependency Mapping

Service dependency maps are among the most valuable observability visualizations. They provide an at-a-glance view of how services interact, which services are critical bottlenecks, and where failures could cascade. Dependency maps are automatically generated from trace data—every parent-child relationship between spans in different services becomes an edge in the dependency graph.

Graph Data Model

The service dependency graph is a directed multigraph where nodes represent services and edges represent call relationships. Each edge carries metrics: request rate, error rate, median latency, and P99 latency. Edges are directional—the edge from Service A to Service B represents A calling B, with separate metrics for each direction.

Graph Computation in C#

C#
public class DependencyGraphService
{
    private readonly ITraceRepository _traceRepo;
    public DependencyGraphService(ITraceRepository traceRepo) { _traceRepo = traceRepo; }

    public async Task<ServiceDependencyGraph> ComputeGraphAsync(
        DateTime startTime, DateTime endTime, CancellationToken ct = default)
    {
        var edges = await _traceRepo.QueryAsync<RawEdge>(@"
            SELECT
                parent.service_name AS source_service,
                child.service_name AS target_service,
                count() AS request_count,
                countIf(child.status_code = 'ERROR') AS error_count,
                avg(child.duration_us) / 1000.0 AS avg_duration_ms,
                quantile(0.99)(child.duration_us) / 1000.0 AS p99_duration_ms,
                count(DISTINCT child.trace_id) AS unique_traces
            FROM traces AS child
            INNER JOIN traces AS parent
                ON child.trace_id = parent.trace_id
                AND child.parent_span_id = parent.span_id
            WHERE child.start_time >= toDateTime64(@start)
              AND child.start_time <= toDateTime64(@end)
              AND parent.service_name != child.service_name
            GROUP BY source_service, target_service
            ORDER BY request_count DESC
        ", startTime, endTime, ct);

        var serviceNames = new HashSet<string>();
        var graphEdges = new List<ServiceEdge>();

        foreach (var edge in edges)
        {
            serviceNames.Add(edge.SourceService);
            serviceNames.Add(edge.TargetService);
            var errorRate = edge.RequestCount > 0
                ? (double)edge.ErrorCount / edge.RequestCount * 100 : 0;

            graphEdges.Add(new ServiceEdge
            {
                Source = edge.SourceService, Target = edge.TargetService,
                RequestRate = edge.RequestCount / (endTime - startTime).TotalSeconds,
                ErrorRate = errorRate,
                AvgDurationMs = edge.AvgDurationMs,
                P99DurationMs = edge.P99DurationMs,
                Health = errorRate > 10 ? EdgeHealth.Critical
                       : errorRate > 5 ? EdgeHealth.Degraded
                       : EdgeHealth.Healthy
            });
        }

        var nodes = serviceNames.Select(name => new ServiceNode
        {
            Name = name,
            TotalRequestRate = graphEdges.Where(e => e.Source == name).Sum(e => e.RequestRate),
        }).ToList();

        return new ServiceDependencyGraph { Nodes = nodes, Edges = graphEdges };
    }
}

public class ServiceDependencyGraph
{
    public List<ServiceNode> Nodes { get; set; } = new();
    public List<ServiceEdge> Edges { get; set; } = new();
}
public class ServiceNode { public string Name { get; set; } = ""; public double TotalRequestRate { get; set; } }
public class ServiceEdge
{
    public string Source { get; set; } = ""; public string Target { get; set; } = "";
    public double RequestRate { get; set; } public double ErrorRate { get; set; }
    public double AvgDurationMs { get; set; } public double P99DurationMs { get; set; }
    public EdgeHealth Health { get; set; }
}
public enum EdgeHealth { Healthy, Warning, Degraded, Critical }

The service map visualization must balance information density with readability. For small service counts, a force-directed layout works well. For larger counts, progressive disclosure is essential—show cluster-level views first, then drill into individual services. Edge thickness encodes request rate, edge color encodes health, and node size encodes throughput. Interactive features include hovering for metrics, clicking to view traces, and filtering by time range. The visualization must update in real-time with a 5-second polling interval.

14. Sampling Strategies

Sampling is the process of deciding which traces to retain and which to discard. At scale—processing billions of traces per day—retaining all traces is neither cost-effective nor necessary. A well-designed sampling strategy retains the traces most valuable for debugging while discarding routine, low-value traces.

Head-Based Sampling

Head-based sampling makes the decision at the root span before any child spans are created. If the root span decision is do not sample, all child spans are also unsampled. This is the simplest strategy: a probabilistic sampler (e.g., sample 10% of traces) is configured in the SDK, and the decision propagates via trace context headers. Head-based sampling is extremely efficient because no coordination is required. The disadvantage is that it cannot make informed decisions based on the complete trace.

Tail-Based Sampling

Tail-based sampling makes the decision after the complete trace has been collected. The Collector holds all spans for a trace in memory, waits for completion, and then decides based on error status, latency, or attribute matching. This ensures 100% of error traces and 100% of slow traces are retained. The disadvantage is memory and complexity—the Collector must buffer complete traces, requiring significant memory for high-throughput systems.

Adaptive Sampling

Adaptive sampling adjusts the rate dynamically based on traffic volume. The goal is to maintain a consistent number of sampled traces per second regardless of traffic fluctuations. During traffic spikes, the rate decreases; during low traffic, it increases. This requires a feedback loop between the storage system and sampler. OpenTelemetry supports this via ParentBasedSampler with TraceIdRatioBasedSampler, with the ratio updated through Collector configuration management.

Hybrid Sampling Implementation in C#

C#
using System;
using System.Collections.Concurrent;

public class HybridSamplingStrategy
{
    private readonly ConcurrentDictionary<string, ServiceSamplingConfig> _configs = new();

    public void ConfigureService(string serviceName, ServiceSamplingConfig config)
    {
        _configs.AddOrUpdate(serviceName, config, (_, _) => config);
    }

    public SamplingResult ShouldSample(string serviceName, ActivityContext parentContext,
        string operationName)
    {
        var config = _configs.GetOrAdd(serviceName, GetDefaultConfig);

        if (config.AlwaysSampleOperations.Contains(operationName))
            return new SamplingResult { Decision = SamplingDecision.RecordAndSample, Reason = "Critical operation" };

        if (config.NeverSampleOperations.Contains(operationName))
            return new SamplingResult { Decision = SamplingDecision.Drop, Reason = "Non-essential operation" };

        if (parentContext.TraceFlags.HasFlag(ActivityTraceFlags.Recorded))
            return new SamplingResult { Decision = SamplingDecision.RecordAndSample, Reason = "Parent sampled" };

        var shouldSample = Random.Shared.NextDouble() < config.HeadSamplingRate;
        return new SamplingResult
        {
            Decision = shouldSample ? SamplingDecision.RecordAndSample : SamplingDecision.Drop,
            Reason = $"Head-sampled at {config.HeadSamplingRate:P0}"
        };
    }

    private static ServiceSamplingConfig GetDefaultConfig(string _) => new()
    {
        HeadSamplingRate = 0.1,
        AlwaysSampleOperations = new HashSet<string>
        {
            "POST /api/v1/payments", "POST /api/v1/orders", "POST /api/v1/auth/login"
        },
        NeverSampleOperations = new HashSet<string>
        {
            "GET /health", "GET /ready", "GET /metrics"
        }
    };
}

public class ServiceSamplingConfig
{
    public double HeadSamplingRate { get; set; } = 0.1;
    public HashSet<string> AlwaysSampleOperations { get; set; } = new();
    public HashSet<string> NeverSampleOperations { get; set; } = new();
}
public class SamplingResult { public SamplingDecision Decision { get; set; } public string Reason { get; set; } = ""; }
public enum SamplingDecision { Drop, RecordOnly, RecordAndSample }

Sampling Rate Selection

A common starting point is 10% for routine services and 100% for critical services (payments, authentication, order processing). Error traces should always be sampled. For low-traffic services (under 100 traces/second), 100% sampling is often affordable. For high-traffic services (over 10,000/second), even 1% produces 100 traces/second. The sampling configuration should be dynamic—adjustable without code deployment—to respond to debugging needs during incidents.

15. Multi-Tenant Observability Platform

Enterprise observability platforms must serve multiple teams, business units, or customers from a single deployment. Multi-tenancy provides cost efficiency through resource sharing and operational simplicity through centralized management. However, it introduces significant complexity in data isolation, resource allocation, and access control.

Tenant Isolation Model

Multi-tenant isolation spans four dimensions. Data isolation ensures Tenant A cannot query Tenant B data—enforced by adding tenant_id to every stored signal and filtering all queries. Resource isolation ensures one tenant ingestion volume does not degrade another query performance through per-tenant rate limits and quotas. Compute isolation ensures expensive queries from one tenant do not starve others. Network isolation ensures infrastructure is segmented per tenant when required for compliance.

Rate Limiting and Quotas

Each tenant has configurable limits on ingestion throughput, storage quota, and query concurrency. The rate limiter in the Collector uses a token bucket algorithm per tenant. When exceeded, excess spans are dropped with a clear metric. Storage quotas are enforced by the storage backend. Query concurrency is managed by the API layer, which limits concurrent queries per tenant.

Tenant Configuration Service in C#

C#
public class TenantConfigurationService
{
    private readonly ITenantRepository _tenantRepo;
    public TenantConfigurationService(ITenantRepository tenantRepo) { _tenantRepo = tenantRepo; }

    public async Task<TenantConfig> GetTenantConfigAsync(string tenantId, CancellationToken ct = default)
    {
        return await _tenantRepo.GetConfigAsync(tenantId, ct) ?? GetDefaultConfig(tenantId);
    }

    public async Task<CollectorConfigPatch> GetCollectorConfigForTenantAsync(
        string tenantId, CancellationToken ct = default)
    {
        var config = await GetTenantConfigAsync(tenantId, ct);
        return new CollectorConfigPatch
        {
            Processors = new Dictionary<string, object>
            {
                ["filter"] = new
                {
                    error_mode = "ignore",
                    traces = new { span = new[] { new { attributes = new[] {
                        new { key = "tenant.id", value = tenantId, op = "equals" } } } } }
                },
                ["attributes"] = new { actions = new[] {
                    new { key = "tenant.id", value = tenantId, action = "upsert" } } }
            },
            RateLimits = new RateLimitConfig
            {
                MaxSpansPerSecond = config.Limits.MaxSpansPerSecond,
                MaxMetricsPerSecond = config.Limits.MaxMetricsPerSecond,
                MaxLogsPerSecond = config.Limits.MaxLogsPerSecond
            },
            Retention = new RetentionConfig
            {
                TraceRetentionDays = config.Retention.TracesDays,
                MetricRetentionDays = config.Retention.MetricsDays,
                LogRetentionDays = config.Retention.LogsDays
            }
        };
    }

    private static TenantConfig GetDefaultConfig(string tenantId) => new()
    {
        TenantId = tenantId, Name = tenantId, Tier = TenantTier.Standard,
        Sampling = new SamplingConfig { HeadRate = 0.1 },
        Limits = new ResourceLimits
        {
            MaxSpansPerSecond = 10000, MaxMetricsPerSecond = 5000,
            MaxLogsPerSecond = 50000, MaxStorageGB = 500, MaxQueryConcurrency = 10
        },
        Retention = new RetentionConfig { TracesDays = 7, MetricsDays = 90, LogsDays = 14 }
    };
}

public class TenantConfig
{
    public string TenantId { get; set; } = "";
    public string Name { get; set; } = "";
    public TenantTier Tier { get; set; }
    public SamplingConfig Sampling { get; set; } = new();
    public ResourceLimits Limits { get; set; } = new();
    public RetentionConfig Retention { get; set; } = new();
}
public enum TenantTier { Free, Standard, Premium, Enterprise }
public class SamplingConfig { public double HeadRate { get; set; } }
public class ResourceLimits { public int MaxSpansPerSecond { get; set; } public int MaxMetricsPerSecond { get; set; } public int MaxLogsPerSecond { get; set; } public int MaxStorageGB { get; set; } public int MaxQueryConcurrency { get; set; } }
public class RetentionConfig { public int TracesDays { get; set; } public int MetricsDays { get; set; } public int LogsDays { get; set; } }
public class CollectorConfigPatch { public Dictionary<string, object> Processors { get; set; } = new(); public RateLimitConfig RateLimits { get; set; } = new(); public RetentionConfig Retention { get; set; } = new(); }
public class RateLimitConfig { public int MaxSpansPerSecond { get; set; } = 10000; public int MaxMetricsPerSecond { get; set; } = 5000; public int MaxLogsPerSecond { get; set; } = 50000; }

Tenant-aware dashboarding ensures each tenant sees only their own data. The dashboard API injects tenant filters into all queries. A global platform team view aggregates data across tenants for capacity planning. Pre-built dashboards for common use cases automatically adapt to each tenant instrumentation. Multi-tenancy transforms an observability tool into an observability platform where teams self-serve within their tenant boundaries.

16. Performance Overhead and Optimization

Observability is not free. Every span created, metric recorded, and log emitted consumes CPU, memory, network, and storage resources. If overhead is too high, teams will resist instrumentation. The goal is to keep observability overhead below 3% of the host application resource consumption while providing comprehensive coverage.

SDK Overhead

The OTel SDK overhead comes from four sources: span creation (memory allocation for the Activity object and its attributes), attribute recording (hash table operations for each SetTag call), context propagation (header injection and extraction on every request), and export (serialization and network I/O for sending spans to the Collector). The SDK mitigates these costs through pre-allocated object pools for span objects, lock-free concurrent collections for attribute storage, and asynchronous batched export that amortizes network overhead across many spans.

Network Overhead

Network overhead is the most visible cost. Each span averages 500 bytes serialized in OTLP protobuf format. A service generating 1,000 requests/second with 5 spans per request produces 2.5 MB/second of span data. Compression (gzip typically achieves 5:1 ratio) reduces this to 500 KB/second. Agent collectors perform compression before forwarding to gateways, significantly reducing network utilization. The batch processor further optimizes by coalescing many small OTLP requests into larger, fewer transmissions.

Storage Overhead

Storage costs dominate the total cost of ownership. ClickHouse with columnar compression typically achieves 10:1 compression for span data. A platform ingesting 1 billion spans per day at 500 bytes per span (500 GB uncompressed) requires approximately 50 GB of storage per day after compression. At $0.03/GB for SSD storage, this is $1.50/day or $45/month—remarkably affordable for the debugging value provided. Retention policies and sampling further reduce storage costs.

Optimization Strategies

Monitoring the Monitor

The observability platform itself must be monitored. Collector health metrics (queue size, export success rate, memory utilization) are exposed on the Collector metrics endpoint. A lightweight Prometheus instance scrapes these metrics and provides basic alerting. Storage health (ingestion lag, query latency, disk usage) is monitored separately. The meta-observability layer uses a different, simpler stack (e.g., Prometheus + Grafana) to avoid circular dependencies—you do not want your observability platform depending on itself for health monitoring.

Performance optimization is an ongoing process, not a one-time effort. As the system grows, new bottlenecks emerge. Regular performance reviews, load testing of the Collector pipeline, and capacity planning for storage growth are essential operational practices. The key metric is the observability tax—the percentage of application resources consumed by instrumentation. Keeping this below 3% requires continuous attention to SDK efficiency, sampling configuration, and storage tiering.

17. Integration with Cloud Provider Tooling

Most organizations operate in multi-cloud or hybrid environments and need their observability platform to integrate with cloud-native monitoring services. This section covers integration patterns with AWS, Azure, and GCP observability offerings.

AWS Integration

AWS provides X-Ray for distributed tracing, CloudWatch for metrics and logs, and CloudWatch Service Lens for unified observability. The OTel Collector can export to X-Ray via the awsxray exporter, to CloudWatch Metrics via the awscloudwatch exporter, and to CloudWatch Logs via the awscloudwatchlogs exporter. For organizations using AWS Distro for OpenTelemetry (ADOT), the pre-built collector image includes all AWS exporters pre-configured. The platform should support dual-export—sending telemetry to both the internal platform and AWS services—to maintain portability while leveraging cloud-native features like CloudWatch Alarms and X-Ray Service Map.

Azure Integration

Azure Monitor provides Application Insights for application monitoring, Log Analytics for log querying, and Prometheus metrics through Azure Monitor Managed Prometheus. The OTel Collector exports to Application Insights via the azuremonitor exporter and to Azure Monitor via the otlp exporter through the Azure Monitor OpenTelemetry ingestion endpoint. Azure Monitor also provides auto-instrumentation agents that complement OTel instrumentation. The integration pattern is similar to AWS: dual-export to both the internal platform and Azure-native services.

GCP Integration

Google Cloud provides Cloud Trace, Cloud Monitoring, and Cloud Logging. The OTel Collector exports to Cloud Trace via the googlecloudtrace exporter and to Cloud Monitoring via the googlecloudmonitoring exporter. Google Operations (formerly Stackdriver) provides a unified interface for all three signals. For GKE environments, the GKE collection option automatically collects metrics from the Kubernetes control plane and integrates with Cloud Monitoring.

Cloud Integration Architecture

Best Practices for Cloud Integration

Several best practices emerge. First, always maintain the internal platform as the primary observability system, with cloud-native tools as supplementary. This ensures portability and avoids vendor lock-in. Second, use OTLP as the universal transport format—every major cloud provider now supports OTLP ingestion natively, eliminating the need for proprietary exporters in many cases. Third, map semantic conventions to cloud-native concepts—OTel resource attributes like cloud.provider, cloud.region, and cloud.availability_zone should be populated automatically via resource detection. Fourth, test failover scenarios—if the cloud provider monitoring service goes down, telemetry should still flow to the internal platform. Fifth, leverage cloud-native features (CloudWatch Alarms, Azure Alerts, Cloud Monitoring Uptime Checks) for alerting while using the internal platform for deep-dive debugging.

Cloud integration is not just about exporting data to multiple destinations. It is about building a cohesive observability strategy that leverages the strengths of each platform while maintaining independence and portability. The OTel Collector dual-export pattern is the foundation of this strategy, enabling organizations to use cloud-native features without sacrificing their investment in a unified internal platform.

18. Interview Q&A

The following questions and answers cover the key concepts discussed in this article. These are typical of system design interviews for senior and staff-level engineering positions focused on observability, platform engineering, and distributed systems.

Q1: How would you design a distributed tracing system that handles 10 billion spans per day?

I would start with the three-layer architecture: agent collectors (DaemonSet per Kubernetes node) performing initial batching and compression, gateway collectors handling tail-based routing and sampling, and ClickHouse as the storage backend. At 10 billion spans/day, that is approximately 115,000 spans/second average. With ClickHouse clusters of 20 nodes, each handling 6,000 spans/second ingestion, this is well within capacity. I would implement 90% head-based sampling for routine traffic, reducing ingestion to 11,500 spans/second, with tail-based sampling retaining 100% of errors and slow traces. Storage at 10:1 compression would be approximately 500 GB/day, or 15 TB/month, well within a multi-node ClickHouse cluster.

Q2: What are the trade-offs between head-based and tail-based sampling?

Head-based sampling is simple and has zero memory overhead but cannot make informed decisions—it might drop an error trace or retain a routine one. Tail-based sampling can make optimal decisions (retain all errors, all slow traces) but requires buffering complete traces in memory, which is expensive at scale. At 100,000 active traces in flight, with each trace averaging 10 KB, the tail sampling buffer requires 1 GB of memory per Collector instance. The hybrid approach uses head-based sampling for initial reduction (10% of traffic) and tail-based sampling within that 10% for intelligent selection.

Q3: How do you ensure trace context propagation works correctly across services written in different languages?

Use W3C Trace Context as the propagation standard, which OpenTelemetry supports natively across all language SDKs. The traceparent header format is language-agnostic—a Go service calling a .NET service propagates context seamlessly. For message queues, inject context into message headers (AMQP, Kafka headers, SQS message attributes). The key is standardizing on a single propagation format across the organization and using the OTel CompositeTextMapPropagator for services that must interoperate with legacy B3 or Jaeger propagation.

Q4: How would you handle multi-tenancy in the observability platform?

Every telemetry signal carries a tenant_id resource attribute, added by the Collector based on the source namespace or API key. All storage queries include a tenant_id filter. The Collector enforces per-tenant rate limits via a token bucket processor. Storage quotas are enforced per tenant at the database level. The API layer authenticates requests and injects the tenant_id filter into all queries. Cross-tenant analytics for the platform team use a separate admin API with appropriate authorization.

Q5: How do you prevent the observability platform from becoming a single point of failure?

The platform itself must be designed with the same reliability requirements as the services it monitors. This means: redundant Collector instances (no single agent per node), replicated storage (ClickHouse with 3-way replication), load-balanced query APIs, and a separate meta-observability stack for monitoring the monitoring system. The Collector memory limiter processor ensures the Collector degrades gracefully under load (dropping data) rather than crashing. For critical workloads, dual-export to an external provider (Grafana Cloud, Datadog) provides failover when the internal platform is unavailable.

Q6: Explain how you would implement SLO-based alerting and why it is better than threshold-based alerting.

SLO-based alerting tracks error budget consumption rather than raw thresholds. Instead of alerting when error rate exceeds 5%, it alerts when the error budget is being consumed at a rate that will exhaust it before the SLO window ends. This automatically accounts for diurnal patterns (5% errors during low traffic is very different from 5% errors during peak traffic). Implementation uses multi-window burn rate: a 5-minute window detects sudden incidents, while a 1-hour window confirms sustained degradation. The burn rate thresholds (14.4x for critical, 6x for warning) correspond to specific time-to-exhaustion targets.

Q7: How do you handle high-cardinality metrics without overwhelming the metrics backend?

High cardinality occurs when metrics include unbounded label values (user IDs, request paths). The defense is three-fold: (1) the OTel cardinality limiter processor drops metrics that exceed a configured label value count threshold, (2) the platform provides guidelines for which labels are safe (bounded enum values like status code, region) versus unsafe (unbounded values like user ID, request path), and (3) the metrics backend (Prometheus/Mimir) is configured with active series limits that reject high-cardinality metrics before they consume resources. For analytics requiring high-cardinality data, use trace attributes or log fields instead of metrics.

Q8: Describe the end-to-end flow of a trace from instrumentation to visualization.

The flow begins when a request enters Service A. The OTel SDK creates a root span, injects trace context into outgoing HTTP headers (traceparent), and starts timing. Service B extracts the context, creates a child span, and continues the chain. When the request completes, each service ends its span. The SDK BatchSpanProcessor buffers spans and exports them via OTLP/gRPC to the agent Collector. The agent compresses, batches, and forwards to the gateway. The gateway applies tail-based sampling, enriches with Kubernetes metadata, and exports to ClickHouse. The trace query API fetches spans by TraceId, builds the span tree in memory, and returns it to the web dashboard, which renders a waterfall chart showing the full request flow with timing, status, and attributes for each span.

Q9: What is the role of OpenTelemetry Collector connectors, and when would you use them?

Connectors link two pipelines within a single Collector instance. The output of one pipeline becomes the input of another. For example, the spanmetrics connector generates RED metrics (rate, error, duration) directly from trace data, eliminating the need for a separate metrics pipeline. The servicegraph connector generates service dependency graph data from traces. Connectors are useful when you want to derive one signal type from another without external processing. They reduce deployment complexity by keeping the derivation within a single Collector binary.

Q10: How would you migrate an existing system from Jaeger to a custom observability platform?

The migration would follow a three-phase approach. Phase 1: Deploy the new platform alongside Jaeger, with both receiving telemetry via OTel Collector dual-export. This allows comparison of data quality and query performance without risk. Phase 2: Migrate dashboards and alerts to the new platform while keeping Jaeger as a fallback. Validate that all debugging workflows function correctly. Phase 3: Remove Jaeger export from the Collector configuration. Throughout the migration, instrumentation code remains unchanged because it targets the OTel API, not any specific backend. The Collector acts as the abstraction layer, making backend changes transparent to instrumented services.