1. Introduction

In modern distributed systems, observability is not a luxury—it is a fundamental requirement. Every microservice, every container, every network call, and every user interaction generates signals that, when properly collected and analyzed, reveal the health, performance, and behavior of the entire system. A Distributed Metrics Logging and Aggregation System is the backbone of this observability stack.

Consider a company like Netflix running thousands of microservices across multiple AWS regions. When a latency spike occurs in their video streaming pipeline, engineers need to identify which service is causing the bottleneck within seconds, not minutes. Without a robust metrics infrastructure, debugging becomes a needle-in-a-haystack exercise that can cost millions in lost revenue and degraded user experience.

The Problem

At scale, metrics present unique engineering challenges:

• Volume: A single service instance can emit hundreds of metrics per second. Multiply by thousands of instances across hundreds of services, and you are looking at billions of data points per day.
• Velocity: Metrics must be ingested in real-time. A 30-second delay in detecting a CPU spike can mean the difference between proactive scaling and reactive firefighting.
• Variety: Metrics come in different shapes—counters, gauges, histograms, summaries, timers, and custom types. Each has different aggregation semantics.
• Distributed Nature: Data originates from hundreds or thousands of machines, across multiple data centers and cloud regions. Centralizing this data without losing fidelity or overwhelming the network is a serious distributed systems problem.

Business Motivation

• Incident Detection: Automated alerting based on metrics reduces mean time to detection (MTTD) from hours to seconds.
• Capacity Planning: Historical metrics enable predictive scaling, reducing both over-provisioning costs and under-provisioning risks.
• Performance Optimization: P50, P95, P99 latency metrics reveal user-perceived performance issues that averages hide.
• SLA Compliance: Metrics provide the evidence needed to demonstrate SLA adherence to customers and regulators.
• Cost Attribution: Per-team or per-service metrics enable chargeback models that drive engineering efficiency.

Real-World Examples

• Google: Borgmon and later Monarch form the backbone of Google's internal monitoring. Monarch processes over 1 billion samples per second.
• Netflix: Atlas, their open-source time-series database, ingests over 1.5 billion measurements per minute.
• Uber: M3, their distributed metrics platform, processes trillions of data points daily.
• LinkedIn: InGraphs and later their migration to Kafka-based pipelines handle billions of metrics per day.
• Amazon: CloudWatch processes trillions of metrics from millions of AWS resources.
• Datadog: Processes over 1 trillion data points per day from their SaaS monitoring platform.

Evolution of Metrics Architecture

1. Monolithic Era (2000s): Centralized collection servers like Nagios. Pull-based, single points of failure.
2. Scalable Time-Series (2010s): Graphite, InfluxDB, OpenTSDB. Better write throughput, but still centralized storage.
3. Agent-Based (2015+): StatsD, Collectd, Telegraf. Local aggregation reduced network overhead.
4. Cloud-Native (2018+): Prometheus, Thanos, Cortex. Pull-based with service discovery. Kubernetes-native.
5. Observability Pipelines (2020+): OpenTelemetry, Fluent Bit, Vector. Vendor-neutral collection with streaming processing.
6. eBPF-Based (2022+): Cilium, Pixie. Kernel-level metrics without code instrumentation.

2. Real Interview Context

This system design question appears frequently at top-tier technology companies because it tests a wide range of distributed systems concepts in a single problem.

Companies That Ask This Question

Why Interviewers Choose This Problem

• Data Modeling: Can the candidate design appropriate schemas for high-write, time-series workloads?
• Scale Handling: Does the candidate understand the challenges of ingesting billions of data points per day?
• Storage Trade-offs: Time-series databases have unique compression, retention, and query patterns.
• Stream Processing: Real-time aggregation requires understanding of windowing, tumbling windows, and state management.
• Distributed Systems Fundamentals: Consistency vs. availability, fault tolerance, and graceful degradation.
• Production Readiness: Monitoring the monitoring system (meta-observability).

Skills Being Evaluated

Common Mistakes Candidates Make

1. Skip Requirements: Jumping straight into architecture without clarifying functional and non-functional requirements.
2. Ignore Scale: Designing for thousands of metrics instead of billions. Not considering write amplification.
3. Over-Engineer Storage: Using a general-purpose database like PostgreSQL for time-series data without considering compression and partitioning.
4. Forget Retention: Not discussing data retention policies, compaction, or tiered storage.
5. Ignore Hot Keys: Popular metric names (like CPU usage for a popular service) can create hot partitions.
6. No Meta-Observability: Not discussing how to monitor the monitoring system itself.
7. Skip Aggregation: Not pre-aggregating data at different granularities (1-minute, 5-minute, 1-hour).

What Distinguishes Senior from Junior Answers

3. Functional Requirements

Before designing any system, we need to clearly define what it must do. The following functional requirements capture the essential capabilities of a distributed metrics logging and aggregation system.

Metrics Collection (FR-01 to FR-10)

1. FR-01: Support multiple metric types: counters (monotonically increasing), gauges (can increase or decrease), histograms (distribution of values), summaries (precomputed quantiles), and timers.
2. FR-02: Accept metrics via both push (agents send to collection endpoints) and pull (system scrapes endpoints) models.
3. FR-03: Support multiple transport protocols: HTTP/HTTPS, gRPC, UDP (for StatsD-compatible agents), and Kafka for high-volume ingestion.
4. FR-04: Support metric tagging/labeling with key-value pairs for dimensional querying (e.g., http_requests_total{method="GET", status="200", service="api-gateway"}).
5. FR-05: Support metric names up to 256 characters and hierarchical naming conventions.
6. FR-06: Support batch ingestion of metrics, accepting up to 10,000 metric samples per request.
7. FR-07: Support metric metadata (description, unit, type) registration.
8. FR-08: Handle out-of-order metrics arrival with configurable grace periods.
9. FR-09: Support monotonically increasing counters with automatic delta computation.
10. FR-10: Support metric value validation (non-negative for counters, valid ranges for gauges).

Metrics Storage (FR-11 to FR-15)

11. FR-11: Store raw metric data for a configurable retention period (default: 30 days high-resolution, 1 year downsampled).
12. FR-12: Support automatic downsampling/aggregation at configurable intervals (1-minute to 1-hour rollups).
13. FR-13: Support tiered storage: hot (last 24h, SSD), warm (7 days, HDD), cold (30+ days, S3).
14. FR-14: Support metric deletion by name pattern, label matchers, or time range.
15. FR-15: Support data export in standard formats (Prometheus, OpenMetrics, CSV, Parquet).

Metrics Querying (FR-16 to FR-23)

16. FR-16: Support time-range queries with arbitrary start and end timestamps.
17. FR-17: Support aggregation functions: sum, avg, min, max, count, rate, delta, percentiles (P50, P95, P99), and standard deviation.
18. FR-18: Support dimensional filtering with label selectors (equals, not-equals, regex, set membership).
19. FR-19: Support grouping by label dimensions.
20. FR-20: Support mathematical operations on metrics: arithmetic (+, -, *, /), derivative, and integral.
21. FR-21: Support metric comparison across time windows (this week vs. last week).
22. FR-22: Return query results within 2 seconds for 99% of queries.
23. FR-23: Support a query language for complex expressions (similar to PromQL or M3QL).

Alerting (FR-24 to FR-30)

24. FR-24: Support rule-based alerting with configurable thresholds, durations, and severities.
25. FR-25: Support alert notification via email, Slack, PagerDuty, webhooks, and SMS.
26. FR-26: Support alert aggregation and deduplication to prevent alert storms.
27. FR-27: Support alert silencing during maintenance windows.
28. FR-28: Support alert escalation policies.
29. FR-29: Provide alert history and audit trail.
30. FR-30: Support composite alerts (alert when multiple conditions are true simultaneously).

Dashboarding (FR-31 to FR-35)

31. FR-31: Web-based dashboard builder with drag-and-drop capabilities.
32. FR-32: Multiple chart types: line, area, bar, heatmap, gauge, table, text annotations.
33. FR-33: Dashboard templating with dynamic variables.
34. FR-34: Dashboard sharing via URL with configurable access permissions.
35. FR-35: Real-time dashboard updates with configurable refresh intervals.

Multi-Tenancy and Access Control (FR-36 to FR-40)

36. FR-36: Multiple tenants with isolated metric namespaces.
37. FR-37: Role-based access control (RBAC) with roles: admin, editor, viewer.
38. FR-38: API key authentication for programmatic access.
39. FR-39: SSO integration via SAML/OIDC.
40. FR-40: Audit logging for all administrative actions.

4. Non-Functional Requirements

5. Requirement Prioritization

Must Have

• Metric ingestion via HTTP and gRPC
• Support for counters, gauges, histograms, and timers
• Time-range queries with aggregation functions
• Label/dimension-based filtering and grouping
• Automatic downsampling at configurable intervals
• Data retention with tiered storage
• Rule-based alerting with notification channels
• Multi-tenant isolation
• 99.99% availability for ingestion pipeline
• At-least-once delivery guarantee

Should Have

• Web-based dashboard builder
• Alert aggregation and deduplication
• PromQL-compatible query language
• Dashboard templating with variables
• API key and SSO authentication
• Rolling deployments without downtime
• Self-monitoring (meta-observability)
• Data export in standard formats

Nice to Have

• eBPF-based auto-discovery of metrics
• Machine learning-based anomaly detection
• Natural language querying
• Cost attribution dashboards
• Custom retention policies per tenant
• OpenTelemetry native support

Out of Scope

• Distributed tracing (separate system)
• Log aggregation (separate system)
• Real-time streaming analytics (use dedicated stream processor)
• Custom plugin ecosystem
• Mobile app for dashboard viewing