1. Introduction

A Distributed Stream Processing System is the backbone of modern data infrastructure. It enables organizations to publish, subscribe to, store, and process streams of records in real-time at massive scale. Apache Kafka, the de facto standard in this space, processes over 7 trillion messages per day at LinkedIn alone.

Consider the scale of Uber's event streaming platform: every ride request, GPS update, payment transaction, and driver location change generates an event. These events flow through a stream processing system that powers real-time pricing, ETA calculations, fraud detection, and driver dispatch. Without a robust stream processing system, these real-time features would be impossible.

The Problem

Traditional message queues (RabbitMQ, ActiveMQ) were designed for task queuing and RPC-style communication. They excel at decoupling producers and consumers but struggle with:

• Throughput: Processing millions of messages per second across distributed systems.
• Replay: Re-reading historical messages for debugging or reprocessing.
• Retention: Storing days or weeks of message history for batch processing.
• Ordering: Maintaining message order across partitions while scaling horizontally.
• Exactly-Once: Ensuring each message is processed exactly once despite failures.

Business Motivation

• Real-Time Analytics: Process clickstream data, calculate metrics in real-time.
• Event Sourcing: Maintain complete audit trail of all state changes.
• Microservices Communication: Decouple services with asynchronous event-driven patterns.
• Data Integration: Connect heterogeneous systems through a common data backbone.
• Machine Learning Pipelines: Feed real-time data to ML models for inference.

Real-World Examples

• LinkedIn: Kafka was invented here. Handles 7 trillion messages/day, 4 petabytes of data daily.
• Netflix: Uses Kafka for real-time monitoring, A/B testing, and content delivery.
• Uber: M3 (metrics) and Kafka process trillions of events for ride matching and pricing.
• Airbnb: Kafka powers real-time search indexing, pricing, and fraud detection.
• Goldman Sachs: Uses Kafka for trade execution, risk management, and compliance.
• Twitter: Kafka processes billions of events for timeline generation and trending topics.

Evolution of Stream Processing

1. Message Queues (1990s): IBM MQ, TIBCO. Point-to-point, low throughput.
2. Enterprise Service Bus (2000s): Centralized routing, transformation, orchestration.
3. Log-Based Messaging (2010s): Kafka introduced distributed commit log. Decoupled storage from processing.
4. Stream Processing Frameworks (2015+): Apache Flink, Spark Streaming, Kafka Streams. Stateful processing at scale.
5. Cloud-Native Streaming (2020+): Confluent Cloud, AWS Kinesis, Azure Event Hubs. Managed services.
6. Real-Time Data Platforms (2023+): Apache Pulsar, Redpanda. Unified batch and stream processing.

2. Real Interview Context

This is one of the most commonly asked system design questions because it tests fundamental distributed systems concepts while being deeply practical.

Companies That Ask This Question

Skills Being Evaluated

• Storage Engine Design: Append-only log, segment files, indexing.
• Distributed Coordination: Partitioning, replication, leader election.
• Consistency Models: At-least-once, exactly-once, ordered delivery.
• Consumer Group Protocol: Partition assignment, rebalancing, offset management.
• High Availability: Replication, failover, data durability.
• Performance Engineering: Zero-copy, batching, compression, page cache.

Common Mistakes

1. Confusing with Message Queue: Designing a task queue instead of a log-based streaming system.
2. Ignoring Ordering: Not discussing how ordering is maintained across partitions.
3. Skip Consumer Groups: Not explaining how consumers coordinate and rebalance.
4. No Durability: Not discussing how data survives node failures.
5. Forget Offset Management: Not explaining how consumers track their position in the log.

3. Functional Requirements

Core Messaging (FR-01 to FR-15)

1. FR-01: Support creating and deleting topics (logical categories of messages).
2. FR-02: Support partitioning of topics across multiple brokers for horizontal scaling.
3. FR-03: Producers can publish messages to any partition within a topic.
4. FR-04: Messages within a partition are strictly ordered by offset.
5. FR-05: Messages are assigned monotonically increasing offsets within each partition.
6. FR-06: Producers can specify a partition key for deterministic partition assignment.
7. FR-07: Support batch publishing of multiple messages in a single request.
8. FR-08: Messages are persisted to disk before acknowledgment (durability guarantee).
9. FR-09: Support configurable replication factor per topic (1, 2, 3, or more).
10. FR-10: Support configurable retention period per topic (time-based and size-based).
11. FR-11: Support compacted topics (retain latest value per key for changelog semantics).
12. FR-12: Support topic-level configuration for compression, cleanup policy, and segment size.
13. FR-13: Support message headers/metadata in addition to key and value.
14. FR-14: Support multiple serialization formats (JSON, Avro, Protobuf, raw bytes).
15. FR-15: Support schema registry for schema evolution and validation.

Consumer Operations (FR-16 to FR-25)

16. FR-16: Support consumer groups where multiple consumers share partition consumption.
17. FR-17: Within a consumer group, each partition is consumed by exactly one consumer.
18. FR-18: Support consumer group rebalancing when consumers join or leave.
19. FR-19: Consumers can read from any offset (beginning, end, specific offset, timestamp).
20. FR-20: Support manual offset commit for at-least-once processing.
21. FR-21: Support automatic offset commit for simpler at-most-once processing.
22. FR-22: Support consumer lag monitoring (distance between latest offset and consumer offset).
23. FR-23: Support consumer pause/resume without leaving the group.
24. FR-24: Support read replicas for consumers that need isolated read throughput.
25. FR-25: Support dead letter queues for messages that fail processing after retries.

Cluster Operations (FR-26 to FR-35)

26. FR-26: Support adding and removing brokers without downtime.
27. FR-27: Support partition reassignment for rebalancing across brokers.
28. FR-28: Support leader election for partition leadership changes.
29. FR-29: Support controlled shutdown of brokers with leader migration.
30. FR-30: Support topic creation with specified partition count and replication factor.
31. FR-31: Support dynamic configuration changes without restart.
32. FR-32: Support cluster metadata management (topic configs, broker configs, ACLs).
33. FR-33: Support multiple data centers with cross-DC replication.
34. FR-34: Support tiered storage (hot data on local disk, cold on object storage).
35. FR-35: Support transactional writes across multiple partitions (atomic multi-partition publish).

4. Non-Functional Requirements

5. Requirement Prioritization

Must Have

• Topic creation with configurable partitions and replication
• Message publishing with partition key routing
• Consumer groups with partition assignment
• Persistent, ordered, replayable commit log
• Consumer offset tracking and management
• Replication for durability and availability
• Leader election for partition failover
• Configurable retention policies

Should Have

• Transactional writes across partitions
• Exactly-once semantics (producer + consumer)
• Schema registry integration
• Consumer lag monitoring
• Dynamic topic configuration
• Compacted topics for changelog semantics
• Dead letter queue support

Nice to Have

• Tiered storage (hot/cold)
• Cross-datacenter replication
• Kafka Streams API for stream processing
• Connect API for source/sink connectors
• Multi-tenancy with resource isolation

Out of Scope

• SQL query engine over streams (use ksqlDB separately)
• Machine learning model serving
• Full-text search over messages
• Complex event processing (CEP)

6. Capacity Estimation

Assumptions

Storage Calculations

Broker Sizing

Infrastructure Summary