Home › Categories › big-data › Apache Druid Tutorial: Real-Time Analytics Datab…
big-data6 min read

Apache Druid Tutorial: Real-Time Analytics Database (2026)

Sandip Mhaske • Ayodhyya • big-data

Apache Druid Tutorial: Real-Time Analytics Database (2026)

Published:  |  Category: Big Data  |  Reading time: ~15 min
Apache Druid Tutorial: Real-Time Analytics Database (2026)

Apache Druid is a real-time analytics database designed for fast slice-and-dice queries on event-oriented data. After running Druid clusters serving dashboards with sub-second response times on billions of events, I appreciate how it combines the flexibility of a columnar store with the speed of a search index for operational analytics.

This tutorial covers Druid's architecture, ingestion methods (real-time and batch), query types, indexing strategies, and production deployment for high-performance time-series analytics.

📖 Table of Contents
  1. Druid Architecture and Data Model
  2. Real-Time and Batch Ingestion
  3. Query Types and SQL Interface
  4. Segment Design and Optimization
  5. Deep Storage and Retention
  6. Production Deployment and Monitoring

Druid Architecture and Data Model

Druid consists of four node types: Historical nodes store and serve immutable data segments. MiddleManager nodes ingest real-time data and create segments. Broker nodes handle query routing and result merging. Coordinator nodes manage segment placement and replication. The metadata store (MySQL or PostgreSQL) tracks segment information.

Data is organized into datasources (similar to tables). Each datasource is partitioned into segments — immutable, compressed columnar files containing a time slice of data. Segments are the unit of distribution, replication, and querying.

# Druid data model
# Datasource: events
# Dimensions: user_id, event_type, region, page
# Measures: count, sum_amount, unique_users (hyperloglog)

# Ingestion spec — streaming from Kafka
{
  "dataSchema": {
    "dataSource": "events",
    "timestampSpec": {"column": "timestamp", "format": "iso"},
    "dimensionsSpec": {
      "dimensions": [
        {"name": "user_id", "type": "string"},
        {"name": "event_type", "type": "string"},
        {"name": "region", "type": "string"},
        {"name": "page", "type": "string"}
      ]
    },
    "metricsSpec": [
      {"type": "count", "name": "count"},
      {"type": "doubleSum", "name": "sum_amount", "fieldName": "amount"},
      {"type": "hyperUnique", "name": "unique_users", "fieldName": "user_id"}
    ]
  },
  "ioConfig": {
    "type": "kafka",
    "consumerProperties": {"bootstrap.servers": "kafka:9092"},
    "topic": "events",
    "inputFormat": {"type": "json"}
  },
  "tuningConfig": {
    "type": "kafka",
    "maxRowsPerSegment": 5000000
  }
}

Real-Time and Batch Ingestion

Druid ingests data via Kafka (real-time streaming) or from files on deep storage (batch). The Kafka indexing service continuously consumes events, creates segments when they reach a size threshold, and publishes them to deep storage. Batch ingestion reads Parquet/CSV/JSON files, creates segments, and publishes them atomically.

Real-time ingestion provides sub-minute latency from event to queryable. Batch ingestion is used for backfill, historical data loading, and periodic re-indexing. Both methods produce identical segment formats.

# Kafka streaming ingestion
POST /druid/indexer/v1/supervisor
{
  "type": "kafka",
  "spec": {
    "ioConfig": {
      "type": "kafka",
      "consumerProperties": {"bootstrap.servers": "kafka:9092"},
      "topic": "events",
      "taskCount": 3,
      "replicas": 2,
      "taskDuration": "PT1H"
    },
    "dataSchema": {
      "dataSource": "events",
      "timestampSpec": {"column": "timestamp"},
      "granularitySpec": {"queryGranularity": "minute", "segmentGranularity": "hour"}
    }
  }
}

# Batch ingestion from files
POST /druid/indexer/v1/supervisor
{
  "type": "native",
  "spec": {
    "dataSchema": {"dataSource": "events"},
    "ioConfig": {
      "type": "static",
      "inputSource": {"type": "local", "baseDir": "/data/events"},
      "inputFormat": {"type": "json"}
    }
  }
}

Query Types and SQL Interface

Druid supports multiple query types: Timeseries (aggregations over time), TopN (top N values by metric), GroupBy (dimension grouping with aggregations), and Scan (full scan for debugging). The SQL interface translates SQL to native Druid queries using Apache Calcite.

Druid's query engine leverages columnar storage for fast aggregations, bitmap indexes for fast filtering, and approximate algorithms (HyperLogLog, quantiles sketches) for cardinality and percentile estimates.

-- Druid SQL (via JDBC/REST)
SELECT 
    TIME_FLOOR(timestamp, 'P1D') AS day,
    region,
    COUNT(*) AS events,
    SUM(amount) AS total_amount,
    APPROX_COUNT_DISTINCT(user_id) AS unique_users
FROM events
WHERE timestamp >= CURRENT_TIMESTAMP - INTERVAL '7' DAY
  AND event_type = 'purchase'
GROUP BY 1, 2
ORDER BY total_amount DESC
LIMIT 20;

-- Native Timeseries query
{
  "queryType": "timeseries",
  "dataSource": "events",
  "granularity": "day",
  "aggregations": [
    {"type": "count", "name": "count"},
    {"type": "doubleSum", "name": "total", "fieldName": "amount"}
  ],
  "intervals": ["2026-01-01/2026-01-08"],
  "filter": {"type": "selector", "dimension": "event_type", "value": "purchase"}
}

Segment Design and Optimization

Segment granularity determines how data is partitioned by time. Hourly segments are common for high-volume data; daily segments for lower volume. The trade-off: smaller segments enable faster time-range queries but increase metadata overhead and segment management complexity.

Column design determines query performance. String dimensions use dictionary encoding with bitmap indexes. Numeric dimensions use front-coded compression. Measures use appropriate aggregation types (count, sum, hyperUnique, thetaSketch).

# Segment design considerations

# Granularity: hourly for >1M events/hour
granularitySpec: {
  "queryGranularity": "minute",
  "segmentGranularity": "hour"
}

# For lower volume data
granularitySpec: {
  "queryGranularity": "hour",
  "segmentGranularity": "day"
}

# Column tuning
"dimensionsSpec": {
  "dimensions": [
    {"name": "user_id", "type": "string"},
    {"name": "amount", "type": "double"},
    {"name": "region", "type": "string", "createBitmapIndex": true}
  ]
}

# HyperLogLog for unique counts
"metricsSpec": [
  {"type": "hyperUnique", "name": "unique_users", "fieldName": "user_id"}
]

# Theta sketches for set operations
"metricsSpec": [
  {"type": "thetaSketch", "name": "user_set", "fieldName": "user_id"}
]

Deep Storage and Retention

Druid stores segments on deep storage (S3, HDFS, local disk). Historical nodes cache segments locally for fast serving. Retention rules define how long data is kept — per datasource or globally. Druid automatically drops old segments and compacts small segments into larger ones.

For cost optimization, use tiered storage: hot tier (SSD, fast serving for recent data), warm tier (HDD, serving for older data), and cold tier (S3, for archival). Retention rules can be per-tier and per-datasource.

# Deep storage configuration
# druid.properties
druid.storage.type=s3
druid.s3.accessKey=AKIA...
druid.s3.secretKey=...
druid.s3.bucket=druid-segments
druid.s3.baseKey=segments/

# Retention rules
POST /druid/coordinator/v1/rules/events
[
  {
    "type": "loadByPeriod",
    "tieredReplicants": {
      "hot": {"numReplicants": 2},
      "warm": {"numReplicants": 1}
    },
    "period": "P90D"
  },
  {
    "type": "dropForever"
  }
]

# Compaction (merge small segments)
POST /druid/coordinator/v1/compact
{
  "dataSource": "events",
  "interval": "2026-01-01/2026-01-08",
  "targetCompactedSizeBytes": 419430400
}

Production Deployment and Monitoring

Deploy Druid on Kubernetes with the Druid Operator for production. Each node type has different resource requirements: Brokers need memory for query processing, Historicals need disk and memory for segment caching, MiddleManagers need CPU for ingestion, and Coordinators need database access for metadata management.

Monitor query latency (p50, p95, p99), segment count, ingestion lag, and deep storage usage. Set alerts for ingestion delays exceeding 5 minutes and query latency exceeding 1 second for dashboard queries.

# Druid Kubernetes deployment
apiVersion: druid.apache.org/v1alpha1
kind: Druid
metadata:
  name: druid-cluster
spec:
  image: apache/druid:27.0.0
  coordinators:
    instances: 2
    resources:
      memory: 4Gi
      cpu: 2
  historicals:
    instances: 3
    resources:
      memory: 8Gi
      cpu: 4
    volumeClaimTemplate:
      storageClassName: fast-ssd
      resources:
        storage: 500Gi
  brokers:
    instances: 2
    resources:
      memory: 8Gi
      cpu: 4
  middleManagers:
    instances: 3
    resources:
      memory: 8Gi
      cpu: 4

# Monitoring endpoints
GET /status          # Cluster health
GET /druid/coordinator/v1/loadstatus  # Segment distribution
GET /druid/broker/v1/readiness    # Broker readiness

Frequently Asked Questions

What is the difference between Druid and ClickHouse?

Druid excels at time-series analytics with sub-second query latency. ClickHouse is a more general-purpose columnar database with richer SQL support. Druid is better for operational dashboards; ClickHouse for analytical queries on arbitrary schemas.

How much storage does Druid need?

Druid compresses data 5-10x. 1 TB of raw event data typically occupies 100-200 GB in Druid. Storage depends on dimension cardinality and metric types. HyperLogLog metrics add ~8 bytes per unique value.

Can Druid handle joins?

Druid supports lookup joins (pre-loaded dimension tables) and inline joins (limited). For complex joins, pre-join data during ingestion or use a separate OLAP engine for queries requiring extensive joins.

What is the ingestion latency?

Kafka ingestion: 1-5 seconds from event to queryable. Batch ingestion: depends on data volume and segment size. Typical batch loads of 1M rows complete in under 1 minute.

Originally published on Ayodhyyya. Last updated June 1, 2026.