Home › Categories › big-data › Apache Kylin Tutorial: OLAP on Hadoop for Big Da…
big-data6 min read

Apache Kylin Tutorial: OLAP on Hadoop for Big Data Analytics (2026)

Sandip Mhaske • Ayodhyya • big-data

Apache Kylin Tutorial: OLAP on Hadoop for Big Data Analytics (2026)

Published:  |  Category: Big Data  |  Reading time: ~15 min
Apache Kylin Tutorial: OLAP on Hadoop for Big Data Analytics (2026)

Apache Kylin provides sub-second OLAP queries on massive datasets by pre-computing and storing data cubes on Hadoop. After building Kylin cubes for interactive analytics on billions of rows, I appreciate how it transforms Hadoop into an OLAP engine that rivals traditional data warehouse performance.

This tutorial covers Kylin's cube model, build pipeline, query engine, REST API, and optimization strategies for delivering fast analytical queries on big data.

📖 Table of Contents
  1. Kylin Architecture and Cube Model
  2. Cube Build Pipeline
  3. Query Engine and OLAP Performance
  4. Model Design and Optimization
  5. REST API and Integration
  6. Production Deployment and Monitoring

Kylin Architecture and Cube Model

Kylin's core concept is the cube: a pre-computed multidimensional aggregation of a source table. Cubes are defined by dimensions (attributes you filter/group by) and measures (aggregations you compute). The build process reads source data from Hive, computes all dimension combinations, and stores results in a columnar format (Parquet or ORC).

The cube model supports hiearchical dimensions, derived dimensions, and row-level cube slices. Cubes are built incrementally — only new data is processed on each build, not the entire dataset.

# Kylin cube model definition
-- Hive source table
CREATE TABLE sales (
    order_id STRING,
    product_id STRING,
    category STRING,
    region STRING,
    amount DECIMAL(10,2),
    quantity INT,
    order_date STRING
);

-- Kylin cube model
-- Dimensions: product_id, category, region, order_date
-- Measures: sum(amount), count(*)
-- Rowkey: [region, category, product_id, order_date]

# Cube build process
# 1. Spark/Hive job reads source data
# 2. Computes all dimension combinations
# 3. Stores pre-aggregated results
# 4. Builds cube segments (time-based partitions)

Cube Build Pipeline

Cube builds run as Spark jobs that read Hive tables, compute aggregates for all dimension combinations, and write results to HDFS. The build is incremental — each build processes only data for a specific time range. Cube segments are the unit of incremental builds, stored as separate Parquet files.

Build jobs have four phases: data preparation (reading and converting), cube aggregation (computing all dimension combinations), cube saving (writing to HDFS), and HBase/ES update (building lookup indexes). The entire process is managed by Kylin's job engine.

# Cube build via REST API
# Build incrementally
curl -X PUT -u admin:ADMIN \
  -H 'Content-Type: application/json' \
  -d '{
    "sourceType": "BUILD",
    "buildRange": {
      "startTime": 1704067200000,
      "endTime": 1704153600000
    }
  }' \
  http://kylin:7070/kylin/api/cubes/sales_cube/build

# Merge segments (consolidate)
curl -X PUT -u admin:ADMIN \
  http://kylin:7070/kylin/api/cubes/sales_cube/merge

# Build status
curl -u admin:ADMIN \
  http://kylin:7070/kylin/api/cubes/sales_cube

# Schedule builds via cron or Airflow
# Build daily at 2 AM
# cron: 0 2 * * * curl -X PUT http://kylin:7070/kylin/api/cubes/sales_cube/build

Query Engine and OLAP Performance

Kylin's query engine translates SQL into cube lookups, computing aggregations directly from pre-computed cubes. For a query filtering on dimensions and requesting measures, Kylin reads the relevant cube segments and computes the result without scanning raw data. This provides sub-second response times on billions of rows.

The query optimizer selects the optimal cube segments to scan, pushes down filters to the cube layer, and uses bitmap indexes for dimension lookups. When a query cannot be fully answered by the cube, Kylin falls back to Hive (real-time OLAP).

-- Kylin SQL (OLAP query)
SELECT 
    region,
    category,
    sum(amount) AS total_revenue,
    count(*) AS order_count
FROM sales_cube
WHERE order_date >= '2026-01-01' 
  AND order_date < '2026-02-01'
  AND region IN ('us-east', 'us-west')
GROUP BY region, category
ORDER BY total_revenue DESC;

-- Query execution plan
# 1. Parse SQL, resolve cube
# 2. Select cube segments matching date filter
# 3. Apply bitmap filter on region dimension
# 4. Compute sum(amount) and count(*) from cube
# 5. Return result (<1 second for billions of rows)

-- Kylin query via REST API
curl -X POST -u admin:ADMIN \
  -H 'Content-Type: application/json' \
  -d '{"sql": "SELECT region, sum(amount) FROM sales_cube GROUP BY region"}' \
  http://kylin:7070/kylin/api/query

Model Design and Optimization

Cube model design determines query performance. Choose dimensions that match your most common query patterns. Avoid high-cardinality dimensions (like user_id) in cubes — use rowkey encoding to reduce cube size. Use aggregation groups to pre-compute only the dimension combinations you actually query.

Rowkey encoding compresses dimension values into byte arrays. Use dictionary encoding for low-cardinality dimensions (region, category) and fixed-length encoding for high-cardinality ones. This dramatically reduces cube size while maintaining lookup speed.

# Cube optimization strategies

# 1. Aggregation groups — reduce cube size
# Instead of 2^n full cube, define aggregation groups:
Aggregation Groups: [
  {dimensions: [region, category, order_date]},  -- 3 dimensions
  {dimensions: [product_id, order_date]},          -- 2 dimensions
  {dimensions: [region, category]}                 -- 2 dimensions
]
# Total: 3+2+2=7 groups instead of 2^4=16 full cube

# 2. Rowkey design
rowkey: {
  region: dictionary_encoding,
  category: dictionary_encoding,
  product_id: fixed_length_8,
  order_date: date_to_long
}

# 3. Cube size estimation
# Low-cardinality cube (3 dimensions): ~100MB per day
# Full cube (4 dimensions, 1M unique): ~10GB per day
# Use Aggregation Groups to keep size manageable

REST API and Integration

Kylin provides a comprehensive REST API for cube management, query execution, and monitoring. The API supports CRUD operations on models, cubes, and build jobs. Integrate Kylin with BI tools (Tableau, Power BI, Looker) via JDBC/ODBC drivers for interactive dashboards.

For programmatic access, use the Kylin JDBC driver to connect standard SQL clients. The driver communicates with Kylin's query API, enabling integration with any JDBC-compatible tool.

# Kylin REST API examples

# List all cubes
curl -u admin:ADMIN \
  http://kylin:7070/kylin/api/cubes

# Get cube details
curl -u admin:ADMIN \
  http://kylin:7070/kylin/api/cubes/sales_cube

# Execute query
curl -X POST -u admin:ADMIN \
  -H 'Content-Type: application/json' \
  -d '{"sql": "SELECT region, sum(amount) FROM sales_cube GROUP BY region"}' \
  http://kylin:7070/kylin/api/query

# Enable JDBC driver
# JDBC URL: jdbc:kylin://kylin-host:7070/project_name
# Driver: org.apache.kylin.jdbc.Driver

# Tableau connection:
# 1. Install Kylin ODBC driver
# 2. Create DSN pointing to Kylin
# 3. Connect Tableau to DSN
# 4. Use Kylin as data source

Production Deployment and Monitoring

Kylin requires Hadoop, Hive, HBase (or Elasticsearch), and Spark. Deploy the Kylin server as a standalone JVM or on YARN. For production, use multiple Kylin nodes behind a load balancer with HBase for metadata storage and Elasticsearch for query acceleration.

Monitor cube build times, query performance, and storage usage. Set alerts for failed builds, long-running queries, and low HDFS capacity. Kylin emits metrics via JMX for integration with Prometheus and Grafana.

# Kylin configuration
# kylin.properties
kylin.metadata.url=kylin_metadata@hbase:kylin
kylin.storage.url=hdfs://namenode:8020/kylin
kylin.job.log.dir=hdfs://namenode:8020/kylin/logs
kylin.query.max_sql_len=10000
kylin.query.max_return_rows=10000

# Spark config for cube builds
spark.master=yarn
spark.executor.memory=8g
spark.executor.cores=4
spark.dynamicAllocation.enabled=true

# Monitoring
curl -u admin:ADMIN http://kylin:7070/kylin/api/system/status

# Key metrics
# - cube_build_time: seconds per build
# - query_response_time: average query latency
# - cube_size_gb: total storage per cube
# - cube_hit_rate: cache hit ratio for queries

Frequently Asked Questions

How does Kylin differ from Apache Druid?

Kylin pre-computes cubes at build time (offline). Druid ingests data in real-time and stores raw data for flexible querying. Kylin excels at pre-defined analytical queries; Druid excels at ad-hoc queries on streaming data.

What is the maximum cube size Kylin can handle?

Kylin handles cubes with trillions of rows. Cube size depends on dimension cardinality and aggregation groups. A well-designed cube compresses data significantly — 100:1 compression ratios are common.

Can Kylin query data in real-time?

Kylin supports hybrid queries: pre-computed cubes for historical data, and Hive fallback for real-time data. For true real-time OLAP, use Druid or ClickHouse.

How do I know if my cube design is optimal?

Check cube hit rate (should be >95%), query response time, and cube size. If queries hit Hive fallback frequently, adjust your cube dimensions and aggregation groups to match actual query patterns.

Originally published on Ayodhyyya. Last updated June 1, 2026.