Apache Atlas Tutorial: Data Governance and Metadata Management (2026)
Apache Atlas provides data governance and metadata management for the Hadoop ecosystem, enabling organizations to catalog, classify, and track data lineage across their data platforms. After implementing Atlas for data governance in enterprises with strict compliance requirements, I appreciate how it creates a searchable, auditable view of all data assets and their relationships.
This tutorial covers Atlas's type system, metadata model, lineage tracking, classification framework, hooks integration, and REST API for building data governance solutions.
Atlas Architecture and Type System
Atlas consists of a core metadata repository, a type system, and hook integrations. The type system defines entity types (datasets, processes, users) and their attributes. The repository stores entity instances and their relationships. Hooks automatically capture metadata from Hive, Kafka, Sqoop, and other ecosystem components.
The type system is extensible — you define custom types that model your organization's data landscape. Built-in types cover common concepts: Hive tables, HDFS paths, Kafka topics, and processing workflows. Custom types extend these for domain-specific metadata.
# Atlas type definition (TypeSystem)
# Custom type for a data product
curl -X POST -u admin:admin -H 'Content-Type: application/json' \
-d '{
"enumTypes": [],
"structTypes": [],
"classTypes": [{
"name": "DataProduct",
"typeVersion": "1.0",
"superTypes": ["DataSet"],
"attributeDefinitions": [
{"name": "owner", "typeName": "string", "isOptional": false},
{"name": "sla", "typeName": "string", "isOptional": true},
{"name": "domain", "typeName": "string", "isOptional": false}
]
}],
"traitTypes": []
}' \
http://atlas:21000/api/v2/types
# Get entity
curl -u admin:admin http://atlas:21000/api/v2/entity/guid/abc-123
# Search entities
curl -u admin:admin \
'http://atlas:21000/api/v2/search/dsl?query=type:hive_table'Metadata Model and Entity Relationships
Atlas models data assets as entities connected by relationships. A Hive table entity has relationships to its database, columns, storage description, and processing processes. Relationships capture data flow: Table A → Process P → Table B shows how data moves through the system.
The entity model supports hierarchical classification, rich metadata attributes, and bidirectional relationships. Entities can have multiple classifications (tags) that describe their properties: PII, sensitive, financial, etc.
# Entity relationships
# hive_table → has_columns → hive_column
# hive_table → created_by_process → process
# process → inputs → hive_table
# process → outputs → hive_table
# Create entity via API
curl -X POST -u admin:admin -H 'Content-Type: application/json' \
-d '{
"typeName": "hive_table",
"attributes": {
"name": "sales_summary",
"qualifiedName": "analytics.sales_summary@production",
"db": {"guid": "db-guid-123"},
"owner": "data_team",
"createTime": 1704067200000,
"tableType": "MANAGED_TABLE"
}
}' \
http://atlas:21000/api/v2/entity
# Get lineage
curl -u admin:admin \
http://atlas:21000/api/v2/lineage/table-guid-123/depth/3Data Lineage Tracking
Atlas tracks data lineage — the path data takes from source to destination. Lineage is captured automatically through hooks: Hive hooks track table reads/writes, Kafka hooks track topic production/consumption, Sqoop hooks track import/export operations. Manual lineage can be registered via API for custom processes.
Lineage queries answer critical questions: where did this data come from? What downstream systems are affected by a change? Which tables feed a specific report? This is essential for impact analysis and regulatory compliance.
# Lineage query via REST API
# Get upstream lineage (inputs)
curl -u admin:admin \
http://atlas:21000/api/v2/lineage/table-guid-123/direction/input/depth/5
# Get downstream lineage (outputs)
curl -u admin:admin \
http://atlas:21000/api/v2/lineage/table-guid-123/direction/output/depth/5
# Lineage graph visualization
# Returns entity relationships as graph
curl -u admin:admin \
http://atlas:21000/api/v2/graph/lineage/table-guid-123
# Impact analysis: what tables are downstream of column X?
# This helps assess impact of schema changes
curl -u admin:admin \
'http://atlas:21000/api/v2/search/dsl?query=column_name:revenue+AND+type:hive_column&sortBy=qualifiedName&sortOrder=ascending'Classification and Tagging
Atlas classifications (tags) are labels applied to entities to describe their properties. Classifications enable policy-based data governance: tag data as PII, and Ranger policies automatically enforce access controls. Classifications can propagate through lineage — if a source table is PII, downstream tables inherit the PII classification.
Custom classifications model organization-specific concepts: data quality, retention policy, business domain, compliance requirements. Classifications support attributes for additional metadata beyond simple tags.
# Add classification to entity
curl -X POST -u admin:admin -H 'Content-Type: application/json' \
-d '{
"typeName": "PII",
"attributes": {
"category": "email",
"retention_days": 365
}
}' \
http://atlas:21000/api/v2/entity/guid/column-guid-456/classification
# Get classified entities
curl -u admin:admin \
'http://atlas:21000/api/v2/search/dsl?query=PII%20AND%20type:hive_column'
# Propagate classification through lineage
# In Atlas config: atlas.lineage.hook.enable.auto.propagation=true
# Classification propagation:
# Source table (PII) → Process → Target table (PII propagated)
# Batch classification
curl -X POST -u admin:admin -H 'Content-Type: application/json' \
-d '{
"entityFilters": "type:hive_table AND db.name:production",
"classifications": [{"typeName": "CONFIDENTIAL", "attributes": {}}]
}' \
http://atlas:21000/api/v2/entity/bulk/classificationHooks and Integration
Atlas hooks are interceptors that capture metadata from Hadoop ecosystem components. The Hive hook captures DDL operations (CREATE TABLE, ALTER TABLE) and DML operations (INSERT, SELECT). The Kafka hook captures topic metadata and producer/consumer relationships. The Sqoop hook captures import/export lineage.
Hooks are configured as lifecycle interceptors in the respective services. When a Hive query runs, the hook extracts table metadata, column definitions, and operation lineage, then sends it to Atlas for indexing.
# Hive hook configuration
# atlas-application.properties
atlas.hook.hive.synchronous=true
atlas.cluster.name=production
atlas.hook.hive.maxRetryCount=3
atlas.hook.hive.websocketMaxRetryCount=3
# atlas-application.properties for Kafka hook
atlas.hook.kafka.synchronous=true
atlas.kafka.bootstrap.servers=kafka:9092
atlas.kafka.zookeeper.connect=zk:2181
# Sqoop hook
atlas.hook.sqoop.synchronous=true
# Verify hooks are sending metadata
# Check Atlas audit logs
curl -u admin:admin http://atlas:21000/api/v2/admin/audit?limit=50
# Check entity count
curl -u admin:admin http://atlas:21000/api/v2/admin/stats
# Returns count of entities by typeREST API and Governance Workflow
Atlas's REST API provides complete metadata management: CRUD operations on entities, lineage queries, classification management, and search. The API supports bulk operations for large-scale metadata operations. Integration with Ranger enables policy-based governance — classification in Atlas triggers access policies in Ranger.
For governance workflows, Atlas supports entity lifecycle events: creation, update, deprecation. Combined with approval processes, organizations can enforce data governance policies through automated workflows.
# Search with DSL
# Full-text search
curl -u admin:admin \
'http://atlas:21000/api/v2/search/full-text?query=sales'
# DSL search
curl -u admin:admin \
'http://atlas:21000/api/v2/search/dsl?query=type:hive_table+AND+name:revenue+AND+owner:finance'
# Delete entity (soft delete)
curl -X DELETE -u admin:admin \
http://atlas:21000/api/v2/entity/guid/entity-guid-789
# Export entity
curl -u admin:admin \
http://atlas:21000/api/v2/entity/guid/entity-guid-789/export
# Import entities (migration)
curl -X POST -u admin:admin -H 'Content-Type: application/json' \
-d @exported_entities.json \
http://atlas:21000/api/v2/entity/importFrequently Asked Questions
What is the difference between Atlas and a data catalog?
Atlas is a metadata store and governance framework. A data catalog (Amundsen, DataHub) adds search, discovery, and collaboration features. Atlas provides the governance backbone; catalogs provide the user-facing discovery layer.
Does Atlas work with non-Hadoop data sources?
Atlas primarily supports Hadoop ecosystem components. For non-Hadoop sources (databases, SaaS), use custom hooks or integration via the REST API. Apache Atlas has community-developed hooks for some RDBMS.
How does Atlas handle schema evolution?
Atlas captures schema changes as entity updates. Each change is versioned with timestamps. You can compare entity versions to see how a table's schema evolved over time.
Can Atlas enforce data quality rules?
Atlas tracks metadata and lineage but does not enforce data quality directly. Combine Atlas with tools like Apache Griffin or Great Expectations for data quality monitoring, and use Atlas to document quality policies.
Originally published on Ayodhyyya. Last updated June 1, 2026.