Getting Started with Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene. It stores data as JSON documents and indexes them for near real-time search. The Elastic Stack (ELK Stack) includes Elasticsearch, Logstash for data ingestion, Kibana for visualization, and Beats for lightweight data shipping.

Installation is straightforward. Download the tarball from elastic.co or use package managers. On macOS, brew install elasticsearch. Start with bin/elasticsearch and verify with curl http://localhost:9200. The response includes cluster name, version, and node information. For a development environment, a single node is sufficient. Production requires a cluster of at least three nodes.

# Start Elasticsearch
bin/elasticsearch

# Verify
curl http://localhost:9200

# Index a document
curl -X PUT "localhost:9200/products/_doc/1" -H 'Content-Type: application/json' -d'
{
    "name": "Wireless Mouse",
    "category": "electronics",
    "price": 29.99,
    "description": "Ergonomic wireless mouse with USB receiver",
    "in_stock": true
}'

# Basic search
curl -X GET "localhost:9200/products/_search" -H 'Content-Type: application/json' -d'
{
    "query": {
        "match": {
            "description": "wireless mouse"
        }
    }
}'

Indexing, Mapping, and Analysis

An index is a logical namespace that holds documents with similar characteristics. When you index a document, Elasticsearch automatically detects field types through dynamic mapping. A string field becomes text (full-text search) and keyword (exact matching and aggregations). This works for prototyping but is inadequate for production where you need explicit control over field types and analysis.

Explicit mapping defines the schema before indexing data. Specify field types like text, keyword, integer, float, date, geo_point, or nested. The text type undergoes analysis: the standard analyzer lowercases, splits on whitespace and punctuation, and removes stop words. Custom analyzers can handle language-specific stemming, synonyms, and character filters for HTML stripping.

Inverted index is the core data structure. Elasticsearch tokenizes text fields into terms and builds a mapping from each term to the documents containing it. When you search for wireless mouse, it looks up both terms and returns documents that match. Relevance scoring uses BM25 to rank results by how frequently the term appears in the document versus how common it is across all documents.

// Explicit mapping
curl -X PUT "localhost:9200/articles" -H 'Content-Type: application/json' -d'
{
    "mappings": {
        "properties": {
            "title": { "type": "text", "analyzer": "english" },
            "body": { "type": "text", "analyzer": "english" },
            "author": { "type": "keyword" },
            "published_at": { "type": "date" },
            "tags": { "type": "keyword" },
            "views": { "type": "integer" }
        }
    }
}'

Search Queries: From Simple to Complex

Elasticsearch queries range from simple match queries to complex boolean combinations. The match query analyzes the input text and searches for matching terms. The term query searches for exact values in keyword fields. Bool queries combine multiple conditions with must (AND), should (OR), filter (non-scoring filter), and must_not (exclusion).

Full-text search capabilities include phrase matching with match_phrase, fuzzy matching for typo tolerance (fuzziness: AUTO), prefix queries, and wildcard queries. The multi_match query searches across multiple fields with field-specific boosting. A search for title^3 and body gives the title field three times the importance of the body field.

Aggregations are Elasticsearch's answer to GROUP BY and analytics. Bucket aggregations group documents by criteria like terms, date ranges, or geolocation. Metric aggregations compute stats like avg, sum, min, max, and cardinality (unique count). You can nest aggregations for drill-down analysis. A common pattern is terms aggregation on category, sub-aggregated by avg price, sub-aggregated by date histogram.

// Bool query with multiple conditions
GET /products/_search
{
    "query": {
        "bool": {
            "must": [
                { "match": { "description": "wireless" } },
                { "term": { "category": "electronics" } }
            ],
            "filter": [
                { "range": { "price": { "gte": 10, "lte": 100 } } },
                { "term": { "in_stock": true } }
            ],
            "should": [
                { "match": { "name": "ergonomic" } }
            ],
            "minimum_should_match": 1
        }
    },
    "highlight": {
        "fields": {
            "description": {}
        }
    }
}

Cluster Architecture and Scalability

An Elasticsearch cluster consists of nodes that each serve a role. Master nodes manage cluster state: tracking nodes, indexes, and shard allocation. Data nodes store data and execute queries. Ingest nodes preprocess documents before indexing. Coordinating nodes route requests and aggregate results. For clusters under 10 nodes, each node can serve multiple roles. Larger clusters benefit from dedicated master and coordinating nodes.

Sharding is Elasticsearch's scaling mechanism. An index is divided into shards, each stored on a different node. Primary shards hold the original data, and replica shards are copies for redundancy and read parallelism. The number of primary shards is fixed at index creation and cannot be changed (without reindexing). Choose shard count based on the index size: aim for shards between 10GB and 50GB each.

Index lifecycle management (ILM) automates index management over time. Define policies that roll over indexes based on size, document count, or age. Hot tier holds actively written indexes on fast storage. Warm tier stores read-only indexes on standard storage. Cold tier archives older indexes on cost-effective storage. Delete tier removes indexes after the retention period expires.

// ILM policy
PUT _ilm/policy/logs_policy
{
    "policy": {
        "phases": {
            "hot": {
                "min_age": "0ms",
                "actions": {
                    "rollover": {
                        "max_size": "50GB",
                        "max_age": "30d"
                    },
                    "set_priority": { "priority": 100 }
                }
            },
            "warm": {
                "min_age": "30d",
                "actions": {
                    "forcemerge": { "max_num_segments": 1 },
                    "shrink": { "number_of_shards": 1 },
                    "set_priority": { "priority": 50 }
                }
            },
            "cold": {
                "min_age": "90d",
                "actions": {
                    "freeze": {},
                    "set_priority": { "priority": 0 }
                }
            },
            "delete": {
                "min_age": "365d",
                "actions": {
                    "delete": {}
                }
            }
        }
    }
}

Performance Tuning and Query Optimization

Elasticsearch performance tuning starts at the hardware level. Elasticsearch is I/O-bound in most cases. Use SSDs, preferably NVMe. Configure the thread pool sizes based on CPU count. The search thread pool handles query operations, the write thread pool handles indexing, and the force merge thread pool handles segment merging. Monitor thread pool queues: if they grow, nodes are overloaded.

Query optimization focuses on reducing the number of shards a query touches. Use the search_type parameter: query_then_fetch is the default, suitable for most cases. The dfs_query_then_fetch performs a global term frequency calculation for more accurate scoring but adds a round trip. For aggregations on large datasets, use the composite aggregation instead of terms for paginated aggregation results.

Segment merging is a background operation that consumes I/O and CPU. Elasticsearch automatically merges smaller segments into larger ones to keep search performance optimal. Tune the merge policy if automatic merging causes I/O spikes during business hours. Force merging read-only indexes into a single segment eliminates merge overhead entirely.

// Cluster health and node stats
GET /_cluster/health
GET /_nodes/stats
GET /_nodes/hot_threads

// Thread pool monitoring
GET /_cat/thread_pool?v

// Force merge a read-only index
POST /logs-2025-01/_forcemerge?max_num_segments=1

// Slow log configuration
PUT /products/_settings
{
    "index.search.slowlog.threshold.query.warn": "5s",
    "index.search.slowlog.threshold.query.info": "1s",
    "index.indexing.slowlog.threshold.index.info": "500ms",
    "index.indexing.slowlog.source": 1000
}

Security, Snapshots, and Operations

Elasticsearch security includes authentication, authorization, and encryption. The basic license includes TLS for transport and HTTP layers, file-based authentication, and role-based access control. Create roles with specific index privileges: read, write, create_index, manage. Users authenticate with username/password or API keys. Always encrypt traffic between nodes and between clients and the cluster.

Snapshots are the backup mechanism. Register a snapshot repository (S3, GCS, Azure, or shared filesystem) and take snapshots of specific indexes or the entire cluster. Snapshots are incremental: only changed data is transferred after the initial full snapshot. Automate snapshots with SLM (Snapshot Lifecycle Management) policies that define schedule, retention, and repository.

Rolling restarts allow upgrading Elasticsearch without downtime. Disable shard allocation, stop the node, upgrade, restart, and re-enable allocation. Repeat for each node. Monitor recovery status during the process. For major version upgrades, use a reindexing approach: create a new cluster, reindex from the old cluster, and switch traffic. This avoids compatibility issues between major versions.

// SLM policy
PUT /_slm/policy/nightly-snapshots
{
    "name": "",
    "schedule": "0 30 2 * * ?",
    "repository": "s3-backup",
    "config": {
        "include_global_state": false
    },
    "retention": {
        "expire_after": "30d",
        "min_count": 5,
        "max_count": 50
    }
}