Object Storage

Object storage is the backbone of Thanos long-term retention. Unlike traditional databases that store data on attached volumes or shared file systems, Thanos leverages the economics, durability, and scalability of cloud object storage services like Amazon S3, Google Cloud Storage, Microsoft Azure Blob Storage, and MinIO for on-premises deployments. Object storage provides virtually unlimited capacity at extremely low cost per gigabyte, with built-in redundancy across multiple availability zones. For Thanos, this means that metrics data can be retained for months or years at a fraction of the cost of maintaining local SSD storage.

The Thanos bucket configuration is defined in a YAML file that specifies the provider, bucket name, region, and authentication credentials. Each supported provider has its own configuration schema, but the general structure is consistent. The configuration file is typically mounted as a Kubernetes secret to avoid committing credentials to version control. Thanos uses the objstore library, which provides a unified interface across all supported providers, making it straightforward to switch between providers or test locally with a filesystem-based implementation.

YAML
# Object Storage Configuration for Amazon S3
type: S3
config:
  bucket: "thanos-metrics-prod"
  endpoint: "s3.us-east-1.amazonaws.com"
  region: "us-east-1"
  access_key: "AKIAIOSFODNN7EXAMPLE"
  secret_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
  insecure: false
  signature_version2: false
  sse_config:
    type: "SSE-S3"
  http_config:
    idle_conn_timeout: 90s
    response_header_timeout: 2m
    insecure_skip_verify: false
  part_size: 134217728
  list_objects_version: "v2"

# Alternative: Google Cloud Storage
# type: GCS
# config:
#   bucket: "thanos-metrics-prod"
#   service_account: "/etc/thanos/gcs-service-account.json"

# Alternative: Azure Blob Storage
# type: AZURE
# config:
#   storage_account: "thanosstorage"
#   storage_account_key: "storage-key"
#   container: "thanos-metrics"
#   endpoint: "blob.core.windows.net"
#   max_retries: 3
#   msi_resource: ""

# Alternative: MinIO (on-premises)
# type: S3
# config:
#   bucket: "thanos-metrics"
#   endpoint: "minio.internal:9000"
#   access_key: "minioadmin"
#   secret_key: "minioadmin"
#   insecure: true
#   signature_version2: false

The bucket structure used by Thanos follows a specific convention that enables efficient metadata queries and block management. Each uploaded block is stored under a prefix derived from its ULID (Universally Unique Lexicographically Sortable Identifier), which ensures that blocks are naturally ordered by time. Within each block directory, Thanos stores the block's chunk files (containing the actual time series data), the index file (enabling efficient lookups), the meta.json file (containing block metadata), and optionally the deletion marker files used for soft-deleting blocks. The Compactor writes additional metadata files during the compaction and downsampling processes, and the bucket store reads these files to build an in-memory index of all available blocks.

Performance considerations for object storage in Thanos revolve around three key metrics: latency, throughput, and cost. S3 and GCS provide excellent throughput for sequential reads, which aligns well with Thanos access patterns (blocks are read sequentially during queries). However, latency for individual object requests can be 50-200ms, which is why the Store Gateway's caching layer is essential for maintaining acceptable query performance. Cost optimization involves understanding the pricing model of your chosen provider — S3 charges for storage, GET/PUT requests, and data transfer — and configuring block sizes and retention policies accordingly. Larger blocks reduce the total number of objects (reducing API call costs) but increase the minimum data transfer per query.

For on-premises or hybrid deployments, MinIO provides a fully S3-compatible object storage solution that can run on commodity hardware. Thanos connects to MinIO using the S3 provider configuration with the insecure: true flag (for non-TLS connections) and a custom endpoint pointing to the MinIO service. MinIO supports erasure coding for data durability, versioning for data protection, and lifecycle policies for automatic data tiering. In multi-datacenter deployments, MinIO's replication feature enables cross-site data protection without relying on cloud providers, which is essential for organizations with strict data residency requirements or air-gapped environments.

The bucket configuration is shared across all Thanos components that interact with object storage. The Sidecar writes to the bucket, the Store Gateway reads from it, the Compactor reads and writes during compaction, and the Ruler may write recording rule results to it. In a Kubernetes environment, the bucket configuration is stored as a Secret and mounted as a volume in each Thanos component pod. Access to the bucket should follow the principle of least privilege: the Sidecar should have write-only permissions, the Store Gateway should have read-only permissions, and the Compactor should have read-write-delete permissions. This is typically implemented using IAM policies on the cloud provider side.

Bucket versioning is a critical operational consideration. When the Compactor deletes or overwrites blocks during compaction, bucket versioning ensures that the previous versions are preserved. This provides a safety net against accidental data loss caused by compaction bugs or misconfigured retention policies. Most cloud providers support bucket versioning natively, and Thanos is designed to work correctly with versioned buckets. It is strongly recommended to enable versioning on production buckets, even though it slightly increases storage costs due to the retention of previous object versions.

Store Gateway

The Store Gateway is Thanos's answer to the challenge of efficiently reading historical data from object storage. Without the Store Gateway, querying data that has been uploaded to S3 or GCS would require downloading entire blocks — each potentially hundreds of megabytes — just to read a small subset of time series within a narrow time window. The Store Gateway solves this problem by maintaining an in-memory index of all blocks in the bucket, selectively downloading only the index files and data chunks needed to satisfy each query, and caching frequently accessed data to minimize repeated downloads.

The Store Gateway operates by periodically syncing the block metadata from the object storage bucket. This sync process lists all blocks in the bucket, downloads their meta.json files, and builds an in-memory index mapping label sets, time ranges, and block IDs. The sync interval is configurable and should be balanced between freshness (shorter intervals ensure new blocks are discovered quickly) and efficiency (longer intervals reduce the number of API calls to the object store). In typical production deployments, a sync interval of 3 to 5 minutes provides a good balance.

When a query arrives at the Store Gateway via the gRPC StoreAPI, it performs several optimization steps before downloading any data. First, it uses the in-memory block index to determine which blocks contain data relevant to the requested time range and label matchers. This pruning step can eliminate the vast majority of blocks from consideration, especially for queries targeting recent data or specific label values. Second, it checks its local disk cache and in-memory cache for any previously downloaded data that matches the query parameters. Only data that is not already cached is fetched from object storage, and even then, only the specific chunks needed for the query are downloaded.

The caching architecture of the Store Gateway operates at multiple levels. The index cache stores decoded index headers in memory, avoiding repeated downloads and parsing of block index files. The bucket cache wraps the object storage client and caches individual object reads, preventing redundant downloads of the same chunk files across different queries. The series cache stores the final decoded time series results, enabling instant responses for repeated or overlapping queries. Each cache level has its own eviction policy and memory budget, and operators can tune these independently based on their workload characteristics.

YAML
# Thanos Store Gateway Deployment with Caching
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-store-gateway
spec:
  replicas: 3
  selector:
    matchLabels:
      app: thanos-store
  template:
    spec:
      containers:
        - name: store-gateway
          image: quay.io/thanos/thanos:v0.35.0
          args:
            - 'store'
            - '--data-dir=/data'
            - '--objstore.config-file=/etc/thanos/bucket.yml'
            - '--grpc-address=0.0.0.0:10901'
            - '--http-address=0.0.0.0:10902'
            - '--index-cache-size=500MB'
            - '--chunk-pool-size=2GB'
            - '--sync-block-duration=3m'
            - '--min-time=-2w'
            - '--max-time=-6h'
            - '--store.grpc.series-max-concurrency=20'
            - '--consistency-delay=30m'
          ports:
            - containerPort: 10901
              name: grpc
            - containerPort: 10902
              name: http
          volumeMounts:
            - name: store-data
              mountPath: /data
            - name: thanos-bucket-config
              mountPath: /etc/thanos
              readOnly: true
          resources:
            requests:
              memory: "4Gi"
              cpu: "1"
            limits:
              memory: "8Gi"
              cpu: "2"
      volumes:
        - name: store-data
          persistentVolumeClaim:
            claimName: thanos-store-pvc
        - name: thanos-bucket-config
          secret:
            secretName: thanos-bucket-credentials

The Store Gateway supports horizontal scaling through hash-ring-based sharding. When multiple Store Gateway replicas are deployed, each instance is assigned a subset of blocks based on a consistent hash ring. This means that each replica only syncs and serves data for its assigned blocks, distributing both memory usage and query load across the fleet. The hash ring membership is communicated through a shared KV store (typically etcd or Consul) or through the Store Gateway's built-in hashing mechanism. Sharding is essential for large buckets with millions of blocks, as a single Store Gateway instance may not have sufficient memory to index all blocks.

The --min-time and --max-time flags on the Store Gateway enable vertical splitting of the data across multiple Store Gateway instances. For example, you might deploy one set of Store Gateways handling data from the last two weeks (for fast access to recent historical data) and another set handling data older than two weeks (for compliance or archival queries). This separation allows you to allocate more cache resources to the recent-data gateways, improving performance for the most common query patterns while keeping the older-data gateways smaller and cheaper.

Operational monitoring of the Store Gateway is essential for maintaining query performance. Key metrics to watch include thanos_bucket_store_series_data_fetched (which chunks are being fetched), thanos_bucket_store_series_result_series (how many series are being returned), thanos_bucket_store_block_load_duration_seconds (how long it takes to load a block index), and cache hit/miss ratios for each cache type. A sudden drop in cache hit ratio may indicate a memory pressure issue causing evictions, while a spike in block load duration may suggest network latency to the object store. These metrics should be monitored through the Store Gateway's built-in Prometheus metrics endpoint and alerted on proactively.

Query Frontend

The Query Frontend is the entry point for all PromQL queries in a Thanos deployment, positioned between client applications (such as Grafana, API consumers, or alerting rules) and the Query component. While the Query component handles the actual PromQL evaluation and fan-out, the Query Frontend adds a critical layer of query optimization, caching, and request management that dramatically improves performance and reliability at scale. Think of it as a reverse proxy for PromQL queries, applying a series of transformations to make each query more efficient before forwarding it downstream.

The most impactful feature of the Query Frontend is query splitting. Large PromQL queries that span many hours or days of data can be extremely resource-intensive for the Query component. The Query Frontend automatically splits these large queries into smaller, time-based sub-queries that can be evaluated independently and in parallel. For example, a query spanning 7 days might be split into 84 two-hour sub-queries, each of which hits a smaller set of blocks and returns results faster. The sub-query results are then merged by the Query Frontend before being returned to the client. This splitting is transparent to the user — they issue a single query and receive a single response, but the backend work is distributed more efficiently.

Query results caching is the second major optimization. When a query is split into sub-queries, the Query Frontend can cache the results of each sub-query in an external cache backend (such as Memcached, Redis, or an in-memory cache). When the same or overlapping query arrives again, the cached results are used for time ranges that have not changed, and only the most recent sub-queries (where new data may have arrived) are sent to the Query component. This is particularly effective for dashboard queries that are refreshed every 30 seconds or every minute, as the majority of the query results will be identical between refreshes.

The Query Frontend also implements request queuing and retry logic with automatic TTL-based eviction. When the Query component is under heavy load, the Query Frontend queues incoming requests rather than rejecting them. The queue is configurable with a maximum size and request TTL — if a request sits in the queue longer than the TTL, it is automatically dropped with an appropriate error response. This prevents cascading failures where slow queries pile up and overwhelm the system. Requests that fail due to transient errors (such as a brief network interruption to a Store Gateway) are automatically retried up to a configurable number of times.

YAML
# Thanos Query Frontend Configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-query-frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: thanos-query-frontend
  template:
    spec:
      containers:
        - name: query-frontend
          image: quay.io/thanos/thanos:v0.35.0
          args:
            - 'query-frontend'
            - '--http-address=0.0.0.0:10902'
            - '--query-frontend.downstream-url=http://thanos-query:10902'
            - '--query-frontend.compress-responses'
            - '--query-frontend.split-interval=24h'
            - '--query-frontend.max-retries-per-request=3'
            - '--query-frontend.default-value=10000'
            - '--query-frontend.log_queries_longer_than=10s'
            - '--query-range.split-interval=2h'
            - '--query-range.repeat-expr=3'
            - '--labels.split-interval=24h'
          ports:
            - containerPort: 10902
          resources:
            requests:
              memory: "1Gi"
              cpu: "500m"

The Query Frontend supports response compression, which can significantly reduce network bandwidth usage when serving large result sets. The --query-frontend.compress-responses flag enables gzip compression of HTTP responses, which is particularly beneficial when the Query Frontend and Grafana are communicating across different availability zones or over WAN links. Compression typically achieves 5-10x reduction in response size for typical PromQL results, at the cost of slightly increased CPU usage for compression and decompression.

Deduplication at the Query Frontend level is another important feature. When the Query component returns results from multiple Prometheus replicas, the Query Frontend can merge duplicate time series using the configured replica label. This is complementary to the Query component's deduplication — the Query Frontend handles deduplication across different Query component instances, while the Query component handles deduplication across different stores within a single Query instance. In a typical deployment with multiple Query component replicas behind a load balancer, the Query Frontend ensures that the client receives deduplicated results regardless of which Query instance processed the request.

The Query Frontend is typically deployed as a lightweight deployment in front of the Query component. It does not maintain any persistent state and can be scaled horizontally without coordination. In a multi-cluster Thanos deployment, you might deploy a Query Frontend in each cluster, each pointing to a local Query component, with a global Query Frontend sitting in front of all regional Query components. This two-tier architecture provides both local query performance (for cluster-specific dashboards) and global query capability (for cross-cluster dashboards). The Query Frontend should be placed behind a standard HTTP load balancer (such as Kubernetes Ingress or an cloud load balancer) for high availability.

Query Component

The Query component is the central nervous system of Thanos, responsible for evaluating PromQL queries against data from multiple heterogeneous sources and returning unified, deduplicated results. Unlike traditional Prometheus, which queries a single local TSDB, the Thanos Query component implements a fan-out query execution model that distributes queries across all registered StoreAPI endpoints in parallel, collects partial results, merges overlapping data, deduplicates time series, and evaluates PromQL expressions against the merged dataset. This architecture enables a single query to transparently access data spanning multiple Prometheus instances, clusters, and even cloud providers.

When a query arrives at the Query component, the first step is store discovery. The Query component maintains connections to a set of StoreAPI endpoints, which can include Sidecars (for recent local data), Store Gateways (for historical object storage data), and other Query components (for hierarchical query federation). The set of stores is configured statically via command-line flags or dynamically via file-based service discovery. The Query component periodically queries each store's Info RPC to determine its available time range and label sets, using this metadata to route queries only to stores that can contribute relevant data.

The second step is query distribution. Based on the requested time range and the metadata from each store, the Query component determines which stores need to be queried. For a query requesting data from the last hour, only Sidecars with recent data would be queried. For a query requesting data from the last year, both Sidecars (for the most recent hours) and Store Gateways (for the older data) would be queried. The query is sent to each relevant store in parallel, and the Query component collects partial results as they arrive.

The third step is result merging and deduplication. When results arrive from multiple stores, there may be overlapping time series — this is expected when you have two Prometheus replicas scraping the same targets. The Query component identifies overlapping series using the configured replica label and merges them into a single series. The deduplication algorithm is configurable: it can use the last-sample-wins strategy (taking the most recent sample from each replica) or an averaging strategy for cases where you want to smooth out differences between replicas. The merged results are then evaluated against the PromQL expression to produce the final result set.

YAML
# Thanos Query Deployment with Fan-out Configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-query
spec:
  replicas: 3
  selector:
    matchLabels:
      app: thanos-query
  template:
    spec:
      containers:
        - name: query
          image: quay.io/thanos/thanos:v0.35.0
          args:
            - 'query'
            - '--grpc-address=0.0.0.0:10901'
            - '--http-address=0.0.0.0:10902'
            - '--store=dnssrv+_grpc._tcp.thanos-sidecar.monitoring.svc:10901'
            - '--store=dnssrv+_grpc._tcp.thanos-store.monitoring.svc:10901'
            - '--store=dnssrv+_grpc._tcp.thanos-query-remote.monitoring.svc:10901'
            - '--query.replica-label=prometheus_replica'
            - '--query.auto-downsampling'
            - '--query.lookback-delta=5m'
            - '--query.max-concurrent=20'
            - '--query.timeout=2m'
            - '--store.grpc.series-max-concurrency=20'
            - '--store.grpc.series.sample-limit=100000'
            - '--store.grpc.max-sample-count=1000000'
          ports:
            - containerPort: 10901
              name: grpc
            - containerPort: 10902
              name: http
          resources:
            requests:
              memory: "2Gi"
              cpu: "1"
            limits:
              memory: "4Gi"
              cpu: "2"

The fan-out parallelism of the Query component is one of its most powerful features. By default, the Query component queries all stores in parallel, with a configurable maximum concurrency limit. This means that a query against a store that is slow to respond does not block results from faster stores — the Query component returns partial results as they become available (when partial response is enabled) or waits for all stores up to the configured timeout. The partial response feature is particularly valuable in multi-cluster deployments where a cross-cluster network issue might slow down one cluster's response without affecting others.

The Query component supports auto-downsampling when the requested time range exceeds a configurable threshold. For example, if you query 30 days of data, the Query component will automatically use the 5m downsampled blocks instead of the raw data, and for 90+ days it will use the 1h downsampled blocks. This dramatically improves query performance for long-range queries without requiring the user to explicitly request downsampled data. The auto-downsampling thresholds are configurable and should be tuned based on your typical query patterns and the accuracy requirements of your use case.

The hierarchical query model in Thanos enables a powerful pattern where regional Query components are deployed in each cluster, and a global Query component connects to all regional Query components. This reduces the number of cross-cluster connections and centralizes the fan-out logic. Each regional Query component handles queries for its local data, while the global Query component provides the unified view across all regions. This pattern also improves reliability — if the global Query component fails, regional queries continue to work, and if a regional component fails, other regions are unaffected.

Ruler Component

The Thanos Ruler component extends Prometheus's rule evaluation capabilities to operate against the global query view provided by Thanos. While standard Prometheus evaluates recording and alerting rules only against its local TSDB data, the Ruler connects to multiple StoreAPI endpoints and evaluates rules against the combined data from all Prometheus instances, Store Gateways, and other data sources in the Thanos ecosystem. This is fundamental for use cases where alerting rules need to consider metrics from multiple clusters — for example, detecting a capacity threshold that requires aggregating CPU usage across all Kubernetes nodes in all regions.

The Ruler operates in a loop, periodically evaluating a set of PromQL rules against the configured rule files. It connects to a set of StoreAPI endpoints (similar to how the Query component connects to stores) and evaluates each rule expression against the merged data from all connected stores. Recording rules produce pre-computed time series that are written to the Thanos Sidecar (for upload to object storage) or directly to a local TSDB. Alerting rules produce alerts that are sent to the configured Alertmanager endpoints. The evaluation interval is configurable and should typically match the Prometheus scrape interval to ensure consistent alert timing.

A key design consideration for the Ruler is the distinction between ruler evaluation and alert routing. The Ruler evaluates PromQL expressions and generates alert objects, but it does not manage alert lifecycle (firing, pending, resolved). That responsibility lies with Alertmanager, which receives alerts from the Ruler and routes them to the appropriate notification channels based on its routing tree. The Ruler should be configured with the same Alertmanager endpoints as the Prometheus instances it replaces for rule evaluation, ensuring that alerts are routed consistently regardless of where they originate.

Recording rules in the Thanos context serve a dual purpose: they pre-compute expensive PromQL expressions to improve query performance, and they enable cross-cluster aggregations that cannot be expressed in a single Prometheus instance's local data. For example, a recording rule that aggregates HTTP request latency across all clusters would be evaluated by the Ruler using data from all connected StoreAPI endpoints, and the result would be written to object storage where it becomes available for dashboards and further queries. This creates a powerful pipeline where raw metrics flow from Prometheus instances to object storage, recording rules pre-compute aggregations, and dashboards consume the pre-computed results for fast rendering.

YAML
# Thanos Ruler with Recording and Alerting Rules
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-ruler
spec:
  replicas: 2
  selector:
    matchLabels:
      app: thanos-ruler
  template:
    spec:
      containers:
        - name: ruler
          image: quay.io/thanos/thanos:v0.35.0
          args:
            - 'rule'
            - '--grpc-address=0.0.0.0:10901'
            - '--http-address=0.0.0.0:10902'
            - '--data-dir=/data'
            - '--rule-file=/etc/thanos/rules/*.yml'
            - '--store=dnssrv+_grpc._tcp.thanos-query.monitoring.svc:10901'
            - '--objstore.config-file=/etc/thanos/bucket.yml'
            - '--alert.label-drop=prometheus_replica'
            - '--alertmanagers.url=http://alertmanager:9093'
            - '--eval-interval=1m'
            - '--query-grpc-address=thanos-query:10901'
          ports:
            - containerPort: 10901
              name: grpc
            - containerPort: 10902
              name: http
          volumeMounts:
            - name: ruler-rules
              mountPath: /etc/thanos/rules
            - name: thanos-bucket-config
              mountPath: /etc/thanos
              readOnly: true
            - name: ruler-data
              mountPath: /data
      volumes:
        - name: ruler-rules
          configMap:
            name: thanos-ruler-rules
        - name: thanos-bucket-config
          secret:
            secretName: thanos-bucket-credentials
        - name: ruler-data
          persistentVolumeClaim:
            claimName: thanos-ruler-data-pvc

The Ruler supports a sophisticated rule evaluation model with groups and intervals. Rules are organized into groups, and each group has its own evaluation interval. The Ruler evaluates all rules within a group sequentially, waits for the group interval, and then re-evaluates. This allows different rule groups to have different evaluation frequencies — for example, critical alerting rules might be evaluated every 30 seconds while cost-saving recording rules are evaluated every 5 minutes. The group evaluation model also ensures that all rules within a group see a consistent snapshot of data, preventing race conditions where one rule depends on the output of another within the same group.

The Ruler's deployment pattern is flexible. It can connect directly to individual StoreAPI endpoints (Sidecars and Store Gateways) for maximum data access, or it can connect to the Query component for the benefit of deduplication and query optimization. The latter approach is generally recommended for production deployments, as it simplifies the Ruler's configuration (it only needs to know about the Query endpoint) and ensures that rule evaluation uses the same deduplicated, optimized query path as dashboards and ad-hoc queries. The Ruler should always be deployed with a persistent volume to store its local TSDB data, ensuring that recording rule results survive pod restarts and can be queried directly from the Ruler's local storage.

Compactor

The Thanos Compactor is a critical background process that maintains the health, performance, and cost-effectiveness of long-term object storage. Without the Compactor, the object storage bucket would accumulate thousands of small blocks (each covering a two-hour window), resulting in slow queries (due to the large number of blocks to inspect), high API costs (due to many individual object requests), and excessive storage consumption (due to redundant data across overlapping blocks). The Compactor addresses all three issues through block merging, downsampling, and retention enforcement.

Block merging is the Compactor's primary function. It identifies groups of small blocks that overlap in time and label space, downloads them, merges their data, and uploads the resulting larger block. The merged block replaces the original blocks, which are then deleted (after a configurable delay to ensure consistency). Block merging reduces the total number of blocks in the bucket, which directly improves query performance — each query needs to inspect fewer block index files. The Compactor also handles resolution merging, where blocks at different resolutions (raw, 5m, 1h) are kept separate and managed independently.

Downsampling is the process of creating lower-resolution copies of raw data blocks. Thanos supports two downsample resolutions: 5-minute (every 300 seconds) and 1-hour (every 3600 seconds). The 5-minute resolution blocks are created from raw data after a configurable age (typically 48 hours), and the 1-hour resolution blocks are created from 5-minute data after a longer age (typically 14 days). Downsampling dramatically improves query performance for long-range queries — instead of scanning millions of individual samples, the query can scan thousands at the downsampled resolution. The accuracy loss from downsampling is minimal for most monitoring use cases, as the aggregation functions (rate, increase, avg, etc.) are designed to work correctly with lower-resolution data.

Retention enforcement is the process of deleting blocks that are older than the configured retention period. The Compactor periodically scans the bucket for blocks whose maximum timestamp exceeds the retention threshold and deletes them. Deletion is performed using soft-delete markers (in versioned buckets) to allow for recovery in case of accidental deletion. The retention period is configurable per resolution — you might retain raw data for 30 days, 5-minute data for 6 months, and 1-hour data for 2 years. This tiered retention strategy optimizes both storage cost and query performance.

The Compactor operates in a single-instance mode with leader election to prevent concurrent compaction operations that could corrupt data. When deployed in a Kubernetes environment, the Compactor uses a leader election mechanism based on the Kubernetes API or the object storage bucket itself to ensure that only one instance is actively compacting at any time. The standby instance takes over if the active instance fails, ensuring continuous compaction with minimal interruption. This leader election is a critical operational requirement — running multiple compactors simultaneously without coordination will result in data corruption and inconsistent bucket state.

YAML
# Thanos Compactor Deployment with Retention
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-compactor
spec:
  replicas: 1
  serviceName: thanos-compactor
  selector:
    matchLabels:
      app: thanos-compactor
  template:
    spec:
      containers:
        - name: compactor
          image: quay.io/thanos/thanos:v0.35.0
          args:
            - 'compact'
            - '--data-dir=/data'
            - '--objstore.config-file=/etc/thanos/bucket.yml'
            - '--http-address=0.0.0.0:10902'
            - '--compact.concurrency=1'
            - '--downsample.concurrency=2'
            - '--delete-delay=48h'
            - '--retention.resolution-raw=30d'
            - '--retention.resolution-5m=180d'
            - '--retention.resolution-1h=365d'
            - '--consistency-delay=30m'
            - '--compact.cleanup-interval=30m'
            - '--wait'
            - '--wait-interval=5m'
          ports:
            - containerPort: 10902
          volumeMounts:
            - name: compactor-data
              mountPath: /data
            - name: thanos-bucket-config
              mountPath: /etc/thanos
              readOnly: true
          resources:
            requests:
              memory: "4Gi"
              cpu: "1"
            limits:
              memory: "8Gi"
              cpu: "2"
      volumes:
        - name: compactor-data
          persistentVolumeClaim:
            claimName: thanos-compactor-data-pvc
        - name: thanos-bucket-config
          secret:
            secretName: thanos-bucket-credentials

The Compactor's performance depends heavily on the volume of blocks to compact and the speed of the object storage backend. In a large-scale deployment with millions of blocks, the Compactor may need several hours to complete a full compaction cycle. Key metrics to monitor include thanos_compact_group_compactions_failures_total (failed compactions indicating data issues), thanos_compact_block_cleanup_failures_total (cleanup failures), and thanos_compact_downsample_total (downsample progress). The Compactor should always run with persistent storage for its local data directory, as it maintains intermediate state during compaction operations.

Block Upload and Retention

Understanding the TSDB block lifecycle is essential for operating Thanos effectively. Prometheus TSDB organizes time series data into blocks, each typically covering a two-hour window. A block consists of several files: chunk segments (containing the actual sample data), an index file (providing efficient lookups by label name and value), a meta.json file (containing block metadata including time range, label sets, and compaction level), and a tombstone file (recording deleted data). Each block is identified by a ULID that encodes its creation time, ensuring natural chronological ordering.

The Thanos Sidecar is responsible for detecting completed blocks and uploading them to object storage. The upload process begins when the Sidecar detects that a block in the local TSDB directory has been finalized — meaning its time range is closed and it will not receive any additional samples. The Sidecar verifies this by checking that the block's maximum timestamp is at least 1.5 times the block duration behind the current time. Once confirmed, the Sidecar streams the block's files to object storage, uploading each file individually to maintain atomicity (a partially uploaded block will not be visible to the Store Gateway because the meta.json file is uploaded last).

The upload strategy has several important characteristics. First, the Sidecar uploads blocks in the background, without blocking Prometheus's normal operation. The upload uses a bounded amount of memory by streaming file contents rather than loading entire files into memory. Second, the Sidecar tracks which blocks have been uploaded using a local state file, ensuring that it does not re-upload blocks after a restart. Third, the Sidecar respects the block duration configuration — it only uploads blocks with the expected two-hour duration, ignoring any blocks that may have been created by Prometheus's internal compaction process.

Retention policies in Thanos are enforced at multiple levels. The Compactor handles the primary retention logic by deleting blocks that exceed the configured retention period. The retention is configurable per resolution, allowing operators to keep raw data for a short period (30 days) while retaining downsampled data for much longer (1 year or more). The Compactor also handles the deletion of partially compacted blocks — during compaction, the original blocks are not immediately deleted but are marked for deletion after a configurable delay (default 48 hours). This delay provides a safety window for detecting and recovering from compaction errors.

Bucket cleanup is an essential operational task that works alongside retention policies. Over time, the object storage bucket may accumulate orphaned blocks, partial uploads, or blocks marked for deletion. The Compactor periodically runs cleanup operations to identify and remove these artifacts. The cleanup process is idempotent and safe to run multiple times, making it resilient to partial failures. Operators should monitor the total number of blocks in the bucket and set up alerts for unexpected growth, which could indicate a compaction failure or a Sidecar uploading duplicate blocks.

For organizations with strict data governance requirements, Thanos supports bucket-level policies that control who can read, write, and delete objects. Cloud provider IAM policies can be configured to grant the Sidecar write-only access to the bucket, the Store Gateway read-only access, and the Compactor full access. This ensures that a compromised Sidecar cannot delete historical data, and a compromised Store Gateway cannot modify or upload new blocks. Audit logging should be enabled on the bucket to track all access patterns and detect potential security issues.

Multi-Tenancy

Multi-tenancy in Thanos is a critical requirement for organizations that need to share a single monitoring infrastructure across multiple teams, business units, or external customers. Unlike dedicated Prometheus deployments where each team runs their own instance, a multi-tenant Thanos deployment allows multiple tenants to share object storage, query infrastructure, and compaction resources while maintaining strict data isolation between tenants. Each tenant's metrics are stored in separate block directories, queried independently, and retained according to their specific requirements.

Thanos implements multi-tenancy through a label-based approach. Each metric is tagged with a tenant identifier (typically a label like tenant_id or __tenant_id__) that is enforced at the ingestion and query layers. At the ingestion side, the Sidecar or Receive component ensures that metrics are uploaded with the correct tenant label, and the object storage path includes the tenant identifier to prevent cross-tenant data leakage. At the query side, the Store Gateway and Query components filter results based on the tenant context, ensuring that each tenant can only access their own data.

The implementation of multi-tenancy typically involves a proxy layer that sits in front of the Thanos Query component and enforces tenant isolation. This proxy intercepts incoming requests, extracts the tenant identifier from a request header (such as X-Tenant-ID), and appends a label matcher to the query that restricts results to the specified tenant. The proxy also handles authentication and authorization, verifying that the requesting user has permission to access the specified tenant's data. This approach is clean because it requires no modifications to the Thanos components themselves — the tenant isolation is enforced entirely at the proxy layer.

YAML
# Multi-Tenant Proxy Configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-multi-tenant-proxy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: thanos-tenant-proxy
  template:
    spec:
      containers:
        - name: proxy
          image: custom/thanos-tenant-proxy:v1.0.0
          env:
            - name: TENANT_HEADER
              value: "X-Tenant-ID"
            - name: UPSTREAM_URL
              value: "http://thanos-query:10902"
            - name: AUTH_ENABLED
              value: "true"
            - name: AUTH_URL
              value: "http://auth-service:8080/verify"
          ports:
            - containerPort: 8080
          resources:
            requests:
              memory: "256Mi"
              cpu: "250m"
---
# Tenant-specific bucket prefixes
# Tenant A blocks: s3://thanos-bucket/tenant-a/
# Tenant B blocks: s3://thanos-bucket/tenant-b/

Object storage organization for multi-tenancy uses a prefix-based approach. Each tenant's blocks are stored under a separate prefix in the bucket (for example, s3://thanos-bucket/tenant-a/ and s3://thanos-bucket/tenant-b/). This organization makes it straightforward to implement per-tenant retention policies, per-tenant storage quotas, and per-tenant access controls. The Compactor can be configured to process each tenant's blocks independently, preventing one tenant's compaction from affecting another tenant's query performance. Storage quotas can be enforced at the cloud provider level using bucket policies or at the application level using the Sidecar's upload validation logic.

Access control for multi-tenant Thanos typically involves three layers: authentication (verifying the identity of the requester), authorization (determining which tenants the requester can access), and data filtering (ensuring that query results are scoped to the authorized tenants). Authentication is typically handled by an identity provider (such as OAuth2 or OIDC), authorization by a policy engine (such as OPA or Casbin), and data filtering by the proxy layer described above. Each layer should be independently configurable and auditable to meet compliance requirements.

A common challenge in multi-tenant Thanos deployments is the "noisy neighbor" problem, where one tenant's high-volume queries or high-cardinality metrics degrade performance for other tenants. This is mitigated through per-tenant resource quotas, query timeouts, and rate limiting at the proxy layer. Each tenant should have a maximum query concurrency, a maximum query time range, and a maximum series count per query. These limits prevent any single tenant from monopolizing shared resources while still allowing legitimate high-volume queries to complete within reasonable time frames.

Query Optimization

Query optimization in Thanos is essential for maintaining acceptable performance as the volume of metrics grows. A naive approach to querying across multiple stores can result in slow queries, high memory usage, and excessive object storage API calls. Thanos provides two primary optimization strategies: vertical splitting (distributing queries across time) and horizontal splitting (distributing queries across stores). These strategies can be applied independently or in combination, and the Query Frontend automates most of the optimization logic.

Vertical splitting divides a query by time range. A query requesting 7 days of data is split into 84 two-hour sub-queries, each of which can be evaluated independently. This is effective because each two-hour sub-query hits a small, well-defined set of blocks, reducing the number of index lookups and chunk reads. The sub-queries are evaluated in parallel across multiple Query component instances (if available), and the results are merged by the Query Frontend before being returned to the client. Vertical splitting also improves cache utilization, as each two-hour sub-query result can be cached independently and reused across different queries that overlap in time.

Horizontal splitting distributes queries across multiple Query component instances, each responsible for a subset of the stores. This is particularly effective in multi-cluster deployments where each Query instance is responsible for a single cluster's stores. A global query is distributed to all regional Query instances, each of which evaluates its portion of the query and returns partial results. The global Query component (or the Query Frontend) merges these partial results into the final response. Horizontal splitting reduces the load on individual Query instances and improves query latency by leveraging parallelism across clusters.

The Query Frontend supports additional optimization techniques including query deduplication (removing overlapping results from multiple stores), response streaming (returning results as they become available rather than waiting for all stores), and query result caching (storing intermediate results in an external cache for reuse). These techniques can be configured independently and should be tuned based on the specific workload characteristics. For example, streaming responses are most beneficial for high-latency queries, while caching is most beneficial for frequently repeated queries.

YAML
# Query Optimization Configuration
# Query Frontend with advanced splitting and caching
apiVersion: v1
kind: ConfigMap
metadata:
  name: thanos-query-config
data:
  query-frontend.yaml: |
    # Vertical splitting: split queries by time into 2-hour chunks
    split_interval: 2h

    # Horizontal splitting: split across multiple Query instances
    horizontal_sharding:
      enabled: true
      shard_count: 4

    # Caching: use Memcached for query results
    cache:
      backend: memcached
      memcached:
        addresses: thanos-memcached:11211
        timeout: 500ms
        max_item_size: 1MB

    # Response optimization
    compress_responses: true
    max_retries_per_request: 3

    # Query logging for performance analysis
    log_queries_longer_than: 10s
    step: 1m

  query.yaml:
    # Query component configuration
    max_concurrent: 40
    timeout: 3m
    lookback_delta: 5m
    auto_downsampling: true

Query performance analysis is a continuous process. The Query component exposes detailed metrics about query execution, including the number of series fetched per store, the total samples processed, the query execution time breakdown (fetching, merging, evaluation), and the number of cached vs. non-cached results. These metrics should be exported to Prometheus and visualized in a dashboard to identify slow queries, underperforming stores, and cache inefficiencies. The Query Frontend's query logging feature (enabled via --query-frontend.log_queries_longer-than) captures detailed information about queries that exceed a configurable threshold, helping operators identify and optimize problematic queries.

Caching Strategies

Caching is one of the most impactful optimizations in a Thanos deployment, capable of reducing query latency by orders of magnitude and significantly lowering object storage API costs. Thanos implements caching at multiple layers, each targeting a different part of the query pipeline. The three primary cache types are index cache (for block index data), bucket cache (for raw object data from the storage backend), and query results cache (for evaluated PromQL results). Understanding when and how to configure each cache type is essential for optimizing Thanos performance.

The index cache is used by the Store Gateway to cache decoded block index headers in memory. Block index headers contain posting lists (inverted index entries mapping label values to series IDs) and symbol tables (mapping string values to numeric references). Decoding these headers is computationally expensive and requires reading the entire header file from object storage. By caching the decoded headers in memory, subsequent queries that need the same block's index can be served instantly without re-downloading or re-decoding. The index cache is especially effective for blocks that are frequently queried, such as blocks containing data for popular services or high-traffic endpoints.

The bucket cache operates at the object storage level, caching the raw bytes of individual objects (chunks, index files, meta.json files) after they are downloaded from the backend. This cache prevents redundant downloads of the same objects across different queries. For example, if two different queries both need data from the same chunk file, the bucket cache ensures that the chunk is downloaded only once from S3 and served from cache for the second query. The bucket cache is implemented as a transparent caching layer that wraps the object storage client, making it invisible to the rest of the Thanos components.

The query results cache is used by the Query Frontend to cache the evaluated results of PromQL queries. When a query is split into sub-queries, each sub-query's result can be cached independently. For repeated queries (such as dashboard refreshes), the cached results are returned directly without re-executing the query against the Query component. The query results cache is the most user-facing cache, as it directly reduces the response time for dashboard loads and API calls. It requires an external cache backend such as Memcached or Redis, as the Query Frontend is stateless and does not persist cached data locally.

Memcached is the most commonly used cache backend for Thanos due to its simplicity, performance, and horizontal scalability. A typical deployment uses a cluster of Memcached instances with consistent hashing to distribute the cache keys evenly across nodes. The Thanos components connect to the Memcached cluster through DNS-based service discovery, which allows Memcached nodes to be added or removed without reconfiguring Thanos. Redis is an alternative that provides additional features such as persistence and pub/sub, but these features are generally unnecessary for Thanos caching use cases where data loss on cache eviction is acceptable.

YAML
# Memcached Deployment for Thanos Caching
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-memcached
spec:
  replicas: 6
  serviceName: thanos-memcached
  selector:
    matchLabels:
      app: thanos-memcached
  template:
    spec:
      containers:
        - name: memcached
          image: memcached:1.6-alpine
          args:
            - '-m'
            - '4096'
            - '-I'
            - '2m'
            - '-c'
            - '4096'
            - '-t'
            - '4'
          ports:
            - containerPort: 11211
          resources:
            requests:
              memory: "4.5Gi"
              cpu: "1"
            limits:
              memory: "4.5Gi"
              cpu: "2"
---
# Store Gateway with Index Cache pointing to Memcached
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-store-gateway
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: store-gateway
          image: quay.io/thanos/thanos:v0.35.0
          args:
            - 'store'
            - '--data-dir=/data'
            - '--objstore.config-file=/etc/thanos/bucket.yml'
            - '--index-cache.config-file=/etc/thanos/index-cache.yml'
            - '--store.grpc.series-max-concurrency=20'

Cache sizing is critical for performance. The index cache should be sized based on the total size of block index headers that are frequently accessed. A rule of thumb is to allocate enough cache memory to hold the index headers for the most recent 1-2 weeks of data. The bucket cache should be sized to hold the most frequently accessed chunk files, which typically represent the most popular services or endpoints. The query results cache should be sized based on the number of unique sub-query results generated by the Query Frontend, which depends on the splitting interval and the number of concurrent dashboards.

Cache invalidation in Thanos is largely handled by TTL-based expiration rather than explicit invalidation signals. Since object storage blocks are immutable once uploaded, cache entries for block metadata and chunks rarely become stale. The primary exception is when the Compactor deletes or replaces blocks during compaction — in this case, the Store Gateway's sync process detects the change and the cache entries for the deleted blocks are naturally evicted. The query results cache uses a shorter TTL (typically 1-5 minutes) to ensure that recently ingested data is reflected in query results without requiring manual cache invalidation.

Security

Security in a Thanos deployment encompasses multiple layers: transport encryption, authentication, authorization, and data protection. Since Thanos components communicate over gRPC and HTTP, and metrics data may contain sensitive information about application behavior and infrastructure state, a comprehensive security posture is essential for production deployments. This section covers the security mechanisms available in Thanos and recommended configurations for enterprise environments.

Transport encryption (TLS) protects data in transit between Thanos components and between clients and the Thanos query endpoint. Thanos supports TLS for both gRPC and HTTP connections. Each Thanos component can be configured with a TLS certificate, private key, and CA certificate to establish mutual TLS (mTLS) connections. mTLS ensures that both the client and server verify each other's identity, preventing man-in-the-middle attacks and unauthorized component impersonation. TLS should be enabled for all inter-component communication, especially in multi-cluster deployments where traffic crosses network boundaries.

Authentication verifies the identity of clients connecting to Thanos endpoints. While Thanos does not have built-in authentication, it can be integrated with external authentication proxies that sit in front of the Query Frontend and enforce authentication policies. Common approaches include using an OAuth2 proxy that validates JWT tokens against an identity provider, or implementing a custom authentication middleware that validates API keys. The authentication proxy should pass the authenticated user's identity to Thanos through HTTP headers, which can then be used for authorization decisions.

Authorization determines what data each authenticated user can access. In a single-tenant deployment, authorization is typically simple — all authenticated users have access to all metrics. In a multi-tenant deployment, authorization is more complex and involves mapping users to tenants and restricting query access accordingly. Authorization can be implemented at the proxy layer using a policy engine like OPA (Open Policy Agent) or at the Thanos level using the multi-tenancy mechanisms described earlier in this guide. Authorization policies should follow the principle of least privilege, granting each user only the access they need for their specific role.

YAML
# TLS Configuration for Thanos Components
# Store Gateway with mTLS
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-store-gateway
spec:
  template:
    spec:
      containers:
        - name: store-gateway
          image: quay.io/thanos/thanos:v0.35.0
          args:
            - 'store'
            - '--grpc-address=0.0.0.0:10901'
            - '--grpc-server-tls-cert=/etc/thanos/tls/server.crt'
            - '--grpc-server-tls-key=/etc/thanos/tls/server.key'
            - '--grpc-server-tls-ca=/etc/thanos/tls/ca.crt'
            - '--objstore.config-file=/etc/thanos/bucket.yml'
          volumeMounts:
            - name: tls-certs
              mountPath: /etc/thanos/tls
              readOnly: true
      volumes:
        - name: tls-certs
          secret:
            secretName: thanos-tls-certs
---
# Query Frontend with HTTPS termination
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: thanos-query-frontend
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/auth-url: "http://auth-proxy:8080/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://auth.example.com/login"
spec:
  tls:
    - hosts:
        - thanos.example.com
      secretName: thanos-tls-secret
  rules:
    - host: thanos.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: thanos-query-frontend
                port:
                  number: 10902

Data protection in Thanos involves securing the object storage bucket where metrics data is stored. The bucket should be encrypted at rest using the cloud provider's server-side encryption (SSE-S3 or SSE-KMS). Access to the bucket should be controlled through IAM policies that grant minimal permissions to each Thanos component. Bucket versioning should be enabled to protect against accidental deletion, and lifecycle policies should be configured to manage version retention. Audit logging should be enabled on the bucket to track all access patterns and detect potential security issues.

Operational security practices for Thanos include regular rotation of TLS certificates and bucket credentials, monitoring of authentication and authorization failures, and periodic security reviews of IAM policies. Secrets should be managed through a secrets management system (such as Kubernetes Secrets with encryption at rest, HashiCorp Vault, or cloud provider Secrets Manager) rather than stored in configuration files or version control. Network policies should be applied to restrict which pods can communicate with Thanos components, following the principle of least network privilege.

Deployment Patterns

Thanos supports several deployment patterns, each optimized for different scale requirements, infrastructure constraints, and operational preferences. The two primary patterns are the Sidecar pattern (where Thanos Sidecar runs alongside Prometheus) and the Receive pattern (where Thanos Receive accepts pushed metrics from Prometheus Remote Write). Within these patterns, additional variations exist for edge computing (minimal resource deployments), gateway architectures (centralized query routing), and hybrid approaches that combine both Sidecar and Receive patterns.

The Sidecar pattern is the most common and recommended starting point for Thanos deployments. In this pattern, each Prometheus instance runs with a co-located Thanos Sidecar that reads the local TSDB data and uploads blocks to object storage. The Sidecar pattern preserves Prometheus's pull-based model, which is well-understood and battle-tested. It also ensures that the Sidecar has direct access to the TSDB directory, enabling efficient block uploads without network overhead. The primary limitation of the Sidecar pattern is that it requires Prometheus to be running and healthy for metrics to be accessible — if a Prometheus instance crashes and loses its local data, the corresponding Sidecar cannot serve data for the gap period (unless the data has already been uploaded to object storage).

The Receive pattern addresses this limitation by accepting metrics via Prometheus Remote Write. In this pattern, Prometheus instances are configured to remote-write their samples to a Thanos Receive endpoint instead of (or in addition to) writing them locally. The Receive component writes the incoming samples to its own TSDB, which can then be queried by the Store Gateway and uploaded to object storage by a Sidecar. The Receive pattern is particularly useful in environments where Prometheus instances have limited local storage, where you want to centralize data ingestion for security or compliance reasons, or where you need to support both pull-based (Prometheus) and push-based (OpenTelemetry Collector, Grafana Agent) metric collection.

The edge pattern is designed for resource-constrained environments such as edge computing sites, retail locations, or remote offices. In this pattern, a minimal Prometheus instance collects metrics locally, and a lightweight Thanos Sidecar uploads them to a central object storage bucket. The central Thanos Query and Store Gateway components handle all querying, while the edge Prometheus instances focus solely on data collection. This pattern minimizes resource requirements at the edge while providing centralized visibility. The edge pattern often uses a bandwidth-efficient upload strategy, such as batching blocks and compressing them before upload, to minimize WAN traffic.

The gateway pattern centralizes the Thanos query layer in a dedicated region or cluster, with all data flowing through a central gateway. This pattern is useful when you need a single query endpoint for all your metrics, regardless of where they originate. The gateway pattern typically deploys multiple Query component replicas behind a load balancer, with regional Sidecars and Store Gateways connecting to the central Query cluster. This centralization simplifies dashboard configuration (all dashboards point to a single endpoint) and enables global deduplication across all regions. The trade-off is increased query latency for remote clusters, as queries must traverse the WAN to reach the central gateway.

A common production deployment combines multiple patterns. For example, a typical multi-cluster Kubernetes deployment uses the Sidecar pattern within each cluster, a regional Query component per cluster for local queries, a global Query component for cross-cluster queries, and a Query Frontend as the single entry point for all dashboards and API consumers. The Receive pattern might be added for non-Kubernetes workloads (such as VMs or bare-metal servers) that cannot run a Prometheus Sidecar. This hybrid approach provides maximum flexibility while maintaining a clean, well-understood architecture.

Comparison with Cortex, Mimir, VictoriaMetrics

The Thanos ecosystem exists within a broader landscape of horizontally scalable, long-term Prometheus-compatible metrics systems. Understanding how Thanos compares to its alternatives — Cortex, Grafana Mimir, and VictoriaMetrics — is essential for making informed architectural decisions. While all four systems aim to solve similar problems (horizontal scalability, long-term storage, multi-cluster querying), they differ significantly in their architecture, operational model, and trade-offs.

Cortex is the original project that pioneered the concept of a distributed Prometheus-compatible metrics system. Developed primarily by Grafana Labs and the community, Cortex uses a push-based model where Prometheus instances remote-write their samples to a central Cortex cluster. Cortex stores data in a combination of object storage (for blocks) and a key-value store (for ruler and alertmanager configurations). Cortex provides multi-tenancy as a first-class feature, with tenant isolation enforced at every layer. However, Cortex has been effectively superseded by Grafana Mimir, which is its successor with improved performance, better operational simplicity, and the same architectural foundations.

Grafana Mimir is the evolution of Cortex, rewritten with a focus on operational simplicity and performance. Mimir uses the same push-based model as Cortex but introduces significant improvements including a unified storage engine based on TSDB, horizontal scalability for the ruler and alertmanager components, and built-in support for exemplars and native histograms. Mimir is tightly integrated with Grafana Cloud and Grafana's commercial offerings, which provides a seamless experience for organizations already using Grafana. However, Mimir's tight coupling with the Grafana ecosystem may be a concern for organizations that prefer vendor-neutral solutions.

VictoriaMetrics takes a different approach from Thanos, Cortex, and Mimir. It is a complete, standalone metrics system written in Go with its own storage engine, ingestion protocol, and query language (MetricsQL, which is a superset of PromQL). VictoriaMetrics supports both pull-based scraping and push-based ingestion (via Prometheus Remote Write, InfluxDB line protocol, and OpenTelemetry). Its storage engine is optimized for compression and query performance, achieving 10x better compression than Prometheus TSDB. VictoriaMetrics can also read Prometheus data from remote storage and reindex it, making it a potential replacement for existing Prometheus deployments.

Choosing between these systems depends on several factors. Choose Thanos if you prefer the pull-based Prometheus model, want maximum composability and vendor independence, and are comfortable managing multiple components. Thanos's sidecar pattern means you can adopt it incrementally, starting with a single cluster and expanding to multi-cluster without architectural changes. Choose Grafana Mimir if you are already deeply invested in the Grafana ecosystem, prefer push-based ingestion, and want a tightly integrated experience with Grafana Cloud. Mimir's built-in multi-tenancy and horizontal scalability make it a strong choice for large-scale SaaS platforms. Choose VictoriaMetrics if you want a simpler operational model with a single binary, need excellent compression for cost savings, or are running on-premises where the lack of cloud dependencies is an advantage.

In practice, the choice often comes down to operational preferences and existing infrastructure. Teams that already run Prometheus with a pull-based model will find Thanos the most natural fit, as it requires minimal changes to existing configurations. Teams that are building new observability platforms from scratch may prefer Mimir or VictoriaMetrics for their more streamlined operational models. Regardless of the choice, all four systems have proven track records in production environments at significant scale, and the differences between them are often less impactful than the quality of the operational practices applied to them.

Interview Q&A

The following questions are designed to test deep understanding of Thanos architecture and are commonly asked in senior platform engineer and SRE interviews. Each answer provides the key technical details that interviewers expect at a senior+ level.

Q1: How does Thanos achieve high availability for Prometheus, and what are the trade-offs?

Thanos achieves HA by running two or more Prometheus instances that scrape the same targets, with each instance having a co-located Sidecar. The Query component connects to both Sidecars, fetches results from both, and deduplicates them using a configured replica label (such as prometheus_replica). The deduplication merges overlapping time series into a single result, with the last-sample-wins strategy by default. The trade-off is increased resource consumption (double the Prometheus instances, double the Sidecar resources, double the object storage uploads) and eventual consistency — during a brief window after one replica writes a sample but before the other, the deduplicated result may temporarily show only one replica's data. Additionally, deduplication adds latency to query execution, as the Query component must collect and merge results from multiple stores before returning.

Q2: Explain the Thanos block lifecycle from creation to deletion.

A block is created by Prometheus TSDB every 2 hours. The block contains chunk files (raw time series data), an index file (posting lists and symbol table), a meta.json (block metadata), and a tombstone file (deleted series). The Thanos Sidecar detects completed blocks and uploads them to object storage, streaming each file individually with meta.json uploaded last for atomicity. The Store Gateway discovers new blocks during its periodic sync (every 3-5 minutes) and adds them to its in-memory index. The Compactor periodically merges small blocks into larger ones (down to ~512MB), creates downsampled copies at 5m and 1h resolution, and deletes blocks older than the configured retention period. Deleted blocks are soft-deleted (using S3 delete markers) for a configurable delay period before hard deletion, providing a recovery window for accidental deletions.

Q3: How does query fan-out work in Thanos, and what factors affect its performance?

When the Query component receives a PromQL query, it determines which stores can contribute data based on the requested time range and each store's reported metadata (minimum/maximum timestamps and label sets). It sends sub-queries to all relevant stores in parallel, collects partial results as they arrive, merges overlapping series, and deduplicates using the replica label. Performance factors include: the number of stores (more stores = more parallel connections), the latency of each store (a slow store delays the entire query unless partial response is enabled), the volume of data per store (high cardinality = more data to transfer and merge), and the complexity of the PromQL expression (aggregations over many series are more expensive than simple lookups). The Query component's concurrency limit (--query.max-concurrent) and timeout (--query.timeout) directly bound query performance.

Q4: What is the difference between the Sidecar and Receive patterns, and when would you choose each?

The Sidecar pattern runs alongside Prometheus, reads its local TSDB data, and uploads blocks to object storage. It preserves Prometheus's pull-based model and provides the lowest latency for querying recent data (since data is served directly from the local volume). Choose Sidecar when you have standard Kubernetes deployments with Prometheus running as a pod and want minimal architectural changes. The Receive pattern accepts pushed metrics via Prometheus Remote Write, writes them to its own TSDB, and makes them available through the StoreAPI. Choose Receive when you cannot run Prometheus in pull mode (e.g., network-restricted environments), need to support push-based collectors (OpenTelemetry, Grafana Agent), want to centralize ingestion for compliance, or need a unified write endpoint for multiple Prometheus clusters. The Receive pattern adds operational complexity (managing Receive TSDBs, ensuring high availability of the Receive endpoints) but provides more flexibility in ingestion.

Q5: How does the Thanos Compactor prevent data corruption during compaction?

The Compactor uses several mechanisms to prevent corruption: (1) Single-instance execution with leader election — only one Compactor instance actively compacts at a time, preventing race conditions. (2) Consistency delay — the Compactor waits a configurable period before processing recently uploaded blocks, ensuring that blocks are fully uploaded and stable. (3) Atomic block replacement — during compaction, the Compactor uploads the merged block first and only deletes the source blocks after the upload succeeds. (4) Soft-delete with delay — source blocks are soft-deleted (using delete markers) rather than immediately removed, providing a recovery window. (5) Idempotent operations — all Compactor operations are designed to be safe to retry, so partial failures can be recovered by restarting the Compactor. (6) Metadata validation — the Compactor validates block metadata before and after compaction, detecting any inconsistencies that might indicate corruption.

Q6: Design a Thanos deployment for a company with 5 Kubernetes clusters across 3 regions.

The deployment would use the Sidecar pattern within each cluster, with 2 Prometheus replicas per cluster for HA (total 10 Prometheus instances). Each cluster has a regional Thanos Query component connected to its local Sidecars, providing cluster-specific queries with low latency. A global Thanos Query component connects to all regional Query components, providing cross-cluster queries. A Query Frontend sits in front of the global Query as the single entry point for Grafana and API consumers. Each cluster has a Store Gateway instance for local historical data, and a shared Compactor runs in one region (with leader election standby in another). Object storage uses a single S3 bucket with the Compactor handling all regions' data. Memcached clusters in each region provide caching for the Store Gateway and Query Frontend. TLS is enabled for all inter-component communication, and the Query Frontend is exposed through an authenticated Ingress.

Q7: Explain the caching architecture in Thanos and how to tune cache sizes.

Thanos uses four cache types: Index Cache (decoded block index headers in Store Gateway memory, sized based on the total index header size of frequently queried blocks), Bucket Cache (raw object bytes in Store Gateway, sized to hold the most frequently accessed chunks), Series Cache (decoded time series in Store Gateway, sized based on typical query cardinality), and Query Results Cache (PromQL results in Query Frontend, backed by Memcached, sized based on the number of unique sub-query results). Cache sizes are tuned by monitoring hit rates: if the hit rate is below 80%, increase the cache size; if above 95%, you may be over-provisioning. The index cache should be the largest, as index headers are the most expensive to re-download. The query results cache provides the highest user-visible impact, as it can serve entire dashboard queries from cache.

Q8: How does Thanos handle cross-cluster network partitions?

Thanos is designed to be resilient to network partitions through several mechanisms. The Query component supports partial response mode, where it returns results from available stores and indicates which stores were unreachable. This means that a partition affecting one cluster does not block queries for data from other clusters. The Sidecar continues uploading blocks to object storage locally (assuming the object storage endpoint is reachable), so data is not lost during a partition. When the partition heals, the Store Gateway syncs newly uploaded blocks and makes them queryable. The Compactor may delay processing blocks from the partitioned cluster until they become visible. For HA Query components, regional Query instances continue serving local queries, and the global Query degrades gracefully by omitting results from unreachable regions.

Q9: What metrics should you monitor to ensure Thanos is healthy?

Key metrics to monitor include: thanos_sidecar_uploaded_blocks_total (block upload rate from Sidecars), thanos_bucket_store_blocks_loaded (blocks loaded in Store Gateway memory), thanos_bucket_store_series_data_fetched (chunks fetched per query), thanos_query_gate_queries_in_flight (concurrent queries in the Query component), thanos_compact_group_compactions_failures_total (compaction failures), thanos_store_index_cache_hits_total / misses_total (cache hit ratio), thanos_query_frontend_cache_hits_total (query results cache hits), thanos_receive_requests_total (if using Receive), and thanos_shipper_uploads_failed_total (upload failures). Alert on: zero block uploads for > 30 minutes (Sidecar issue), compaction failures > 0 (data integrity risk), cache hit ratio < 70% (performance degradation), and query timeout rate > 5% (capacity issue).

Q10: How would you migrate from a single Prometheus instance to Thanos without downtime?

The migration follows a phased approach. Phase 1: Deploy a second Prometheus instance scraping the same targets alongside the first, with both writing to separate storage volumes. This establishes HA at the Prometheus level without Thanos. Phase 2: Add Thanos Sidecar to both Prometheus instances, pointing them at the same object storage bucket but with different block upload configurations to avoid conflicts. Phase 3: Deploy a Thanos Query component connected to both Sidecars, with the replica label configured for deduplication. Phase 4: Update Grafana dashboards and API consumers to query the Thanos Query endpoint instead of individual Prometheus endpoints. Phase 5: Configure the Query Frontend as the single entry point with caching and query splitting. Phase 6: Deploy the Compactor for long-term block management. Phase 7: Gradually extend Prometheus local retention down to a few hours (since Thanos now handles long-term storage) and tune the Compactor's retention policies. Each phase can be validated independently before proceeding, ensuring zero-downtime migration.