Elasticsearch in Production: A Practical Operations Guide

Elasticsearch powers search at GitHub, Wikipedia, and Shopify. Production operation requires understanding characteristics that differ significantly from traditional databases.

Index Mapping Design

Unlike databases, changing Elasticsearch mappings on existing indices requires full reindexing. Get it right upfront:

PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "index.refresh_interval": "30s",
    "analysis": {
      "analyzer": {
        "product_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "stop", "snowball", "synonym_filter"]
        }
      },
      "filter": {
        "synonym_filter": {
          "type": "synonym",
          "synonyms": ["iphone,apple phone => phone", "laptop,notebook,computer"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "product_id": { "type": "keyword" },
      "name": {
        "type": "text",
        "analyzer": "product_analyzer",
        "fields": {
          "keyword": { "type": "keyword" },
          "completion": { "type": "completion" }
        }
      },
      "price": { "type": "scaled_float", "scaling_factor": 100 },
      "category": { "type": "keyword" },
      "in_stock": { "type": "boolean" },
      "embedding": {
        "type": "dense_vector",
        "dims": 768,
        "index": true,
        "similarity": "cosine"
      }
    }
  }
}

Index Lifecycle Management (ILM)

Automate rollover, optimization, and deletion for time-series indices:

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": { "max_size": "50gb", "max_age": "1d", "max_docs": 100000000 }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "cold": { "min_age": "30d", "actions": { "freeze": {} } },
      "delete": { "min_age": "90d", "actions": { "delete": {} } }
    }
  }
}

Search Relevance Tuning

GET /products/_search
{
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": [{
            "multi_match": {
              "query": "wireless headphones",
              "fields": ["name^3", "description^1", "tags^2"],
              "type": "best_fields",
              "fuzziness": "AUTO",
              "minimum_should_match": "75%"
            }
          }],
          "filter": [
            { "term": { "in_stock": true } },
            { "range": { "price": { "gte": 20, "lte": 500 } } }
          ]
        }
      },
      "functions": [
        { "gauss": { "rating": { "origin": 5.0, "scale": 2.0, "decay": 0.5 } }, "weight": 2 },
        { "gauss": { "created_at": { "origin": "now", "scale": "30d", "decay": 0.5 } }, "weight": 0.5 }
      ],
      "score_mode": "multiply"
    }
  }
}

Hybrid Search: BM25 + Vector Embeddings

from elasticsearch import Elasticsearch
from sentence_transformers import SentenceTransformer

es = Elasticsearch("https://localhost:9200")
model = SentenceTransformer('all-MiniLM-L6-v2')

def semantic_search(query: str, size: int = 10):
    embedding = model.encode(query).tolist()
    return es.search(
        index='products',
        body={
            "query": {
                "bool": {
                    "should": [{
                        "multi_match": {
                            "query": query,
                            "fields": ["name^2", "description"],
                            "boost": 0.7
                        }
                    }]
                }
            },
            "knn": {
                "field": "embedding",
                "query_vector": embedding,
                "k": 10,
                "num_candidates": 100,
                "boost": 0.3
            },
            "size": size
        }
    )["hits"]["hits"]

Bulk Indexing

from elasticsearch.helpers import parallel_bulk

def generate_actions(products):
    for p in products:
        yield {"_index": "products", "_id": p["product_id"], "_source": p}

for ok, action in parallel_bulk(
    es, generate_actions(products),
    thread_count=4, chunk_size=1000,
    max_chunk_bytes=10 * 1024 * 1024
):
    if not ok:
        handle_error(action)

Cluster Health Monitoring

GET /_cluster/health
# green: all good | yellow: replica unassigned | red: data loss risk

GET /_cat/nodes?v&h=name,heapPercent,cpu,load_1m
# heapPercent > 85%: GC pressure warning

# Fix unassigned replicas (single-node dev)
PUT /my_index/_settings
{ "number_of_replicas": 0 }

# Explain allocation problems
GET /_cluster/allocation/explain

# Circuit breaker (prevents OOM)
PUT /_cluster/settings
{
  "persistent": { "indices.breaker.total.limit": "70%" }
}

Slow Log

PUT /my_index/_settings
{
  "index.search.slowlog.threshold.query.warn": "5s",
  "index.search.slowlog.threshold.query.info": "1s",
  "index.search.slowlog.threshold.fetch.warn": "500ms"
}

Forcemerge for Read-Only Indices

POST /my_old_index/_forcemerge?max_num_segments=1
# Reduces segment count, reclaims disk space for cold indices

Elasticsearch clusters that run smoothly are those where teams define mappings upfront, implement ILM for time-series, tune relevance for their domain, and monitor heap usage proactively.