Elasticsearch Cluster Setup for Web Applications

Elasticsearch Cluster Setup for Web Applications

When moving from a single Elasticsearch instance to a cluster, you often run into non‑obvious problems: split‑brain, unbalanced shard distribution, memory leaks. We've prepared a step‑by‑step guide for setting up a production cluster on 3 nodes — from role configuration to ILM. Our experience shows that a properly configured cluster pays for itself after just a month under load.

Why a Three‑Node Cluster?

A single node is only suitable for development. Production requires a cluster for fault tolerance, horizontal scaling, and load isolation. A 3‑node cluster is 10 times more reliable than a single instance: it survives the loss of one node without data loss and handles parallel queries efficiently. For most web applications, this is the optimal balance of cost and performance.

Node Roles

In Elasticsearch 8.x each node can perform multiple roles. For small clusters (3–5 nodes) all nodes typically combine all roles. For clusters of 10+ nodes, separation is mandatory.

Master‑eligible node – participates in master election, manages cluster state. Minimum 3 master‑eligible nodes for quorum.

Data node – stores shards, executes searches. Most resource‑intensive: needs fast disks (NVMe) and plenty of RAM for JVM heap and OS file cache.

Coordinating node (client node) – receives client requests, distributes to data nodes, collects results. Relieves data nodes during the scatter‑gather phase.

Ingest node – processes documents through ingest pipelines (parsing, enrichment, transformation). Role configuration in elasticsearch.yml:

# Master-only node node.roles: [ master ] # Data node node.roles: [ data, data_content, data_hot, data_warm, data_cold ] # Coordinating only node.roles: [] 

Minimum Production Configuration: 3 Nodes

All three nodes are master‑eligible + data. This provides quorum (2 out of 3) and data storage. elasticsearch.yml for node 1:

cluster.name: myapp-prod node.name: es-node-01 node.roles: [master, data, ingest] network.host: 0.0.0.0 http.port: 9200 transport.port: 9300 discovery.seed_hosts: - es-node-01:9300 - es-node-02:9300 - es-node-03:9300 cluster.initial_master_nodes: - es-node-01 - es-node-02 - es-node-03 path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.keystore.path: elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: elastic-certificates.p12 

On nodes 2 and 3 only node.name changes.

cluster.initial_master_nodes is used only on the first cluster boot. After the cluster forms, comment out this line — otherwise a restart might cause split‑brain.

JVM and Memory

Elasticsearch defaults to 1 GB heap — critically low for production. Rule: heap = half of available RAM, but not more than 31 GB (above 32 GB JVM loses compressed oops).

In /etc/elasticsearch/jvm.options.d/heap.options:

-Xms16g -Xmx16g 

The remaining memory goes to OS file cache — Elasticsearch uses mmap heavily for reading Lucene segments. On a 64 GB RAM server: 31 GB heap + 30+ GB OS cache is optimal.

System tuning:

# /etc/sysctl.conf vm.max_map_count=262144 vm.swappiness=1 # /etc/security/limits.conf elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited elasticsearch soft nofile 65536 elasticsearch hard nofile 65536 

TLS Certificate Generation and Security Setup

# Generate CA and cluster certificates /usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /etc/elasticsearch/elastic-ca.p12 /usr/share/elasticsearch/bin/elasticsearch-certutil cert \ --ca /etc/elasticsearch/elastic-ca.p12 \ --out /etc/elasticsearch/elastic-certificates.p12 # Set elastic user password /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto 

HTTP TLS (for client connections) uses a separate certificate:

xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: http.p12 

How to Configure ILM to Save Resources?

For logs and temporary data, an ILM policy is mandatory. Without it, indices grow forever and fill the disk. Example policy with four phases:

PUT _ilm/policy/logs-policy { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_age": "7d", "max_size": "50gb" }, "set_priority": { "priority": 100 } } }, "warm": { "min_age": "7d", "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 }, "set_priority": { "priority": 50 } } }, "cold": { "min_age": "30d", "actions": { "freeze": {}, "set_priority": { "priority": 0 } } }, "delete": { "min_age": "90d", "actions": { "delete": {} } } } } } 

Step‑by‑Step Cluster Setup

1. Analysis: determine load, data volume, required node roles.
2. Design: choose number of nodes, disk size, JVM settings.
3. Installation: deploy nodes on servers or containers, configure network.
4. Security configuration: generate TLS, set passwords.
5. ILM and templates: create index lifecycle policies.
6. Testing: verify cluster health, shard distribution, fault tolerance.
7. Deployment: connect the application, enable monitoring via Kibana.

What’s Included in Turnkey Setup

• Cluster deployment on bare metal or cloud
• TLS and basic security configuration
• ILM policy and index template setup
• Integration with Kibana for monitoring
• Operations documentation
• Team training on basic administration
• 2 weeks of post‑launch support

Checking Cluster Health

# Cluster status (green/yellow/red) curl -u elastic:changeme http://localhost:9200/_cluster/health?pretty # Node list curl -u elastic:changeme http://localhost:9200/_cat/nodes?v # Unassigned shards and reasons curl -u elastic:changeme "http://localhost:9200/_cluster/allocation/explain?pretty" 

Status yellow means all primary shards are assigned but some replicas are not. On a 1‑node cluster this is normal. On a 3‑node cluster yellow indicates a problem.

Connecting from an Application

use Elastic\Elasticsearch\ClientBuilder; $client = ClientBuilder::create() ->setHosts(['https://es-node-01:9200', 'https://es-node-02:9200', 'https://es-node-03:9200']) ->setBasicAuthentication('elastic', 'changeme') ->setCABundle('/path/to/ca.crt') ->build(); 

from elasticsearch import Elasticsearch es = Elasticsearch( ['https://es-node-01:9200', 'https://es-node-02:9200'], basic_auth=('elastic', 'changeme'), ca_certs='/path/to/ca.crt', retry_on_timeout=True, max_retries=3, )