An Elasticsearch data node is experiencing high load and yet may even have a lower shard count than other nodes in the cluster. How does this scenario happen?

I follow a rollover pattern in my cluster, which means the majority of my indices are actually inactive when it comes to write operations. Not all shards imply an equal amount of load on a node in this case. The shard of an active write index will induce a much higher load than one that is only used in searches.

What happens when not all shards are created equal?

Elasticsearch’s default settings will attempt to keep shards balanced across data nodes in the cluster. But it balances only using shard count as a metric.

Common Scenario: a node leaves the cluster and rejoins. When it rejoins, all of its shards have been allocated to other nodes. Now, the node that left and rejoined has very few shards. This also occurs when a brand new node is added. Elasticsearch will automatically rebalance and send shards from other nodes to maintain a similar shard count per node. This may take quite some time depending on the shard sizes and how many concurrent recoveries are allowed in the cluster settings (2 by default).

If index creation happens frequently, through rollovers or other means, all new “active” indices will schedule their shards on the node with the lowest shard count to help balance out the cluster. This is undesirable because all the shards being allocated to the node are extremely load-heavy shards, and the cluster has resources allocated assuming the load will be spread across all the data nodes.

For example, let’s say each of 20 data nodes houses 200 shards. Of these 200 shards about 10% are active for writes. An index with 10 shards and 1 replica gets created and it puts 1 shard on each of the data nodes as they are already balanced. However if 19 of the 20 data nodes have 200 shards and one of them has 50, then it will allocate all 20 shards from the index creation to the node with the 50 shard count. Now, from the cluster’s perspective the data node has 70 shards and is still underutilized. It continues to schedules new shards there until it reaches 200. Unfortunately, 150 of the 200 shards are extremely active and load-heavy. So, naturally, this node will be under immense load (75% active shards) compared to all other data nodes in the cluster (10% active shards).

How can we achieve better shard balance?

One way to maintain balance, even if a node leaves like the scenario above, is to enforce a limit on the number of shards one index can put on a single node. Using an index template like the one below:

{ “order”: 1, “index_patterns”: [“*”], “settings”: {“index”: { “routing”:{ “allocation”: {“total_shards_per_node”:”1″}}}}

The downsides of this template example is that you need enough data nodes to contain only 1 shard from each index, and to be tolerant of nodes occasionally leaving. If unassigned shards have no where to go because every node already has 1 of their index’s shards, then the cluster will go into YELLOW state.

To reduce the number of data nodes needed, you can increase the shards per node in the template but allocate more resources to the nodes to handle a higher number of active shards.