Possible Cause
The possible causes are as follows:
- Shards were unevenly distributed in previous index allocations, and the predominate parameter in the latest indexed shard allocation was balance.shard. To balance the shard distribution across nodes, the new shards were allocated to the node with only a small number of shards.
- After a new node was added to a cluster and before the automatic cluster rebalancing completes, the predominate parameter was balance.shard. The shards of a new index are allocated to the new node, where there are no shards yet.
The following two parameters are used to balance the shard allocation in a cluster:
cluster.routing.allocation.balance.index (default value: 0.45f)
cluster.routing.allocation.balance.shard (default value: 0.55f)
- balance.index: A larger value indicates that all the shards of an index are more evenly distributed across nodes. For example, if an index has six shards and there are three data nodes, two shards will be distributed on each node.
- balance.shard: A larger value indicates that all the shards of all the indexes are more evenly distributed across nodes. For example, if index a has two shards, index b has four, and there are three data nodes, two shards will be distributed on each node.
- You can specify both balance.index and balance.shard to balance the shard allocation.
Solution
To prevent the all the shards of an index from being allocated to a single node, use either of the following methods:
- To create an index during cluster scale-out, configure the following parameter:
"index.routing.allocation.total_shards_per_node": 2
That is, allow no more than two shards of an index to be allocated on each node. Determine the maximum number of shards allocated to each node based on the number of data nodes in your cluster and the number of index shards (both primary and secondary).
- If too many shards are distributed on only a few nodes, you can move some of the shards to other nodes to balance the distribution. Run the move command of POST _cluster/reroute. The rebalance module will automatically exchange the shard with a shard on the destination node. Determine the values of balance.index and balance.shard as needed.