I noticed some alerts in icinga today about shards not allocated for some logstash100x nodes, this is the status:
```
elukey@logstash1008:~$ curl -s http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "production-logstash-eqiad",
"status" : "yellow", <======================================
"timed_out" : false,
"number_of_nodes" : 6,
"number_of_data_nodes" : 3,
"active_primary_shards" : 483,
"active_shards" : 1103,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 18, <================================
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 98.39429081177519
}
```
Some shards are not assigned:
```
elukey@logstash1008:~$ curl -s 'http://localhost:9200/_cat/shards' | grep -v STARTED | sort
logstash-2020.04.13 0 r UNASSIGNED
logstash-2020.04.14 0 r UNASSIGNED
logstash-2020.04.14 0 r UNASSIGNED
logstash-deploy-2020.04.12 0 r UNASSIGNED
logstash-deploy-2020.04.13 0 r UNASSIGNED
logstash-deploy-2020.04.14 0 r UNASSIGNED
logstash-deploy-2020.04.14 0 r UNASSIGNED
logstash-mediawiki-2020.04.12 0 r UNASSIGNED
logstash-mediawiki-2020.04.13 0 r UNASSIGNED
logstash-mediawiki-2020.04.14 0 r UNASSIGNED
logstash-mediawiki-2020.04.14 0 r UNASSIGNED
logstash-restbase-2020.04.13 0 r UNASSIGNED
logstash-restbase-2020.04.14 0 r UNASSIGNED
logstash-restbase-2020.04.14 0 r UNASSIGNED
logstash-syslog-2020.04.12 0 r UNASSIGNED
logstash-syslog-2020.04.13 0 r UNASSIGNED
logstash-syslog-2020.04.14 0 r UNASSIGNED
logstash-syslog-2020.04.14 0 r UNASSIGNED
```
With `_cluster/allocation/explain?pretty` I can see the following two reasons:
* "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[logstash-2020.04.14][0]
* "explanation" : "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=0.91], using more disk space than the maximum allowed [91.0%], actual free: [8.086810601726357%]"
This is the cluster allocation status:
```
elukey@logstash1008:~$ curl -s -k 'http://localhost:9200/_cat/allocation?v'
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
377 7.5tb 13.2tb 1.1tb 14.3tb 92 10.64.0.181 10.64.0.181 logstash1010-production-logstash-eqiad
358 6.5tb 12.2tb 2tb 14.3tb 85 10.64.48.177 10.64.48.177 logstash1012-production-logstash-eqiad
368 7.4tb 13.2tb 1.1tb 14.3tb 91 10.64.16.30 10.64.16.30 logstash1011-production-logstash-eqiad
18 UNASSIGNED
```
From `_cluster/settings` this is the current config:
```
"disk" : {
"watermark" : {
"low" : "0.91",
"high" : "0.93"
```