I will keep the log files in my home under T108180.
At a glance there's a lot of instability when a node leave, some nodes have some difficulties to stay in the cluster.
Some ping response (zend discovery) have a 90sec delay and are considered timed-out.

Maybe the fast recovery cannot work with these conditions.

Best bet so far: the master and/or the restarted node is not able to handle all cluster state communication in time.
Replicas are starting to be promoted as primary and new replicas are created on other nodes. This explain why the cluster needs to rebalance, this seems to be confirmed by ganglia by looking at the free disk space : https://ganglia.wikimedia.org/latest/?r=custom&cs=08%2F04%2F2015+00%3A00&ce=08%2F07%2F2015+00%3A00&c=Elasticsearch+cluster+eqiad&h=&tab=m&vn=&hide-hf=false&m=disk_free&sh=1&z=small&hc=4&host_regex=&max_graphs=0&s=by+name

The peaks seems to coincide with restarts.

@dcausse Can you give some more information on the status of this task?

One thing i noticed that may help reduce the time it takes to shuffle is the elasticsearch optimize API. I added this ticket to investigate: https://phabricator.wikimedia.org/T110547

Stalled for now, there is a lot of stuff to do listed in https://phabricator.wikimedia.org/T109089
Maybe we could just remove this task from the sprint?
I don't think I'll be able to find a solution any time soon :(

In T108180#1580560, @dcausse wrote:

Stalled for now, there is a lot of stuff to do listed in https://phabricator.wikimedia.org/T109089
Maybe we could just remove this task from the sprint?
I don't think I'll be able to find a solution any time soon :(

Thanks for the update! Will take it out of the sprint.