Some notes of a discussion with @EBernhardson and @dcausse:

• Our most promising option is probably tuning delayed allocation. Delayed allocation needs to be long enough to allow shards to fully recover before being moved, but short enough so that if a node is down for real, shards are relocated. Raising it to 10 minutes is probably a good start.
• Freezing writes and flushing indices before restarting a node should help. This step was removed from the restart procedure as it did not improve the situation, we need to revisit this.
• Tuning of initial recovery settings (number of recoveries, throttling, ...) might help.
• While restarting a full row at a time is carries a large risk, and probably some performance impact on reads, restarting 2 nodes in the same row at the same time should be safe. We need some experimentation to find the optimal number of nodes to restart at the same time.

Mentioned in SAL (#wikimedia-operations) [2016-12-05T10:27:41Z] <gehel> enabling trace logging on indices recovery on elasticsearch codfw - T145065

In the latest cluster restart, we did manage to restart a few nodes in < 3 minutes when writes are disabled. There is evidence that writes are happening in the cluster during write freeze (the sync_id changes while writes are frozen). The current scripts have a lot of wait time to ensure writes are processed between node restarts, this needs to be improved to actually gain time during cluster restarts.