Page MenuHomePhabricator

unattended elasticsearch restarts
Closed, ResolvedPublic

Description

salt call should be enough, done serially, context:

16:38  <^d> godog: I wonder if we could come up with some script (using salt maybe?) that could make it completely 
            automated?
16:38  <^d> Or at least very hands-off
16:38  <godog> ^d: we most certainly can, haven't got around it but basically yeah a "serial salt" call would do it
16:38  <^d> Yeah, that's my thought
16:39  <^d> es-tool does the hard work, so salt could just call that probably
16:39  <godog> yep, one thing to think about is how to safely stop salt so that it simply doesn't go to the next host 
               and call es-tool
16:39  <godog> and the next, and the next

Event Timeline

fgiunchedi claimed this task.
fgiunchedi raised the priority of this task from to Medium.
fgiunchedi updated the task description. (Show Details)
fgiunchedi subscribed.

restart is easily handled by es-tool for single node (see https://gerrit.wikimedia.org/r/#/c/164401/) orchestration across hosts is still missing and can be handled by salt for example

Restricted Application added a subscriber: Matanya. · View Herald Transcript

after upgrade to 1.6 this should also take way less time

chasemp subscribed.

I'm not sure what this ticket entails at this point. We know we have some issues with mass cluster update and I imagine those get dealt with in T109089 and this follows but for now this issue isn't tracking any work I can quantify.

the idea was to track work towards having unattended cluster restarts as much as possible, at this point I don't know where it stands though