More like a sketch than a full on solution.
It would be quite valuable for network maintenance and outage response to have a cookbook allowing to depool all servers in a given rack.
Full row is not necessary as we're moving toward a per rack redundancy model (and we could in extreme cases run it multiple times).
I think most of the building blocks are there (restart cookbooks, restart doc), and what's needed is the glue between them.
An idea is to have per team or server types cookbooks, that are themselves called by this meta cookbook. Hosts with no downtime cookbook would be listed in the script output.
Many hosts also have a depool command line tool, that takes care of all the depool action, for such hosts the cookbook could do it via cumin (which would only work for maintenance and not when the rack is down).
Cookbook could also check etcd if it manages that host's pooled status.
Current status.
I've spent a bit more time on it, and the current (still WIP) cookbook is on Gerrit : https://gerrit.wikimedia.org/r/c/1239896
The cookbook requires Hiera definitions to tie a pool or depool action to a Puppet role. You can see examples in this CR: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1249958
This will not try to depool the host, it will display: skipping host (no depool needed). It's also possible to specify a message that will be displayed.
profile::server_depool: policy: skip
This will (eventually) connect to the host and run the local "depool" command. And display it when running it as --show
profile::server_depool: policy: local_command command: depool
Some additional information :
- There is also the possibility to call a 3rd party cookbook
- k8s hosts will be handled a bit differently as they already have a cookbook to depool a full rack.
- The same key have a profile::server_pool equivalent
- If the Hiera key is not present, it will show : Couldn't get or parse depool Hiera key
- If the --teams parameter is used, it will group the hosts by teams (see full run example bellow).
cumin1003:~$ time test-cookbook -c 1239896 --no-sal-logging sre.network.depool-rack --site eqiad --rack D7 --show --team depool [...] an-master1004: Couldn't get or parse depool Hiera key an-worker1152: skipping host (no depool needed) an-worker1232: skipping host (no depool needed) aux-k8s-worker1007: Couldn't get or parse depool Hiera key backup1007: Couldn't get or parse depool Hiera key cirrussearch1120: Couldn't get or parse depool Hiera key cirrussearch1121: Couldn't get or parse depool Hiera key cirrussearch1122: Couldn't get or parse depool Hiera key dbprov1004: Couldn't get or parse depool Hiera key dse-k8s-worker1021: Couldn't get or parse depool Hiera key ganeti1051: skipping host (Use sre.ganeti.drain-node, primaries need to be failed-over too) ganeti1052: skipping host (Use sre.ganeti.drain-node, primaries need to be failed-over too) logging-hd1003: skipping host (No cookbook, no depool needed but there's a switch we can flip to mitigate the churn caused when the cluster detects a down node) lvs1020: Couldn't get or parse depool Hiera key mc-gp1006: Couldn't get or parse depool Hiera key ms-be1093: skipping host (Can't be depooled, need to go down one at a time with special care) prometheus1007: skipping host (doesn't require any depooling) wikikube-worker1370: skipping host (insetup role) ['Data Platform']: ['an-master1004', 'an-worker1152', 'an-worker1232', 'dse-k8s-worker1021'] ['Infrastructure Foundations']: ['aux-k8s-worker1007', 'ganeti1051', 'ganeti1052'] ['Data Persistence']: ['backup1007', 'dbprov1004', 'ms-be1093'] ['Search Platform']: ['cirrussearch1120', 'cirrussearch1121', 'cirrussearch1122'] ['Observability']: ['logging-hd1003', 'prometheus1007'] ['Traffic']: ['lvs1020'] ['ServiceOps']: ['mc-gp1006', 'wikikube-worker1370'] real 7m34.104s
In its current state, only in show mode, the cookbook has been useful to see what can and can't be depooled easily.
Next steps are to validate the Hiera keys and overall data structure and get SRE participation to cover as many servers roles as possible.
Then continue improving the cookbook :
- Add full k8s support
- Add the possibility to update a task
- Add the possibility to downtime all the servers
- Test thoroughly (both pool and depool actions)
- Longer term: tackle servers that are connected to the ToR switches instead of the rack (for special cases like E12 rack).