Event Timeline
Mentioned in SAL (#wikimedia-operations) [2021-05-20T05:24:42Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223
Mentioned in SAL (#wikimedia-operations) [2021-05-20T05:27:12Z] <ryankemper@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223
Mentioned in SAL (#wikimedia-operations) [2021-05-20T05:33:05Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation reboot without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223
Mentioned in SAL (#wikimedia-operations) [2021-05-20T05:33:21Z] <ryankemper> T283223 sudo -i cookbook sre.elasticsearch.rolling-operation cloudelastic "cloudelastic reboot" --reboot --nodes-per-run 1 --start-datetime 2021-05-20T05:16:40 --task-id T283223 on ryankemper@cumin1001 tmux session restart_cloudelastic
Mentioned in SAL (#wikimedia-operations) [2021-05-20T06:50:08Z] <ryankemper> T283223 Write queue not draining fast enough for the next node to reboot, will finish reboot tomorrow
Mentioned in SAL (#wikimedia-operations) [2021-05-20T06:50:12Z] <ryankemper@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) reboot without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223
Mentioned in SAL (#wikimedia-operations) [2021-06-01T15:16:21Z] <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation reboot without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223
Mentioned in SAL (#wikimedia-operations) [2021-06-01T15:16:30Z] <ryankemper> T283223 sudo -i cookbook sre.elasticsearch.rolling-operation cloudelastic "cloudelastic reboot" --reboot --nodes-per-run 1 --start-datetime 2021-05-20T05:16:40 --task-id T283223 on ryankemper@cumin1001 tmux session restart_cloudelastic
Mentioned in SAL (#wikimedia-operations) [2021-06-01T15:23:36Z] <ryankemper@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) reboot without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223
ryankemper@cumin1001:~$ sudo -E cumin -b 6 'P{cloudelastic*}' 'uname -r'
6 hosts will be targeted:
cloudelastic[1001-1006].wikimedia.org
Ok to proceed on 6 hosts? Enter the number of affected hosts to confirm or "q" to quit 6
===== NODE GROUP =====
(6) cloudelastic[1001-1006].wikimedia.org
----- OUTPUT of 'uname -r' -----
4.9.0-15-amd64
================Reboot is complete.