Page MenuHomePhabricator
Create Task
Maniphest T344798

Write a cookbook for rolling reboot/restart of datahubsearch servers
Closed, ResolvedPublic
Actions

Description

We currently have three servers called datahubsearch100[1-3] in production use as the Opensearch cluster behind https://datahub.wikimedia.org

At present there is no cookbook that can help us to perform maintenance on these servers.

It would be useful if we could perform a rolling reboot of the cluster, as well as a rolling restart of the opensearch services

Acceptance criteria:

  • We can perform an automated rolling restart of the opensearch cluster
  • We can perform an automated rolling reboot of the cluster

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
sre.opensearch.roll-restart-reboot: Define the opensearch service name as a patternoperations/cookbooksmaster+10 -4
Enable cumin hosts to reach the opensearch API on logstash clustersoperations/puppetproduction+6 -0
Define datacenter-local cumin aliases for the logstash clusteroperations/puppetproduction+2 -0
Add: roll restart/reboot command for opensearch cluster nodesoperations/cookbooksmaster+93 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

BTullis created this task.Aug 23 2023, 8:28 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 23 2023, 8:28 AM
BTullis moved this task from Incoming to Misc on the Data-Platform-SRE board.Aug 23 2023, 8:39 AM
Gehel updated the task description. (Show Details)Aug 23 2023, 8:40 AM
brouberol claimed this task.Sep 5 2023, 12:01 PM
brouberol added a subscriber: Gehel.Sep 6 2023, 2:39 PM

@BTullis I see that we already have a rolling-operation cookbook for elasticsearch that seems to perform extensive satefy operations:

  • set an alerting downtime
  • stop puppet
  • stop the service
  • wait for the service to be back up (green indices)

Do you think we could reuse this, as was hinted by @Gehel yesterday?

BTullis added subscribers: andrea.denisse, colewhite, herron.Sep 6 2023, 3:27 PM
In T344798#9146488, @brouberol wrote:

@BTullis I see that we already have a rolling-operation cookbook for elasticsearch that seems to perform extensive satefy operations:

  • set an alerting downtime
  • stop puppet
  • stop the service
  • wait for the service to be back up (green indices)

Do you think we could reuse this, as was hinted by @Gehel yesterday?

It's definitely a possibility, but it strikes me that it might be a bit heavyweight and might need more modification than we're comfortable with.
However, I'm happy if you want to look into it and you find that it's manageable

One of the main differences is that we use opensearch, rather than elasticsearch, so the commands such as this arent' going to work:

stop_es_cmd = 'systemctl list-units elasticsearch_* --plain --no-legend --all | ' + \
  'awk \' { print $1 } \' | xargs systemctl stop'

We would need to have some kind of conditional, so that we could work with our opensearch_1@datahub.service instead, which makes me a bit uneasy.

Plus we only have three nodes, so we don't need:

  • The requirement to carry out operations on batches of hosts
  • The rolling reimaging feature
  • The rolling package upgrade feature
  • Stopping and starting puppet

So it strikes me that it would perhaps be overkill to modify the existing cookbook to work with this small cluster.

However, there are some nice things about it, which it would be great to be able to reuse.

  • Setting and clearing downtime
  • Pooling and depooling each server
  • Checking for green indices

It might be worth checking with the observability team, because as far as I know they use opensearch on the logstash clusters.
e.g.

btullis@logstash1010:~$ systemctl list-units opensearch* --plain --no-legend
opensearch_2@production-elk7-eqiad.service            loaded active running OpenSearch (cluster production-elk7-eqiad)                                   
opensearch-production-elk7-eqiad-gc-log-cleanup.timer loaded active waiting Periodic execution of opensearch-production-elk7-eqiad-gc-log-cleanup.service

Perhaps they have their own cookbook that they use, or perhaps this one we've been discussing is already suitable and I'm unuaware of it.
I'll tag people like @andrea.denisse , @colewhite, and @herron - to see if they have any suggestions.

colewhite added a comment.Sep 7 2023, 12:01 AM

Related: T255864: Use/adopt search cluster ES management cookbooks for logging ES too

gerritbot added a comment.Sep 7 2023, 11:52 AM

Change 955717 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/cookbooks@master] Add: roll restart/reboot command for opensearch cluster nodes

https://gerrit.wikimedia.org/r/955717

gerritbot added a project: Patch-For-Review.Sep 7 2023, 11:52 AM
brouberol added a comment.Sep 7 2023, 11:56 AM

As requested by @Gehel, I pushed a boilerplate patch to gerrit, to make sure my accesses were working correctly. I've marked the change request as WIP.

BTullis added subscribers: bking, Volans.Sep 7 2023, 12:03 PM
In T344798#9149149, @brouberol wrote:

As requested by @Gehel, I pushed a boilerplate patch to gerrit, to make sure my accesses were working correctly. I've marked the change request as WIP.

Great! I will check it out. Although you won't be able to get to the cumin hosts yet (until T345633 is complete), there is a new test framework for in-progress cookbooks which you might like to read up on.
https://wikitech.wikimedia.org/wiki/Spicerack/Cookbooks#Test_before_merging

I've not used this myself yet, but I believe that @bking might have done so recently and may be able to share his experience of it.

Also, @Volans is always keen to help out on spicerack and cookbook related matters.

Volans added a comment.Sep 7 2023, 1:24 PM

@BTullis For context the sre.elasticsearch.rolling-operation is currently not using the existing framework for batch actions and IMHO it should be migrated to it to take advantages of the shared base classes, simplify the cookbook itself and its future maintenance.

As for the WIP patch I'll comment directly on Gerrit. I see that the batch classes where used there, thanks for that!

brouberol added a comment.Sep 7 2023, 1:32 PM

⚠ This is currently being implemented, meaning it's missing aliases, selectors, service names etc, that I was hoping to glean after I get SSH access :)

BTullis added a subscriber: RKemper.Sep 7 2023, 1:42 PM
In T344798#9149548, @Volans wrote:

@BTullis For context the sre.elasticsearch.rolling-operation is currently not using the existing framework for batch actions and IMHO it should be migrated to it to take advantages of the shared base classes, simplify the cookbook itself and its future maintenance.

Thanks @Volans. I agree that it would be beneficial to migrate that script to the new framework. It's not a cookbook that I've ever actually needed to run yet, although I suspect that @RKemper, @bking and @Gehel are much more familiar with it. The only [elastic|open]search servers that I have had to work on so far are the three datahubsearch100[1-3] VMs. These form a standalone cluster behind datahub.wikimedia.org

I'd be a little nervous about getting into a migration for a cookbook that I've never used, which is why I suggested that perhaps @brouberol might like to start with a clean sheet for this small, independent cluster. However, as always, I'm very open to advice and suggestions. If you think that this is the time to create a more unified cookbook for all clusters, then we can plan accordingly.

brouberol added a comment.EditedSep 7 2023, 2:25 PM

Also, qq for @BTullis: what does: "depooling" mean? I see on https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Server_actions that it "removes the server from active production". How does it work? Do we remove the host FQDN from an CNAME DNS?

Edit: seems like it https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/spicerack/+/master/spicerack/dnsdisc.py#251, assuming these configuration are then mapped into our DNS system, à la Consul.

bking added a comment.EditedSep 7 2023, 7:39 PM

@brouberol "depooling" is more accurately defined as "remove host from load balancer pool". confctl has a CLI as well which works from cumin (our shared SRE hosts from where we run cookbooks and other operations ).

bking mentioned this in T345880: Refactor sre.elasticsearch.rolling-operation to use spicerack improvements.Sep 7 2023, 8:00 PM
brouberol mentioned this in T345900: Add a dependency on the opensearch-py client.Sep 8 2023, 9:51 AM
brouberol added a subtask: T345900: Add a dependency on the opensearch-py client.Sep 8 2023, 9:54 AM
brouberol closed subtask T345900: Add a dependency on the opensearch-py client as Declined.Sep 11 2023, 10:21 AM
brouberol added a comment.Sep 11 2023, 12:02 PM

Alright, I seem to be getting working results, at least in dry-run mode: https://phabricator.wikimedia.org/P52407 !

brouberol added a comment.Sep 11 2023, 12:04 PM

Note: due to the reasons explained out here, we're swapping using the official opensearch client with a simple curl request. We deemed the complexity we were onboarding not worth the effort, given the simplicity of our usecase.

gerritbot added a comment.Sep 12 2023, 1:35 PM

Change 955717 merged by jenkins-bot:

[operations/cookbooks@master] Add: roll restart/reboot command for opensearch cluster nodes

https://gerrit.wikimedia.org/r/955717

brouberol updated the task description. (Show Details)Sep 12 2023, 1:36 PM
Maintenance_bot removed a project: Patch-For-Review.Sep 12 2023, 2:11 PM
brouberol closed this task as Resolved.Sep 12 2023, 2:19 PM
Gehel reopened this task as Open.Sep 12 2023, 3:22 PM
Gehel moved this task from Misc to Needs Review on the Data-Platform-SRE board.
brouberol added a comment.Sep 12 2023, 3:25 PM

Needs to be tested in the wild before calling it done

gerritbot added a comment.Sep 12 2023, 3:42 PM

Change 956916 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/cookbooks@master] Fix typo in allowed cluster group aliases

https://gerrit.wikimedia.org/r/956916

gerritbot added a project: Patch-For-Review.Sep 12 2023, 3:42 PM
brouberol changed the task status from Open to In Progress.Sep 13 2023, 9:12 AM

I'm working on some more improvements before we can test.

gerritbot added a comment.Sep 13 2023, 12:58 PM

Change 957287 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Define dataceneter-local cumin aliases for the logstash cluster

https://gerrit.wikimedia.org/r/957287

gerritbot added a comment.Sep 13 2023, 2:49 PM

Change 957287 merged by Brouberol:

[operations/puppet@production] Define datacenter-local cumin aliases for the logstash cluster

https://gerrit.wikimedia.org/r/957287

gerritbot added a comment.Sep 13 2023, 4:34 PM

Change 957324 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] Enable cumin hosts to reach the opensearch API on logstash clusters

https://gerrit.wikimedia.org/r/957324

gerritbot added a comment.Sep 14 2023, 8:03 AM

Change 957324 merged by Brouberol:

[operations/puppet@production] Enable cumin hosts to reach the opensearch API on logstash clusters

https://gerrit.wikimedia.org/r/957324

brouberol added a comment.Sep 14 2023, 8:27 AM

I added documentation about how to roll restart/reboot OS daemons/hosts in https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/DataHub and https://wikitech.wikimedia.org/wiki/OpenSearch#Operations

gerritbot added a comment.Sep 14 2023, 9:04 AM

Change 956916 merged by Brouberol:

[operations/cookbooks@master] sre.opensearch.roll-restart-reboot: Define the opensearch service name as a pattern

https://gerrit.wikimedia.org/r/956916

brouberol moved this task from Needs Review to To be Deployed on the Data-Platform-SRE board.Sep 14 2023, 9:09 AM
Maintenance_bot removed a project: Patch-For-Review.Sep 14 2023, 9:10 AM
brouberol mentioned this in T255864: Use/adopt search cluster ES management cookbooks for logging ES too.Sep 14 2023, 11:00 AM
Stashbot added a comment.Sep 14 2023, 11:04 AM

Mentioned in SAL (#wikimedia-operations) [2023-09-14T11:04:53Z] <brouberol> brouberol@cumin2002 START - Cookbook sre.opensearch.roll-restart-reboot rolling restart_daemons on A:datahubsearch - T344798

brouberol closed this task as Resolved.Sep 14 2023, 11:13 AM

Rolling restart worked on datahubsearch:

brouberol@cumin2002:~$ sudo cookbook sre.opensearch.roll-restart-reboot --reason 'Rolling restart by brouberol to make sure the cookbook works as intended' --alias datahubsearch restart_daemons
START - Cookbook sre.opensearch.roll-restart-reboot rolling restart_daemons on A:datahubsearch
Cluster health is green
Scheduling downtime on Icinga server alert1001.wikimedia.org for hosts: datahubsearch1001
[1/12, retrying in 10.00s] Unable to verify all hosts got downtimed: Some hosts are not yet downtimed: ['datahubsearch1001']
Created silence ID ff94e83d-582a-4a01-9c62-88c1b45310c8
----- OUTPUT of '/bin/systemctl r..._[1-2]@*.service' -----
================
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:14<00:00, 14.74s/hosts]
FAIL |                                                                                                                                                                                                                                         |   0% (0/1) [00:14<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: '/bin/systemctl r..._[1-2]@*.service'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Deleted silence ID ff94e83d-582a-4a01-9c62-88c1b45310c8
[1/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1001.eqiad.wmnet] Cluster is in status yellow
[2/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1001.eqiad.wmnet] Cluster is in status yellow
[3/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1001.eqiad.wmnet] Cluster is in status yellow
[4/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1001.eqiad.wmnet] Cluster is in status yellow
[5/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1001.eqiad.wmnet] Cluster is in status yellow
[6/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1001.eqiad.wmnet] Cluster is in status yellow
Cluster health is green
Sleeping for 120 seconds
Cluster health is green
Scheduling downtime on Icinga server alert1001.wikimedia.org for hosts: datahubsearch1002
[1/12, retrying in 10.00s] Unable to verify all hosts got downtimed: Some hosts are not yet downtimed: ['datahubsearch1002']
Created silence ID c0b41e09-59d2-412b-a6ba-7f1edf350652
----- OUTPUT of '/bin/systemctl r..._[1-2]@*.service' -----
================
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:18<00:00, 18.16s/hosts]
FAIL |                                                                                                                                                                                                                                         |   0% (0/1) [00:18<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: '/bin/systemctl r..._[1-2]@*.service'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Deleted silence ID c0b41e09-59d2-412b-a6ba-7f1edf350652
[1/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1002.eqiad.wmnet] Cluster is in status yellow
[2/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1002.eqiad.wmnet] Cluster is in status yellow
[3/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1002.eqiad.wmnet] Cluster is in status yellow
[4/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1002.eqiad.wmnet] Cluster is in status yellow
[5/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1002.eqiad.wmnet] Cluster is in status yellow
[6/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1002.eqiad.wmnet] Cluster is in status yellow
[7/60, retrying in 10.00s] Attempt to run 'cookbooks.sre.opensearch.roll-restart-reboot.RollingOperationRunner.check_for_green_indices' raised: [datahubsearch1002.eqiad.wmnet] Cluster is in status yellow
Cluster health is green
Sleeping for 120 seconds
Cluster health is green
Scheduling downtime on Icinga server alert1001.wikimedia.org for hosts: datahubsearch1003
[1/12, retrying in 10.00s] Unable to verify all hosts got downtimed: Some hosts are not yet downtimed: ['datahubsearch1003']
Created silence ID d7480f4c-c68d-44af-acdf-b7cedd9b261a
----- OUTPUT of '/bin/systemctl r..._[1-2]@*.service' -----
================
PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:16<00:00, 16.54s/hosts]
FAIL |                                                                                                                                                                                                                                         |   0% (0/1) [00:16<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: '/bin/systemctl r..._[1-2]@*.service'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
[1/15, retrying in 3.00s] Attempt to run 'spicerack.icinga.IcingaHosts.wait_for_optimal.<locals>.check' raised: Not all services are recovered: datahubsearch1003:OpenSearch health check for shards on 9200
[2/15, retrying in 6.00s] Attempt to run 'spicerack.icinga.IcingaHosts.wait_for_optimal.<locals>.check' raised: Not all services are recovered: datahubsearch1003:OpenSearch health check for shards on 9200
[3/15, retrying in 9.00s] Attempt to run 'spicerack.icinga.IcingaHosts.wait_for_optimal.<locals>.check' raised: Not all services are recovered: datahubsearch1003:OpenSearch health check for shards on 9200
[4/15, retrying in 12.00s] Attempt to run 'spicerack.icinga.IcingaHosts.wait_for_optimal.<locals>.check' raised: Not all services are recovered: datahubsearch1003:OpenSearch health check for shards on 9200
[5/15, retrying in 15.00s] Attempt to run 'spicerack.icinga.IcingaHosts.wait_for_optimal.<locals>.check' raised: Not all services are recovered: datahubsearch1003:OpenSearch health check for shards on 9200
[6/15, retrying in 18.00s] Attempt to run 'spicerack.icinga.IcingaHosts.wait_for_optimal.<locals>.check' raised: Not all services are recovered: datahubsearch1003:OpenSearch health check for shards on 9200
[7/15, retrying in 21.00s] Attempt to run 'spicerack.icinga.IcingaHosts.wait_for_optimal.<locals>.check' raised: Not all services are recovered: datahubsearch1003:OpenSearch health check for shards on 9200
[8/15, retrying in 24.00s] Attempt to run 'spicerack.icinga.IcingaHosts.wait_for_optimal.<locals>.check' raised: Not all services are recovered: datahubsearch1003:OpenSearch health check for shards on 9200
Deleted silence ID d7480f4c-c68d-44af-acdf-b7cedd9b261a
Cluster health is green
All restart_daemons were successful
END (PASS) - Cookbook sre.opensearch.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:datahubsearch
brouberol reopened this task as Open.Sep 14 2023, 11:14 AM

Oops, I change the status by mistake. I still have to test the reboot action.

brouberol closed this task as Resolved.Sep 14 2023, 11:38 AM

Rolling reboot worked on datahubsearch:

brouberol@cumin2002:~$ sudo cookbook sre.opensearch.roll-restart-reboot --reason 'Rolling reboot by brouberol to make sure the cookbook works as intended' --task-id T344798 --alias datahubsearch reboot
START - Cookbook sre.opensearch.roll-restart-reboot rolling reboot on A:datahubsearch
Cluster health is green
Scheduling downtime on Icinga server alert1001.wikimedia.org for hosts: datahubsearch1001
[1/12, retrying in 10.00s] Unable to verify all hosts got downtimed: Some hosts are not yet downtimed: ['datahubsearch1001']
Created silence ID 774926b5-eec6-49a9-9e4f-f5316f7d4dd6
Rebooting 1 hosts in batches of 1 with 0.0s of sleep in between: datahubsearch1001.eqiad.wmnet
----- OUTPUT of 'reboot-host' -----                                                                                                                                                                                                          
================                                                                                                                                                                                                                             
PASS |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00,  1.10hosts/s]
FAIL |                                                                                                                                                                                                       |   0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'reboot-host'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
[1/240, retrying in 10.00s] Attempt to run 'spicerack.remote.RemoteHosts.wait_reboot_since' raised: Reboot for datahubsearch1001.eqiad.wmnet not found yet, keep polling for it: unable to get uptime
Caused by: Cumin execution failed (exit_code=2)
[2/240, retrying in 10.00s] Attempt to run 'spicerack.remote.RemoteHosts.wait_reboot_since' raised: Reboot for datahubsearch1001.eqiad.wmnet not found yet, keep polling for it: unable to get uptime
Caused by: Cumin execution failed (exit_code=2)
Found reboot since 2023-09-14 11:21:55.468494 for hosts datahubsearch1001.eqiad.wmnet
[1/60, retrying in 30.00s] Attempt to run 'spicerack.puppet.PuppetHosts.wait_since' raised: Successful Puppet run too old (2023-09-14 11:04:13 <= 2023-09-14 11:21:55.468494) on: datahubsearch1001.eqiad.wmnet
[2/60, retrying in 60.00s] Attempt to run 'spicerack.puppet.PuppetHosts.wait_since' raised: Successful Puppet run too old (2023-09-14 11:04:13 <= 2023-09-14 11:21:55.468494) on: datahubsearch1001.eqiad.wmnet
[3/60, retrying in 90.00s] Attempt to run 'spicerack.puppet.PuppetHosts.wait_since' raised: Successful Puppet run too old (2023-09-14 11:04:13 <= 2023-09-14 11:21:55.468494) on: datahubsearch1001.eqiad.wmnet
Successful Puppet run found
Deleted silence ID 774926b5-eec6-49a9-9e4f-f5316f7d4dd6
Cluster health is green
Sleeping for 120 seconds
Cluster health is green
Scheduling downtime on Icinga server alert1001.wikimedia.org for hosts: datahubsearch1002
[1/12, retrying in 10.00s] Unable to verify all hosts got downtimed: Some hosts are not yet downtimed: ['datahubsearch1002']
Created silence ID 0709120f-8625-4b08-aefe-0decda8f074e
Rebooting 1 hosts in batches of 1 with 0.0s of sleep in between: datahubsearch1002.eqiad.wmnet
----- OUTPUT of 'reboot-host' -----                                                                                                                                                                                                          
================                                                                                                                                                                                                                             
PASS |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:01<00:00,  1.03s/hosts]
FAIL |                                                                                                                                                                                                       |   0% (0/1) [00:01<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'reboot-host'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
[1/240, retrying in 10.00s] Attempt to run 'spicerack.remote.RemoteHosts.wait_reboot_since' raised: Reboot for datahubsearch1002.eqiad.wmnet not found yet, keep polling for it: unable to get uptime
Caused by: Cumin execution failed (exit_code=2)
[2/240, retrying in 10.00s] Attempt to run 'spicerack.remote.RemoteHosts.wait_reboot_since' raised: Reboot for datahubsearch1002.eqiad.wmnet not found yet, keep polling for it: unable to get uptime
Caused by: Cumin execution failed (exit_code=2)
Found reboot since 2023-09-14 11:27:54.202575 for hosts datahubsearch1002.eqiad.wmnet
[1/60, retrying in 30.00s] Attempt to run 'spicerack.puppet.PuppetHosts.wait_since' raised: Successful Puppet run too old (2023-09-14 11:23:24 <= 2023-09-14 11:27:54.202575) on: datahubsearch1002.eqiad.wmnet
[2/60, retrying in 60.00s] Attempt to run 'spicerack.puppet.PuppetHosts.wait_since' raised: Successful Puppet run too old (2023-09-14 11:23:24 <= 2023-09-14 11:27:54.202575) on: datahubsearch1002.eqiad.wmnet
[3/60, retrying in 90.00s] Attempt to run 'spicerack.puppet.PuppetHosts.wait_since' raised: Successful Puppet run too old (2023-09-14 11:23:24 <= 2023-09-14 11:27:54.202575) on: datahubsearch1002.eqiad.wmnet
Successful Puppet run found
Deleted silence ID 0709120f-8625-4b08-aefe-0decda8f074e
Cluster health is green
Sleeping for 120 seconds
Cluster health is green
Scheduling downtime on Icinga server alert1001.wikimedia.org for hosts: datahubsearch1003
[1/12, retrying in 10.00s] Unable to verify all hosts got downtimed: Some hosts are not yet downtimed: ['datahubsearch1003']
Created silence ID 3ffe3831-475d-4c59-a4e9-b498dfeb2bcc
Rebooting 1 hosts in batches of 1 with 0.0s of sleep in between: datahubsearch1003.eqiad.wmnet
----- OUTPUT of 'reboot-host' -----                                                                                                                                                                                                          
================                                                                                                                                                                                                                             
PASS |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:01<00:00,  1.02s/hosts]
FAIL |                                                                                                                                                                                                       |   0% (0/1) [00:01<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'reboot-host'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
[1/240, retrying in 10.00s] Attempt to run 'spicerack.remote.RemoteHosts.wait_reboot_since' raised: Reboot for datahubsearch1003.eqiad.wmnet not found yet, keep polling for it: unable to get uptime
Caused by: Cumin execution failed (exit_code=2)
Found reboot since 2023-09-14 11:33:52.747657 for hosts datahubsearch1003.eqiad.wmnet
[1/60, retrying in 30.00s] Attempt to run 'spicerack.puppet.PuppetHosts.wait_since' raised: Successful Puppet run too old (2023-09-14 11:05:04 <= 2023-09-14 11:33:52.747657) on: datahubsearch1003.eqiad.wmnet
[2/60, retrying in 60.00s] Attempt to run 'spicerack.puppet.PuppetHosts.wait_since' raised: Successful Puppet run too old (2023-09-14 11:05:04 <= 2023-09-14 11:33:52.747657) on: datahubsearch1003.eqiad.wmnet
[3/60, retrying in 90.00s] Attempt to run 'spicerack.puppet.PuppetHosts.wait_since' raised: Successful Puppet run too old (2023-09-14 11:05:04 <= 2023-09-14 11:33:52.747657) on: datahubsearch1003.eqiad.wmnet
Successful Puppet run found
Deleted silence ID 3ffe3831-475d-4c59-a4e9-b498dfeb2bcc
Cluster health is green
All reboot were successful
END (PASS) - Cookbook sre.opensearch.roll-restart-reboot (exit_code=0) rolling reboot on A:datahubsearch
Gehel edited projects, added Data-Platform-SRE (2023.12.01 - 2023.12.31); removed Data-Platform-SRE.May 5 2025, 12:53 PM
Gehel moved this task from Backlog to Done on the Data-Platform-SRE (2023.12.01 - 2023.12.31) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits