Page MenuHomePhabricator

ceph.operationalization: Add cookbooks to do an all-round reboot of the service (mon/mgr/osd)
Closed, ResolvedPublic

Description

    1. Write the description below
  • reboot osds
  • reboot mons/mgrs

Event Timeline

dcaro triaged this task as High priority.Apr 27 2021, 12:50 PM
dcaro created this task.

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T16:17:51Z] <wm-bot> Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T16:18:52Z] <wm-bot> Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T16:18:56Z] <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T16:21:57Z] <wm-bot> Finished rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T16:22:03Z] <wm-bot> Rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T16:24:52Z] <wm-bot> Finished rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T16:24:56Z] <wm-bot> Rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T16:27:49Z] <wm-bot> Finished rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-08T16:28:05Z] <wm-bot> Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus

Change 698819 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] ceph: add cookbooks to reboot osds

https://gerrit.wikimedia.org/r/698819

Mentioned in SAL (#wikimedia-cloud) [2021-06-09T12:53:34Z] <wm-bot> Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-09T12:53:47Z] <wm-bot> Rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-09T13:01:03Z] <wm-bot> Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-09T13:01:08Z] <wm-bot> Rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-09T13:21:53Z] <wm-bot> Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-09T13:21:57Z] <wm-bot> Rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-09T13:24:27Z] <wm-bot> Finished rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-09T13:24:30Z] <wm-bot> Rebooting node cloudcephmon2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-09T13:27:25Z] <wm-bot> Finished rebooting node cloudcephmon2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-09T13:27:30Z] <wm-bot> Rebooting node cloudcephmon2004-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-09T13:30:21Z] <wm-bot> Finished rebooting node cloudcephmon2004-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-09T13:30:38Z] <wm-bot> Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] (T281248) - cookbook ran by dcaro@vulcanus

Change 698987 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/software/spicerack@master] icinga.icinga_hosts: use bash wrapper to allow sudo

https://gerrit.wikimedia.org/r/698987

Change 698989 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] wmcs.ceph: Add cookbook to reboot mons

https://gerrit.wikimedia.org/r/698989

Change 698987 merged by jenkins-bot:

[operations/software/spicerack@master] icinga.icinga_hosts: use bash wrapper to allow sudo

https://gerrit.wikimedia.org/r/698987

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T09:24:36Z] <wm-bot> Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T09:24:48Z] <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T09:26:15Z] <wm-bot> Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T09:26:20Z] <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T09:29:47Z] <wm-bot> Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T09:29:51Z] <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T09:32:43Z] <wm-bot> Finished rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T09:32:47Z] <wm-bot> Rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T09:35:35Z] <wm-bot> Finished rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T09:35:43Z] <wm-bot> Rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T09:38:42Z] <wm-bot> Finished rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T09:39:01Z] <wm-bot> Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus

Change 698819 merged by jenkins-bot:

[operations/cookbooks@wmcs] ceph: add cookbooks to reboot osds

https://gerrit.wikimedia.org/r/698819

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:39:13Z] <wm-bot> Rebooting the nodes cloudcephosd2001-dev,cloudcephosd2002-dev,cloudcephosd2003-dev (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:39:26Z] <wm-bot> Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:42:23Z] <wm-bot> Finished rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:42:27Z] <wm-bot> Rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:45:25Z] <wm-bot> Finished rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:45:30Z] <wm-bot> Rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:48:28Z] <wm-bot> Finished rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:48:48Z] <wm-bot> Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:49:06Z] <wm-bot> Rebooting the nodes cloudcephmon2002-dev,cloudcephmon2003-dev,cloudcephmon2004-dev (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:49:21Z] <wm-bot> Rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:52:00Z] <wm-bot> Finished rebooting node cloudcephmon2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:52:06Z] <wm-bot> Rebooting node cloudcephmon2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:55:39Z] <wm-bot> Finished rebooting node cloudcephmon2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:55:43Z] <wm-bot> Rebooting node cloudcephmon2004-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:58:18Z] <wm-bot> Finished rebooting node cloudcephmon2004-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-06-10T10:58:35Z] <wm-bot> Finished rebooting the nodes ['cloudcephmon2002-dev', 'cloudcephmon2003-dev', 'cloudcephmon2004-dev'] (T281248) - cookbook ran by dcaro@vulcanus

Change 698989 merged by jenkins-bot:

[operations/cookbooks@wmcs] wmcs.ceph: Add cookbook to reboot mons

https://gerrit.wikimedia.org/r/698989