Page MenuHomePhabricator

Install the new ceph osd machines cloudcephosd10(1[6-9]|20) using cookbooks
Closed, ResolvedPublic

Description

Write the description below

Now that we have the new machines there (T274945), install and incorporate them to the eqiad cluster using cookbooks.

Note that there's some part of it that has to be done manually:

  • Check if the disk hardware model has support to disable write cache, if so:
    • Add entry to profile::ceph::osd::disk_models_without_write_cache
  • Change host role and add puppet hiera info:
    • manifests/site.pp (to the wmcs::ceph::osd role)
    • hieradata/eqiad/profile/ceph/osd.yaml

The above will add the packages, configuration and credentials necesary to configure the OSD.

Event Timeline

dcaro triaged this task as High priority.Jun 30 2021, 1:15 PM
dcaro created this task.
dcaro renamed this task from Install the new ceph osd machines cloudcephosd101[6-9] using cookbooks to Install the new ceph osd machines cloudcephosd10(1[6-9]|20) using cookbooks.Jun 30 2021, 1:16 PM

Change 702370 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmcs.ceph: update disk models should disable write cache

https://gerrit.wikimedia.org/r/702370

Change 702374 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmcs.ceph: Add cloudcephosd1016 to the ceph osd role

https://gerrit.wikimedia.org/r/702374

Change 702370 merged by David Caro:

[operations/puppet@production] wmcs.ceph: update disk models should disable write cache

https://gerrit.wikimedia.org/r/702370

Change 702374 merged by David Caro:

[operations/puppet@production] wmcs.ceph: Add cloudcephosd1016 to the ceph osd role

https://gerrit.wikimedia.org/r/702374

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T10:40:43Z] <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T10:41:25Z] <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T10:42:45Z] <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T10:44:52Z] <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T10:47:46Z] <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T10:58:53Z] <dcaro> rebooting cloudcephosd1016 (T285858)

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T11:13:32Z] <wm-bot> Adding new OSD cloudcephosd1016.eqiad.wmnet to the cluster (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T11:16:17Z] <wm-bot> Added new OSD node cloudcephosd1016.eqiad.wmnet (T285858) - cookbook ran by dcaro@vulcanus

Change 702655 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmcs.ceph: Add the new 17, 19 and 20 OSDs

https://gerrit.wikimedia.org/r/702655

Change 702655 merged by David Caro:

[operations/puppet@production] wmcs.ceph: Add the new 17, 19 and 20 OSDs

https://gerrit.wikimedia.org/r/702655

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T14:16:58Z] <wm-bot> Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T14:17:04Z] <wm-bot> Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T14:20:33Z] <wm-bot> Adding new OSDs ['cloudcephosd1017.eqiad.wmnet', 'cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T14:20:37Z] <wm-bot> Adding OSD cloudcephosd1017.eqiad.wmnet... (1/3) (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T14:25:52Z] <wm-bot> Added OSD cloudcephosd1017.eqiad.wmnet... (1/3) (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-01T14:25:57Z] <wm-bot> Adding OSD cloudcephosd1019.eqiad.wmnet... (2/3) (T285858) - cookbook ran by dcaro@vulcanus

Change 702677 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] ceph.keyring: ensure that the bootstrap dir exists

https://gerrit.wikimedia.org/r/702677

Change 702677 merged by David Caro:

[operations/puppet@production] ceph.keyring: ensure that the bootstrap dir exists

https://gerrit.wikimedia.org/r/702677

Mentioned in SAL (#wikimedia-cloud) [2021-07-02T09:13:53Z] <wm-bot> Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-02T09:13:58Z] <wm-bot> Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) (T285858) - cookbook ran by dcaro@vulcanus

Change 702897 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] ceph.keyring: make sure the bootstrap dir exists

https://gerrit.wikimedia.org/r/702897

Change 702897 merged by David Caro:

[operations/puppet@production] ceph.keyring: make sure the bootstrap dir exists

https://gerrit.wikimedia.org/r/702897

Mentioned in SAL (#wikimedia-cloud) [2021-07-02T10:01:54Z] <wm-bot> Adding new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-02T10:01:58Z] <wm-bot> Adding OSD cloudcephosd1019.eqiad.wmnet... (1/2) (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-02T10:07:02Z] <wm-bot> Added OSD cloudcephosd1019.eqiad.wmnet... (1/2) (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-02T10:07:06Z] <wm-bot> Adding OSD cloudcephosd1020.eqiad.wmnet... (2/2) (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-02T10:12:13Z] <wm-bot> Added OSD cloudcephosd1020.eqiad.wmnet... (2/2) (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-02T10:12:19Z] <wm-bot> Added 2 new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-07-02T10:12:35Z] <wm-bot> The cluster is not rebalance after adding the new OSDs ['cloudcephosd1019.eqiad.wmnet', 'cloudcephosd1020.eqiad.wmnet'] (T285858) - cookbook ran by dcaro@vulcanus

Change 702930 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] wmcs.ceph: add cookbook to bootstrap and add OSDs

https://gerrit.wikimedia.org/r/702930

Change 702929 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] wmcs.ceph: rename the ceph controller to CephClusterController

https://gerrit.wikimedia.org/r/702929

Change 702929 merged by jenkins-bot:

[operations/cookbooks@wmcs] wmcs.ceph: rename the ceph controller to CephClusterController

https://gerrit.wikimedia.org/r/702929

Change 702930 merged by jenkins-bot:

[operations/cookbooks@wmcs] wmcs.ceph: add cookbook to bootstrap and add OSDs

https://gerrit.wikimedia.org/r/702930

Mentioned in SAL (#wikimedia-cloud) [2021-08-11T13:47:45Z] <wm-bot> Adding new OSDs ['cloudcephosd1018.eqiad.wmnet'] to the cluster (T285858) - cookbook ran by dcaro@vulcanus

Mentioned in SAL (#wikimedia-cloud) [2021-08-11T13:47:50Z] <wm-bot> Adding OSD cloudcephosd1018.eqiad.wmnet... (1/1) (T285858) - cookbook ran by dcaro@vulcanus

Change 711499 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] wmcs.ceph: add cloudcephosd1018 as osd

https://gerrit.wikimedia.org/r/711499

Change 711499 merged by Andrew Bogott:

[operations/puppet@production] wmcs.ceph: add cloudcephosd1018 as osd

https://gerrit.wikimedia.org/r/711499

Re-assign to me once the new OSD is deployed to followup with the docs.

Andrew subscribed.

nodes are installed -- back to you!