Page MenuHomePhabricator

Setup cloudcephosd10[25-34] into the ceph eqiad cluster
Open, In Progress, HighPublic

Description

We have to set them up (configure and such) and make them join the eqiad ceph cluster.

Note that these hosts will have a different network setup than usual, having their private network in the ranges:

cloudcephosd10[25-29] (row E) -> 192.168.5.0/24
cloudcephosd10[30-34] (row F) -> 192.168.6.0/24

So there might be more changes than usual needed.

For a guideline see:
https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Ceph#Adding_OSDs

Progress

  • cloudcephosd1025
  • cloudcephosd1026
  • cloudcephosd1027
  • cloudcephosd1028
  • cloudcephosd1029
  • cloudcephosd1030
  • cloudcephosd1031
  • cloudcephosd1032
  • cloudcephosd1033
  • cloudcephosd1034 < hardware issues (see T316673)

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -6
operations/cookbookswmcs+215 -3
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/cookbookswmcs+11 -16
operations/puppetproduction+2 -2
operations/puppetproduction+1 -0
operations/puppetproduction+82 -94
operations/puppetproduction+96 -97
operations/cookbookswmcs+19 -3
operations/cookbookswmcs+125 -0
operations/cookbookswmcs+6 -13
operations/cookbookswmcs+3 -3
operations/cookbookswmcs+2 -2
operations/cookbookswmcs+276 -280
operations/cookbookswmcs+90 -142
operations/cookbookswmcs+699 -201
operations/puppetproduction+2 -2
operations/puppetproduction+2 -0
operations/puppetproduction+20 -20
operations/puppetproduction+443 -348
operations/puppetproduction+2 -2
operations/puppetproduction+2 -0
operations/puppetproduction+15 -6
operations/puppetproduction+2 -1
operations/puppetproduction+2 -2
operations/puppetproduction+86 -3
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-cloud) [2022-08-18T07:29:56Z] <dcaro> Starting up all the osd daemons on cloudcephosd1025 (T314870)

Change 824421 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] cloud: reformat cloud.yaml with prettier

https://gerrit.wikimedia.org/r/824421

Change 824422 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] p:ceph::osd: get the os disks by size

https://gerrit.wikimedia.org/r/824422

Change 824423 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/puppet@production] ceph::osd: add new disks model to disable write caches for

https://gerrit.wikimedia.org/r/824423

Change 823169 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] global: add inventory module

https://gerrit.wikimedia.org/r/823169

Change 823666 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] Openstack: use cluster_name instead of control node

https://gerrit.wikimedia.org/r/823666

Change 823667 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] ceph: use cluster_name instead of control node

https://gerrit.wikimedia.org/r/823667

Change 823668 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] ceph: use human-readable names for ceph clusters

https://gerrit.wikimedia.org/r/823668

Change 823669 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] ceph: use the correct codfw ceph mon hosts

https://gerrit.wikimedia.org/r/823669

Change 823670 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] ceph,opensatck: use the inventory to get the nodes domain

https://gerrit.wikimedia.org/r/823670

Change 823671 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] ceph: add roll_restart_osd_daemons cookbook

https://gerrit.wikimedia.org/r/823671

Change 824149 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] ceph.bootstrap_and_add: add --force option

https://gerrit.wikimedia.org/r/824149

Change 824153 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] WIP: adding support to change the osd class type

https://gerrit.wikimedia.org/r/824153

Change 824457 had a related patch set uploaded (by David Caro; author: David Caro):

[operations/cookbooks@wmcs] ceph.bootstrapp_and_add: don't rely on sda/sdb being the os disks

https://gerrit.wikimedia.org/r/824457

Change 824489 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] Add cloudcephosd1026 to the Ceph pool

https://gerrit.wikimedia.org/r/824489

Change 824489 merged by FNegri:

[operations/puppet@production] Add cloudcephosd1026 to the Ceph pool

https://gerrit.wikimedia.org/r/824489

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-18T13:10:54Z] <wm-bot2> Adding new OSDs ['cloudcephosd1026.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-18T13:10:57Z] <wm-bot2> Adding OSD cloudcephosd1026.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-18T13:12:22Z] <wm-bot2> Rebooting node cloudcephosd1026.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-18T13:15:23Z] <wm-bot2> Finished rebooting node cloudcephosd1026.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-18T13:25:34Z] <wm-bot2> Added OSD cloudcephosd1026.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-18T13:25:37Z] <wm-bot2> Added 1 new OSDs ['cloudcephosd1026.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Change 823169 merged by jenkins-bot:

[operations/cookbooks@wmcs] global: add inventory module

https://gerrit.wikimedia.org/r/823169

Change 823666 merged by jenkins-bot:

[operations/cookbooks@wmcs] Openstack: use cluster_name instead of control node

https://gerrit.wikimedia.org/r/823666

Change 823667 merged by jenkins-bot:

[operations/cookbooks@wmcs] ceph: use cluster_name instead of control node

https://gerrit.wikimedia.org/r/823667

Change 823668 merged by jenkins-bot:

[operations/cookbooks@wmcs] ceph: use human-readable names for ceph clusters

https://gerrit.wikimedia.org/r/823668

Change 823669 merged by jenkins-bot:

[operations/cookbooks@wmcs] ceph: use the correct codfw ceph mon hosts

https://gerrit.wikimedia.org/r/823669

Change 823670 merged by jenkins-bot:

[operations/cookbooks@wmcs] ceph,opensatck: use the inventory to get the nodes domain

https://gerrit.wikimedia.org/r/823670

Change 823671 merged by jenkins-bot:

[operations/cookbooks@wmcs] ceph: add roll_restart_osd_daemons cookbook

https://gerrit.wikimedia.org/r/823671

Change 824149 merged by jenkins-bot:

[operations/cookbooks@wmcs] ceph.bootstrap_and_add: add --force option

https://gerrit.wikimedia.org/r/824149

Change 824421 merged by FNegri:

[operations/puppet@production] cloud: reformat cloud.yaml with prettier

https://gerrit.wikimedia.org/r/824421

Change 824422 merged by David Caro:

[operations/puppet@production] p:ceph::osd: get the os disks by size

https://gerrit.wikimedia.org/r/824422

Change 824423 merged by David Caro:

[operations/puppet@production] ceph::osd: add new disks model to disable write caches for

https://gerrit.wikimedia.org/r/824423

Change 825722 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] Add cloudcephosd1027 to the Ceph pool

https://gerrit.wikimedia.org/r/825722

Change 825722 merged by FNegri:

[operations/puppet@production] Add cloudcephosd1027 to the Ceph pool

https://gerrit.wikimedia.org/r/825722

Change 824457 merged by jenkins-bot:

[operations/cookbooks@wmcs] ceph.bootstrapp_and_add: don't rely on sda/sdb being the os disks

https://gerrit.wikimedia.org/r/824457

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-23T13:22:47Z] <wm-bot2> Adding new OSDs ['cloudcephosd1027.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-23T13:22:51Z] <wm-bot2> Adding OSD cloudcephosd1027.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-23T13:24:26Z] <wm-bot2> Rebooting node cloudcephosd1027.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-23T13:27:49Z] <wm-bot2> Finished rebooting node cloudcephosd1027.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-23T13:46:28Z] <wm-bot2> Added OSD cloudcephosd1027.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-23T13:46:33Z] <wm-bot2> Added 1 new OSDs ['cloudcephosd1027.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Change 826208 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] Add cloudcephosd1028 to the Ceph pool

https://gerrit.wikimedia.org/r/826208

Change 826208 merged by FNegri:

[operations/puppet@production] Add cloudcephosd1028 to the Ceph pool

https://gerrit.wikimedia.org/r/826208

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-24T10:33:29Z] <wm-bot2> Adding new OSDs ['cloudcephosd1028.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-24T10:33:33Z] <wm-bot2> Adding OSD cloudcephosd1028.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-24T10:34:08Z] <wm-bot2> Rebooting node cloudcephosd1028.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-24T10:37:15Z] <wm-bot2> Finished rebooting node cloudcephosd1028.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-24T10:45:38Z] <wm-bot2> Added OSD cloudcephosd1028.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-24T10:45:42Z] <wm-bot2> Added 1 new OSDs ['cloudcephosd1028.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Change 826577 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] Add cloudcephosd1029 to the Ceph pool

https://gerrit.wikimedia.org/r/826577

Change 826577 merged by FNegri:

[operations/puppet@production] Add cloudcephosd1029 to the Ceph pool

https://gerrit.wikimedia.org/r/826577

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-25T14:58:20Z] <wm-bot2> Adding new OSDs ['cloudcephosd1029.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-25T14:58:25Z] <wm-bot2> Adding OSD cloudcephosd1029.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-25T14:59:31Z] <wm-bot2> Rebooting node cloudcephosd1029.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-25T15:02:36Z] <wm-bot2> Finished rebooting node cloudcephosd1029.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-25T15:14:16Z] <wm-bot2> Added OSD cloudcephosd1029.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-25T15:14:21Z] <wm-bot2> Added 1 new OSDs ['cloudcephosd1029.eqiad.wmnet'] (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Change 826843 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] Add cloudcephosd1030 to the Ceph pool

https://gerrit.wikimedia.org/r/826843

Change 826843 merged by FNegri:

[operations/puppet@production] Add cloudcephosd1030 to the Ceph pool

https://gerrit.wikimedia.org/r/826843

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-30T09:10:34Z] <wm-bot2> Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-30T09:10:38Z] <wm-bot2> Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-30T09:11:44Z] <wm-bot2> Rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-30T09:14:56Z] <wm-bot2> Finished rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-30T09:32:17Z] <wm-bot2> Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-30T09:32:21Z] <wm-bot2> Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-30T09:32:57Z] <wm-bot2> Rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-30T09:39:00Z] <wm-bot2> Finished rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-30T10:38:52Z] <wm-bot2> Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-30T10:38:55Z] <wm-bot2> Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-30T10:39:31Z] <wm-bot2> Rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-08-30T10:43:02Z] <wm-bot2> Finished rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Change 824153 merged by jenkins-bot:

[operations/cookbooks@wmcs] ceph.bootstrap_and_add: add support to change the osd class type

https://gerrit.wikimedia.org/r/824153

Change 830147 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/puppet@production] Add cloudcephosd103[1-4] to the Ceph pool

https://gerrit.wikimedia.org/r/830147

Change 830147 merged by FNegri:

[operations/puppet@production] Add cloudcephosd103[1-4] to the Ceph pool

https://gerrit.wikimedia.org/r/830147

Mentioned in SAL (#wikimedia-cloud) [2022-09-07T09:41:52Z] <dhinus> Temporarily removing cloudcephosd1030 from Ceph cluster (https://phabricator.wikimedia.org/T314870)

cloudcephosd1030 and cloudcephosd1031 have hardware issues (see related tasks). I'm going to remove osd.231, osd.232 and osd.233 from the cluster (they're in cloudcephosd1030)

Following these instructions: https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/#removing-osds-manual

$ sudo ceph osd out osd.231
marked out osd.231.
$ sudo ceph osd out osd.232
marked out osd.232.
$ sudo ceph osd out osd.233
marked out osd.233.

$ sudo systemctl stop ceph-osd@231.service
$ sudo systemctl stop ceph-osd@232.service
$ sudo systemctl stop ceph-osd@233.service

$ sudo ceph osd purge osd.231 --yes-i-really-mean-it
purged osd.231
$ sudo ceph osd purge osd.232 --yes-i-really-mean-it
purged osd.232
$ sudo ceph osd purge osd.233 --yes-i-really-mean-it
purged osd.233

Mentioned in SAL (#wikimedia-cloud-feed) [2022-09-07T10:10:07Z] <wm-bot2> Adding new OSDs ['cloudcephosd1032.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-09-07T10:10:10Z] <wm-bot2> Adding OSD cloudcephosd1032.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-09-07T10:10:51Z] <wm-bot2> Rebooting node cloudcephosd1032.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-09-07T10:14:01Z] <wm-bot2> Finished rebooting node cloudcephosd1032.eqiad.wmnet (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

Mentioned in SAL (#wikimedia-cloud-feed) [2022-09-07T10:18:01Z] <wm-bot2> Added OSD cloudcephosd1032.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@Francesco’s-MacBook-Pro

fnegri removed a project: Patch-For-Review.
fnegri removed a subscriber: cmooney.

Mentioned in SAL (#wikimedia-cloud-feed) [2022-09-27T10:32:03Z] <wm-bot2> Adding new OSDs ['cloudcephosd1030.eqiad.wmnet'] to the cluster (T314870) - cookbook ran by fran@wmf3169

Mentioned in SAL (#wikimedia-cloud-feed) [2022-09-27T10:32:07Z] <wm-bot2> Adding OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169

Mentioned in SAL (#wikimedia-cloud-feed) [2022-09-27T10:32:41Z] <wm-bot2> Rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@wmf3169

Mentioned in SAL (#wikimedia-cloud-feed) [2022-09-27T10:35:56Z] <wm-bot2> Finished rebooting node cloudcephosd1030.eqiad.wmnet (T314870) - cookbook ran by fran@wmf3169

Mentioned in SAL (#wikimedia-cloud-feed) [2022-09-27T10:48:13Z] <wm-bot2> Added OSD cloudcephosd1030.eqiad.wmnet... (1/1) (T314870) - cookbook ran by fran@wmf3169