Page MenuHomePhabricator

cloud ceph: include cloudcephosd102[1-4].eqiad.wmnet in the farm
Closed, ResolvedPublic

Assigned To
Authored By
aborrero
Nov 4 2021, 9:01 AM
Referenced Files
F34731278: image.png
Nov 5 2021, 11:56 AM
F34731237: image.png
Nov 5 2021, 11:33 AM
F34731064: image.png
Nov 5 2021, 10:16 AM
F34729964: image.png
Nov 4 2021, 10:02 AM

Description

This task is to track work to enroll cloudcephosd102[1-4].eqiad.wmnet in the eqiad WMCS ceph farm.

There is a nice automation created by @dcaro in the form of a spicerack cookbook to do this:

arturo@endurance:~/git/wmf/operations/cookbooks wmcs $ cookbook -l
[..]
`-- wmcs
    |-- wmcs.ceph
    |   |-- wmcs.ceph.osd
    |   |   `-- wmcs.ceph.osd.bootstrap_and_add
[..]

Related docs: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Ceph#Post_Installation_Procedures

Event Timeline

aborrero triaged this task as Medium priority.Nov 4 2021, 9:59 AM
aborrero updated the task description. (Show Details)

Change 736719 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: ceph: eqiad: include new nodes in the farm

https://gerrit.wikimedia.org/r/736719

Change 736719 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: ceph: eqiad: include new nodes in the farm

https://gerrit.wikimedia.org/r/736719

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T11:16:04Z] <wm-bot> Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T11:19:47Z] <wm-bot> Adding new OSDs ['cloudcephosd1021.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T11:19:51Z] <wm-bot> Adding OSD cloudcephosd1021.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T11:26:44Z] <wm-bot> Added OSD cloudcephosd1021.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T11:26:49Z] <wm-bot> Added 1 new OSDs ['cloudcephosd1021.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance

Change 736733 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloud: ceph: fix interface name on newest osd servers

https://gerrit.wikimedia.org/r/736733

Change 736733 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloud: ceph: fix interface name on newest osd servers

https://gerrit.wikimedia.org/r/736733

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T16:00:19Z] <wm-bot> Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T16:00:23Z] <wm-bot> Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T16:12:28Z] <wm-bot> Adding new OSDs ['cloudcephosd1022.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T16:12:32Z] <wm-bot> Adding OSD cloudcephosd1022.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T16:17:42Z] <wm-bot> Added OSD cloudcephosd1022.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T16:17:46Z] <wm-bot> Added 1 new OSDs ['cloudcephosd1022.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T16:33:47Z] <wm-bot> Adding new OSDs ['cloudcephosd1023.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T16:33:51Z] <wm-bot> Adding OSD cloudcephosd1023.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T16:39:01Z] <wm-bot> Added OSD cloudcephosd1023.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-04T16:39:05Z] <wm-bot> Added 1 new OSDs ['cloudcephosd1023.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance

This is done!

image.png (230×301 px, 17 KB)

Per the dashboard we've gained about 41TB effective storage space.

We reduced our storage usage from 59% to 52%, 7% gain!

I missed cloudcephosd1024.eqiad.wmnet, adding it now

Mentioned in SAL (#wikimedia-cloud) [2021-11-05T11:12:05Z] <wm-bot> Adding new OSDs ['cloudcephosd1024.eqiad.wmnet'] to the cluster (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-05T11:12:10Z] <wm-bot> Adding OSD cloudcephosd1024.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-05T11:17:55Z] <wm-bot> Added OSD cloudcephosd1024.eqiad.wmnet... (1/1) (T295012) - cookbook ran by arturo@endurance

Mentioned in SAL (#wikimedia-cloud) [2021-11-05T11:18:01Z] <wm-bot> Added 1 new OSDs ['cloudcephosd1024.eqiad.wmnet'] (T295012) - cookbook ran by arturo@endurance

Corrected metric after accounting for 3x replication and 90% max storage usage for OSDs:

image.png (233×317 px, 16 KB)