Maniphest T329709

[cookbooks.ceph] Add a cookbook to drain a ceph osd in a safe manner
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dcaro
	Feb 15 2023, 10:01 AM

Tags

Referenced Files

None

Subscribers

Description

Things that it should take into account:

The cluster will not become read only due to space issues
That there's no placement group missing an osd placed in this osd (so it does not become read only)
That there's no placement only in this osd (so it does not become missing)
Optionally wait until the rebalance is done

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	ceph: drain and undrain in chunks	cloud/wmcs-cookbooks	main	+323 -172
	ceph: add drain/undrain host and rack cookbooks	cloud/wmcs-cookbooks	main	+743 -13
	ceph.osd: Add a cookbook to depool_and_destroy osd daemons	cloud/wmcs-cookbooks	main	+1 K -3

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		MoritzMuehlenhoff	T253824 planned upstream deprecation of the ssh-rsa signing algorithm (RSA with SHA-1)
		Resolved		ayounsi	T254013 all network devices must run OpenSSH >= 7.2p1 but != 7.4p1
		Resolved		ayounsi	T317175 Junos: resolve DNS through mgmt_junos
		Resolved		ayounsi	T327862 Use mgmt_junos on all network devices
					Restricted Task
		Resolved		ayounsi	T316539 Upgrade network devices to Junos 20+
		Resolved		cmooney	T316544 Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+
		Resolved		dcaro	T297083 [ceph] Getting rack level HA
		Resolved		• nskaggs	T329498 [ceph] Move cloudcephosd1001 (b7) and cloudcephosd1002 (b4) to rack e4
		Resolved		None	T329592 beta cluster down
		Resolved		taavi	T329581 PAWS down
		Resolved	BUG REPORT	Lydia_Pintscher	T329555 WDQS tutorial on toolforge is down
		Resolved	BUG REPORT	Andrew	T329590 grafana.wmcloud.org offline following cloud wide outage
		Resolved	BUG REPORT	taavi	T329589 gerrit copy of cloud/instance-puppet stopped replicating
		Resolved		• aborrero	T329611 Toolforge grid: start webservices after outage
		Resolved	BUG REPORT	Ladsgroup	T329934 Wikimedia Chat (Mattermost instance) is down
		Resolved	BUG REPORT	dcaro	T329535 Cloud Ceph outage 2023-02-13
		Resolved		dcaro	T329709 [cookbooks.ceph] Add a cookbook to drain a ceph osd in a safe manner

Event Timeline

dcaro created this task.Feb 15 2023, 10:01 AM

dcaro changed the task status from Open to In Progress.

dcaro triaged this task as High priority.

dcaro added a parent task: T329535: Cloud Ceph outage 2023-02-13.

dcaro moved this task from To refine to Doing on the User-dcaro board.

Mentioned in SAL (#wikimedia-cloud) [2023-02-16T10:15:33Z] <dcaro> purges osd daemons 48 and 40 from eqiad ceph cluster (T329709)

Change 889775 had a related patch set uploaded (by David Caro; author: David Caro):

[cloud/wmcs-cookbooks@main] ceph.osd: Add a cookbook to depool_and_destroy osd daemons

https://gerrit.wikimedia.org/r/889775

gerritbot added a project: Patch-For-Review.Feb 16 2023, 11:31 AM

Change 889775 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] ceph.osd: Add a cookbook to depool_and_destroy osd daemons

https://gerrit.wikimedia.org/r/889775

dcaro mentioned this in rCCKB580f4eff91cd: ceph.osd: Add a cookbook to depool_and_destroy osd daemons.Mar 3 2023, 9:44 AM

dcaro edited projects, added cloud-services-team (FY2022/2023-Q4); removed cloud-services-team (FY2022/2023-Q3).Apr 26 2023, 7:56 AM

fnegri moved this task from FY2022/2023-Q4 to FY2023/2024-Q1-Q2 on the cloud-services-team board.Jul 26 2023, 5:46 PM

fnegri edited projects, added cloud-services-team (FY2023/2024-Q1-Q2); removed cloud-services-team (FY2022/2023-Q4).

fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2023/2024-Q1-Q2) board.

fnegri added a project: Cloud-VPS.Aug 4 2023, 12:34 PM

fnegri moved this task from Unsorted to wmcs-cookbooks on the Cloud-VPS board.

Change 966136 had a related patch set uploaded (by David Caro; author: David Caro):

[cloud/wmcs-cookbooks@main] ceph: add drain/undrain host and rack cookbooks

https://gerrit.wikimedia.org/r/966136

Change 966136 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] ceph: add drain/undrain host and rack cookbooks

https://gerrit.wikimedia.org/r/966136

dcaro mentioned this in rCCKB023c445c682f: ceph: add drain/undrain host and rack cookbooks.Oct 30 2023, 3:46 PM

fnegri edited projects, added cloud-services-team (FY2023/2024-Q3-Q4); removed cloud-services-team (FY2023/2024-Q1-Q2).Feb 1 2024, 11:13 AM

fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2023/2024-Q3-Q4) board.

Change #1013369 had a related patch set uploaded (by David Caro; author: David Caro):

[cloud/wmcs-cookbooks@main] ceph: drain and undrain in chunks

https://gerrit.wikimedia.org/r/1013369

Change #1013369 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] ceph: drain and undrain in chunks

https://gerrit.wikimedia.org/r/1013369

dcaro mentioned this in rCCKB6518f64e0b88: ceph: drain and undrain in chunks.Jul 8 2024, 1:01 PM

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2024, 8:40 PM

fnegri edited projects, added cloud-services-team (FY2024/2025-Q1-Q2); removed cloud-services-team (FY2023/2024-Q3-Q4).Aug 19 2024, 9:18 AM

fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2024/2025-Q1-Q2) board.

dcaro closed this task as Resolved.Sep 17 2024, 8:17 AM

fnegri moved this task from In progress to Done on the cloud-services-team (FY2024/2025-Q1-Q2) board.Sep 24 2024, 1:50 PM