Ceph cookbook fails waiting for OSDs to show up
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fnegri
	Sep 27 2022, 3:26 PM

Description

A new check for OSDs to be present in the OSD tree was added in this patch but it's not working correctly:

Exception raised while executing cookbook wmcs.ceph.osd.bootstrap_and_add:
Traceback (most recent call last):
  File "/Users/fran/.virtualenvs/cookbooks/lib/python3.10/site-packages/spicerack/_menu.py", line 234, in run
    raw_ret = runner.run()
  File "/Users/fran/wmf/cookbooks/cookbooks/wmcs/ceph/osd/bootstrap_and_add.py", line 210, in run
    new_osds = _wait_for_osds_to_show_up(
  File "/Users/fran/wmf/cookbooks/cookbooks/wmcs/ceph/osd/bootstrap_and_add.py", line 117, in _wait_for_osds_to_show_up
    raise Exception(f"Unable to find the new OSD node ({ceph_hostname}) in the osd tree")
Exception: Unable to find the new OSD node (cloudcephosd1030) in the osd tree
END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99)

Details

	Subject	Repo	Branch	Lines +/-
	ceph.bootstrap_and_add: fix _wait_for_osds (take 2)	operations/cookbooks	wmcs	+2 -1
	ceph.bootstrap_and_add: fix _wait_for_osds	operations/cookbooks	wmcs	+196 -11

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	cmooney	T304989 Finalise design extension of WMCS networks to new cloudsw in Eqiad rows E/F
Resolved	cmooney	T304936 Configure cloudsw1-e4-eqiad and cloudsw1-f4-eqiad
		Unknown Object (Task)
Resolved	• Cmjohnson	T294972 Q2:(Need By: TBD) rack/setup/install cloudcephosd10[25-34]
Resolved	fnegri	T314870 Setup cloudcephosd10[25-34] into the ceph eqiad cluster
Resolved	fnegri	T318723 Ceph cookbook fails waiting for OSDs to show up

Event Timeline

fnegri changed the task status from Open to In Progress.Sep 27 2022, 3:26 PM

fnegri triaged this task as Medium priority.

fnegri created this task.

fnegri moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

Change 835643 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/cookbooks@wmcs] ceph.bootstrap_and_add: fix _wait_for_osds

https://gerrit.wikimedia.org/r/835643

gerritbot added a project: Patch-For-Review.Sep 27 2022, 3:30 PM

Change 835643 merged by jenkins-bot:

[operations/cookbooks@wmcs] ceph.bootstrap_and_add: fix _wait_for_osds

https://gerrit.wikimedia.org/r/835643

Maintenance_bot removed a project: Patch-For-Review.Sep 30 2022, 11:30 AM

This is still not working correctly, because I forgot to fix the value that is returned by the method _wait_for_osds_to_show_up. New error is:

Exception raised while executing cookbook wmcs.ceph.osd.bootstrap_and_add:
Traceback (most recent call last):
  File "/Users/fran/.virtualenvs/cookbooks/lib/python3.10/site-packages/spicerack/_menu.py", line 234, in run
    raw_ret = runner.run()
  File "/Users/fran/wmf/cookbooks/cookbooks/wmcs/ceph/osd/bootstrap_and_add.py", line 205, in run
    new_osds = _wait_for_osds_to_show_up(
  File "/Users/fran/wmf/cookbooks/cookbooks/wmcs/ceph/osd/bootstrap_and_add.py", line 120, in _wait_for_osds_to_show_up
    return osd_tree["children"][ceph_hostname]["children"]
KeyError: 'children'

Putting up a new patch to fix this.

Change 837112 had a related patch set uploaded (by FNegri; author: FNegri):

[operations/cookbooks@wmcs] ceph.bootstrap_and_add: fix _wait_for_osds (take 2)

https://gerrit.wikimedia.org/r/837112

gerritbot added a project: Patch-For-Review.Sep 30 2022, 1:19 PM

fnegri closed this task as Resolved.Sep 30 2022, 3:17 PM

fnegri removed a project: Patch-For-Review.

Change 837112 merged by jenkins-bot:

[operations/cookbooks@wmcs] ceph.bootstrap_and_add: fix _wait_for_osds (take 2)

https://gerrit.wikimedia.org/r/837112

dcaro mentioned this in rCCKBa4d4d5cee961: ceph.bootstrap_and_add: fix _wait_for_osds.Dec 14 2022, 3:33 PM

dcaro mentioned this in rCCKB13cf4974fcf4: ceph.bootstrap_and_add: fix _wait_for_osds (take 2).