Page MenuHomePhabricator
Create Task
Maniphest T401693

Put cloudcephosd10[42-47] in service
Closed, ResolvedPublic
Actions

Description

  • cloudcephosd1042
  • cloudcephosd1043
  • cloudcephosd1044
  • cloudcephosd1045
  • cloudcephosd1046
  • cloudcephosd1047

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Replace cloudvirt1045operations/puppetproduction+9 -13
cloudceph: remove cloudcephosd1045, againoperations/puppetproduction+13 -9
New attempt to put new cloudcephosd hosts onlineoperations/puppetproduction+34 -34
cloudceph: further mark out OSD nodes not yet ready for actionoperations/puppetproduction+2 -2
cloudceph: mark out some OSD nodes not yet ready for actionoperations/puppetproduction+32 -32
cloudceph: add new OSDs: cloudcephosd1042-1051operations/puppetproduction+60 -5
osd.yaml: add entries for cloudcephosd1042, cloudcephosd104[67]operations/puppetproduction+24 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.Aug 12 2025, 2:17 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptAug 12 2025, 2:17 PM
ops-monitoring-bot added a comment.Aug 12 2025, 3:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudcephosd1042.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 12 2025, 6:24 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudcephosd1042.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1042 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1042.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 12 2025, 6:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudcephosd1042.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 12 2025, 7:41 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudcephosd1042.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1042 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1042.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 12 2025, 7:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudcephosd1042.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 12 2025, 8:55 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudcephosd1042.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1042 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1042.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 12 2025, 8:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudcephosd1042.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 12 2025, 9:14 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudcephosd1042.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1042 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1042.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Andrew updated the task description. (Show Details)Aug 14 2025, 9:26 PM
Stashbot added a comment.Aug 15 2025, 7:26 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-15T19:26:54Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Stashbot added a comment.Aug 15 2025, 7:33 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-15T19:32:57Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T401693)

gerritbot added a comment.Aug 15 2025, 7:40 PM

Change #1179218 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] osd.yaml: add entries for cloudcephosd1042, cloudcephosd104[67]

https://gerrit.wikimedia.org/r/1179218

gerritbot added a project: Patch-For-Review.Aug 15 2025, 7:40 PM

Change #1179218 merged by Andrew Bogott:

[operations/puppet@production] osd.yaml: add entries for cloudcephosd1042, cloudcephosd104[67]

https://gerrit.wikimedia.org/r/1179218

Stashbot added a comment.Aug 15 2025, 7:51 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-15T19:51:21Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Stashbot added a comment.Aug 15 2025, 7:58 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-15T19:57:58Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T401693)

Stashbot added a comment.Aug 15 2025, 8:18 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-15T20:18:03Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Stashbot added a comment.Aug 15 2025, 8:24 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-15T20:24:28Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T401693)

Maintenance_bot removed a project: Patch-For-Review.Aug 15 2025, 8:30 PM
Andrew triaged this task as Medium priority.Aug 20 2025, 2:12 PM
Andrew updated the task description. (Show Details)Aug 20 2025, 10:25 PM
gerritbot added a comment.Aug 21 2025, 2:05 AM

Change #1180693 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudceph: add new OSDs: cloudcephosd1042-1051

https://gerrit.wikimedia.org/r/1180693

gerritbot added a project: Patch-For-Review.Aug 21 2025, 2:05 AM
gerritbot added a comment.Aug 21 2025, 2:10 AM

Change #1180693 merged by Andrew Bogott:

[operations/puppet@production] cloudceph: add new OSDs: cloudcephosd1042-1051

https://gerrit.wikimedia.org/r/1180693

Andrew updated the task description. (Show Details)Aug 21 2025, 2:21 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 21 2025, 2:31 AM
gerritbot added a comment.Aug 21 2025, 2:39 AM

Change #1180696 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudceph: mark out some OSD nodes not yet ready for action

https://gerrit.wikimedia.org/r/1180696

gerritbot added a project: Patch-For-Review.Aug 21 2025, 2:39 AM

Change #1180696 merged by Andrew Bogott:

[operations/puppet@production] cloudceph: mark out some OSD nodes not yet ready for action

https://gerrit.wikimedia.org/r/1180696

gerritbot added a comment.Aug 21 2025, 2:49 AM

Change #1180697 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudceph: further mark out OSD nodes not yet ready for action

https://gerrit.wikimedia.org/r/1180697

gerritbot added a comment.Aug 21 2025, 2:57 AM

Change #1180697 merged by Andrew Bogott:

[operations/puppet@production] cloudceph: further mark out OSD nodes not yet ready for action

https://gerrit.wikimedia.org/r/1180697

Andrew updated the task description. (Show Details)Aug 21 2025, 3:06 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 21 2025, 3:38 AM
gerritbot added a comment.Aug 22 2025, 6:42 PM

Change #1181178 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] New attempt to put new cloudcephosd hosts online

https://gerrit.wikimedia.org/r/1181178

gerritbot added a project: Patch-For-Review.Aug 22 2025, 6:42 PM
gerritbot added a comment.Aug 22 2025, 7:47 PM

Change #1181178 merged by Andrew Bogott:

[operations/puppet@production] New attempt to put new cloudcephosd hosts online

https://gerrit.wikimedia.org/r/1181178

Andrew updated the task description. (Show Details)Aug 22 2025, 7:57 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 22 2025, 8:31 PM
Stashbot added a comment.Aug 23 2025, 4:02 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-23T04:02:17Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Stashbot added a comment.Aug 23 2025, 4:08 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-23T04:08:27Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T401693)

gerritbot added a comment.Aug 23 2025, 4:13 AM

Change #1181221 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudceph: remove cloudcephosd1045, again

https://gerrit.wikimedia.org/r/1181221

gerritbot added a project: Patch-For-Review.Aug 23 2025, 4:13 AM
gerritbot added a comment.Aug 23 2025, 4:18 AM

Change #1181221 merged by Andrew Bogott:

[operations/puppet@production] cloudceph: remove cloudcephosd1045, again

https://gerrit.wikimedia.org/r/1181221

Stashbot added a comment.Aug 23 2025, 4:21 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-23T04:21:20Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Maintenance_bot removed a project: Patch-For-Review.Aug 23 2025, 4:30 AM
Stashbot added a comment.Aug 23 2025, 7:24 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-23T19:24:04Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T401693)

Stashbot added a comment.Aug 23 2025, 7:27 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-23T19:27:31Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Andrew updated the task description. (Show Details)Aug 23 2025, 7:33 PM
Stashbot added a comment.Aug 24 2025, 10:35 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-24T10:35:52Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T401693)

Stashbot added a comment.Aug 24 2025, 5:19 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-24T17:19:31Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Andrew updated the task description. (Show Details)Aug 24 2025, 5:23 PM
Stashbot added a comment.Aug 25 2025, 4:05 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-25T04:05:28Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T401693)

Stashbot added a comment.Aug 25 2025, 4:06 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-25T04:06:33Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Andrew updated the task description. (Show Details)Aug 25 2025, 1:06 PM
Stashbot added a comment.Aug 25 2025, 1:36 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-25T13:36:54Z] <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97) (T401693)

Andrew updated the task description. (Show Details)Aug 25 2025, 1:59 PM
Andrew updated the task description. (Show Details)Aug 25 2025, 2:39 PM
gerritbot added a comment.Aug 25 2025, 7:39 PM

Change #1181767 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Replace cloudvirt1045

https://gerrit.wikimedia.org/r/1181767

gerritbot added a project: Patch-For-Review.Aug 25 2025, 7:39 PM
gerritbot added a comment.Aug 25 2025, 9:03 PM

Change #1181767 merged by Andrew Bogott:

[operations/puppet@production] Replace cloudvirt1045

https://gerrit.wikimedia.org/r/1181767

Stashbot added a comment.Aug 25 2025, 9:07 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-25T21:07:14Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Stashbot added a comment.Aug 25 2025, 9:17 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-25T21:17:49Z] <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=97) (T401693)

Maintenance_bot removed a project: Patch-For-Review.Aug 25 2025, 9:31 PM
Stashbot added a comment.Aug 26 2025, 1:11 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-26T01:11:01Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Stashbot added a comment.Aug 26 2025, 1:20 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-26T01:20:41Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T401693)

Stashbot added a comment.Aug 26 2025, 1:20 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-26T01:20:49Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Andrew closed this task as Resolved.Aug 26 2025, 1:08 PM
Andrew updated the task description. (Show Details)
Stashbot added a comment.Aug 26 2025, 2:02 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-08-26T14:02:35Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T401693)

Stashbot added a comment.Sep 5 2025, 3:10 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-05T15:10:24Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Stashbot added a comment.Sep 5 2025, 3:15 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-05T15:15:12Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T401693)

Stashbot added a comment.Sep 5 2025, 3:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-05T15:30:28Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Stashbot added a comment.Sep 5 2025, 3:37 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-05T15:37:10Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) (T401693)

Stashbot added a comment.Sep 5 2025, 3:37 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-05T15:37:47Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Stashbot added a comment.Sep 5 2025, 3:37 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-05T15:37:54Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T401693)

Stashbot added a comment.Sep 5 2025, 3:38 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-05T15:37:59Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693)

Stashbot added a comment.Sep 5 2025, 3:38 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-09-05T15:38:05Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T401693)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits