Page MenuHomePhabricator
Create Task
Maniphest T309789

[ceph] Upgrade hosts to bullseye
Closed, ResolvedPublic
Actions

Description

This was blocked by not having the packages, but with T309786 we unblock this one.

codfw

  • cloudcephmon2004-dev
  • cloudcephmon2005-dev
  • cloudcephmon2006-dev
  • cloudcephosd2001-dev (replaced with new hardware)
  • cloudcephosd2002-dev (replaced with new hardware)
  • cloudcephosd2003-dev (replaced with new hardware)

eqiad

  • cloudcephosd1006
  • cloudcephosd1007
  • cloudcephosd1008
  • cloudcephosd1009
  • cloudcephosd1011
  • cloudcephosd1012
  • cloudcephosd1013
  • cloudcephosd1014
  • cloudcephosd1015
  • cloudcephosd1016
  • cloudcephosd1017
  • cloudcephosd1018
  • cloudcephosd1019
  • cloudcephosd1020
  • cloudcephosd1022
  • cloudcephosd1023
  • cloudcephosd1024
  • cloudcephmon1001 -- replaced with cloudcephmon1004
  • cloudcephmon1002 -- replaced with cloudcephmon1005
  • cloudcephmon1003 -- replaced with cloudcephmon1006

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Jan 24 2025, 3:02 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1013 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1013.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 24 2025, 3:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2025, 3:38 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1013 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1013.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 24 2025, 4:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 25 2025, 3:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 25 2025, 3:12 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1013 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1013.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 25 2025, 3:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 25 2025, 3:27 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1013 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1013.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 25 2025, 3:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 25 2025, 3:34 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1013 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1013.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 26 2025, 3:22 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1013 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1013.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 26 2025, 3:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 26 2025, 4:44 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1013 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1013.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 26 2025, 4:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 26 2025, 5:18 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1013 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1013.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 26 2025, 5:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 27 2025, 1:10 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye executed with errors:

  • cloudcephosd1013 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudcephosd1013.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 27 2025, 1:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 27 2025, 1:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 27 2025, 2:30 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1002 for host cloudcephosd1013.eqiad.wmnet with OS bullseye completed:

  • cloudcephosd1013 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501271413_andrew_2581557_cloudcephosd1013.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Aklapper changed the task status from In Progress to Open.Mar 22 2025, 7:23 AM
Aklapper subscribed.

Resetting task status from "In Progress" to "Open" as this task has been "in progress" for more than two years.

dcaro removed dcaro as the assignee of this task.May 22 2025, 7:39 AM

Wa

dcaro claimed this task.May 22 2025, 7:40 AM

This is still ongoing work :/

Andrew updated the task description. (Show Details)Jun 12 2025, 2:13 AM
Stashbot added a comment.Jun 12 2025, 2:19 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-12T02:19:02Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

Stashbot added a comment.Jun 12 2025, 5:23 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-12T05:23:14Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T309789)

gerritbot added a comment.Jun 12 2025, 11:13 AM

Change #1156302 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudcephosd1015 -> puppet 7

https://gerrit.wikimedia.org/r/1156302

gerritbot added a project: Patch-For-Review.Jun 12 2025, 11:13 AM

Change #1156303 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudcephosd1016 -> puppet 7

https://gerrit.wikimedia.org/r/1156303

gerritbot added a comment.Jun 12 2025, 11:13 AM

Change #1156304 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudcephosd1017 -> puppet 7

https://gerrit.wikimedia.org/r/1156304

gerritbot added a comment.Jun 12 2025, 11:13 AM

Change #1156305 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudcephosd1018 -> puppet 7

https://gerrit.wikimedia.org/r/1156305

gerritbot added a comment.Jun 12 2025, 11:13 AM

Change #1156306 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudcephosd1019 -> puppet 7

https://gerrit.wikimedia.org/r/1156306

gerritbot added a comment.Jun 12 2025, 11:13 AM

Change #1156307 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudcephosd1020 -> puppet 7

https://gerrit.wikimedia.org/r/1156307

Stashbot added a comment.Jun 12 2025, 11:41 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-12T11:41:35Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

gerritbot added a comment.Jun 12 2025, 2:44 PM

Change #1156375 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudcephosd1014: update nic names

https://gerrit.wikimedia.org/r/1156375

gerritbot added a comment.Jun 12 2025, 2:45 PM

Change #1156375 merged by Andrew Bogott:

[operations/puppet@production] cloudcephosd1014: update nic names

https://gerrit.wikimedia.org/r/1156375

gerritbot added a comment.Jun 12 2025, 3:13 PM

Change #1156302 merged by Andrew Bogott:

[operations/puppet@production] cloudcephosd1015 -> puppet 7

https://gerrit.wikimedia.org/r/1156302

Andrew updated the task description. (Show Details)Jun 12 2025, 3:32 PM
RobH subscribed.EditedJun 12 2025, 4:57 PM

cloudcephosd1015 reimage troubleshooting things I checked:

  • The PXE boot setting is indeed set to the 10G NIC's port 0 as expected.
  • Boot system with: sudo cookbook sre.hosts.reimage --os bookworm cloudcephosd1015
  • System fires into reimage script, but when sending for DHCP the following shows on the install1004 syslog, but no DHCP accept, it then fails into the normal OS load:
Jun 12 16:51:29 install1004 dhcpd[60964]: DHCPDISCOVER from bc:97:e1:28:3a:10 via 10.64.20.252
Jun 12 16:51:29 install1004 dhcpd[60964]: DHCPOFFER on 10.64.20.66 to bc:97:e1:28:3a:10 via 10.64.20.252

So it appears its sending in the request, and our install host is sending back the offer, but the offer is not being received by cloudcephosd1015 and it fails to booting off disk instead.

Stashbot added a comment.Jun 12 2025, 5:51 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-12T17:51:05Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T309789)

cmooney subscribed.Jun 12 2025, 6:44 PM
In T309789#10910380, @RobH wrote:

So it appears its sending in the request, and our install host is sending back the offer, but the offer is not being received by cloudcephosd1015 and it fails to booting off disk instead.

Thanks Rob. Yeah to check I connected to the host over mgmt console, and then deleted its static IP and manually ran dhclient. Which worked fine, i.e. the DHCP OFFER made it back to the host, the host processed it and replied with REQ etc. You can see the flow in P77858

So the issue is not network comms from the switch/port to the install server and back, or due to any misconfig with vlan etc. Not sure why the OFFER would be ignored at PXEboot stage, not sure I've seen that one before.

Stashbot added a comment.Jun 12 2025, 7:18 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-12T19:18:00Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

Stashbot added a comment.Jun 12 2025, 7:49 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-12T19:49:02Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T309789)

Andrew updated the task description. (Show Details)Jun 12 2025, 8:29 PM
gerritbot added a comment.Jun 12 2025, 8:46 PM

Change #1156303 merged by Andrew Bogott:

[operations/puppet@production] cloudcephosd1016 -> puppet 7

https://gerrit.wikimedia.org/r/1156303

Andrew updated the task description. (Show Details)Jun 12 2025, 9:43 PM
Stashbot added a comment.Jun 13 2025, 12:36 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-13T00:36:46Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

Stashbot added a comment.Jun 13 2025, 12:57 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-13T00:57:38Z] <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (T309789)

gerritbot added a comment.Jun 13 2025, 3:02 AM

Change #1156304 merged by Andrew Bogott:

[operations/puppet@production] cloudcephosd1017 -> puppet 7

https://gerrit.wikimedia.org/r/1156304

Stashbot added a comment.Jun 13 2025, 4:07 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-13T04:07:00Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

Andrew updated the task description. (Show Details)Jun 13 2025, 4:07 AM
Stashbot added a comment.Jun 13 2025, 6:31 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-13T06:31:21Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T309789)

Stashbot added a comment.Jun 13 2025, 11:46 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-13T11:46:18Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

gerritbot added a comment.Jun 13 2025, 12:18 PM

Change #1156305 merged by Andrew Bogott:

[operations/puppet@production] cloudcephosd1018 -> puppet 7

https://gerrit.wikimedia.org/r/1156305

Stashbot added a comment.Jun 13 2025, 3:50 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-13T15:50:14Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T309789)

gerritbot added a comment.Jun 13 2025, 7:15 PM

Change #1156306 merged by Andrew Bogott:

[operations/puppet@production] cloudcephosd1019 -> puppet 7

https://gerrit.wikimedia.org/r/1156306

RobH unsubscribed.Jun 13 2025, 7:40 PM
Stashbot added a comment.Jun 13 2025, 8:13 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-13T20:13:15Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

Andrew updated the task description. (Show Details)Jun 13 2025, 10:27 PM
Stashbot added a comment.Jun 14 2025, 12:18 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T00:17:58Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T309789)

Stashbot added a comment.Jun 14 2025, 1:38 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T01:38:44Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

Stashbot added a comment.Jun 14 2025, 3:51 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T03:51:35Z] <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.ceph.osd.drain_node (exit_code=97) (T309789)

Stashbot added a comment.Jun 14 2025, 3:55 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T03:55:25Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

Stashbot added a comment.Jun 14 2025, 3:56 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T03:56:17Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T309789)

Stashbot added a comment.Jun 14 2025, 3:56 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T03:56:47Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

Stashbot added a comment.Jun 14 2025, 3:57 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T03:57:47Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T309789)

Stashbot added a comment.Jun 14 2025, 4:23 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T04:23:01Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

gerritbot added a comment.Jun 14 2025, 4:23 AM

Change #1156307 merged by Andrew Bogott:

[operations/puppet@production] cloudcephosd1020 -> puppet 7

https://gerrit.wikimedia.org/r/1156307

Stashbot added a comment.Jun 14 2025, 4:23 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T04:23:51Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) (T309789)

Stashbot added a comment.Jun 14 2025, 4:25 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T04:25:54Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

Maintenance_bot removed a project: Patch-For-Review.Jun 14 2025, 4:30 AM
Stashbot added a comment.Jun 14 2025, 7:29 AM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T07:29:48Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T309789)

gerritbot added a comment.Jun 14 2025, 12:48 PM

Change #1157505 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudcephosd1022: prepare for Bullseye upgrade

https://gerrit.wikimedia.org/r/1157505

gerritbot added a project: Patch-For-Review.Jun 14 2025, 12:48 PM

Change #1157506 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudcephosd1023: prepare for Bullseye upgrade

https://gerrit.wikimedia.org/r/1157506

gerritbot added a comment.Jun 14 2025, 12:48 PM

Change #1157507 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudcephosd1024: prepare for Bullseye upgrade

https://gerrit.wikimedia.org/r/1157507

gerritbot added a comment.Jun 14 2025, 12:51 PM

Change #1157505 merged by Andrew Bogott:

[operations/puppet@production] cloudcephosd1022: prepare for Bullseye upgrade

https://gerrit.wikimedia.org/r/1157505

Stashbot added a comment.Jun 14 2025, 1:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T13:30:57Z] <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.drain_node (T309789)

Stashbot added a comment.Jun 14 2025, 6:44 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2025-06-14T18:44:34Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.drain_node (exit_code=0) (T309789)

gerritbot added a comment.Jun 14 2025, 6:53 PM

Change #1157506 merged by Andrew Bogott:

[operations/puppet@production] cloudcephosd1023: prepare for Bullseye upgrade

https://gerrit.wikimedia.org/r/1157506

gerritbot added a comment.Jun 14 2025, 9:16 PM

Change #1157507 merged by Andrew Bogott:

[operations/puppet@production] cloudcephosd1024: prepare for Bullseye upgrade

https://gerrit.wikimedia.org/r/1157507

Maintenance_bot removed a project: Patch-For-Review.Jun 14 2025, 9:30 PM
Andrew updated the task description. (Show Details)Jun 14 2025, 10:40 PM
gerritbot added a comment.Jun 16 2025, 12:19 PM

Change #1159425 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] codfw1dev ceph: cloudcephmons -> puppet 7

https://gerrit.wikimedia.org/r/1159425

gerritbot added a project: Patch-For-Review.Jun 16 2025, 12:20 PM
Andrew updated the task description. (Show Details)Jun 16 2025, 12:21 PM
gerritbot added a comment.Jun 16 2025, 1:18 PM

Change #1159425 merged by Andrew Bogott:

[operations/puppet@production] codfw1dev ceph: cloudcephmons -> puppet 7

https://gerrit.wikimedia.org/r/1159425

Maintenance_bot removed a project: Patch-For-Review.Jun 16 2025, 1:31 PM
Andrew mentioned this in T393614: Q4:rack/setup/install cloudcephosd200[567]-dev.Jun 16 2025, 2:23 PM
Andrew updated the task description. (Show Details)Jun 16 2025, 9:08 PM
Andrew updated the task description. (Show Details)Jul 2 2025, 2:03 AM
taavi closed this task as Resolved.Jul 2 2025, 8:44 AM
fnegri moved this task from In progress to Done on the cloud-services-team (FY2024/2025-Q3-Q4) board.Jul 9 2025, 2:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits