⚓ T400877 Install new disk controllers to SM swift backends (eqiad)

Subject	Repo	Branch	Lines +/-
Return ms-be10{89,90} to the rings	operations/puppet	production	+2 -0
swift: remove ms-be10{89,90} for controller swap	operations/puppet	production	+0 -4
swift: re-add 2 nodes, drain the final 2, leave 1 for testing	operations/puppet	production	+6 -2
swift: remove 3 drained eqiad nodes for disk controller swap	operations/puppet	production	+0 -6
swift: re-add 3 nodes, drain the next 3	operations/puppet	production	+9 -3
swift: remove 3 drained eqiad nodes for disk controller swap	operations/puppet	production	+0 -6
thanos: add thanos-be1005 (JBOD), drain thanos-be2005	operations/puppet	production	+14 -1
thanos: remove thanos-be1005 from rings	operations/puppet	production	+0 -2
swift: re-add SM C-J hosts with JBOD card	operations/puppet	production	+4 -0
swift: remove old nodes, drain & reweight SM C-J nodes	operations/puppet	production	+18 -9
thanos: drain thanos-be1005 for disk controller swap	operations/puppet	production	+2 -1

Status	Assigned	Task
Resolved	MatthewVernon	T400878 Install new JBOD disk controllers into SM swift backends
Resolved	VRiley-WMF	T400877 Install new disk controllers to SM swift backends (eqiad)
Resolved	MatthewVernon	T401127 Swift device facts / names for new JBOD controllers
Invalid	None	T401203 PCC 7 regression "Error: Evaluation Error: Unknown function: 'index'"
Resolved	MatthewVernon	T401387 Swift device names should not contain underscores

VRiley-WMF moved this task from Hardware Failure / Troubleshoot to Decommission on the ops-eqiad board.Aug 21 2025, 4:29 PM

VRiley-WMF moved this task from Decommission to Blocked on the ops-eqiad board.

Change #1183627 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: remove 3 drained eqiad nodes for disk controller swap

https://gerrit.wikimedia.org/r/1183627

Change #1183628 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: re-add 3 nodes, drain the next 3

https://gerrit.wikimedia.org/r/1183628

Change #1183627 merged by MVernon:

[operations/puppet@production] swift: remove 3 drained eqiad nodes for disk controller swap

https://gerrit.wikimedia.org/r/1183627

MatthewVernon updated the task description. (Show Details)Sep 1 2025, 11:25 AM

Icinga downtime and Alertmanager silence (ID=5d9cb26e-171b-4940-aeef-3b79dd0f568e) set by mvernon@cumin2002 for 2 days, 0:00:00 on 3 host(s) and their services with reason: awaiting controller swap

ms-be[1083-1085].eqiad.wmnet

@VRiley-WMF three nodes - ms-be1083 ms-be1084 ms-be1085 are now ready for disk swaps, as soon as you've some time, please. I've downtimed them for 2 days.

Starting on ms-be1083

VRiley-WMF changed the task status from Open to In Progress.Sep 1 2025, 7:04 PM

VRiley-WMF moved this task from Blocked to Hardware Failure / Troubleshoot on the ops-eqiad board.Sep 1 2025, 8:33 PM

VRiley-WMF updated the task description. (Show Details)

ms-be1083 has been completed. moving onto ms-be1084

ms-be1084 completed. Moving onto ms-be1085

ms-be1085 is completed

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1083.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1084.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1085.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1083.eqiad.wmnet with OS bullseye executed with errors:

ms-be1083 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be1083.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1083.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1084.eqiad.wmnet with OS bullseye completed:

ms-be1084 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509020914_mvernon_647760_ms-be1084.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1085.eqiad.wmnet with OS bullseye completed:

ms-be1085 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509020917_mvernon_648375_ms-be1085.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1083.eqiad.wmnet with OS bullseye completed:

ms-be1083 (PASS)
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509021019_mvernon_655251_ms-be1083.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Change #1183628 merged by MVernon:

[operations/puppet@production] swift: re-add 3 nodes, drain the next 3

https://gerrit.wikimedia.org/r/1183628

MatthewVernon updated the task description. (Show Details)Sep 2 2025, 10:59 AM

Maintenance_bot removed a project: Patch-For-Review.Sep 2 2025, 11:32 AM

Hey @MatthewVernon I wanted to check back in with this ticket and see if any of these are available to commence with the swap. No rush, just wanted to check. Thanks!

Change #1190673 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: remove 3 drained eqiad nodes for disk controller swap

https://gerrit.wikimedia.org/r/1190673

Change #1190674 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: re-add 3 nodes, drain the final 2

https://gerrit.wikimedia.org/r/1190674

Change #1190673 merged by MVernon:

[operations/puppet@production] swift: remove 3 drained eqiad nodes for disk controller swap

https://gerrit.wikimedia.org/r/1190673

MatthewVernon updated the task description. (Show Details)Sep 23 2025, 1:51 PM

Icinga downtime and Alertmanager silence (ID=08fe2c84-09f5-45bc-a5f4-ae2812070ff6) set by mvernon@cumin2002 for 2 days, 0:00:00 on 3 host(s) and their services with reason: awaiting controller swap

ms-be[1086-1088].eqiad.wmnet

Hi @VRiley-WMF, yes there are - 3 nodes are now ready ms-be1086, ms-be1087, ms-be1088. Please swap when you've some time :)

Apologies for the delayed response, I was on leave last week.

Hi @VRiley-WMF do you think you'll be able to do these swaps this week, please?

Icinga downtime and Alertmanager silence (ID=f05e1660-c13c-4689-a96d-eaccf6967088) set by mvernon@cumin2002 for 4 days, 0:00:00 on 3 host(s) and their services with reason: awaiting controller swap

ms-be[1086-1088].eqiad.wmnet

Hey @MatthewVernon Yes, I am planning on doing this today. I apologize as I was out for two days last week.

Cool, thanks :)

Starting work on ms-be1087 (will get to ms-be1086 in a bit. starting with the cage I'm currently in)

Finished updating ms-be1087, moving onto ms-be1088

moving onto ms-be1086

These are all done! will await for the next two. Thanks @MatthewVernon

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-be1086.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-be1086.eqiad.wmnet with OS bullseye completed:

ms-be1086 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509300909_mvernon_3766436_ms-be1086.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-be1087.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-be1087.eqiad.wmnet with OS bullseye completed:

ms-be1087 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509301044_mvernon_3817060_ms-be1087.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Change #1190674 merged by MVernon:

[operations/puppet@production] swift: re-add 2 nodes, drain the final 2, leave 1 for testing

https://gerrit.wikimedia.org/r/1190674

Maintenance_bot removed a project: Patch-For-Review.Sep 30 2025, 12:32 PM

MatthewVernon updated the task description. (Show Details)Sep 30 2025, 1:38 PM

Hey @MatthewVernon Just wanted to check in and see if the other two maybe be ready? Let us now, thanks!

Hi @VRiley-WMF I'm afraid not (filesystems still about 25% full, so a little way to go yet).

Change #1198005 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: remove ms-be10{89,90} for controller swap

https://gerrit.wikimedia.org/r/1198005

gerritbot added a project: Patch-For-Review.Oct 22 2025, 9:25 AM

Change #1198005 merged by MVernon:

[operations/puppet@production] swift: remove ms-be10{89,90} for controller swap

https://gerrit.wikimedia.org/r/1198005

Maintenance_bot removed a project: Patch-For-Review.Oct 22 2025, 10:31 AM

MatthewVernon updated the task description. (Show Details)Oct 22 2025, 11:37 AM

Icinga downtime and Alertmanager silence (ID=cea00150-47a1-46ce-a142-ec46d9e47678) set by mvernon@cumin1003 for 3 days, 0:00:00 on 2 host(s) and their services with reason: awaiting controller swap

ms-be[1089-1090].eqiad.wmnet

@VRiley-WMF the last two nodes ms-be1089 and ms-be1090 are ready for controller swap, please; I've downtimed them for a couple of days.

Starting on ms-be1089

ms-be1089 is completed, moving onto the next server ms-be1090

I'm having issues bringing ms-be1090 back up. Will continue to work on this

VRiley-WMF updated the task description. (Show Details)Oct 23 2025, 1:49 AM

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1089.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1089.eqiad.wmnet with OS bullseye completed:

ms-be1089 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510231015_mvernon_3273107_ms-be1089.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Created a ticket for ms-be1090 for Supermicro to assist

Case #00061744

Closed by mistake

Jclark-ctr mentioned this in T408478: Unresponsive management for ms-be1090.mgmt:22.Oct 28 2025, 11:43 AM

Attempted to swap the unit and it wouldn't power back on.
Swapped it back out with the old one, and it still won't power on.
Checked the cables and reseated them with no change.
When looking at the unit, it seems to not detect the PSU.
Support came back with the following and I have responded to them with the following.

Ensure that all cables, including power and SAS cables from drives to the backplane, are properly connected. - This has been completed with no change.

Check for the power button and ribbon cable issues. - This has been completed with no change.

Please remove the Controller and try to power on the server - This has been completed, no change.

Please share the server health event log and sensor log - I have attached it to the email.

awaiting their response.

Removed all RAM from unit (except 1) to see if it would boot. Found that it did boot normally. I'm slowly adding more RAM to find out which one of these is causing the issue.

After reseating the RAM, it seems lke everything has come back up and it's showing a healthy status. @MatthewVernon Can you please verify if everything looks good on your end for ms-be1090?

@VRiley-WMF the host is up, but it can't reach any of its spinning disks (the OS sees none, and the BMC says 0 physical disks). Could you take another look, please?

Of course, I'm looking into this now.

Hey @MatthewVernon I apologize about that. It seems the cable slipped out of the card while I was trying to diagnose the issue. It's been reseated. Please check it when possible

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1003 for host ms-be1090.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1003 for host ms-be1090.eqiad.wmnet with OS bullseye completed:

ms-be1090 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced UEFI HTTP Boot for next reboot
- Host rebooted via Redfish
- Host up (Debian installer)
- Add puppet_version metadata (7) to Debian installer
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510310917_mvernon_1484055_ms-be1090.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB