Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-14T11:57:33Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-14T11:57:39Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-14T11:58:11Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1048' (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-14T12:12:28Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1048' (T431682)

@VRiley-WMF I thought we could start with one host, cloudvirt1048 and test-run the whole procedure there. Once we have nailed the process then we can do the rest in batch, what do you think?

cloudvirt1048 is now in maintenance and drained of VMs, I also silenced CloudVirtDown page for this host. We are ready to start https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Move_existing_server_between_rows/racks,_changing_IPs

I'm going through https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Move_existing_server_between_rows/racks,_changing_IPs and figuring out ownership for the various steps, cc @cmooney @ayounsi

• Remove the host from active production (depool, failover, etc.)
• Run the <code>'''sre.hosts.decommission'''</code> cookbook with the <code>--keep-mgmt-dns</code>, see [[Spicerack/Cookbooks#Run_a_single_Cookbook]]

I'm happy to do this part, in fact we can do it at any time now

• Physically relocate the host now.
• Update Netbox
  • Update Netbox device page with new rack location
  • Delete the cables connected to the device's interfaces (write down their ID if going to reuse them)
  • Delete *all* Netbox interfaces for the device, except for the 'mgmt' one. '''DO NOT USE THE "DELETE" BUTTON IN THE UPPER-RIGHT-HAND CORNER, THIS WILL COMPLETELY DELETE THE DEVICE!'''
  • Change Netbox status for the device from 'decommissioned' to 'planned'
  • Run the [https://netbox.wikimedia.org/extras/scripts/9/ Provision Server] Netbox script, providing the new switch and port, along with the cable ID and vlan detail (as normal, ask for help if in doubt).
  • Verify that the device now has 2 interfaces in Netbox, mgmt and PRIMARY , each with an IP assigned
• Run the <code>'''sre.dns.netbox'''</code> cookbook: [[DNS/Netbox#Update_generated_records]]
• Run the <code>'''sre.network.configure-switch-interfaces'''</code> cookbook to configure the switch interface
• If any firmware upgrades are required to reimage the server, apply them now (this is no different than any reimage/upgrade)

I'd prefer if someone more expert on netbox would assist with the steps up until here, after that I'm happy to take over the steps below. I'll also note that yes these servers are in more than one vlan, I'll put them back on cloud-private.

• Patch puppet:
  • This steps here only adjust Netbox records and those generated from them.
  • Any hardcoded references to the old IP addresses in the puppet repo need to be changed also
  • This will vary from server to server depending on what it does, what talks to it
  • Bear in mind interface names can also change after reimage (if OS or firmware upgraded during process), so references to those may also need updating
• Run puppet on the install servers: <code>cumin 'A:installserver' 'run-puppet-agent -q'</code>
• Run the <code>'''sre.hosts.reimage'''</code> cookbook with the <code>--new</code> option, following the procedure at [[Server Lifecycle/Reimage]]
• If the server has more than a single network interface:
  • The reimage will run the [https://netbox.wikimedia.org/extras/scripts/interface_automation.ImportPuppetDB/ PuppetDB import script] to add any additional host interfaces / switch connections in Netbox
  • In Netbox set the correct Vlan(s) on the switch ports for the 2nd and subsequent interfaces (if unsure ask netops what is needed)
  • Run the the <code>'''sre.network.configure-switch-interfaces'''</code> cookbook to configure the additional interfaces on the switch

please let me know what you think

Thanks @fgiunchedi! I will be running through this today. I just got back from vacation and catching up on a few different things here. I will let you know when this first one is done.

In T431682#12119607, @VRiley-WMF wrote:

Thanks @fgiunchedi! I will be running through this today. I just got back from vacation and catching up on a few different things here. I will let you know when this first one is done.

Sweet, thank you so much! Please let me know how I can help/assist, also this is not super urgent and I'll be OOO Thurs afternoon and all day Friday JFYI

Hey @fgiunchedi thanks for your patience. As it turns out there is a lot going on here at eqiad. I know you're currently out, but I'm planning on hitting this next week.

cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: cloudvirt1048.eqiad.wmnet

• cloudvirt1048.eqiad.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Wiped all swraid, partition-table and filesystem signatures
  • Powered off
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Configured the linked switch interface(s)
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB

@VRiley-WMF I realized I should be doing the decommission cloudvirt1048 myself, host is now decom'd (mgmt dns is left intact)

Please relocate cloudvirt1048 to C8 when you can, also would you mind doing this procedure up to and including the firmware upgrades? I'll take over from there (reimage, add extra vlans, etc) https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Move_existing_server_between_rows/racks,_changing_IPs ? thank you!

Sure thing, I'm planning on this tomorrow. Thank you!

Moved unit into

C8
U26
cable ID: 5346
Port 26

Updating netbox now.

Updating firmware on cloudvirt1048

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1048.eqiad.wmnet with OS trixie

@fgiunchedi I was able to get through a good majority of this process, but I'm getting a hangup from the reimage. It seems like it's seeing all the correct ports and the fiber has activity. Should I try the reimage with something other than trixie?

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1048.eqiad.wmnet with OS trixie executed with errors:

• cloudvirt1048 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudvirt1048.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1048.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloudvirt1048.eqiad.wmnet with OS trixie

Change #1319749 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] installserver: move cloudvirt1048 to efi

https://gerrit.wikimedia.org/r/1319749

Change #1319749 merged by Filippo Giunchedi:

[operations/puppet@production] installserver: move cloudvirt1048 to efi

https://gerrit.wikimedia.org/r/1319749

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1048.eqiad.wmnet with OS trixie executed with errors:

• cloudvirt1048 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudvirt1048.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1003 for host cloudvirt1048.eqiad.wmnet with OS trixie

I am working on getting cloudvirt1048 up an running in C8 after the move. The current situation is the following:

• host is 'active' in netbox, sync-hiera happened
• I have assigned addresses for cloud-private and run the dns cookbook
• reimage is running, waiting for a successful puppet run
• puppet runs, though fails to complete because cloud-private hosts are unreachable (e.g. rabbitmq01.eqiad1.wikimediacloud.org)
• vlans are configured on the host, though not in netbox and therefore not on the switches either

I am not quite sure what to do next and/or how to make vlan interfaces appear in https://netbox.wikimedia.org/dcim/devices/4126/interfaces/ similar to what happens with e.g. cloudvirt1047 https://netbox.wikimedia.org/dcim/devices/3937/interfaces/

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1003 for host cloudvirt1048.eqiad.wmnet with OS trixie completed:

• cloudvirt1048 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202607310719_filippo_3941183_cloudvirt1048.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Ok baby steps: reimage wasn't completing because puppet wasn't able to complete a successful run. Specifically nova-compute.service can't start because it can't talk to rabbit, for now I hacked my way around it because I wanted to see the reimage cookbook finish:

# cat /etc/systemd/system/nova-compute.service.d/override.conf 
[Service]
Type=simple
ExecStart=
ExecStart=/usr/bin/true

Indeed reimage also fixed up the interfaces in netbox:

[info] Processing baremetal device cloudvirt1048 (WMF11480)
[success] cloudvirt1048: renamed ##PRIMARY## interface to eno1np0
[info] Skipping br-int as we do not import openvswitch interfaces.
[info] Skipping br-internal as we do not import openvswitch interfaces.
[info] Skipping br-tun as we do not import openvswitch interfaces.
[info] Skipping ovs-system as we do not import openvswitch interfaces.
[info] cloudvirt1048.eqiad.wmnet assign_name: 10.64.151.12/24 already has correct DNS name.
[info] 10.64.151.12/24 is already primary for cloudvirt1048 (WMF11480)
[info] cloudvirt1048.eqiad.wmnet assign_name: 2620:0:861:11f:10:64:151:12/64 already has correct DNS name.
[info] 2620:0:861:11f:10:64:151:12/64 is already primary for cloudvirt1048 (WMF11480)
[info] Creating interface br-int for device cloudvirt1048 (WMF11480)
[info] Creating interface br-internal for device cloudvirt1048 (WMF11480)
[info] Creating interface br-tun for device cloudvirt1048 (WMF11480)
[info] Creating interface eno2np1 for device cloudvirt1048 (WMF11480)
[info] Creating interface eno3 for device cloudvirt1048 (WMF11480)
[info] Creating interface eno4 for device cloudvirt1048 (WMF11480)
[info] Creating interface ovs-system for device cloudvirt1048 (WMF11480)
[info] Added vlan 1105 to eno1np0 tagged vlans
[info] Set vlan 1105 as untagged vlan on vlan1105
[info] Added vlan 1151 to eno1np0 tagged vlans
[info] Set vlan 1151 as untagged vlan on vlan1151
[info] Assigning 172.20.1.3/24 to cloudvirt1048 (WMF11480):vlan1151
[info] Assigning 2a02:ec80:a000:201::3/64 to cloudvirt1048 (WMF11480):vlan1151
[info] Set cloudsw1-c8-eqiad xe-0/0/26 tagged vlans to [<VLAN: cloud-instances2-b-eqiad (1105)>, <VLAN: cloud-private-c8-eqiad (1151)>] matching eno1np0
[info] cloudvirt1048 (WMF11480) done.

pinging hosts on cloud-private though still doesn't work:

root@cloudvirt1048:~# ping -c2 -4 rabbitmq01.eqiad1.wikimediacloud.org
PING rabbitmq01.eqiad1.wikimediacloud.org (172.20.1.17) 56(84) bytes of data.
From cloudvirt1048.private.eqiad.wikimedia.cloud (172.20.1.3) icmp_seq=1 Destination Host Unreachable
From cloudvirt1048.private.eqiad.wikimedia.cloud (172.20.1.3) icmp_seq=2 Destination Host Unreachable

--- rabbitmq01.eqiad1.wikimediacloud.org ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1014ms
pipe 2
root@cloudvirt1048:~# ping -c2 -6 rabbitmq01.eqiad1.wikimediacloud.org
PING rabbitmq01.eqiad1.wikimediacloud.org (2a02:ec80:a000:201::17) 56 data bytes
From cloudvirt1048.private.eqiad.wikimedia.cloud (2a02:ec80:a000:201::3) icmp_seq=1 Destination unreachable: Address unreachable
From cloudvirt1048.private.eqiad.wikimedia.cloud (2a02:ec80:a000:201::3) icmp_seq=2 Destination unreachable: Address unreachable

--- rabbitmq01.eqiad1.wikimediacloud.org ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1017ms

root@cloudvirt1048:~# ip -6 r
2620:0:861:11f::/64 dev eno1np0 proto kernel metric 256 pref medium
2a02:ec80:a000:1::/64 via 2a02:ec80:a000:201::1 dev vlan1151 metric 1024 pref medium
2a02:ec80:a000:100::/64 via 2a02:ec80:a000:201::1 dev vlan1151 metric 1024 pref medium
2a02:ec80:a000:201::/64 dev vlan1151 proto kernel metric 256 pref medium
2a02:ec80:a000:200::/56 via 2a02:ec80:a000:201::1 dev vlan1151 metric 1024 pref medium
2a02:ec80:a000:4000::/64 via 2a02:ec80:a000:201::1 dev vlan1151 metric 1024 pref medium
fe80::/64 dev eno1np0 proto kernel metric 256 pref medium
fe80::/64 dev vlan1151 proto kernel metric 256 pref medium
fe80::/64 dev vlan1105 proto kernel metric 256 pref medium
default via fe80::8ad9:8f04:68b6:e0e0 dev eno1np0 proto ra metric 1024 expires 596sec hoplimit 64 pref medium
root@cloudvirt1048:~# ip -4 r
default via 10.64.151.1 dev eno1np0 onlink 
10.64.151.0/24 dev eno1np0 proto kernel scope link src 10.64.151.12 
172.16.0.0/21 via 172.20.1.1 dev vlan1151 
172.16.8.0/21 via 172.20.1.1 dev vlan1151 
172.16.16.0/21 via 172.20.1.1 dev vlan1151 
172.16.24.0/24 via 172.20.1.1 dev vlan1151 
172.20.0.0/16 via 172.20.1.1 dev vlan1151 
172.20.1.0/24 dev vlan1151 proto kernel scope link src 172.20.1.3 
185.15.56.0/24 via 172.20.1.1 dev vlan1151 
root@cloudvirt1048:~#

And I thought switch interfaces needed to be updated, though homer reports no diffs:

root@cumin1003:~# cookbook sre.network.configure-switch-interfaces --homer cloudvirt1048
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='netbox.discovery.wmnet', port=443): Read timed out. (read timeout=5.0)")': /api/dcim/devices/?name=cloudvirt1048&limit=0
Acquired lock for key /spicerack/locks/cookbooks/sre.network.configure-switch-interfaces: {'concurrency': 20, 'created': '2026-07-31 14:04:27.746844+00:00', 'owner': 'filippo@cumin1003 [3995186]', 'ttl': 1800}
START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1048
INFO:homer.devices:Initialized 131 devices
INFO:homer:Committing config for query cloudsw1-c8-eqiad.* with message: Ran from cookbook cookbooks.sre.network
INFO:homer:Gathering global Netbox data
INFO:homer.devices:Matched 1 device(s) for query 'cloudsw1-c8-eqiad.*'
INFO:homer:Generating configuration for cloudsw1-c8-eqiad.mgmt.eqiad.wmnet
INFO:homer.transports.junos:Empty diff for cloudsw1-c8-eqiad.mgmt.eqiad.wmnet, skipping device.
INFO:homer:Homer run completed successfully on 1 devices: ['cloudsw1-c8-eqiad.mgmt.eqiad.wmnet']
Released lock for key /spicerack/locks/cookbooks/sre.network.configure-switch-interfaces: {'concurrency': 20, 'created': '2026-07-31 14:04:27.746844+00:00', 'owner': 'filippo@cumin1003 [3995186]', 'ttl': 1800}
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1048

I feel like I'm missing something obvious here, any idea of what that might be @cmooney @ayounsi ?

The switch port was missing the required tagged VLANs (cloud-instances and cloud-private), I fixed that with https://netbox.wikimedia.org/extras/changelog/290186/ and ran Homer which applied the following diff:

[edit interfaces xe-0/0/26]
+   native-vlan-id 1128;
[edit interfaces xe-0/0/26 unit 0 family ethernet-switching]
-      interface-mode access;
+      interface-mode trunk;
[edit interfaces xe-0/0/26 unit 0 family ethernet-switching vlan]
-       members cloud-hosts1-c8-eqiad;
+       members [ cloud-hosts1-c8-eqiad cloud-instances2-b-eqiad cloud-private-c8-eqiad ];

After that I can successfully ping cloud-private targets:

taavi@cloudvirt1048 ~ $ ping -c2 -6 rabbitmq01.eqiad1.wikimediacloud.org
PING rabbitmq01.eqiad1.wikimediacloud.org (2a02:ec80:a000:201::17) 56 data bytes
64 bytes from cloudrabbit1001.private.eqiad.wikimedia.cloud (2a02:ec80:a000:201::17): icmp_seq=1 ttl=64 time=0.147 ms
64 bytes from cloudrabbit1001.private.eqiad.wikimedia.cloud (2a02:ec80:a000:201::17): icmp_seq=2 ttl=64 time=0.143 ms

--- rabbitmq01.eqiad1.wikimediacloud.org ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.143/0.145/0.147/0.002 ms

Thank you @taavi, appreciate it!

I don't understand what this line in the reimage cookbook output did or didn't do:

[info] Set cloudsw1-c8-eqiad xe-0/0/26 tagged vlans to [<VLAN: cloud-instances2-b-eqiad (1105)>, <VLAN: cloud-private-c8-eqiad (1151)>] matching eno1np0
``

Good question. Looking at the Netbox changelog that change was first made in 290184 and then reverted as 290185. What's interesting is that the change and the revert happened on the same second and with the same request ID?

It seems like the code (https://gerrit.wikimedia.org/g/operations/software/netbox-extras/+/3887a7014907cd10325e14ae697efc721e9f3e96/scripts_imports/wmf_scripts_imports/common.py#486) is at least not updating the interface mode from access to tagged (which is needed in the interface to have tagged VLANs in addition to the untagged one).

In the meantime I have restore nova-compute and brought it up:

| 79d5cff2-1f5b-423d-bd9a-6119597f8aaa | nova-compute   | cloudvirt1048      | nova     | disabled | up    | 2026-07-31T16:09:42.000000 |

Host works as expected and IMHO can be brought out of maint. I'm going to hold off on a Fri afternoon just in case, though everything seems in order.

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-03T14:50:14Z] <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-03T14:50:20Z] <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T431682)

I put 1048 back in service and it seems to be working fine.

@Andrew Awesome, sounds good. Would we like to start on 1049?

In T431682#12174859, @taavi wrote:

Good question. Looking at the Netbox changelog that change was first made in 290184 and then reverted as 290185. What's interesting is that the change and the revert happened on the same second and with the same request ID?

It seems like the code (https://gerrit.wikimedia.org/g/operations/software/netbox-extras/+/3887a7014907cd10325e14ae697efc721e9f3e96/scripts_imports/wmf_scripts_imports/common.py#486) is at least not updating the interface mode from access to tagged (which is needed in the interface to have tagged VLANs in addition to the untagged one).

Fascinating, thank you for digging into that.

From what I could gather so far the procedure is as follows, applies for a single or multiple hosts:

1. wmcs sets insetup_noferm puppet role for the host in question and efi partman recipes
2. wmcs runs decom cookbook
3. dcops relocates the host
4. dcops updates netbox, firmware upgrade, etc, and reimage
5. wmcs sets wmcs::openstack::eqiad1::virt_ceph and any required dns names
6. wmcs reimages host with the new role and puts it back in service

It would be nice if we could avoid the "change the role" dance, though IMHO we don't exercise the "relocate a server" procedure often enough to justify spending time now on flawless move + reimage.

In T431682#12180113, @VRiley-WMF wrote:

@Andrew Awesome, sounds good. Would we like to start on 1049?

I was thinking of doing the remaining six hosts in one batch, meaning we would take all of them out of service and they would all be ready for you. Would that work for you? If all moves could be completed within a week of starting that would be awesome

Hey @fgiunchedi Yes, that would work for me! Would monday next week work for you?

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T07:29:57Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1049' (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T07:38:04Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1049' (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T07:39:50Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1051' (T431682)

In T431682#12209331, @VRiley-WMF wrote:

Hey @fgiunchedi Yes, that would work for me! Would monday next week work for you?

Monday sounds great! I'll get started on draining the hosts and getting them ready for you

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T07:56:27Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1051' (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T08:02:00Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1054' (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T08:19:48Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1054' (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T08:22:12Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1055' (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T08:42:41Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1055' (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T09:01:38Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1056' (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T09:18:18Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1056' (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T09:30:04Z] <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1057' (T431682)

Mentioned in SAL (#wikimedia-cloud-feed) [2026-08-13T09:46:23Z] <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1057' (T431682)

Change #1325458 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] site: get cloudvirts ready for E4 -> C8 move

https://gerrit.wikimedia.org/r/1325458

Change #1325459 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] installserver: switch cloudvirt to efi standard recipes

https://gerrit.wikimedia.org/r/1325459

Change #1325458 merged by Filippo Giunchedi:

[operations/puppet@production] site: get cloudvirts ready for E4 -> C8 move

https://gerrit.wikimedia.org/r/1325458

cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: cloudvirt1049.eqiad.wmnet

• cloudvirt1049.eqiad.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Wiped all swraid, partition-table and filesystem signatures
  • Powered off
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Configured the linked switch interface(s)
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB

cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: cloudvirt1051.eqiad.wmnet

• cloudvirt1051.eqiad.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Wiped all swraid, partition-table and filesystem signatures
  • Powered off
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Configured the linked switch interface(s)
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB

cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: cloudvirt1054.eqiad.wmnet

• cloudvirt1054.eqiad.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Wiped all swraid, partition-table and filesystem signatures
  • Powered off
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Configured the linked switch interface(s)
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB

cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: cloudvirt1055.eqiad.wmnet

• cloudvirt1055.eqiad.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Wiped all swraid, partition-table and filesystem signatures
  • Powered off
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Configured the linked switch interface(s)
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB

cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: cloudvirt1056.eqiad.wmnet

• cloudvirt1056.eqiad.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Wiped all swraid, partition-table and filesystem signatures
  • Powered off
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Configured the linked switch interface(s)
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB

cookbooks.sre.hosts.decommission executed by filippo@cumin1003 for hosts: cloudvirt1057.eqiad.wmnet

• cloudvirt1057.eqiad.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Wiped all swraid, partition-table and filesystem signatures
  • Powered off
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Configured the linked switch interface(s)
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB

In T431682#12210883, @fgiunchedi wrote:

In T431682#12209331, @VRiley-WMF wrote:

Hey @fgiunchedi Yes, that would work for me! Would monday next week work for you?

Monday sounds great! I'll get started on draining the hosts and getting them ready for you

Hosts are now decom'd and ready for you. reimage might fail but that's okay we can take it from there once physical move/netbox/provision/fw-update are done

Perfect. I will be starting to take these down and moving them now.

Servers have been physically relocated. Will start powering on the units and updating netbox shortly.

1049
CableID 5295
Row 14
Port 9

1051
CableID 5297
Row 15
Port 7

1054
CableID 5352
Row 16
Port 36

1055
CableID 5355
Row 18
Port 39

1056
CableID 5331
Row 27
Port 45

1057
CableID 5328
Row 29
Port 42

Change #1325459 merged by Filippo Giunchedi:

[operations/puppet@production] installserver: switch cloudvirt to efi standard recipes

https://gerrit.wikimedia.org/r/1325459

Updated Netbox device page with new rack location
Deleted the cables connected to the device's interfaces
Deleted *all* Netbox interfaces for the device, except for the 'mgmt' one.

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1049.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1049.eqiad.wmnet with OS trixie executed with errors:

• cloudvirt1049 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudvirt1049.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1049.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1049.eqiad.wmnet with OS trixie completed:

• cloudvirt1049 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608171352_vriley_466291_cloudvirt1049.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1051.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1055.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1051.eqiad.wmnet with OS trixie completed:

• cloudvirt1051 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced UEFI HTTP Boot for next reboot
  • Host rebooted via Redfish
  • Host up (Debian installer)
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202608171448_vriley_476700_cloudvirt1051.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1056.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1054.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1055.eqiad.wmnet with OS trixie executed with errors:

• cloudvirt1055 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudvirt1055.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Hey @fgiunchedi it seems as though that cloudvirt1054, cloudvirt1055, cloudvirt1056, cloudvirt1057 are having issues. I did noticed that these servers were a bit different than cloudvirt1048, cloudvirt1049, cloudvirt1051. Would you be able to look into this a bit?

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host cloudvirt1057.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1056.eqiad.wmnet with OS trixie executed with errors:

• cloudvirt1056 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudvirt1056.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1054.eqiad.wmnet with OS trixie executed with errors:

• cloudvirt1054 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudvirt1054.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host cloudvirt1057.eqiad.wmnet with OS trixie executed with errors:

• cloudvirt1057 (FAIL)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cloudvirt1057.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.