Are ganeti4006/4008 the only servers which will be touched in this maintenance? Then I would already go ahead and drain/remove ganeti4006

Only hosts in this one rack will be touched: https://netbox.wikimedia.org/dcim/racks/34/ but we'll depool the entire site before work starts.

So that should only be ganeti4006 and ganeti4008 according to Netbox.

Ack, thanks!

Draining ganeti4006.ulsfo.wmnet of running VMs

Draining ganeti4006.ulsfo.wmnet of running VMs

Mentioned in SAL (#wikimedia-operations) [2026-04-30T09:24:28Z] <moritzm> temporarily remove ganeti4006 from the ulsfo02 Ganeti cluster in preparation of forthcoming switch maintenance in ulsfo T424686

Mentioned in SAL (#wikimedia-operations) [2026-04-30T09:27:21Z] <moritzm> failover Ganeti master in ulsfo02 to ganeti4005 in preparation of forthcoming switch maintenance in ulsfo T424686

Draining ganeti4008.ulsfo.wmnet of running VMs

Draining ganeti4008.ulsfo.wmnet of running VMs

VM ncredir4003.ulsfo.wmnet switching disk type to plain

VM ncredir4004.ulsfo.wmnet switching disk type to plain

VM prometheus4003.ulsfo.wmnet switching disk type to plain

VM netflow4003.ulsfo.wmnet switching disk type to plain

VM hcaptcha-proxy4003.wikimedia.org switching disk type to plain

Mentioned in SAL (#wikimedia-operations) [2026-04-30T13:00:40Z] <moritzm> temporarily remove ganeti4008 from the ulsfo02 Ganeti cluster in preparation of forthcoming switch maintenance in ulsfo T424686

ganeti4006 and ganeti4008 are free of VMs and temporarily removed as nodes from the ulsfo Ganeti cluster. They can be freely shutdown, moved around and re-cabled. When each of them is done from the DC ops side, just drop me a node and I'll reimage them and re-add them to the cluster.

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ganeti4006.ulsfo.wmnet

• ganeti4006.ulsfo.wmnet (FAIL)
  • Downtimed host on Icinga/Alertmanager
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
  • Powered off
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Configured the linked switch interface(s)
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB

ERROR: some step on some host failed, check the bolded items above

Change #1283536 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] Hieradata: Update IPs for ULSFO CP hosts in rack 23

https://gerrit.wikimedia.org/r/1283536

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ganeti4008.ulsfo.wmnet

• ganeti4008.ulsfo.wmnet (FAIL)
  • Downtimed host on Icinga/Alertmanager
  • Found physical host
  • Downtimed management interface on Alertmanager
  • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
  • Powered off
  • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
  • Configured the linked switch interface(s)
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB

ERROR: some step on some host failed, check the bolded items above

Change #1283545 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] Hieradata: Update IPs for ULSFO lvs hosts in rack 23

https://gerrit.wikimedia.org/r/1283545

Change #1283548 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] Hieradata: Update IPs for ULSFO dns host in rack 23

https://gerrit.wikimedia.org/r/1283548

Change #1283536 merged by Slyngshede:

[operations/puppet@production] Hieradata: Update IPs for ULSFO CP hosts in rack 23

https://gerrit.wikimedia.org/r/1283536

Re-image command:

sudo cookbook sre.hosts.reimage --os trixie -t T424686 cp4040

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4040.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4040.ulsfo.wmnet with OS trixie completed:

• cp4040 (WARN)
  • Downtimed on Icinga/Alertmanager
  • Unable to disable Puppet, the host may have been unreachable
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605060945_slyngshede_501048_cp4040.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4042.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4044.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti4006.ulsfo.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4042.ulsfo.wmnet with OS trixie completed:

• cp4042 (WARN)
  • Downtimed on Icinga/Alertmanager
  • Unable to disable Puppet, the host may have been unreachable
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061044_slyngshede_524487_cp4042.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4044.ulsfo.wmnet with OS trixie completed:

• cp4044 (WARN)
  • Downtimed on Icinga/Alertmanager
  • Unable to disable Puppet, the host may have been unreachable
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061048_slyngshede_525567_cp4044.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4048.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4050.ulsfo.wmnet with OS trixie

cp4046 refused to be reimaged and seems to be missing some configuration in Netbox from what I can tell. I've pinged @Papaul and asked to have it investigated as I'm not entirely sure how to fix it.

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti4006.ulsfo.wmnet with OS bookworm completed:

• ganeti4006 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061059_jmm_2656424_ganeti4006.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4050.ulsfo.wmnet with OS trixie completed:

• cp4050 (WARN)
  • Downtimed on Icinga/Alertmanager
  • Unable to disable Puppet, the host may have been unreachable
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061150_slyngshede_573909_cp4050.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Same issue for cp4052:

slyngshede@cumin1003:~$ sudo cookbook sre.hosts.reimage --os trixie -t T424686 cp4052
==> ATTENTION: Destructive action for cp4052. Proceed?
Type "go" to proceed or "abort" to interrupt the execution
> go
User input is: "go"
Exception raised while initializing the Cookbook sre.hosts.reimage:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 205, in run
    runner = self.instance.get_runner(args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 121, in get_runner
    return ReimageRunner(args, self.spicerack)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 183, in __init__
    self.netbox_server.access_vlan in LEGACY_VLANS and \
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/netbox.py", line 519, in access_vlan
    netbox_switch_iface = self._find_primary_switch_iface()
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/spicerack/netbox.py", line 504, in _find_primary_switch_iface
    raise NetboxError("Primary interface not connected.")
spicerack.netbox.NetboxError: Primary interface not connected.
slyngshede@cumin1003:~$

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4048.ulsfo.wmnet with OS trixie completed:

• cp4048 (WARN)
  • Downtimed on Icinga/Alertmanager
  • Unable to disable Puppet, the host may have been unreachable
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061153_slyngshede_573594_cp4048.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Change #1283545 merged by Slyngshede:

[operations/puppet@production] Hieradata: Update IPs for ULSFO lvs hosts in rack 23

https://gerrit.wikimedia.org/r/1283545

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host lvs4008.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host lvs4010.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4046.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host ganeti4008.ulsfo.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp4052.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4046.ulsfo.wmnet with OS trixie completed:

• cp4046 (WARN)
  • Downtimed on Icinga/Alertmanager
  • Unable to disable Puppet, the host may have been unreachable
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061345_slyngshede_669687_cp4046.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host ganeti4008.ulsfo.wmnet with OS bookworm completed:

• ganeti4008 (PASS)
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061400_jmm_2782428_ganeti4008.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB
  • Updated Netbox status planned -> active
  • The sre.puppet.sync-netbox-hiera cookbook was run successfully

Change #1283548 merged by Slyngshede:

[operations/puppet@production] Hieradata: Update IPs for ULSFO dns host in rack 23

https://gerrit.wikimedia.org/r/1283548

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp4052.ulsfo.wmnet with OS trixie completed:

• cp4052 (WARN)
  • Downtimed on Icinga/Alertmanager
  • Unable to disable Puppet, the host may have been unreachable
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061428_slyngshede_699047_cp4052.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host lvs4008.ulsfo.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host lvs4008.ulsfo.wmnet with OS bookworm completed:

• lvs4008 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061609_sukhe_775691_lvs4008.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host lvs4008.ulsfo.wmnet with OS trixie completed:

• lvs4008 (WARN)
  • Downtimed on Icinga/Alertmanager
  • Unable to disable Puppet, the host may have been unreachable
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061314_slyngshede_648464_lvs4008.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host lvs4010.ulsfo.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1003 for host cp4050.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host lvs4010.ulsfo.wmnet with OS bookworm completed:

• lvs4010 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061854_sukhe_848892_lvs4010.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1003 for host cp4050.ulsfo.wmnet with OS trixie completed:

• cp4050 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061859_sukhe_849053_cp4050.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host lvs4010.ulsfo.wmnet with OS trixie completed:

• lvs4010 (WARN)
  • Downtimed on Icinga/Alertmanager
  • Unable to disable Puppet, the host may have been unreachable
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605061339_slyngshede_665287_lvs4010.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

VM ncredir4003.ulsfo.wmnet switching disk type to drbd

VM ncredir4003.ulsfo.wmnet switching disk type to drbd

VM netflow4003.ulsfo.wmnet switching disk type to drbd

VM ncredir4004.ulsfo.wmnet switching disk type to drbd

VM prometheus4003.ulsfo.wmnet switching disk type to drbd

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host bast4006.wikimedia.org with OS trixie

VM hcaptcha-proxy4003.wikimedia.org switching disk type to drbd

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host bast4006.wikimedia.org with OS trixie completed:

• bast4006 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via gnt-instance
  • Host up (Debian installer)
  • Set boot media to disk
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605070953_jmm_3550036_bast4006.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Mentioned in SAL (#wikimedia-operations) [2026-05-07T10:13:58Z] <moritzm> rebalance ganti cluster in ulsfo following host reimages T424686