Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1001 for host testvm2002.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1001 for host testvm2002.codfw.wmnet with OS bullseye completed:

• testvm2002 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via gnt-instance
  • Host up (Debian installer)
  • Set boot media to disk
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202305111329_slyngshede_4027206_testvm2002.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Change 920192 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/cookbooks@master] k8s upgrade cluster: use sre.hosts.reimage

https://gerrit.wikimedia.org/r/920192

Change 920203 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/cookbooks@master] sre.ganeti.makevm call reimage after VM creation

https://gerrit.wikimedia.org/r/920203

Change 920192 merged by Slyngshede:

[operations/cookbooks@master] k8s upgrade cluster: use sre.hosts.reimage

https://gerrit.wikimedia.org/r/920192

Change 922065 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/cookbooks@master] sre.ganeti.reimage: Remove specialised cookbook.

https://gerrit.wikimedia.org/r/922065

I've updated https://wikitech.wikimedia.org/wiki/Ganeti to point to the new cookbook

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1001 for host testvm2006.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1001 for host testvm2006.codfw.wmnet with OS bookworm completed:

• testvm2006 (PASS)
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via gnt-instance
  • Host up (Debian installer)
  • Set boot media to disk
  • Host up (new fresh bookworm OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202305301053_slyngshede_818964_testvm2006.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

cookbooks.sre.hosts.decommission executed by slyngshede@cumin1001 for hosts: testvm2006.codfw.wmnet

• testvm2006.codfw.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed
  • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox

Change 920203 merged by Slyngshede:

[operations/cookbooks@master] sre.ganeti.makevm call reimage after VM creation

https://gerrit.wikimedia.org/r/920203

Change 922065 merged by Slyngshede:

[operations/cookbooks@master] sre.ganeti.reimage: Remove specialised cookbook.

https://gerrit.wikimedia.org/r/922065