Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2042.codfw.wmnet with OS trixie completed:

• es2042 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606081449_marostegui_1980850_es2042.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Icinga downtime and Alertmanager silence (ID=6c545cca-39bb-4652-9cd8-5da4fc40f265) set by jynus@cumin2002 for 4:00:00 on 2 host(s) and their services with reason: Switchover db

db[2183-2184].codfw.wmnet

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2042.codfw.wmnet with OS trixie

Completed depooling of es2042 by marostegui@cumin1003: Upgrading es2042.codfw.wmnet

Upgrading es2042.codfw.wmnet

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1041.eqiad.wmnet with OS trixie completed:

• es1041 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606081318_marostegui_1968582_es1041.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1041.eqiad.wmnet with OS trixie

Completed depooling of es1041 by marostegui@cumin1003: Upgrading es1041.eqiad.wmnet

Upgrading es1041.eqiad.wmnet

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-be2063.codfw.wmnet with OS bullseye completed:

• ms-be2063 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Host successfully migrated to the new VLAN
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606081219_mvernon_1095115_ms-be2063.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-be2062.codfw.wmnet with OS bullseye completed:

• ms-be2062 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Host successfully migrated to the new VLAN
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh bullseye OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606081213_mvernon_1093975_ms-be2062.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2041.codfw.wmnet with OS trixie completed:

• es2041 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606081209_marostegui_1957788_es2041.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-be2063.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2041.codfw.wmnet with OS trixie

Completed depooling of es2041 by marostegui@cumin1003: Upgrading es2041.codfw.wmnet

Upgrading es2041.codfw.wmnet

Upgrading es2042.codfw.wmnet

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-be2062.codfw.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1042.eqiad.wmnet with OS trixie completed:

• es1042 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606081039_marostegui_1894481_es1042.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1042.eqiad.wmnet with OS trixie

Completed depooling of es1042 by marostegui@cumin1003: Upgrading es1042.eqiad.wmnet

Upgrading es1042.eqiad.wmnet

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2043.codfw.wmnet with OS trixie completed:

• es2043 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606080929_marostegui_1846847_es2043.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Host gitlab1004.wikimedia.org rebooted by jelto@cumin1003 with reason: gitlab restart is stuck

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2043.codfw.wmnet with OS trixie

Completed depooling of es2043 by marostegui@cumin1003: Upgrading es2043.codfw.wmnet

Upgrading es2043.codfw.wmnet

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db1217.eqiad.wmnet with OS trixie completed:

• db1217 (WARN)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606080819_marostegui_1813634_db1217.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Updated Netbox data from PuppetDB

Section s5: Wikis urwikisource set up on clouddb - fceratto@cumin1003

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db1217.eqiad.wmnet with OS trixie

Section s5: Wikis urwikisource redacted - fceratto@cumin1003

Section s5: Wikis urwikisource redacted - fceratto@cumin1003

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1051.eqiad.wmnet with OS trixie completed:

• es1051 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606080620_marostegui_1789103_es1051.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1051.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2052.codfw.wmnet with OS trixie completed:

• es2052 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606080539_marostegui_1786432_es2052.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Upgrading es1051.eqiad.wmnet

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2052.codfw.wmnet with OS trixie

Completed depooling of es2052 by marostegui@cumin1003: Upgrading es2052.codfw.wmnet

Upgrading es2052.codfw.wmnet

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1054.eqiad.wmnet with OS trixie completed:

• es1054 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606050545_marostegui_1236596_es1054.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1054.eqiad.wmnet with OS trixie

Completed depooling of es1054 by marostegui@cumin1003: Upgrading es1054.eqiad.wmnet

Upgrading es1054.eqiad.wmnet

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main1010.eqiad.wmnet with OS trixie completed:

• kafka-main1010 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606050139_jasmine_114239_kafka-main1010.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main1010.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main1007.eqiad.wmnet with OS trixie completed:

• kafka-main1007 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606050040_jasmine_100324_kafka-main1007.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main1007.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by jasmine@cumin2002 for host kafka-main1006.eqiad.wmnet with OS trixie completed:

• kafka-main1006 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606042340_jasmine_86465_kafka-main1006.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by jasmine@cumin2002 for host kafka-main1006.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host cp5030.eqsin.wmnet with OS trixie completed:

• cp5030 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Host successfully migrated to the new VLAN
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606041949_cmooney_1168445_cp5030.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host cp5030.eqsin.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1003 for host kafka-main2007.codfw.wmnet with OS trixie completed:

• kafka-main2007 (PASS)
  • Downtimed on Icinga/Alertmanager
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • Removed previous downtime on Alertmanager (old OS)
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606041524_jayme_1135974_kafka-main2007.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is optimal
  • Icinga downtime removed
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.wikireplicas.update-views started by ladsgroup completed:

• an-redacteddb1001.eqiad.wmnet (PASS)
  • Ran Puppet agent
  • Ran 'maintain-views --replace --auto-depool --all-databases --table filerevision_userindex'

Cookbook cookbooks.sre.wikireplicas.update-views run by ladsgroup: Started updating wiki replica views

Cookbook cookbooks.sre.wikireplicas.update-views started by ladsgroup completed:

• an-redacteddb1001.eqiad.wmnet (PASS)
  • Ran Puppet agent
  • Ran 'maintain-views --replace --auto-depool --all-databases --table filerevision'

Cookbook cookbooks.sre.wikireplicas.update-views run by ladsgroup: Started updating wiki replica views

Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1003 for host kafka-main2007.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1057.eqiad.wmnet with OS trixie completed:

• es1057 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606041325_marostegui_1118960_es1057.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Migration of db1220.eqiad.wmnet completed

Completed pooling of db1220 by marostegui@cumin1003: Migration of db1220.eqiad.wmnet completed

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1057.eqiad.wmnet with OS trixie

Completed depooling of es1057 by marostegui@cumin1003: Upgrading es1057.eqiad.wmnet

Upgrading es1057.eqiad.wmnet

Starting pool of db1220 by marostegui@cumin1003: Migration of db1220.eqiad.wmnet completed

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db1220.eqiad.wmnet with OS trixie completed:

• db1220 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606041204_marostegui_1097239_db1220.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db1220.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2050.codfw.wmnet with OS trixie completed:

• es2050 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606041123_marostegui_1091846_es2050.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Completed depooling of db1220 by marostegui@cumin1003: Upgrading db1220.eqiad.wmnet

Upgrading db1220.eqiad.wmnet

Migration of db1179.eqiad.wmnet completed

Completed pooling of db1179 by marostegui@cumin1003: Migration of db1179.eqiad.wmnet completed

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2050.codfw.wmnet with OS trixie

Completed depooling of es2050 by marostegui@cumin1003: Upgrading es2050.codfw.wmnet

Upgrading es2050.codfw.wmnet

Starting pool of db1179 by marostegui@cumin1003: Migration of db1179.eqiad.wmnet completed

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db1179.eqiad.wmnet with OS trixie completed:

• db1179 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606041019_marostegui_1081578_db1179.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es2057.codfw.wmnet with OS trixie completed:

• es2057 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606040954_marostegui_1078552_es2057.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db1179.eqiad.wmnet with OS trixie

Completed depooling of db1179 by marostegui@cumin1003: Upgrading db1179.eqiad.wmnet

Upgrading db1179.eqiad.wmnet

Migration of db1224.eqiad.wmnet completed

Completed pooling of db1224 by marostegui@cumin1003: Migration of db1224.eqiad.wmnet completed

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es2057.codfw.wmnet with OS trixie

Completed depooling of es2057 by marostegui@cumin1003: Upgrading es2057.codfw.wmnet

Upgrading es2057.codfw.wmnet

Starting pool of db1224 by marostegui@cumin1003: Migration of db1224.eqiad.wmnet completed

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db1224.eqiad.wmnet with OS trixie completed:

• db1224 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606040804_marostegui_1052418_db1224.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host db1224.eqiad.wmnet with OS trixie

Completed depooling of db1224 by marostegui@cumin1003: Upgrading db1224.eqiad.wmnet

Upgrading db1224.eqiad.wmnet

Migration of db1255.eqiad.wmnet completed

Completed pooling of db1255 by cwilliams@cumin1003: Migration of db1255.eqiad.wmnet completed

Migration of db2191.codfw.wmnet completed

Completed pooling of db2191 by marostegui@cumin1003: Migration of db2191.codfw.wmnet completed

Starting pool of db1255 by cwilliams@cumin1003: Migration of db1255.eqiad.wmnet completed

Cookbook cookbooks.sre.hosts.reimage started by cwilliams@cumin1003 for host db1255.eqiad.wmnet with OS trixie completed:

• db1255 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606040635_cwilliams_1028532_db1255.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Starting pool of db2191 by marostegui@cumin1003: Migration of db2191.codfw.wmnet completed

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host db2191.codfw.wmnet with OS trixie completed:

• db2191 (WARN)
  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present and deleted any certificates
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Checked BIOS boot parameters are back to normal
  • Host up (new fresh trixie OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga/Alertmanager
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606040615_marostegui_1025774_db2191.out
  • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
  • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by cwilliams@cumin1003 for host db1255.eqiad.wmnet with OS trixie

Completed depooling of db1255 by cwilliams@cumin1003: Upgrading db1255.eqiad.wmnet