Page MenuHomePhabricator
Create Task
Maniphest T293216

Upgrade mc* and mc-gp* hosts to Debian Bullseye
Closed, ResolvedPublic
Actions

Description

Debian Bullseye comes with Memcached 1.6.9. Things to consider before and during the upgrade:

  • T258779 must be Resolved
  • Reimage servers two at a time per DC, reimage next batch after reimaged servers reach a hit ratio of 0.85
    • ensure that each gutter is picking up the traffic

eqiad

  • mc1037.eqiad.wmnet
  • mc1038.eqiad.wmnet
  • mc1039.eqiad.wmnet
  • mc1040.eqiad.wmnet
  • mc1041.eqiad.wmnet
  • mc1042.eqiad.wmnet
  • mc1043.eqiad.wmnet
  • mc1044.eqiad.wmnet
  • mc1045.eqiad.wmnet
  • mc1046.eqiad.wmnet
  • mc1047.eqiad.wmnet
  • mc1048.eqiad.wmnet
  • mc1049.eqiad.wmnet
  • mc1050.eqiad.wmnet
  • mc1051.eqiad.wmnet
  • mc1052.eqiad.wmnet
  • mc1053.eqiad.wmnet
  • mc1054.eqiad.wmnet
  • mc-gp1001.eqiad.wmnet
  • mc-gp1002.eqiad.wmnet
  • mc-gp1003.eqiad.wmnet

codfw

  • mc2038.codfw.wmnet
  • mc2039.codfw.wmnet
  • mc2040.codfw.wmnet
  • mc2041.codfw.wmnet
  • mc2042.codfw.wmnet
  • mc2043.codfw.wmnet
  • mc2044.codfw.wmnet
  • mc2045.codfw.wmnet
  • mc2046.codfw.wmnet
  • mc2047.codfw.wmnet
  • mc2048.codfw.wmnet
  • mc2049.codfw.wmnet
  • mc2050.codfw.wmnet
  • mc2051.codfw.wmnet
  • mc2052.codfw.wmnet
  • mc2053.codfw.wmnet
  • mc2054.codfw.wmnet
  • mc2055.codfw.wmnet
  • mc-gp2001.codfw.wmnet
  • mc-gp2002.codfw.wmnet
  • mc-gp2003.codfw.wmnet

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Jan 16 2023, 4:16 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1042.eqiad.wmnet with OS bullseye completed:

  • mc1042 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301161547_jiji_1923106_mc1042.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 16 2023, 4:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1044.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 16 2023, 4:53 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1044.eqiad.wmnet with OS bullseye completed:

  • mc1044 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301161623_jiji_1930382_mc1044.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 17 2023, 11:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1048.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 17 2023, 11:32 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1048.eqiad.wmnet with OS bullseye completed:

  • mc1048 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301171104_jiji_2132112_mc1048.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
jijiki updated the task description. (Show Details)Jan 17 2023, 7:04 PM
jijiki updated the task description. (Show Details)
ops-monitoring-bot added a comment.Jan 18 2023, 10:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1050.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 18 2023, 11:12 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1050.eqiad.wmnet with OS bullseye completed:

  • mc1050 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301181043_jiji_2406585_mc1050.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 18 2023, 3:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1052.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 18 2023, 3:37 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1052.eqiad.wmnet with OS bullseye completed:

  • mc1052 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301181507_jiji_2456735_mc1052.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 19 2023, 11:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1054.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 19 2023, 11:36 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1054.eqiad.wmnet with OS bullseye completed:

  • mc1054 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301191106_jiji_2922592_mc1054.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 19 2023, 4:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2038.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 19 2023, 5:17 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2038.codfw.wmnet with OS bullseye completed:

  • mc2038 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301191642_jiji_993484_mc2038.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 19 2023, 9:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2039.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 19 2023, 9:46 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2039.codfw.wmnet with OS bullseye completed:

  • mc2039 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301192111_jiji_1122699_mc2039.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 20 2023, 12:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2040.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 20 2023, 12:38 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2040.codfw.wmnet with OS bullseye completed:

  • mc2040 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301201204_jiji_1542309_mc2040.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 24 2023, 9:39 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2041.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2023, 10:13 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2041.codfw.wmnet with OS bullseye completed:

  • mc2041 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301240939_jiji_20498_mc2041.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 24 2023, 3:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2042.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2023, 4:28 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2042.codfw.wmnet with OS bullseye completed:

  • mc2042 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202301241553_jiji_207205_mc2042.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 2 2023, 5:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1037.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 2 2023, 5:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2043.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 2 2023, 6:03 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1037.eqiad.wmnet with OS bullseye completed:

  • mc1037 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302021732_jiji_1352768_mc1037.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 2 2023, 6:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2043.codfw.wmnet with OS bullseye completed:

  • mc2043 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302021732_jiji_1352727_mc2043.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 11:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2044.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 11:29 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1041.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 11:56 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2044.codfw.wmnet with OS bullseye completed:

  • mc2044 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302071121_jiji_2627844_mc2044.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 12:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1041.eqiad.wmnet with OS bullseye completed:

  • mc1041 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302071129_jiji_2631097_mc1041.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 3:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2045.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 3:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1043.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 3:43 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1043.eqiad.wmnet with OS bullseye completed:

  • mc1043 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302071513_jiji_2694013_mc1043.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 3:48 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2045.codfw.wmnet with OS bullseye completed:

  • mc2045 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302071513_jiji_2693965_mc2045.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 4:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1045.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 4:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2046.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 4:46 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1045.eqiad.wmnet with OS bullseye completed:

  • mc1045 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302071614_jiji_2709915_mc1045.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 4:50 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2046.codfw.wmnet with OS bullseye completed:

  • mc2046 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302071615_jiji_2709951_mc2046.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 5:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2047.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 5:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1046.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 5:51 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1046.eqiad.wmnet with OS bullseye completed:

  • mc1046 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302071722_jiji_2725521_mc1046.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 5:53 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2047.codfw.wmnet with OS bullseye completed:

  • mc2047 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302071721_jiji_2725342_mc2047.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 6:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1047.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 6:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2048.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 6:47 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1047.eqiad.wmnet with OS bullseye completed:

  • mc1047 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302071817_jiji_2743718_mc1047.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 6:53 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2048.codfw.wmnet with OS bullseye completed:

  • mc2048 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302071817_jiji_2744000_mc2048.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 7:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2049.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 7:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1049.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 7:33 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1049.eqiad.wmnet with OS bullseye completed:

  • mc1049 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302071904_jiji_2769481_mc1049.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 7:39 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2049.codfw.wmnet with OS bullseye completed:

  • mc2049 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302071903_jiji_2769309_mc2049.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 8:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1051.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 8:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2050.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 8:50 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1051.eqiad.wmnet with OS bullseye completed:

  • mc1051 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302072021_jiji_2807732_mc1051.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 8:57 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2050.codfw.wmnet with OS bullseye completed:

  • mc2050 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302072021_jiji_2807823_mc2050.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 8:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2051.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 9:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1053.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 9:29 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1053.eqiad.wmnet with OS bullseye completed:

  • mc1053 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302072100_jiji_2831017_mc1053.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 9:34 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2051.codfw.wmnet with OS bullseye completed:

  • mc2051 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302072058_jiji_2829852_mc2051.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 9 2023, 10:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2052.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 10:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc-gp1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 11:08 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2052.codfw.wmnet with OS bullseye completed:

  • mc2052 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091033_jiji_3610310_mc2052.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 9 2023, 11:20 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc-gp1001.eqiad.wmnet with OS bullseye executed with errors:

  • mc-gp1001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 9 2023, 11:20 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc-gp1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 11:52 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc-gp1001.eqiad.wmnet with OS bullseye executed with errors:

  • mc-gp1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 9 2023, 11:52 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc-gp1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 11:57 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc-gp1001.eqiad.wmnet with OS bullseye executed with errors:

  • mc-gp1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 9 2023, 11:57 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc-gp1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 12:17 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc-gp1001.eqiad.wmnet with OS bullseye executed with errors:

  • mc-gp1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 9 2023, 12:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc-gp1001.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 9 2023, 12:52 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc-gp1001.eqiad.wmnet with OS buster completed:

  • mc-gp1001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091218_jiji_3634175_mc-gp1001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 9 2023, 2:27 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2053.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 3:03 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2053.codfw.wmnet with OS bullseye completed:

  • mc2053 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091427_jiji_3667594_mc2053.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 9 2023, 3:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2054.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 3:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc-gp1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 3:42 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2054.codfw.wmnet with OS bullseye completed:

  • mc2054 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091506_jiji_3680653_mc2054.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 9 2023, 3:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2055.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 4:09 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2055.codfw.wmnet with OS bullseye executed with errors:

  • mc2055 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 9 2023, 4:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc2055.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 4:10 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc-gp1001.eqiad.wmnet with OS bullseye completed:

  • mc-gp1001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091539_jiji_3688306_mc-gp1001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 9 2023, 4:44 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc2055.codfw.wmnet with OS bullseye completed:

  • mc2055 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091609_jiji_3698426_mc2055.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
jijiki updated the task description. (Show Details)Feb 9 2023, 5:36 PM
ops-monitoring-bot added a comment.Feb 9 2023, 6:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc-gp2001.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 9 2023, 6:55 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc-gp2001.codfw.wmnet with OS bullseye completed:

  • mc-gp2001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302091822_jiji_3729295_mc-gp2001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
jijiki added a subtask: T329323: Update iDRAC and NIC firmware on mc-gp* hosts .Feb 9 2023, 9:16 PM
Papaul closed subtask T329323: Update iDRAC and NIC firmware on mc-gp* hosts as Resolved.Feb 14 2023, 7:39 PM
ops-monitoring-bot added a comment.Feb 20 2023, 12:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc-gp2002.codfw.wmnet with OS bullseye

jijiki added a subtask: T330072: Reset management module of mc1039.Feb 20 2023, 12:23 PM
ops-monitoring-bot added a comment.Feb 20 2023, 12:50 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc-gp2002.codfw.wmnet with OS bullseye completed:

  • mc-gp2002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302201215_jiji_3892969_mc-gp2002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 20 2023, 1:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc-gp2003.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 20 2023, 2:25 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc-gp2003.codfw.wmnet with OS bullseye completed:

  • mc-gp2003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302201350_jiji_3912848_mc-gp2003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 20 2023, 8:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc-gp1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 20 2023, 8:58 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc-gp1002.eqiad.wmnet with OS bullseye completed:

  • mc-gp1002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302202026_jiji_4136425_mc-gp1002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 21 2023, 10:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc-gp1003.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 21 2023, 11:16 AM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc-gp1003.eqiad.wmnet with OS bullseye completed:

  • mc-gp1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302211043_jiji_2075359_mc-gp1003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• Cmjohnson closed subtask T330072: Reset management module of mc1039 as Resolved.Mar 2 2023, 3:56 PM
ops-monitoring-bot added a comment.Mar 8 2023, 11:44 AM

Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1001 for host mc1039.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 8 2023, 12:17 PM

Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1001 for host mc1039.eqiad.wmnet with OS bullseye completed:

  • mc1039 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202303081144_jiji_2906973_mc1039.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
jijiki closed this task as Resolved.Mar 8 2023, 12:39 PM
jijiki updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL