Page MenuHomePhabricator
Create Task
Maniphest T363792

Upgrade s8 to MariaDB 10.6
Open, MediumPublic
Actions

Description

  • dbstore1009
  • db2200
  • db2198
  • db2195
  • db2186
  • db2181
  • db2167
  • db2166
  • db2165 master
  • db2164
  • db2163
  • db2162
  • db2161 candidate master
  • db2154
  • db2152
  • db2098 backup source T360751
  • db1226
  • db1216 (backup source)
  • db1214
  • db1211
  • db1209 master
  • db1203
  • db1193
  • db1192 candidate master
  • db1178
  • db1177
  • db1172
  • db1171 backup source T360751
  • db1167 sanitarium master
  • db1154
  • clouddb1021
  • clouddb1020
  • clouddb1016

Related Objects
Search...

Event Timeline

Marostegui created this task.Tue, Apr 30, 6:44 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Apr 30, 6:44 AM
Marostegui triaged this task as Medium priority.Tue, Apr 30, 6:44 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui mentioned this in T356960: Upgrade hosts to MariaDB 10.6.
ops-monitoring-bot added a comment.Wed, May 1, 5:58 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2166.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Wed, May 1, 6:30 AM
ops-monitoring-bot added a comment.Wed, May 1, 6:38 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2166.codfw.wmnet with OS bookworm completed:

  • db2166 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405010617_marostegui_3706838_db2166.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Wed, May 1, 7:52 AM
ops-monitoring-bot added a comment.Wed, May 1, 7:59 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2164.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Wed, May 1, 8:36 AM
ops-monitoring-bot added a comment.Wed, May 1, 8:44 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2164.codfw.wmnet with OS bookworm completed:

  • db2164 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405010820_marostegui_3724003_db2164.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Marostegui updated the task description. (Show Details)Wed, May 1, 9:29 AM
Marostegui updated the task description. (Show Details)
ops-monitoring-bot added a comment.Wed, May 1, 10:00 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2163.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Wed, May 1, 10:34 AM
ops-monitoring-bot added a comment.Wed, May 1, 10:42 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2163.codfw.wmnet with OS bookworm completed:

  • db2163 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405011020_marostegui_3745756_db2163.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Wed, May 1, 12:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2154.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Wed, May 1, 12:48 PM
ops-monitoring-bot added a comment.Wed, May 1, 12:55 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2154.codfw.wmnet with OS bookworm completed:

  • db2154 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405011235_marostegui_3763527_db2154.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Thu, May 2, 4:35 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2162.codfw.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Thu, May 2, 5:09 AM
ops-monitoring-bot added a comment.Thu, May 2, 5:17 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2162.codfw.wmnet with OS bookworm completed:

  • db2162 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405020455_marostegui_3896600_db2162.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Thu, May 2, 8:49 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2152.codfw.wmnet with OS bookworm

Marostegui added a subtask: T363977: Switchover s8 master (db2165 -> db2161).Thu, May 2, 9:12 AM
Marostegui updated the task description. (Show Details)Thu, May 2, 9:25 AM
ops-monitoring-bot added a comment.Thu, May 2, 9:29 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2152.codfw.wmnet with OS bookworm completed:

  • db2152 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405020909_marostegui_3930667_db2152.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, May 2, 12:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2161.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, May 2, 12:25 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2161.codfw.wmnet with OS bookworm executed with errors:

  • db2161 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" db2161.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Thu, May 2, 12:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2161.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, May 2, 1:08 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2161.codfw.wmnet with OS bookworm completed:

  • db2161 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405021246_marostegui_3975163_db2161.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Marostegui updated the task description. (Show Details)Thu, May 2, 1:12 PM
ops-monitoring-bot added a comment.Fri, May 3, 5:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1214.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Fri, May 3, 5:29 AM
Marostegui updated the task description. (Show Details)
ops-monitoring-bot added a comment.Fri, May 3, 5:47 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1214.eqiad.wmnet with OS bookworm completed:

  • db1214 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405030527_marostegui_29636_db1214.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Fri, May 3, 7:12 AM
ops-monitoring-bot added a comment.Fri, May 3, 10:39 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1203.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Fri, May 3, 11:11 AM
ops-monitoring-bot added a comment.Fri, May 3, 11:16 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1203.eqiad.wmnet with OS bookworm completed:

  • db1203 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405031055_marostegui_71720_db1203.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Marostegui updated the task description. (Show Details)Fri, May 3, 11:17 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)
Marostegui closed subtask T363977: Switchover s8 master (db2165 -> db2161) as Resolved.Mon, May 6, 6:15 AM
ops-monitoring-bot added a comment.Mon, May 6, 6:17 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2165.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, May 6, 6:30 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1193.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Mon, May 6, 6:50 AM
ops-monitoring-bot added a comment.Mon, May 6, 6:58 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2165.codfw.wmnet with OS bookworm completed:

  • db2165 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405060637_marostegui_566511_db2165.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Mon, May 6, 6:59 AM
ops-monitoring-bot added a comment.Mon, May 6, 7:07 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1193.eqiad.wmnet with OS bookworm completed:

  • db1193 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405060646_marostegui_569435_db1193.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Mon, May 6, 9:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1178.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Mon, May 6, 10:31 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1178.eqiad.wmnet with OS bookworm executed with errors:

  • db1178 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" db1178.eqiad.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Wed, May 8, 4:52 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1178.eqiad.wmnet with OS bookworm

Marostegui closed subtask T364300: db1178 not booting up as Resolved.Wed, May 8, 4:57 AM
Marostegui updated the task description. (Show Details)Wed, May 8, 5:19 AM
ops-monitoring-bot added a comment.Wed, May 8, 5:27 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1178.eqiad.wmnet with OS bookworm completed:

  • db1178 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405080507_marostegui_1243840_db1178.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Wed, May 8, 9:09 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-08T09:09:26Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1177 T363792', diff saved to https://phabricator.wikimedia.org/P62041 and previous config saved to /var/cache/conftool/dbconfig/20240508-090925-root.json

ops-monitoring-bot added a comment.Wed, May 8, 9:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1177.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, May 8, 9:49 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1177.eqiad.wmnet with OS bookworm completed:

  • db1177 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405080928_marostegui_1284473_db1177.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Wed, May 8, 9:50 AM
Stashbot added a comment.Thu, May 9, 5:34 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-09T05:34:43Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1172 T363792', diff saved to https://phabricator.wikimedia.org/P62166 and previous config saved to /var/cache/conftool/dbconfig/20240509-053442-marostegui.json

ops-monitoring-bot added a comment.Thu, May 9, 5:37 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1172.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Thu, May 9, 6:06 AM
ops-monitoring-bot added a comment.Thu, May 9, 6:14 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1172.eqiad.wmnet with OS bookworm completed:

  • db1172 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405090552_marostegui_1665916_db1172.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, May 9, 9:29 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1167.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, May 9, 10:06 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1167.eqiad.wmnet with OS bookworm completed:

  • db1167 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405090945_marostegui_1700589_db1167.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Marostegui updated the task description. (Show Details)Thu, May 9, 10:08 AM
ops-monitoring-bot added a comment.Thu, May 9, 12:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1192.eqiad.wmnet with OS bookworm

Marostegui updated the task description. (Show Details)Thu, May 9, 12:14 PM
Marostegui updated the task description. (Show Details)Thu, May 9, 12:41 PM
ops-monitoring-bot added a comment.Thu, May 9, 12:49 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1192.eqiad.wmnet with OS bookworm completed:

  • db1192 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405091227_marostegui_1722061_db1192.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Marostegui added a subtask: T364541: Switchover s8 master (db1209 -> db1192).Tue, May 14, 6:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL