Page MenuHomePhabricator
Create Task
Maniphest T426725

Migrate x3 section to Debian Trixie
Closed, ResolvedPublic
Actions

Description

  • db2244
  • db2243
  • db2242
  • db2241 T426936
  • db2200 backup source T424541
  • db2187
  • db2162
  • db1258
  • db1257
  • db1256
  • db1255 master
  • db1216 backup source T424541
  • db1211 sanitarium master
  • db1154 sanitarium
  • clouddb1023 T415165
  • clouddb1022 T415165
  • clouddb1020 T415165
  • clouddb1016 T415165
sudo cookbook sre.mysql.major-upgrade -t T426725 --reimage trixie --repool $HOST wmf-mariadb1011

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Tue, May 19, 2:17 PM

Completed depooling of db2243 by cwilliams@cumin1003: Upgrading db2243.codfw.wmnet

ops-monitoring-bot added a comment.Tue, May 19, 2:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwilliams@cumin1003 for host db2243.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Tue, May 19, 3:02 PM

Cookbook cookbooks.sre.hosts.reimage started by cwilliams@cumin1003 for host db2243.codfw.wmnet with OS trixie completed:

  • db2243 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605191444_cwilliams_1534026_db2243.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Tue, May 19, 3:04 PM

Starting pool of db2243 by cwilliams@cumin1003: Migration of db2243.codfw.wmnet completed

CWilliams-WMF updated the task description. (Show Details)Tue, May 19, 3:04 PM
ops-monitoring-bot added a comment.Tue, May 19, 3:49 PM

Completed pooling of db2243 by cwilliams@cumin1003: Migration of db2243.codfw.wmnet completed

ops-monitoring-bot added a comment.Tue, May 19, 3:49 PM

Migration of db2243.codfw.wmnet completed

ops-monitoring-bot added a comment.Wed, May 20, 7:27 AM

Upgrading db2187.codfw.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 7:27 AM

Completed depooling of db2187 by cwilliams@cumin1003: Upgrading db2187.codfw.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 7:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by cwilliams@cumin1003 for host db2187.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, May 20, 8:25 AM

Cookbook cookbooks.sre.hosts.reimage started by cwilliams@cumin1003 for host db2187.codfw.wmnet with OS trixie completed:

  • db2187 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605200807_cwilliams_1820489_db2187.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, May 20, 8:27 AM

Starting pool of db2187 by cwilliams@cumin1003: Migration of db2187.codfw.wmnet completed

CWilliams-WMF updated the task description. (Show Details)Wed, May 20, 8:29 AM
ops-monitoring-bot added a comment.Wed, May 20, 9:02 AM

Upgrading db2242.codfw.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 9:03 AM

Completed depooling of db2242 by cwilliams@cumin1003: Upgrading db2242.codfw.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 9:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by cwilliams@cumin1003 for host db2242.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, May 20, 9:13 AM

Completed pooling of db2187 by cwilliams@cumin1003: Migration of db2187.codfw.wmnet completed

ops-monitoring-bot added a comment.Wed, May 20, 9:13 AM

Migration of db2187.codfw.wmnet completed

ops-monitoring-bot added a comment.Wed, May 20, 9:43 AM

Cookbook cookbooks.sre.hosts.reimage started by cwilliams@cumin1003 for host db2242.codfw.wmnet with OS trixie completed:

  • db2242 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605200924_cwilliams_1890620_db2242.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, May 20, 9:46 AM

Starting pool of db2242 by cwilliams@cumin1003: Migration of db2242.codfw.wmnet completed

ops-monitoring-bot added a comment.Wed, May 20, 10:31 AM

Completed pooling of db2242 by cwilliams@cumin1003: Migration of db2242.codfw.wmnet completed

ops-monitoring-bot added a comment.Wed, May 20, 10:31 AM

Migration of db2242.codfw.wmnet completed

CWilliams-WMF changed the task status from Open to In Progress.Wed, May 20, 10:37 AM
CWilliams-WMF updated the task description. (Show Details)
ops-monitoring-bot added a comment.Wed, May 20, 10:39 AM

Upgrading db2162.codfw.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 10:40 AM

Completed depooling of db2162 by cwilliams@cumin1003: Upgrading db2162.codfw.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 10:42 AM

Cookbook cookbooks.sre.hosts.reimage was started by cwilliams@cumin1003 for host db2162.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, May 20, 11:22 AM

Cookbook cookbooks.sre.hosts.reimage started by cwilliams@cumin1003 for host db2162.codfw.wmnet with OS trixie completed:

  • db2162 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605201104_cwilliams_1951032_db2162.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, May 20, 11:24 AM

Starting pool of db2162 by cwilliams@cumin1003: Migration of db2162.codfw.wmnet completed

CWilliams-WMF updated the task description. (Show Details)Wed, May 20, 11:41 AM
ops-monitoring-bot added a comment.Wed, May 20, 11:43 AM

Upgrading db1258.eqiad.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 11:43 AM

Completed depooling of db1258 by cwilliams@cumin1003: Upgrading db1258.eqiad.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 12:09 PM

Completed pooling of db2162 by cwilliams@cumin1003: Migration of db2162.codfw.wmnet completed

ops-monitoring-bot added a comment.Wed, May 20, 12:09 PM

Migration of db2162.codfw.wmnet completed

CWilliams-WMF added a comment.EditedWed, May 20, 12:26 PM

Upgrading db1258.eqiad.wmnet

This ended up in a broken state as the management password that was entered was incorrect.
See https://phabricator.wikimedia.org/P92681 for related output.

Remedied manually, restarted the replica so that the cookbook could run and now proceeding with the reimage

ops-monitoring-bot added a comment.Wed, May 20, 12:33 PM

Upgrading db1258.eqiad.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 12:33 PM

Completed depooling of db1258 by cwilliams@cumin1003: Upgrading db1258.eqiad.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 12:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwilliams@cumin1003 for host db1258.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, May 20, 1:17 PM

Cookbook cookbooks.sre.hosts.reimage started by cwilliams@cumin1003 for host db1258.eqiad.wmnet with OS trixie completed:

  • db1258 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605201259_cwilliams_2035922_db1258.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, May 20, 1:19 PM

Starting pool of db1258 by cwilliams@cumin1003: Migration of db1258.eqiad.wmnet completed

CWilliams-WMF updated the task description. (Show Details)Wed, May 20, 1:23 PM
ops-monitoring-bot added a comment.Wed, May 20, 2:04 PM

Completed pooling of db1258 by cwilliams@cumin1003: Migration of db1258.eqiad.wmnet completed

ops-monitoring-bot added a comment.Wed, May 20, 2:04 PM

Migration of db1258.eqiad.wmnet completed

ops-monitoring-bot added a comment.Wed, May 20, 2:38 PM

Upgrading db2162.codfw.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 2:44 PM

Upgrading db1257.eqiad.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 2:44 PM

Completed depooling of db1257 by cwilliams@cumin1003: Upgrading db1257.eqiad.wmnet

ops-monitoring-bot added a comment.Wed, May 20, 2:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwilliams@cumin1003 for host db1257.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, May 20, 3:25 PM

Cookbook cookbooks.sre.hosts.reimage started by cwilliams@cumin1003 for host db1257.eqiad.wmnet with OS trixie completed:

  • db1257 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605201508_cwilliams_2118406_db1257.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, May 20, 3:27 PM

Starting pool of db1257 by cwilliams@cumin1003: Migration of db1257.eqiad.wmnet completed

ops-monitoring-bot added a comment.Wed, May 20, 4:12 PM

Completed pooling of db1257 by cwilliams@cumin1003: Migration of db1257.eqiad.wmnet completed

ops-monitoring-bot added a comment.Wed, May 20, 4:12 PM

Migration of db1257.eqiad.wmnet completed

CWilliams-WMF updated the task description. (Show Details)Wed, May 20, 4:16 PM
ops-monitoring-bot added a comment.Thu, May 21, 7:21 AM

Upgrading db1256.eqiad.wmnet

ops-monitoring-bot added a comment.Thu, May 21, 7:22 AM

Completed depooling of db1256 by cwilliams@cumin1003: Upgrading db1256.eqiad.wmnet

ops-monitoring-bot added a comment.Thu, May 21, 7:24 AM

Cookbook cookbooks.sre.hosts.reimage was started by cwilliams@cumin1003 for host db1256.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Thu, May 21, 8:00 AM

Cookbook cookbooks.sre.hosts.reimage started by cwilliams@cumin1003 for host db1256.eqiad.wmnet with OS trixie completed:

  • db1256 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605210743_cwilliams_2327330_db1256.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, May 21, 8:02 AM

Starting pool of db1256 by cwilliams@cumin1003: Migration of db1256.eqiad.wmnet completed

CWilliams-WMF updated the task description. (Show Details)Thu, May 21, 8:02 AM
CWilliams-WMF added a subtask: T426936: Switchover x3 master (db2241 -> db2162).Thu, May 21, 8:12 AM
CWilliams-WMF updated the task description. (Show Details)
ops-monitoring-bot added a comment.Thu, May 21, 8:47 AM

Completed pooling of db1256 by cwilliams@cumin1003: Migration of db1256.eqiad.wmnet completed

ops-monitoring-bot added a comment.Thu, May 21, 8:47 AM

Migration of db1256.eqiad.wmnet completed

CWilliams-WMF mentioned this in T426936: Switchover x3 master (db2241 -> db2162).Thu, May 21, 1:15 PM
CWilliams-WMF updated the task description. (Show Details)Thu, May 21, 1:21 PM
ops-monitoring-bot added a comment.Thu, May 21, 1:23 PM

Upgrading db2241.codfw.wmnet

ops-monitoring-bot added a comment.Thu, May 21, 1:23 PM

Completed depooling of db2241 by cwilliams@cumin1003: Upgrading db2241.codfw.wmnet

ops-monitoring-bot added a comment.Thu, May 21, 1:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by cwilliams@cumin1003 for host db2241.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Thu, May 21, 2:02 PM

Cookbook cookbooks.sre.hosts.reimage started by cwilliams@cumin1003 for host db2241.codfw.wmnet with OS trixie completed:

  • db2241 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605211344_cwilliams_2728357_db2241.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, May 21, 2:04 PM

Starting pool of db2241 by cwilliams@cumin1003: Migration of db2241.codfw.wmnet completed

CWilliams-WMF closed subtask T426936: Switchover x3 master (db2241 -> db2162) as Resolved.Thu, May 21, 2:13 PM
CWilliams-WMF updated the task description. (Show Details)
ops-monitoring-bot added a comment.Thu, May 21, 2:49 PM

Completed pooling of db2241 by cwilliams@cumin1003: Migration of db2241.codfw.wmnet completed

ops-monitoring-bot added a comment.Thu, May 21, 2:49 PM

Migration of db2241.codfw.wmnet completed

Marostegui mentioned this in T422365: Migration to Debian Trixie of production database-related hosts.Fri, May 22, 5:34 AM
ops-monitoring-bot added a comment.Wed, Jun 3, 7:56 AM

Icinga downtime and Alertmanager silence (ID=6e84fd59-a589-467f-8a5e-0390a870ff03) set by cwilliams@cumin1003 for 3:00:00 on 3 host(s) and their services with reason: Reimaging upstream server

clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet
ops-monitoring-bot added a comment.Wed, Jun 3, 7:57 AM

Icinga downtime and Alertmanager silence (ID=af05fe7f-cadf-4b93-ae0d-e10a72afaae6) set by cwilliams@cumin1003 for 3:00:00 on 2 host(s) and their services with reason: Reimaging upstream server

clouddb[1022-1023].eqiad.wmnet
ops-monitoring-bot added a comment.Wed, Jun 3, 8:01 AM

Upgrading db1211.eqiad.wmnet

ops-monitoring-bot added a comment.Wed, Jun 3, 8:02 AM

Completed depooling of db1211 by cwilliams@cumin1003: Upgrading db1211.eqiad.wmnet

ops-monitoring-bot added a comment.Wed, Jun 3, 8:03 AM

Cookbook cookbooks.sre.hosts.reimage was started by cwilliams@cumin1003 for host db1211.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, Jun 3, 8:41 AM

Cookbook cookbooks.sre.hosts.reimage started by cwilliams@cumin1003 for host db1211.eqiad.wmnet with OS trixie completed:

  • db1211 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606030824_cwilliams_537045_db1211.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Wed, Jun 3, 8:43 AM

Starting pool of db1211 by cwilliams@cumin1003: Migration of db1211.eqiad.wmnet completed

CWilliams-WMF updated the task description. (Show Details)Wed, Jun 3, 8:47 AM
ops-monitoring-bot added a comment.Wed, Jun 3, 9:28 AM

Completed pooling of db1211 by cwilliams@cumin1003: Migration of db1211.eqiad.wmnet completed

ops-monitoring-bot added a comment.Wed, Jun 3, 9:28 AM

Migration of db1211.eqiad.wmnet completed

Marostegui added a subtask: T427895: Switchover x3 master (db1255 -> db1258).Thu, Jun 4, 5:58 AM
CWilliams-WMF closed subtask T427895: Switchover x3 master (db1255 -> db1258) as Resolved.Thu, Jun 4, 6:10 AM
CWilliams-WMF mentioned this in T427895: Switchover x3 master (db1255 -> db1258).
ops-monitoring-bot added a comment.Thu, Jun 4, 6:12 AM

Upgrading db1255.eqiad.wmnet

ops-monitoring-bot added a comment.Thu, Jun 4, 6:13 AM

Completed depooling of db1255 by cwilliams@cumin1003: Upgrading db1255.eqiad.wmnet

ops-monitoring-bot added a comment.Thu, Jun 4, 6:16 AM

Cookbook cookbooks.sre.hosts.reimage was started by cwilliams@cumin1003 for host db1255.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Thu, Jun 4, 6:51 AM

Cookbook cookbooks.sre.hosts.reimage started by cwilliams@cumin1003 for host db1255.eqiad.wmnet with OS trixie completed:

  • db1255 (WARN)
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606040635_cwilliams_1028532_db1255.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Skipping waiting for Icinga optimal status and not removing the downtime, --no-check-icinga was set
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, Jun 4, 6:53 AM

Starting pool of db1255 by cwilliams@cumin1003: Migration of db1255.eqiad.wmnet completed

CWilliams-WMF updated the task description. (Show Details)Thu, Jun 4, 6:54 AM
ops-monitoring-bot added a comment.Thu, Jun 4, 7:39 AM

Completed pooling of db1255 by cwilliams@cumin1003: Migration of db1255.eqiad.wmnet completed

ops-monitoring-bot added a comment.Thu, Jun 4, 7:39 AM

Migration of db1255.eqiad.wmnet completed

CWilliams-WMF closed this task as Resolved.Thu, Jun 4, 8:26 AM

The remaining hosts are out of scope for this ticket:

% sudo cumin 'A:db-section-x3 and A:bookworm'
6 hosts will be targeted:
clouddb[1016,1020,1022-1023].eqiad.wmnet,db2200.codfw.wmnet,db1216.eqiad.wmnet
DRY-RUN mode enabled, aborting
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits