Page MenuHomePhabricator
Create Task
Maniphest T364290

Upgrade s1 to MariaDB 10.6
Closed, ResolvedPublic
Actions

Description

The following hosts need to be upgraded to Bookworm and 10.6

  • dbstore1008
  • db2216 candidate master
  • db2212
  • db2203 dc master
  • db2202
  • db2188
  • db2186
  • db2176
  • db2174
  • db2173
  • db2170
  • db2153
  • db2146
  • db2145
  • db2141 backup source T360751
  • db2130
  • db2116
  • db2112 missing
  • db2103 missing
  • db2102
  • db1240 backup source T360751
  • db1239 backup source T360751
  • db1235
  • db1234
  • db1232 T365093
  • db1228
  • db1219
  • db1218
  • db1207
  • db1206
  • db1196 sanitarium master
  • db1186
  • db1184
  • db1169
  • db1163 candidate master
  • db1154
  • clouddb1021
  • clouddb1017
  • clouddb1013

Details

SubjectRepoBranchLines +/-
dbbackups: Migrate s1 backups to MariaDB 10.6operations/puppetproduction+13 -10
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedMarosteguiT356960 Upgrade hosts to MariaDB 10.6
 ResolvedABran-WMFT364290 Upgrade s1 to MariaDB 10.6

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Marostegui moved this task from Ready to In progress on the DBA board.Mon, May 20, 5:20 AM
Stashbot added a comment.Wed, May 22, 7:23 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T07:23:08Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'T364290 db1232', diff saved to https://phabricator.wikimedia.org/P62841 and previous config saved to /var/cache/conftool/dbconfig/20240522-072307-arnaudb.json

ops-monitoring-bot added a comment.Wed, May 22, 7:26 AM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1232.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, May 22, 8:02 AM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db1232.eqiad.wmnet with OS bookworm completed:

  • db1232 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405220742_arnaudb_1812486_db1232.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ABran-WMF updated the task description. (Show Details)Wed, May 22, 8:09 AM
Stashbot added a comment.Wed, May 22, 8:11 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T08:11:00Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'T364290 db2173', diff saved to https://phabricator.wikimedia.org/P62845 and previous config saved to /var/cache/conftool/dbconfig/20240522-081059-arnaudb.json

ops-monitoring-bot added a comment.Wed, May 22, 9:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db2173.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, May 22, 10:06 AM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db2173.codfw.wmnet with OS bookworm completed:

  • db2173 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405220944_arnaudb_1829126_db2173.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ABran-WMF updated the task description. (Show Details)Wed, May 22, 10:08 AM
Stashbot added a comment.Wed, May 22, 10:08 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T10:08:34Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'T364290 db2153', diff saved to https://phabricator.wikimedia.org/P62856 and previous config saved to /var/cache/conftool/dbconfig/20240522-100834-arnaudb.json

ops-monitoring-bot added a comment.Wed, May 22, 10:10 AM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db2153.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, May 22, 10:53 AM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db2153.codfw.wmnet with OS bookworm completed:

  • db2153 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405221032_arnaudb_1840377_db2153.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ABran-WMF updated the task description. (Show Details)Wed, May 22, 12:01 PM
Stashbot added a comment.Wed, May 22, 12:02 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T12:02:24Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'T364290 db2145', diff saved to https://phabricator.wikimedia.org/P62876 and previous config saved to /var/cache/conftool/dbconfig/20240522-120223-arnaudb.json

ops-monitoring-bot added a comment.Wed, May 22, 12:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db2145.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, May 22, 12:44 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db2145.codfw.wmnet with OS bookworm completed:

  • db2145 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405221223_arnaudb_1958764_db2145.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ABran-WMF updated the task description. (Show Details)Wed, May 22, 12:46 PM
Stashbot added a comment.Wed, May 22, 3:05 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T15:05:18Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'T364290 db2130', diff saved to https://phabricator.wikimedia.org/P62914 and previous config saved to /var/cache/conftool/dbconfig/20240522-150516-arnaudb.json

ops-monitoring-bot added a comment.Wed, May 22, 3:06 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db2130.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, May 22, 3:50 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db2130.codfw.wmnet with OS bookworm completed:

  • db2130 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405221527_arnaudb_2125243_db2130.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Thu, May 23, 7:56 AM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1238.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, May 23, 8:34 AM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db1238.eqiad.wmnet with OS bookworm completed:

  • db1238 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405230813_arnaudb_2254717_db1238.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Thu, May 23, 10:04 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-23T10:04:53Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db2130 T364290', diff saved to https://phabricator.wikimedia.org/P62970 and previous config saved to /var/cache/conftool/dbconfig/20240523-100452-arnaudb.json

ops-monitoring-bot added a comment.Thu, May 23, 10:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db2130.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, May 23, 10:52 AM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db2130.codfw.wmnet with OS bookworm completed:

  • db2130 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405231028_arnaudb_2279242_db2130.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ABran-WMF updated the task description. (Show Details)Thu, May 23, 11:03 AM
Stashbot added a comment.Thu, May 23, 12:56 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-23T12:56:42Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db2116 T364290', diff saved to https://phabricator.wikimedia.org/P62988 and previous config saved to /var/cache/conftool/dbconfig/20240523-125641-arnaudb.json

ops-monitoring-bot added a comment.Thu, May 23, 12:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db2116.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, May 23, 1:36 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db2116.codfw.wmnet with OS bookworm executed with errors:

  • db2116 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405231319_arnaudb_2309040_db2116.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" db2116.codfw.wmnet to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Thu, May 23, 1:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db2116.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, May 23, 2:28 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db2116.codfw.wmnet with OS bookworm completed:

  • db2116 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405231406_arnaudb_2315121_db2116.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ABran-WMF updated the task description. (Show Details)Thu, May 23, 2:35 PM
ABran-WMF updated the task description. (Show Details)
Stashbot added a comment.Thu, May 23, 2:37 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-23T14:37:43Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1235 T364290', diff saved to https://phabricator.wikimedia.org/P62995 and previous config saved to /var/cache/conftool/dbconfig/20240523-143742-arnaudb.json

ops-monitoring-bot added a comment.Thu, May 23, 2:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1235.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Thu, May 23, 3:16 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db1235.eqiad.wmnet with OS bookworm completed:

  • db1235 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405231455_arnaudb_2324652_db1235.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ABran-WMF updated the task description. (Show Details)Thu, May 23, 3:25 PM
Stashbot added a comment.Tue, May 28, 8:08 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-28T08:08:36Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1228 T364290', diff saved to https://phabricator.wikimedia.org/P63404 and previous config saved to /var/cache/conftool/dbconfig/20240528-080835-arnaudb.json

ops-monitoring-bot added a comment.Tue, May 28, 8:10 AM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1228.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, May 28, 8:45 AM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db1228.eqiad.wmnet with OS bookworm completed:

  • db1228 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405280825_arnaudb_3258839_db1228.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ABran-WMF updated the task description. (Show Details)Tue, May 28, 8:52 AM
Stashbot added a comment.Tue, May 28, 9:07 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-28T09:07:24Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1219 T364290', diff saved to https://phabricator.wikimedia.org/P63410 and previous config saved to /var/cache/conftool/dbconfig/20240528-090724-arnaudb.json

ops-monitoring-bot added a comment.Tue, May 28, 9:08 AM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1219.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, May 28, 9:46 AM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db1219.eqiad.wmnet with OS bookworm completed:

  • db1219 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405280924_arnaudb_3269839_db1219.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ABran-WMF updated the task description. (Show Details)Tue, May 28, 12:24 PM
Stashbot added a comment.Tue, May 28, 12:24 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-28T12:24:43Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1218 T364290', diff saved to https://phabricator.wikimedia.org/P63431 and previous config saved to /var/cache/conftool/dbconfig/20240528-122442-arnaudb.json

ops-monitoring-bot added a comment.Tue, May 28, 12:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1218.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, May 28, 1:03 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db1218.eqiad.wmnet with OS bookworm completed:

  • db1218 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405281242_arnaudb_3302484_db1218.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Tue, May 28, 1:41 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-28T13:41:51Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1207 T364290', diff saved to https://phabricator.wikimedia.org/P63436 and previous config saved to /var/cache/conftool/dbconfig/20240528-134150-arnaudb.json

ops-monitoring-bot added a comment.Tue, May 28, 1:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1207.eqiad.wmnet with OS bookworm

ABran-WMF updated the task description. (Show Details)Tue, May 28, 2:08 PM
ops-monitoring-bot added a comment.Tue, May 28, 2:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1207.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, May 28, 3:07 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db1207.eqiad.wmnet with OS bookworm completed:

  • db1207 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405281446_arnaudb_3320802_db1207.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ABran-WMF updated the task description. (Show Details)Tue, May 28, 3:39 PM
Stashbot added a comment.Tue, May 28, 3:53 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-28T15:53:10Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1206 T364290', diff saved to https://phabricator.wikimedia.org/P63449 and previous config saved to /var/cache/conftool/dbconfig/20240528-155309-arnaudb.json

ops-monitoring-bot added a comment.Tue, May 28, 3:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1206.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, May 28, 4:33 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db1206.eqiad.wmnet with OS bookworm completed:

  • db1206 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405281612_arnaudb_3335262_db1206.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ABran-WMF updated the task description. (Show Details)Wed, May 29, 6:52 AM
Stashbot added a comment.Wed, May 29, 12:43 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-29T12:43:53Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1196 T364290', diff saved to https://phabricator.wikimedia.org/P63529 and previous config saved to /var/cache/conftool/dbconfig/20240529-124352-arnaudb.json

ops-monitoring-bot added a comment.Wed, May 29, 12:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1196.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, May 29, 1:26 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db1196.eqiad.wmnet with OS bookworm completed:

  • db1196 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405291304_arnaudb_3499111_db1196.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ABran-WMF updated the task description. (Show Details)Wed, May 29, 1:27 PM
Stashbot added a comment.Wed, May 29, 1:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-29T13:28:19Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1169 T364290', diff saved to https://phabricator.wikimedia.org/P63535 and previous config saved to /var/cache/conftool/dbconfig/20240529-132818-arnaudb.json

ops-monitoring-bot added a comment.Wed, May 29, 1:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1169.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, May 29, 2:08 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db1169.eqiad.wmnet with OS bookworm completed:

  • db1169 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405291346_arnaudb_3509654_db1169.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ABran-WMF updated the task description. (Show Details)Wed, May 29, 2:11 PM
Stashbot added a comment.Wed, May 29, 2:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-29T14:28:30Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'Depool db1163 T364290', diff saved to https://phabricator.wikimedia.org/P63549 and previous config saved to /var/cache/conftool/dbconfig/20240529-142830-arnaudb.json

ops-monitoring-bot added a comment.Wed, May 29, 2:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1163.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, May 29, 3:08 PM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db1163.eqiad.wmnet with OS bookworm completed:

  • db1163 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405291447_arnaudb_3519736_db1163.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ABran-WMF updated the task description. (Show Details)Wed, May 29, 3:15 PM

only remains the master and the backup sources

jcrespo added a comment.Wed, May 29, 3:28 PM

I will migrate the backups to 10.6 without removing yet the 10.4 backup sources.

gerritbot added a comment.Wed, May 29, 3:46 PM

Change #1037107 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Migrate s1 backups to MariaDB 10.6

https://gerrit.wikimedia.org/r/1037107

gerritbot added a project: Patch-For-Review.Wed, May 29, 3:46 PM
gerritbot added a comment.Wed, May 29, 4:18 PM

Change #1037107 merged by Jcrespo:

[operations/puppet@production] dbbackups: Migrate s1 backups to MariaDB 10.6

https://gerrit.wikimedia.org/r/1037107

jcrespo updated the task description. (Show Details)Wed, May 29, 4:27 PM
Maintenance_bot removed a project: Patch-For-Review.Wed, May 29, 4:31 PM
ops-monitoring-bot added a comment.Tue, Jun 4, 6:26 AM

Cookbook cookbooks.sre.hosts.reimage was started by arnaudb@cumin1002 for host db1184.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Tue, Jun 4, 7:05 AM

Cookbook cookbooks.sre.hosts.reimage started by arnaudb@cumin1002 for host db1184.eqiad.wmnet with OS bookworm completed:

  • db1184 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202406040643_arnaudb_429246_db1184.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ABran-WMF closed this task as Resolved.Tue, Jun 4, 7:07 AM
ABran-WMF updated the task description. (Show Details)
ABran-WMF moved this task from In progress to Done on the DBA board.
ops-monitoring-bot added a comment.Wed, Jun 5, 7:38 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db1186.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, Jun 5, 8:19 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db1186.eqiad.wmnet with OS bookworm completed:

  • db1186 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202406050756_marostegui_649385_db1186.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL