Page MenuHomePhabricator
Create Task
Maniphest T383902

Upgrade backup source or mediabackup database host os to Debian bookworm or decommission them
Closed, ResolvedPublic
Actions

Description

Backup sources

  • db1150
  • db1171
  • db1216
  • db1225
  • db1239
  • db1240
  • db1245
  • db2139 - to be decommed
  • db2141
  • db2197
  • db2198
  • db2199
  • db2200
  • db2201
  • db2239

backup1 hosts

  • db1204
  • db1205
  • db2183
  • db2184

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
dbbackups: Remove last references to dbprov[12]00[12]operations/puppetproduction+0 -12
dbbackups: Fix m5 backup grant issuesoperations/puppetproduction+12 -12
dbbackups: Update grants for misc hosts other than m1operations/puppetproduction+202 -210
dbbackups: Fix dump grants for backup sources and m1operations/puppetproduction+45 -50
dbbackups: Decommission db2139operations/puppetproduction+8 -12
installserver: Review backup and db hostsoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedjcrespoT376916 Upgrade backup hosts to Debian Bookworm 12.X
 ResolvedjcrespoT383902 Upgrade backup source or mediabackup database host os to Debian bookworm or decommission them
 ResolvedjcrespoT366092 Upgrade eqiad mediabackups database hosts to Debian Bookworm
 ResolvedMarosteguiT373579 Productionize db22[21-40]
 ResolvedMarosteguiT374951 find a viable replication source on codfw
DeclinedABran-WMFT374088 Switchover s4 master (db2140 -> db2179)
 ResolvedABran-WMFT372893 Testing environment for mysql cookbooks
 ResolvedABran-WMFT374086 Switchover s5 master (db2123 -> db2213)
 ResolvedLadsgroupT374087 Switchover s6 master (db2129 -> db2214)
 ResolvedMarosteguiT374623 Decommission db21[21-40]
 ResolvedMarosteguiT368874 Productionize dbproxy102[89]
 ResolvedRequestJclark-ctrT383025 decommission dbproxy1020.eqiad.wmnet
 ResolvedRequestJclark-ctrT383033 decommission dbproxy1021.eqiad.wmnet
 ResolvedMarosteguiT367380 Productionize dbproxy200[5-8]
 ResolvedMarosteguiT381962 Decommission dbproxy200[1-4]
 ResolvedRequestJhancock.wmT382867 decommission dbproxy2001.codfw.wmnet
 ResolvedRequestJhancock.wmT382868 decommission dbproxy2002.codfw.wmnet
 ResolvedRequestJhancock.wmT382875 decommission dbproxy2003.codfw.wmnet
 ResolvedRequestJhancock.wmT382877 decommission dbproxy2004.codfw.wmnet
 ResolvedRequestJhancock.wmT383388 decommission db2123.codfw.wmnet
 ResolvedRequestJhancock.wmT383395 decommission db2126.codfw.wmnet
 ResolvedRequestJhancock.wmT383426 decommission db2135.codfw.wmnet
 ResolvedRequestJhancock.wmT383572 decommission db2128.codfw.wmnet
 ResolvedRequestJhancock.wmT383697 decommission db2132.codfw.wmnet
 ResolvedRequestJhancock.wmT383766 decommission db2130.codfw.wmnet
 ResolvedJhancock.wmT383971 decommission db2139
 ResolvedRequestJhancock.wmT384001 decommission db2131.codfw.wmnet
 ResolvedRequestJhancock.wmT384343 decommission db2133.codfw.wmnet
 ResolvedRequestJhancock.wmT384476 decommission db2134.codfw.wmnet
 ResolvedRequestJhancock.wmT384479 decommission db2136.codfw.wmnet
 ResolvedRequestJhancock.wmT384480 decommission db2140.codfw.wmnet
 ResolvedABran-WMFT378503 mariadb::binlog_format: 'ROW' not effective on m5/db1175

Event Timeline

jcrespo created this task.Jan 16 2025, 3:54 PM
jcrespo changed the task status from Open to In Progress.
jcrespo claimed this task.
jcrespo triaged this task as High priority.
jcrespo edited projects, added database-backups, media-backups; removed Epic.
jcrespo added a subtask: T366092: Upgrade eqiad mediabackups database hosts to Debian Bookworm.
jcrespo changed the status of subtask T366092: Upgrade eqiad mediabackups database hosts to Debian Bookworm from Open to In Progress.
jcrespo removed a subscriber: Aklapper.
jcrespo added subtasks: T374623: Decommission db21[21-40], T373579: Productionize db22[21-40].Jan 16 2025, 3:57 PM
jcrespo updated the task description. (Show Details)Jan 16 2025, 4:06 PM
gerritbot added a comment.Jan 17 2025, 10:25 AM

Change #1112184 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] installserver: Review backup and db hosts

https://gerrit.wikimedia.org/r/1112184

gerritbot added a project: Patch-For-Review.Jan 17 2025, 10:25 AM
Marostegui closed subtask T373579: Productionize db22[21-40] as Resolved.Jan 17 2025, 11:00 AM
gerritbot added a comment.Jan 17 2025, 11:30 AM

Change #1112184 merged by Jcrespo:

[operations/puppet@production] installserver: Review backup and db hosts

https://gerrit.wikimedia.org/r/1112184

Maintenance_bot removed a project: Patch-For-Review.Jan 17 2025, 11:31 AM
ops-monitoring-bot added a comment.Jan 17 2025, 11:37 AM

Icinga downtime and Alertmanager silence (ID=2ec27167-237c-4fd7-9ccb-4486e0a3234c) set by jynus@cumin2002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: reimage

db2141.codfw.wmnet
ops-monitoring-bot added a comment.Jan 17 2025, 11:41 AM

Cookbook cookbooks.sre.hosts.reimage was started by jynus@cumin2002 for host db2141.codfw.wmnet with OS bookworm

jcrespo updated the task description. (Show Details)Jan 17 2025, 12:11 PM
ops-monitoring-bot added a comment.Jan 17 2025, 12:25 PM

Cookbook cookbooks.sre.hosts.reimage started by jynus@cumin2002 for host db2141.codfw.wmnet with OS bookworm completed:

  • db2141 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501171201_jynus_262088_db2141.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
jcrespo added a comment.Jan 17 2025, 12:27 PM

Icinga status is not optimal, downtime not removed

That was because I am rebuilding the tables and thus replication is stopped.

ops-monitoring-bot added a comment.Jan 20 2025, 10:27 AM

Icinga downtime and Alertmanager silence (ID=d7245c24-f67b-4f43-ae17-b2ef80f610ed) set by jynus@cumin1002 for 2:00:00 on 1 host(s) and their services with reason: os upgrade

db1245.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 20 2025, 10:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host db1245.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 20 2025, 11:16 AM

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host db1245.eqiad.wmnet with OS bookworm completed:

  • db1245 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501201054_root_52321_db1245.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Jan 20 2025, 4:28 PM

Change #1112802 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Remove set user permissions from m1 backup user grants

https://gerrit.wikimedia.org/r/1112802

gerritbot added a project: Patch-For-Review.Jan 20 2025, 4:28 PM
jcrespo updated the task description. (Show Details)Jan 21 2025, 8:53 PM
ops-monitoring-bot added a comment.Jan 22 2025, 9:16 AM

Icinga downtime and Alertmanager silence (ID=f63b7dc3-cd57-40da-aafb-d98d09fe8ad8) set by jynus@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: os upgrade

db1240.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 22 2025, 9:21 AM

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host db1240.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 22 2025, 10:03 AM

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host db1240.eqiad.wmnet with OS bookworm completed:

  • db1240 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501220940_root_844917_db1240.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 22 2025, 12:02 PM

Icinga downtime and Alertmanager silence (ID=8324c4ec-57d8-4460-ab4e-364dd23824da) set by jynus@cumin1002 for 2:00:00 on 1 host(s) and their services with reason: os upgrade

db1205.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 22 2025, 12:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host db1205.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 22 2025, 12:48 PM

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host db1205.eqiad.wmnet with OS bookworm completed:

  • db1205 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501221225_root_909257_db1205.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
jcrespo updated the task description. (Show Details)Jan 22 2025, 1:01 PM
jcrespo updated the task description. (Show Details)Jan 22 2025, 1:45 PM
Stashbot added a comment.Jan 23 2025, 10:57 AM

Mentioned in SAL (#wikimedia-operations) [2025-01-23T10:57:52Z] <jynus> pausing media backups on eqiad for maintenance T383902

ops-monitoring-bot added a comment.Jan 23 2025, 11:03 AM

Icinga downtime and Alertmanager silence (ID=8968cdca-1368-4a8b-8d7b-88380f4a6dfe) set by jynus@cumin1002 for 4:00:00 on 1 host(s) and their services with reason: os upgrade

db1204.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 23 2025, 11:04 AM

Icinga downtime and Alertmanager silence (ID=8e426345-55db-4d8d-97b1-479f633bd115) set by jynus@cumin1002 for 4:00:00 on 1 host(s) and their services with reason: os upgrade

db1205.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 23 2025, 11:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host db1204.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 23 2025, 11:49 AM

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host db1204.eqiad.wmnet with OS bookworm completed:

  • db1204 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501231126_root_1355894_db1204.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
jcrespo closed subtask T366092: Upgrade eqiad mediabackups database hosts to Debian Bookworm as Resolved.Jan 23 2025, 12:19 PM
jcrespo mentioned this in T366092: Upgrade eqiad mediabackups database hosts to Debian Bookworm.
jcrespo updated the task description. (Show Details)
ops-monitoring-bot added a comment.Jan 23 2025, 1:46 PM

Icinga downtime and Alertmanager silence (ID=6f470676-dea0-4f69-80aa-826d9c313d20) set by jynus@cumin1002 for 2:00:00 on 1 host(s) and their services with reason: reimage

db1239.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 23 2025, 1:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host db1239.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 23 2025, 2:33 PM

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host db1239.eqiad.wmnet with OS bookworm completed:

  • db1239 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501231410_root_1505629_db1239.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
jcrespo updated the task description. (Show Details)Jan 24 2025, 7:50 AM
ops-monitoring-bot added a comment.Jan 24 2025, 7:51 AM

Icinga downtime and Alertmanager silence (ID=9b068a8d-a257-44ae-b3a9-aa92c844556a) set by jynus@cumin1002 for 6:00:00 on 1 host(s) and their services with reason: os upgrade

db1225.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 24 2025, 8:04 AM

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host db1225.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 24 2025, 8:11 AM

Icinga downtime and Alertmanager silence (ID=fef62d94-be78-4342-b8f7-11aec550c58d) set by jynus@cumin1002 for 4:00:00 on 1 host(s) and their services with reason: os upgrade

db1216.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 24 2025, 8:30 AM

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host db1216.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 24 2025, 8:49 AM

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host db1225.eqiad.wmnet with OS bookworm completed:

  • db1225 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501240825_root_1884836_db1225.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 24 2025, 9:14 AM

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host db1216.eqiad.wmnet with OS bookworm completed:

  • db1216 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501240851_root_1888873_db1216.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
jcrespo updated the task description. (Show Details)Jan 24 2025, 1:01 PM
jcrespo updated the task description. (Show Details)Jan 24 2025, 1:07 PM
ops-monitoring-bot added a comment.Jan 24 2025, 1:30 PM

Icinga downtime and Alertmanager silence (ID=850d9949-25a8-4e40-abfc-8a9183ee0279) set by jynus@cumin1002 for 3 days, 0:00:00 on 1 host(s) and their services with reason: rebuilding tables

db1216.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 27 2025, 10:37 AM

Icinga downtime and Alertmanager silence (ID=da5694f7-58b7-4217-8b3e-f5bde76d7490) set by jynus@cumin1002 for 3 days, 0:00:00 on 1 host(s) and their services with reason: reimage

db1171.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 27 2025, 10:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host db1171.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 27 2025, 11:56 AM

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host db1171.eqiad.wmnet with OS bookworm executed with errors:

  • db1171 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console db1171.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jan 27 2025, 11:58 AM

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host db1171.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 27 2025, 12:39 PM

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host db1171.eqiad.wmnet with OS bookworm completed:

  • db1171 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501271216_root_2535731_db1171.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
jcrespo updated the task description. (Show Details)Jan 27 2025, 2:31 PM
ops-monitoring-bot added a comment.Jan 28 2025, 9:00 AM

Icinga downtime and Alertmanager silence (ID=cd3eaa16-18c2-4399-afd5-a1186d100dc4) set by jynus@cumin1002 for 3 days, 0:00:00 on 1 host(s) and their services with reason: reimage

db1150.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 28 2025, 9:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by root@cumin1002 for host db1150.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Jan 28 2025, 9:50 AM

Cookbook cookbooks.sre.hosts.reimage started by root@cumin1002 for host db1150.eqiad.wmnet with OS bookworm completed:

  • db1150 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202501280926_root_2794567_db1150.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui closed subtask T374623: Decommission db21[21-40] as Resolved.Jan 28 2025, 11:19 AM
jcrespo updated the task description. (Show Details)Jan 28 2025, 7:48 PM
ops-monitoring-bot added a comment.Jan 30 2025, 10:37 AM

Icinga downtime and Alertmanager silence (ID=ac63cc5e-30f2-4726-b368-8ac5c3ba5641) set by jynus@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: test new s4 backups

db2201.codfw.wmnet
ops-monitoring-bot added a comment.Jan 30 2025, 10:38 AM

Icinga downtime and Alertmanager silence (ID=0f2ee58b-8e61-458b-8036-54ff80c6dc78) set by jynus@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: prepare for decom

db2202.codfw.wmnet
gerritbot added a comment.Jan 31 2025, 9:14 AM

Change #1115774 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Decommission db2139

https://gerrit.wikimedia.org/r/1115774

gerritbot added a comment.Jan 31 2025, 9:40 AM

Change #1115774 merged by Jcrespo:

[operations/puppet@production] dbbackups: Decommission db2139

https://gerrit.wikimedia.org/r/1115774

jcrespo updated the task description. (Show Details)Jan 31 2025, 10:21 AM

This is technically all done (all hosts have been upgraded), but I still need to review grants and hopefully remove and unify them among all hosts.

gerritbot added a comment.Feb 3 2025, 5:38 PM

Change #1112802 merged by Jcrespo:

[operations/puppet@production] dbbackups: Fix dump grants for backup sources and m1

https://gerrit.wikimedia.org/r/1112802

gerritbot added a comment.Feb 3 2025, 5:40 PM

Change #1116845 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Update grants for misc hosts other than m1

https://gerrit.wikimedia.org/r/1116845

gerritbot added a comment.Feb 3 2025, 5:40 PM

Change #1116846 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Remove last references to dbprov[12]00[12]

https://gerrit.wikimedia.org/r/1116846

Stashbot added a comment.Feb 4 2025, 11:48 AM

Mentioned in SAL (#wikimedia-operations) [2025-02-04T11:48:11Z] <jynus> deploying new backup grants for matomo and analytics_meta T383902

Stashbot added a comment.Feb 4 2025, 12:38 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-04T12:38:18Z] <jynus> deploying new backup grants for ES hosts T383902

gerritbot added a comment.Feb 4 2025, 12:49 PM

Change #1117182 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Fix m5 backup grant issues

https://gerrit.wikimedia.org/r/1117182

gerritbot added a comment.Feb 4 2025, 12:53 PM

Change #1116845 merged by Jcrespo:

[operations/puppet@production] dbbackups: Update grants for misc hosts other than m1

https://gerrit.wikimedia.org/r/1116845

gerritbot added a comment.Feb 4 2025, 12:58 PM

Change #1117182 merged by Jcrespo:

[operations/puppet@production] dbbackups: Fix m5 backup grant issues

https://gerrit.wikimedia.org/r/1117182

jcrespo closed this task as Resolved.Feb 4 2025, 1:15 PM

This is now done, all backup user grants reviewed & updated.

gerritbot added a comment.Feb 5 2025, 2:29 PM

Change #1116846 merged by Jcrespo:

[operations/puppet@production] dbbackups: Remove last references to dbprov[12]00[12]

https://gerrit.wikimedia.org/r/1116846

Maintenance_bot removed a project: Patch-For-Review.Feb 5 2025, 2:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits