Let's upgrade m3 to Bullseye.
- db2134
- db2078
- db1107
- db1117
- Reimage and move db1183 from m2 to m3 so it can later become m3 master.
- Promote db1183 to m3 master: T301219
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T291916 Tracking task for Bullseye migrations in production | |||
| Resolved | Marostegui | T298585 Upgrade WMF database-and-backup-related hosts to bullseye | |||
| Resolved | Marostegui | T300835 Upgrade m3 to Bullseye | |||
| Resolved | Marostegui | T301219 Switchover m3 master (db1107 -> db1183) |
Event Timeline
Comment Actions
Change 759450 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] mariadb: Move db1183 from m2 to m3
Comment Actions
Change 759450 merged by Marostegui:
[operations/puppet@production] mariadb: Move db1183 from m2 to m3
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1183.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1183.eqiad.wmnet with OS bullseye completed:
- db1183 (WARN)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202030931_marostegui_1553_db1183.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Change 759472 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2134: Disable notifications
Comment Actions
Change 759472 merged by Marostegui:
[operations/puppet@production] db2134: Disable notifications
Comment Actions
Change 760814 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2134: Disable notifications
Comment Actions
Change 760814 merged by Marostegui:
[operations/puppet@production] db2134: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2134.codfw.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2134.codfw.wmnet with OS bullseye completed:
- db2134 (WARN)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202080622_marostegui_31510_db2134.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB