- db1159 (master)
- db1217
- db2134 (master)
- db2160
- Move db1119 to m3 to be able to become m3 temporary master.
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Marostegui | T356960 Upgrade hosts to MariaDB 10.6 | |||
| Resolved | Marostegui | T351990 Migrate m3 (phabricator) databases to Bookworm and MariaDB 10.6 | |||
| Resolved | Marostegui | T352149 Switchover m3 master db1159 -> db1119 | |||
| Resolved | Marostegui | T352360 Switchover m3 master db1119 -> db1159 |
Event Timeline
Comment Actions
Change 977321 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1119: Disable notifications
Comment Actions
Change 977321 merged by Marostegui:
[operations/puppet@production] db1119: Disable notifications
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2134.codfw.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2134.codfw.wmnet with OS bookworm completed:
- db2134 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311270711_marostegui_178619_db2134.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB
Comment Actions
Change 977991 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1119: Move it to m3
Comment Actions
Change 977991 merged by Marostegui:
[operations/puppet@production] db1119: Move it to m3
Comment Actions
Change 978722 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1159: Disable notifications
Comment Actions
Change 978722 merged by Marostegui:
[operations/puppet@production] db1159: Disable notifications
Comment Actions
db1159 has been reimaged. I am going to give it 24h before promoting it back to master, which will be tracked at T352360