After finishing clouddb hosts (T356838) sanitarium hosts can now be upgraded
- db1154
- db1155
- db2186
- db2187
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| db1154: Migrate to mariadb 10.6 | operations/puppet | production | +1 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Marostegui | T356960 Upgrade hosts to MariaDB 10.6 | |||
| Resolved | Marostegui | T358638 Upgrade sanitarium hosts to MariaDB 10.6 | |||
| Resolved | Marostegui | T358640 Reclone db2186:3313 (sanitarium) |
Event Timeline
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2186.codfw.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1002 for host db2187.codfw.wmnet with OS bookworm
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2186.codfw.wmnet with OS bookworm completed:
- db2186 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402280720_marostegui_3431477_db2186.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1002 for host db2187.codfw.wmnet with OS bookworm completed:
- db2187 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Checked BIOS boot parameters are back to normal
- Host up (new fresh bookworm OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202402280731_marostegui_3433600_db2187.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Change 1012467 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1154: Migrate to mariadb 10.6
Comment Actions
Change 1012467 merged by Marostegui:
[operations/puppet@production] db1154: Migrate to mariadb 10.6