db1132 is no longer m5 master (T302190), let's reimage it to Bullseye + MariaDB 10.6 and let it replicate in s1 to measure its replication throughput and eventually let it serve queries in s1 in order to look for strange query plans and/or regressions
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Marostegui | T301879 Test MariaDB 10.6 on Bullseye | |||
| Resolved | Marostegui | T303395 Move db1132 from m5 to s1 |
Event Timeline
Comment Actions
Change 769612 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1132: Move it from m5 to s1
Comment Actions
Change 769612 merged by Marostegui:
[operations/puppet@production] db1132: Move it from m5 to s1
Comment Actions
Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1132.eqiad.wmnet with OS bullseye
Comment Actions
Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1132.eqiad.wmnet with OS bullseye completed:
- db1132 (WARN)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203100633_marostegui_164022_db1132.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB
Comment Actions
Change 769655 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1099: Disable notifications
Comment Actions
Change 769655 merged by Marostegui:
[operations/puppet@production] db1099: Disable notifications
Comment Actions
Change 769677 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1132: Install MariaDB 10.6
Comment Actions
Change 769677 merged by Marostegui:
[operations/puppet@production] db1132: Install MariaDB 10.6
Comment Actions
db1132 is now replicating on s1 running bullseye and mariadb 10.6. It is not serving traffic yet.
Comment Actions
Change 770443 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] site.pp: Specify db1132 status
Comment Actions
Change 770443 merged by Marostegui:
[operations/puppet@production] site.pp: Specify db1132 status
Comment Actions
The move itself is done, I will carry on the tests and track them at the main task T301879