Page MenuHomePhabricator

Migrate m3 (phabricator) databases to Bookworm and MariaDB 10.6
Closed, ResolvedPublic

Description

  • db1159 (master)
  • db1217
  • db2134 (master)
  • db2160
  • Move db1119 to m3 to be able to become m3 temporary master.

Event Timeline

Marostegui triaged this task as Medium priority.Nov 27 2023, 6:45 AM
Marostegui moved this task from Triage to In progress on the DBA board.

Change 977321 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1119: Disable notifications

https://gerrit.wikimedia.org/r/977321

Change 977321 merged by Marostegui:

[operations/puppet@production] db1119: Disable notifications

https://gerrit.wikimedia.org/r/977321

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2134.codfw.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2134.codfw.wmnet with OS bookworm completed:

  • db2134 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311270711_marostegui_178619_db2134.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Change 977991 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1119: Move it to m3

https://gerrit.wikimedia.org/r/977991

Change 977991 merged by Marostegui:

[operations/puppet@production] db1119: Move it to m3

https://gerrit.wikimedia.org/r/977991

Change 978722 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1159: Disable notifications

https://gerrit.wikimedia.org/r/978722

Change 978722 merged by Marostegui:

[operations/puppet@production] db1159: Disable notifications

https://gerrit.wikimedia.org/r/978722

db1159 has been reimaged. I am going to give it 24h before promoting it back to master, which will be tracked at T352360