Page MenuHomePhabricator
Create Task
Maniphest T290865

Upgrade s1 to Buster + MariaDB 10.4
Closed, ResolvedPublic
Actions

Description

After the DC switch (scheduled for 14th Sept 2021)

  • Available backup source on the standby DC (db2141 - all sections have sources everywhere now)
  • Switchover backup generation standby DC (Ready to deploy: https://gerrit.wikimedia.org/r/c/operations/puppet/+/721285 )
  • Candidate master on the standby DC (db2112) blocked on T293740
  • Master on the standby DC (db2103)
  • Candidate master on the primary DC
  • Available backup source on the primary DC (db1140 - all sections have sources everywhere now)
  • Switchover backup generation Primary DC (Ready to deploy: https://gerrit.wikimedia.org/r/c/operations/puppet/+/721286 )
  • Switchover on the primary DC to promote a Buster+10.4 host to master: T293964
  • Upgrade the old master and make it a candidate master, pool it
  • Cleanup (remove) old backup sources from both DCs @jcrespo

Please read the doc about procedure for more details.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Sep 15 2021, 11:47 AM

Change 721285 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Switch s1 backup generation from db2097 to db2141

https://gerrit.wikimedia.org/r/721285

gerritbot added a project: Patch-For-Review.Sep 15 2021, 11:47 AM
jcrespo updated the task description. (Show Details)Sep 15 2021, 11:49 AM
gerritbot added a comment.Sep 15 2021, 11:53 AM

Change 721286 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Switch s1 backup generation from db1139 to db1140

https://gerrit.wikimedia.org/r/721286

jcrespo updated the task description. (Show Details)Sep 15 2021, 11:55 AM
gerritbot added a comment.Sep 15 2021, 12:13 PM

Change 721288 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Migrate s8 backups db2100 -> db2098; reimage dbprov2001

https://gerrit.wikimedia.org/r/721288

gerritbot added a comment.Sep 17 2021, 5:12 AM

Change 721652 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Reimage db2103 to Buster

https://gerrit.wikimedia.org/r/721652

gerritbot added a comment.Sep 17 2021, 5:13 AM

Change 721652 merged by Marostegui:

[operations/puppet@production] install_server: Reimage db2103 to Buster

https://gerrit.wikimedia.org/r/721652

gerritbot added a comment.Sep 23 2021, 6:57 AM

Change 723038 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2103: Disable notifications

https://gerrit.wikimedia.org/r/723038

gerritbot added a comment.Sep 23 2021, 6:58 AM

Change 723038 merged by Marostegui:

[operations/puppet@production] db2103: Disable notifications

https://gerrit.wikimedia.org/r/723038

Marostegui added a comment.EditedSep 23 2021, 10:59 AM

All s1 core codfw replica hosts upgraded to the minor 10.4.21
@jcrespo you might want to also upgrade the source backup (I can do so if you give me green light)

Stashbot added a comment.Sep 23 2021, 11:10 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-23T11:10:35Z] <jynus> restart and upgrade db2141 T290865

jcrespo added a comment.Sep 23 2021, 11:23 AM
In T290865#7373914, @Marostegui wrote:

@jcrespo you might want to also upgrade the source backup (I can do so if you give me green light)

Done.

Marostegui moved this task from Ready to In progress on the DBA board.Sep 24 2021, 9:42 AM
gerritbot added a comment.Sep 28 2021, 12:56 PM

Change 724414 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Remove db2103

https://gerrit.wikimedia.org/r/724414

gerritbot added a comment.Sep 28 2021, 12:57 PM

Change 724414 merged by Marostegui:

[operations/puppet@production] install_server: Remove db2103

https://gerrit.wikimedia.org/r/724414

ops-monitoring-bot added a comment.Sep 28 2021, 1:03 PM

Cookbook cookbooks.sre.experimental.reimage was started by marostegui@cumin1001 for host db2103.codfw.wmnet

ops-monitoring-bot added a comment.Sep 28 2021, 1:36 PM

Cookbook cookbooks.sre.experimental.reimage completed:

  • db2103 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB
    • Removed from Debmonitor
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/wmf-auto-reimage/202109281303_marostegui_5029_db2103.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
Marostegui added a comment.Sep 28 2021, 1:38 PM

db2103 (candidate master) reimaged to buster - checking its tables now.

Stashbot added a comment.Sep 28 2021, 1:40 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-28T13:40:12Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2103 T290865', diff saved to https://phabricator.wikimedia.org/P17337 and previous config saved to /var/cache/conftool/dbconfig/20210928-134012-marostegui.json

Marostegui added a comment.Sep 29 2021, 4:49 AM

db2103's check came clean, so I have restarted replication.

Stashbot added a comment.Sep 29 2021, 5:56 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-29T05:56:47Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2103 T290865', diff saved to https://phabricator.wikimedia.org/P17344 and previous config saved to /var/cache/conftool/dbconfig/20210929-055645-marostegui.json

Marostegui updated the task description. (Show Details)Sep 29 2021, 5:58 AM
Marostegui mentioned this in T254688: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing.Sep 30 2021, 7:07 AM
gerritbot added a comment.Oct 19 2021, 5:45 AM

Change 731872 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2112: Disable notifications

https://gerrit.wikimedia.org/r/731872

gerritbot added a comment.Oct 19 2021, 5:46 AM

Change 731872 merged by Marostegui:

[operations/puppet@production] db2112: Disable notifications

https://gerrit.wikimedia.org/r/731872

Stashbot added a comment.Oct 19 2021, 5:46 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-19T05:46:56Z] <marostegui> Reimage db2112 (s1 codfw master) T290865

ops-monitoring-bot added a comment.Oct 19 2021, 5:51 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2112.codfw.wmnet with OS buster

Marostegui added a comment.Oct 19 2021, 6:19 AM

I am having issues with db2112 reimage (codfw master) - going to try to see if I can overcome them and if not, I will do a master switchover as the candidate master is already reimaged.

Marostegui added a comment.Oct 19 2021, 6:38 AM

Looks like it fails to completely load the debian installer after loading the initrd and then after the timeout it reboots itself.
So I think I am going to proceed with the DC switchover and then ask Papaul to upgrade firmwares in case we are bing hit by T216240 again

ops-monitoring-bot added a comment.Oct 19 2021, 6:38 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2112.codfw.wmnet with OS buster executed with errors:

  • db2112 (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Oct 19 2021, 8:49 AM

Change 731882 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Promote db2103 to s1 codfw master

https://gerrit.wikimedia.org/r/731882

Kormat mentioned this in T281058: Rename AbuseFilter indexes for consistency.Oct 19 2021, 9:04 AM
gerritbot added a comment.Oct 19 2021, 9:13 AM

Change 731882 merged by Marostegui:

[operations/puppet@production] mariadb: Promote db2103 to s1 codfw master

https://gerrit.wikimedia.org/r/731882

gerritbot added a comment.Oct 19 2021, 9:17 AM

Change 721285 merged by Marostegui:

[operations/puppet@production] dbbackups: Switch s1 backup generation from db2097 to db2141

https://gerrit.wikimedia.org/r/721285

Marostegui updated the task description. (Show Details)Oct 19 2021, 9:17 AM

I have switched over codfw s1 master, the new master is db2103.

Stashbot added a comment.Oct 19 2021, 9:19 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-19T09:19:58Z] <marostegui> Stop slave on db2112 T290865

ops-monitoring-bot added a comment.Oct 19 2021, 9:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2112.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 19 2021, 9:50 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2112.codfw.wmnet with OS buster executed with errors:

  • db2112 (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 19 2021, 9:52 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2112.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 19 2021, 10:16 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2112.codfw.wmnet with OS buster executed with errors:

  • db2112 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 19 2021, 10:18 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2112.codfw.wmnet with OS stretch

ops-monitoring-bot added a comment.Oct 19 2021, 10:49 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2112.codfw.wmnet with OS stretch completed:

  • db2112 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh stretch OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110191018_marostegui_22737_db2112.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Oct 19 2021, 12:24 PM

Change 731938 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1018: Depool clouddb1017

https://gerrit.wikimedia.org/r/731938

gerritbot added a comment.Oct 19 2021, 12:26 PM

Change 731938 merged by Marostegui:

[operations/puppet@production] dbproxy1018: Depool clouddb1017

https://gerrit.wikimedia.org/r/731938

Marostegui added a comment.Oct 19 2021, 1:29 PM

Upgraded all s1 eqiad replicas (apart from backup and candidate master) to 10.4.21.
Pending one clouddb* hosts which I will do today and tomorrow (to avoid having two of them fully cold the same day).

Marostegui updated the task description. (Show Details)Oct 19 2021, 1:29 PM
Marostegui updated the task description. (Show Details)
Papaul closed subtask T293740: Upgrade db2112 firmware/BIOS as Resolved.Oct 19 2021, 2:47 PM
ops-monitoring-bot added a comment.Oct 20 2021, 5:26 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2112.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 20 2021, 5:59 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2112.codfw.wmnet with OS buster completed:

  • db2112 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110200526_marostegui_1881_db2112.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui added a comment.Oct 20 2021, 6:01 AM

db2112 (old master and now candidate master) has been reimaged to Buster. Checking its tables now.

gerritbot added a comment.Oct 20 2021, 6:19 AM

Change 732118 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1019: Depool clouddb1013

https://gerrit.wikimedia.org/r/732118

gerritbot added a comment.Oct 20 2021, 6:20 AM

Change 732118 merged by Marostegui:

[operations/puppet@production] dbproxy1019: Depool clouddb1013

https://gerrit.wikimedia.org/r/732118

gerritbot added a comment.Oct 20 2021, 6:45 AM

Change 732248 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1118: Disable notifications

https://gerrit.wikimedia.org/r/732248

Stashbot added a comment.Oct 20 2021, 6:45 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-20T06:45:30Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1118 (s1) for reimage T290865', diff saved to https://phabricator.wikimedia.org/P17552 and previous config saved to /var/cache/conftool/dbconfig/20211020-064529-marostegui.json

gerritbot added a comment.Oct 20 2021, 6:46 AM

Change 732248 merged by Marostegui:

[operations/puppet@production] db1118: Disable notifications

https://gerrit.wikimedia.org/r/732248

ops-monitoring-bot added a comment.Oct 20 2021, 6:49 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1118.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 20 2021, 7:16 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1118.eqiad.wmnet with OS buster completed:

  • db1118 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110200648_marostegui_17828_db1118.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui added a comment.Oct 20 2021, 7:17 AM

db1118 (candidate master on eqiad) reimaged, checking its tables now.

Marostegui updated the task description. (Show Details)Oct 21 2021, 4:29 AM

db1118 is fine, restarted replication.

Marostegui added a comment.Oct 21 2021, 4:31 AM

db2112 is fine, restarted replication

Marostegui updated the task description. (Show Details)Oct 21 2021, 4:31 AM
Marostegui updated the task description. (Show Details)Oct 21 2021, 4:52 AM
Marostegui added a comment.Oct 21 2021, 5:25 AM

clouddb hosts for s1 are fully upgraded

Marostegui added a subtask: T254688: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing.Oct 22 2021, 6:33 AM
Marostegui added a comment.Oct 25 2021, 4:28 AM

This is now blocked on the switchover: T293964

Marostegui closed subtask T254688: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing as Resolved.Oct 25 2021, 5:28 AM
Marostegui closed subtask T293964: Switchover s1 from db1163 to db1118 as Resolved.Nov 3 2021, 6:09 AM
Marostegui mentioned this in T293964: Switchover s1 from db1163 to db1118.
Stashbot added a comment.Nov 3 2021, 6:10 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-03T06:10:18Z] <marostegui> Stop replication on db1163 T290865

gerritbot added a comment.Nov 3 2021, 6:11 AM

Change 736328 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1163: Disable notifications

https://gerrit.wikimedia.org/r/736328

gerritbot added a comment.Nov 3 2021, 6:12 AM

Change 736328 merged by Marostegui:

[operations/puppet@production] db1163: Disable notifications

https://gerrit.wikimedia.org/r/736328

Marostegui updated the task description. (Show Details)Nov 3 2021, 6:12 AM
ops-monitoring-bot added a comment.Nov 3 2021, 6:13 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1163.eqiad.wmnet with OS buster

gerritbot added a comment.Nov 3 2021, 6:22 AM

Change 721286 merged by Marostegui:

[operations/puppet@production] dbbackups: Switch s1 backup generation from db1139 to db1140

https://gerrit.wikimedia.org/r/721286

Marostegui added a comment.Nov 3 2021, 6:23 AM

Merged the backup change as well. We had a s1 snapshot that was taken successfully past night so nothing in progress.

Marostegui updated the task description. (Show Details)Nov 3 2021, 6:23 AM
ops-monitoring-bot added a comment.Nov 3 2021, 6:43 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1163.eqiad.wmnet with OS buster completed:

  • db1163 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111030613_marostegui_15409_db1163.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui added a comment.Nov 3 2021, 7:00 AM

db1163 has been reimaged to Buster. I am now checking its tables.

Marostegui added a comment.Nov 4 2021, 5:42 AM

db1163 came back clean - started replication.

Marostegui added a comment.Nov 4 2021, 7:44 AM

I am slowly repooling db1163

Marostegui updated the task description. (Show Details)Nov 4 2021, 7:44 AM
Marostegui moved this task from In progress to Done on the DBA board.

Only left the backup sources cleanup by @jcrespo

Marostegui updated the task description. (Show Details)Nov 4 2021, 7:45 AM
gerritbot added a comment.Nov 11 2021, 11:13 AM

Change 738195 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Reimage db1116, db1139, db2097, db2100 to buster

https://gerrit.wikimedia.org/r/738195

gerritbot added a comment.Nov 11 2021, 11:39 AM

Change 738195 merged by Jcrespo:

[operations/puppet@production] dbbackups: Reimage db1116, db1139, db2097, db2100 to buster

https://gerrit.wikimedia.org/r/738195

jcrespo updated the task description. (Show Details)Nov 12 2021, 11:26 AM

@Marostegui backup upgrade/cleanup is done, I will later reorganize backup distribution (but that is out of scope).

Marostegui closed this task as Resolved.Nov 12 2021, 11:27 AM

Closing! thank you!!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL