Page MenuHomePhabricator
Create Task
Maniphest T290868

Upgrade s8 to Buster + MariaDB 10.4
Closed, ResolvedPublic
Actions

Description

After the DC switch (scheduled for 14th Sept 2021)

  • Available backup source on the standby DC (db2098 - all sections have sources everywhere now)
  • Switchover backup generation standby DC (Ready to deploy: https://gerrit.wikimedia.org/r/c/operations/puppet/+/731404 )
  • Candidate master on the standby DC
  • Master on the standby DC
  • Candidate master on the primary DC
  • Available backup source on the primary DC (db1171 - all sections have sources everywhere now)
  • Switchover backup generation Primary DC @jcrespo https://gerrit.wikimedia.org/r/c/operations/puppet/+/736946
  • Switchover on the primary DC to promote a Buster+10.4 host to master: T294321
  • Upgrade the old master and make it a candidate master, pool it
  • Cleanup (remove) old backup sources from both DCs

Please read the doc about procedure for more details.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Sep 15 2021, 12:39 PM

Change 721288 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Migrate s8 backups db2100 -> db2098; reimage dbprov2001

https://gerrit.wikimedia.org/r/721288

gerritbot added a project: Patch-For-Review.Sep 15 2021, 12:39 PM
Stashbot added a comment.Sep 23 2021, 11:19 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-23T11:19:53Z] <marostegui> Upgrade db2081 db2082 db2083 db2084 db2091 db2152 T290868

Marostegui added a comment.Sep 23 2021, 11:25 AM

All s8 core codfw replica hosts upgraded to the minor 10.4.21

Stashbot added a comment.Sep 24 2021, 8:22 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-24T08:22:19Z] <jynus> upgrade and restart db2098 T290868

Marostegui moved this task from Ready to In progress on the DBA board.Sep 24 2021, 9:42 AM
gerritbot added a comment.Sep 28 2021, 5:05 AM

Change 724237 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] install_server: Reimage db2080

https://gerrit.wikimedia.org/r/724237

gerritbot added a comment.Sep 28 2021, 5:06 AM

Change 724237 merged by Marostegui:

[operations/puppet@production] install_server: Reimage db2080

https://gerrit.wikimedia.org/r/724237

gerritbot added a comment.Sep 28 2021, 6:19 AM

Change 724321 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2080: Disable notifications

https://gerrit.wikimedia.org/r/724321

gerritbot added a comment.Sep 28 2021, 6:22 AM

Change 724321 merged by Marostegui:

[operations/puppet@production] db2080: Disable notifications

https://gerrit.wikimedia.org/r/724321

ops-monitoring-bot added a comment.Sep 28 2021, 8:30 AM

Cookbook cookbooks.sre.experimental.reimage was started by marostegui@cumin1001 for host db2080.codfw.wmnet

ops-monitoring-bot added a comment.Sep 28 2021, 9:00 AM

Cookbook cookbooks.sre.experimental.reimage completed:

  • db2080 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB
    • Removed from Debmonitor
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/wmf-auto-reimage/202109280830_marostegui_16212_db2080.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal
Marostegui added a comment.Sep 28 2021, 9:02 AM

db2080 (candidate master) reimaged - now checking its tables.

Stashbot added a comment.Sep 28 2021, 1:40 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-28T13:40:31Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2080 T290868', diff saved to https://phabricator.wikimedia.org/P17339 and previous config saved to /var/cache/conftool/dbconfig/20210928-134030-marostegui.json

Marostegui added a comment.Sep 29 2021, 4:49 AM

db2080 crashed while checking its tables - so it needs to be recloned.

Stashbot added a comment.Sep 29 2021, 4:50 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-29T04:50:34Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2081 T290868', diff saved to https://phabricator.wikimedia.org/P17342 and previous config saved to /var/cache/conftool/dbconfig/20210929-045033-marostegui.json

gerritbot added a comment.Sep 29 2021, 4:51 AM

Change 724584 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2081: Disable notifications

https://gerrit.wikimedia.org/r/724584

gerritbot added a comment.Sep 29 2021, 4:53 AM

Change 724584 merged by Marostegui:

[operations/puppet@production] db2081: Disable notifications

https://gerrit.wikimedia.org/r/724584

Marostegui added a comment.Sep 29 2021, 6:39 AM

db2080 has been recloned - checking its tables again.
Replication has been configured but not started.

Stashbot added a comment.Sep 29 2021, 7:25 AM

Mentioned in SAL (#wikimedia-operations) [2021-09-29T07:25:21Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2081 T290868', diff saved to https://phabricator.wikimedia.org/P17345 and previous config saved to /var/cache/conftool/dbconfig/20210929-072520-marostegui.json

Marostegui added a comment.Sep 30 2021, 5:24 AM

db2080 has crashed again, I am going to restore this from a logical backup. Given that it is the candidate master, I don't want to take any risks.

Marostegui added a comment.Sep 30 2021, 5:29 AM

Actually, I am going to try to restore from snapshots and see if that works (and we can double check if the backup source host is ok too)

Marostegui added a comment.EditedSep 30 2021, 5:34 AM

Using: dbprov2001.codfw.wmnet:/srv/backups/snapshots/latest/snapshot.s8.2021-09-29--19-00-03.tar.gz (This is a 10.1 snapshot)

Marostegui added a comment.Sep 30 2021, 6:57 AM

db2080 has been recloned from the backup. I have configured replication but NOT started it.
Now I am checking its tables again to see if something breaks.

Marostegui mentioned this in T254688: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing.Sep 30 2021, 7:07 AM
Marostegui added a comment.Sep 30 2021, 7:26 AM

So far both crashes are related to wbt_item_terms, which is almost 200GB. I am checking it right now to see if it makes it crash again.

Marostegui added a comment.Sep 30 2021, 11:08 AM

It worked fine:

mysql:root@localhost [wikidatawiki]> check table wbt_item_terms;



+-----------------------------+-------+----------+----------+
| Table                       | Op    | Msg_type | Msg_text |
+-----------------------------+-------+----------+----------+
| wikidatawiki.wbt_item_terms | check | status   | OK       |
+-----------------------------+-------+----------+----------+
1 row in set (3 hours 39 min 14.537 sec)

mysql:root@localhost [wikidatawiki]>
mysql:root@localhost [wikidatawiki]>
mysql:root@localhost [wikidatawiki]>
mysql:root@localhost [wikidatawiki]>

I am letting the automatic check to finish though (it will do another check to that table)

Marostegui added a comment.Oct 1 2021, 4:52 AM

db2080's check finally came back clean. I have started replication and so far so good.

Marostegui updated the task description. (Show Details)Oct 1 2021, 5:19 AM
Stashbot added a comment.Oct 1 2021, 8:43 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-01T08:43:46Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2080 T290868', diff saved to https://phabricator.wikimedia.org/P17390 and previous config saved to /var/cache/conftool/dbconfig/20211001-084345-marostegui.json

gerritbot added a comment.Oct 18 2021, 11:46 AM

Change 731401 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2079: Disable notifications

https://gerrit.wikimedia.org/r/731401

gerritbot added a comment.Oct 18 2021, 11:47 AM

Change 731401 merged by Marostegui:

[operations/puppet@production] db2079: Disable notifications

https://gerrit.wikimedia.org/r/731401

Stashbot added a comment.Oct 18 2021, 11:49 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-18T11:49:11Z] <marostegui> Reimage db2079 (codfw s8 master) T290868

ops-monitoring-bot added a comment.Oct 18 2021, 11:51 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db2079.codfw.wmnet with OS buster

gerritbot added a comment.Oct 18 2021, 11:55 AM

Change 731404 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbbackups: Migrate s8 backups db2100 -> db2098

https://gerrit.wikimedia.org/r/731404

gerritbot added a comment.Oct 18 2021, 11:55 AM

Change 721288 abandoned by Marostegui:

[operations/puppet@production] dbbackups: Migrate s8 backups db2100 -> db2098; reimage dbprov2001

Reason:

Using this https://gerrit.wikimedia.org/r/c/operations/puppet/+/731404/ instead

https://gerrit.wikimedia.org/r/721288

Marostegui updated the task description. (Show Details)Oct 18 2021, 12:13 PM
ops-monitoring-bot added a comment.Oct 18 2021, 12:22 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db2079.codfw.wmnet with OS buster completed:

  • db2079 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110181151_marostegui_20209_db2079.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui added a comment.Oct 18 2021, 12:24 PM

db2079 (s8 codfw master) has been reimaged - checking its tables now.

gerritbot added a comment.Oct 18 2021, 12:39 PM

Change 731404 merged by Marostegui:

[operations/puppet@production] dbbackups: Migrate s8 backups db2100 -> db2098

https://gerrit.wikimedia.org/r/731404

Marostegui updated the task description. (Show Details)Oct 18 2021, 12:39 PM
Marostegui added a comment.Oct 19 2021, 5:43 AM

db2079's check was ok

Marostegui updated the task description. (Show Details)Oct 19 2021, 5:43 AM
Marostegui added a comment.Oct 19 2021, 11:48 AM

Upgraded all s8 eqiad replicas (apart from backup and candidate master) to 10.4.21.
Pending two clouddb* hosts which I will do today and tomorrow (to avoid having two of them fully cold the same day).

gerritbot added a comment.Oct 20 2021, 6:11 AM

Change 732115 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1126: Disable notifications

https://gerrit.wikimedia.org/r/732115

gerritbot added a comment.Oct 20 2021, 6:12 AM

Change 732115 merged by Marostegui:

[operations/puppet@production] db1126: Disable notifications

https://gerrit.wikimedia.org/r/732115

ops-monitoring-bot added a comment.Oct 20 2021, 6:14 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1126.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 20 2021, 6:41 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1126.eqiad.wmnet with OS buster completed:

  • db1126 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110200614_marostegui_11291_db1126.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui added a comment.Oct 20 2021, 6:43 AM

Checking db1126's tables.

Marostegui added a comment.Oct 21 2021, 4:28 AM

db1126's table showed a corruption on wbt_item_terms's index, so I am fixing that one.

gerritbot added a comment.Oct 21 2021, 5:27 AM

Change 732472 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1018: Depool clouddb1020

https://gerrit.wikimedia.org/r/732472

gerritbot added a comment.Oct 21 2021, 5:28 AM

Change 732472 merged by Marostegui:

[operations/puppet@production] dbproxy1018: Depool clouddb1020

https://gerrit.wikimedia.org/r/732472

Marostegui added a comment.Oct 21 2021, 11:39 AM

Table rebuilt, re-checking it again.

Marostegui added a comment.Oct 21 2021, 1:48 PM
mysql:root@localhost [wikidatawiki]> check table wbt_item_terms;
+-----------------------------+-------+----------+----------+
| Table                       | Op    | Msg_type | Msg_text |
+-----------------------------+-------+----------+----------+
| wikidatawiki.wbt_item_terms | check | status   | OK       |
+-----------------------------+-------+----------+----------+
1 row in set (2 hours 6 min 2.990 sec)

So going to start replication again.

gerritbot added a comment.Oct 21 2021, 3:00 PM

Change 732717 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] backups: Send db2098:3318 to dbprov2002

https://gerrit.wikimedia.org/r/732717

gerritbot added a comment.Oct 21 2021, 3:13 PM

Change 732717 merged by Marostegui:

[operations/puppet@production] backups: Send db2098:3318 to dbprov2002

https://gerrit.wikimedia.org/r/732717

Marostegui added a comment.Oct 22 2021, 4:38 AM

Repooling db1126

Marostegui added a subtask: T254688: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing.Oct 22 2021, 6:33 AM
gerritbot added a comment.Oct 25 2021, 4:30 AM

Change 734060 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1109: Disable notifications

https://gerrit.wikimedia.org/r/734060

Stashbot added a comment.Oct 25 2021, 4:30 AM

Mentioned in SAL (#wikimedia-operations) [2021-10-25T04:30:29Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1109 (s8) for reimage T290868', diff saved to https://phabricator.wikimedia.org/P17590 and previous config saved to /var/cache/conftool/dbconfig/20211025-043028-marostegui.json

gerritbot added a comment.Oct 25 2021, 4:31 AM

Change 734060 merged by Marostegui:

[operations/puppet@production] db1109: Disable notifications

https://gerrit.wikimedia.org/r/734060

ops-monitoring-bot added a comment.Oct 25 2021, 5:01 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1109.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 25 2021, 5:26 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1109.eqiad.wmnet with OS buster completed:

  • db1109 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110250501_marostegui_307_db1109.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui added a comment.Oct 25 2021, 5:26 AM

db1109 (candidate master) has been reimaged - now getting its tables checked.

Marostegui closed subtask T254688: SpecialWhatLinksHere::showIndirectLinks on wikidatawiki needs optimizing as Resolved.Oct 25 2021, 5:28 AM
Marostegui added a comment.Oct 26 2021, 4:18 AM

db1109's check was fine. Restarted replication

Marostegui updated the task description. (Show Details)Oct 26 2021, 4:19 AM
Marostegui added a comment.Oct 26 2021, 5:22 AM

db1109 is being repooled

Marostegui updated the task description. (Show Details)Oct 26 2021, 5:22 AM
gerritbot added a comment.Nov 5 2021, 5:24 AM

Change 736945 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbproxy1019: Depool clouddb1016

https://gerrit.wikimedia.org/r/736945

gerritbot added a comment.Nov 5 2021, 5:29 AM

Change 736945 merged by Marostegui:

[operations/puppet@production] dbproxy1019: Depool clouddb1016

https://gerrit.wikimedia.org/r/736945

Marostegui added a comment.Nov 5 2021, 5:37 AM

Upgraded clouddb1016 - the last host pending on the section.

gerritbot added a comment.Nov 5 2021, 6:12 AM

Change 736946 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] dbbackups: Switch s8 backup generation from db1116 to db1171

https://gerrit.wikimedia.org/r/736946

Marostegui added a comment.Nov 5 2021, 6:14 AM

@jcrespo I have put this patch up to switch s8 backups from 10.1 to 10.4, please review it and amend if it needed: https://gerrit.wikimedia.org/r/c/operations/puppet/+/736946/

Marostegui updated the task description. (Show Details)Nov 5 2021, 6:14 AM
gerritbot added a comment.Nov 11 2021, 5:46 AM

Change 738057 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1104: Disable notifications

https://gerrit.wikimedia.org/r/738057

gerritbot added a comment.Nov 11 2021, 6:03 AM

Change 738057 merged by Marostegui:

[operations/puppet@production] db1104: Disable notifications

https://gerrit.wikimedia.org/r/738057

Marostegui closed subtask T294321: Switchover s8 from db1104 to db1109 as Resolved.Nov 11 2021, 6:06 AM
ops-monitoring-bot added a comment.Nov 11 2021, 6:10 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1104.eqiad.wmnet with OS buster

Marostegui updated the task description. (Show Details)Nov 11 2021, 6:20 AM
ops-monitoring-bot added a comment.Nov 11 2021, 6:37 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1104.eqiad.wmnet with OS buster completed:

  • db1104 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111110610_marostegui_10265_db1104.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Marostegui added a comment.Nov 11 2021, 6:51 AM

old master (db1104) reimaged to Buster - checking its tables now.

gerritbot added a comment.Nov 11 2021, 8:59 AM

Change 736946 merged by Jcrespo:

[operations/puppet@production] dbbackups: Switch s8 backup generation from db1116 to db1171

https://gerrit.wikimedia.org/r/736946

jcrespo updated the task description. (Show Details)Nov 11 2021, 9:00 AM
gerritbot added a comment.Nov 11 2021, 11:13 AM

Change 738195 had a related patch set uploaded (by Jcrespo; author: Jcrespo):

[operations/puppet@production] dbbackups: Reimage db1116, db1139, db2097, db2100 to buster

https://gerrit.wikimedia.org/r/738195

gerritbot added a comment.Nov 11 2021, 11:39 AM

Change 738195 merged by Jcrespo:

[operations/puppet@production] dbbackups: Reimage db1116, db1139, db2097, db2100 to buster

https://gerrit.wikimedia.org/r/738195

Marostegui added a comment.Nov 12 2021, 5:59 AM

db1104's check is fine. I am going to start repooling.

Marostegui updated the task description. (Show Details)Nov 12 2021, 7:03 AM

@jcrespo you can proceed with the backup sources clean up

Marostegui moved this task from In progress to Done on the DBA board.Nov 12 2021, 7:03 AM
jcrespo updated the task description. (Show Details)Nov 12 2021, 11:24 AM

@Marostegui backup upgrade/cleanup is done, I will later reorganize backup distribution (but that is out of scope).

Marostegui closed this task as Resolved.Nov 12 2021, 11:26 AM

Nice! Closing this then!
Thank you

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits