Page MenuHomePhabricator
Create Task
Maniphest T259589

Upgrade m3 to Buster and MariaDB 10.4
Closed, ResolvedPublic
Actions

Description

m3 needs to be upgraded to Buster and MariaDB 10.4

These are the hosts:

  • db2134
  • db2078
  • db1128 (to be replaced by db1132)
  • db1132
  • db1117

Details

SubjectRepoBranchLines +/-
site.pp: Specify the correct role for db1132operations/puppetproduction+3 -2
dbproxy1016,dbproxy1020: Change m3 masteroperations/puppetproduction+6 -7
dbproxy1016,20: Temporary test db1132operations/puppetproduction+4 -4
db1132: Enable notificationsoperations/puppetproduction+0 -1
db1132: Set binlog format to ROWoperations/puppetproduction+1 -0
mariadb: Reimage db1132 to Busteroperations/puppetproduction+8 -12
db2134: Enable notificationsoperations/puppetproduction+0 -1
mariadb: Reimage db2134 to Busteroperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.Aug 4 2020, 8:35 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2020, 8:35 AM
• Marostegui triaged this task as Medium priority.Aug 4 2020, 8:35 AM
• Marostegui moved this task from Triage to In progress on the DBA board.
gerritbot added a comment.Aug 4 2020, 8:42 AM

Change 618238 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Reimage db2134 to Buster

https://gerrit.wikimedia.org/r/618238

gerritbot added a project: Patch-For-Review.Aug 4 2020, 8:42 AM

Change 618238 merged by Marostegui:
[operations/puppet@production] mariadb: Reimage db2134 to Buster

https://gerrit.wikimedia.org/r/618238

ops-monitoring-bot added a comment.Aug 4 2020, 8:48 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2134.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202008040848_marostegui_23648.log.

Maintenance_bot removed a project: Patch-For-Review.Aug 4 2020, 9:10 AM
ops-monitoring-bot added a comment.Aug 4 2020, 9:27 AM

Completed auto-reimage of hosts:

['db2134.codfw.wmnet']

and were ALL successful.

• Marostegui updated the task description. (Show Details)Aug 4 2020, 9:30 AM
gerritbot added a comment.Aug 4 2020, 9:42 AM

Change 618249 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2134: Enable notifications

https://gerrit.wikimedia.org/r/618249

gerritbot added a project: Patch-For-Review.Aug 4 2020, 9:42 AM

Change 618249 merged by Marostegui:
[operations/puppet@production] db2134: Enable notifications

https://gerrit.wikimedia.org/r/618249

Maintenance_bot removed a project: Patch-For-Review.Aug 4 2020, 10:10 AM
• Marostegui added a project: Phabricator.Aug 4 2020, 10:55 AM
• Marostegui added a subscriber: • mmodell.

@mmodell would you be available on Thursday 13th at 05:00 AM UTC for failing over phabricator master? (I can do it earlier if it is easier for you)
Ideally, we should set phabricator into read only for it.
m1 and m2 failover took just a few seconds.

• mmodell added a comment.Aug 4 2020, 10:58 AM

@Marostegui sure I can do that.

• Marostegui added a comment.Aug 4 2020, 10:58 AM

Thank you - I will block that maintenance window on the deployments page and send you a google calendar invite.

RhinosF1 subscribed.Aug 4 2020, 10:58 AM
• mmodell added a comment.Aug 4 2020, 10:59 AM

@Marostegui: in case you ever need to do it without me, it's documented here:

https://wikitech.wikimedia.org/wiki/Phabricator#read-only_mode_/_restarting_mariadb

• Marostegui added a comment.Aug 4 2020, 11:00 AM

Oh, great. Thank you. I can do it myself for learning purposes, but if you are around to support it, just in case, it would be appreciated!

gerritbot added a comment.Aug 5 2020, 4:46 AM

Change 618427 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Reimage db1132 to Buster

https://gerrit.wikimedia.org/r/618427

gerritbot added a project: Patch-For-Review.Aug 5 2020, 4:46 AM
gerritbot added a comment.Aug 5 2020, 7:12 AM

Change 618427 merged by Marostegui:
[operations/puppet@production] mariadb: Reimage db1132 to Buster

https://gerrit.wikimedia.org/r/618427

ops-monitoring-bot added a comment.Aug 5 2020, 7:15 AM

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1132.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202008050715_marostegui_6589.log.

Stashbot added a comment.Aug 5 2020, 7:49 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-05T07:49:15Z] <marostegui> Stop mysql on db1117:3323 (this will generate haproxy irc alerts) T259589

ops-monitoring-bot added a comment.Aug 5 2020, 7:50 AM

Completed auto-reimage of hosts:

['db1132.eqiad.wmnet']

and were ALL successful.

Maintenance_bot removed a project: Patch-For-Review.Aug 5 2020, 8:10 AM
• Marostegui updated the task description. (Show Details)Aug 5 2020, 8:32 AM
gerritbot added a comment.Aug 5 2020, 8:34 AM

Change 618494 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1132: Set binlog format to ROW

https://gerrit.wikimedia.org/r/618494

gerritbot added a project: Patch-For-Review.Aug 5 2020, 8:34 AM

Change 618494 merged by Marostegui:
[operations/puppet@production] db1132: Set binlog format to ROW

https://gerrit.wikimedia.org/r/618494

Maintenance_bot removed a project: Patch-For-Review.Aug 5 2020, 9:10 AM
gerritbot added a comment.Aug 6 2020, 5:10 AM

Change 618655 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1132: Enable notifications

https://gerrit.wikimedia.org/r/618655

gerritbot added a project: Patch-For-Review.Aug 6 2020, 5:10 AM

Change 618655 merged by Marostegui:
[operations/puppet@production] db1132: Enable notifications

https://gerrit.wikimedia.org/r/618655

Maintenance_bot removed a project: Patch-For-Review.Aug 6 2020, 6:10 AM
• Marostegui added a comment.Aug 10 2020, 4:55 AM

@mmodell Thursday is a non deployment day, so let's move this to Tuesday 18th if that's ok?
So that would be Tuesday 18th at 05:00 AM UTC?

• mmodell added a comment.Aug 10 2020, 9:17 PM

@Marostegui: That works for me.

• Marostegui added a comment.Aug 11 2020, 5:11 AM
In T259589#6374543, @mmodell wrote:

@Marostegui: That works for me.

Thanks - I have updated the calendar event.

• mmodell moved this task from To Triage to Infrastructure on the Phabricator board.Aug 11 2020, 8:24 PM
jcrespo mentioned this in T250361: replace phabricator db passwords with longer passwords.Aug 12 2020, 6:31 AM
• Marostegui mentioned this in T260324: Upgrade m5 to Buster and MariaDB 10.4.Aug 13 2020, 5:35 AM
gerritbot added a comment.Aug 17 2020, 7:39 AM

Change 620648 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1016,20: Temporary test db1132

https://gerrit.wikimedia.org/r/620648

gerritbot added a project: Patch-For-Review.Aug 17 2020, 7:39 AM
gerritbot added a comment.Aug 17 2020, 7:54 AM

Change 620648 merged by Marostegui:
[operations/puppet@production] dbproxy1016,20: Temporary test db1132

https://gerrit.wikimedia.org/r/620648

gerritbot added a comment.Aug 17 2020, 9:38 AM

Change 620664 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1016,dbproxy1020: Change m3 master

https://gerrit.wikimedia.org/r/620664

gerritbot added a comment.Aug 18 2020, 4:49 AM

Change 620664 merged by Marostegui:
[operations/puppet@production] dbproxy1016,dbproxy1020: Change m3 master

https://gerrit.wikimedia.org/r/620664

Stashbot added a comment.Aug 18 2020, 5:00 AM

Mentioned in SAL (#wikimedia-operations) [2020-08-18T05:00:11Z] <marostegui> Failover m3 (phabricator) database master from db1128 to db1132 - T259589

• Marostegui added a comment.Aug 18 2020, 5:02 AM

Test read-only off

• mmodell awarded a token.Aug 18 2020, 5:02 AM
• Marostegui removed a project: Patch-For-Review.Aug 18 2020, 5:02 AM
• Marostegui added a project: SRE.
• Marostegui closed this task as Resolved.Aug 18 2020, 5:09 AM

This was done successfully. m3 fully runs Buster and MariaDB 10.4
db1128 will be moved to m5, and that will be tracked at T260324

Thanks @mmodell for helping out during the switchover!

gerritbot added a comment.Aug 18 2020, 5:29 AM

Change 620821 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] site.pp: Specify the correct role for db1132

https://gerrit.wikimedia.org/r/620821

gerritbot added a project: Patch-For-Review.Aug 18 2020, 5:29 AM
gerritbot added a comment.Aug 18 2020, 5:35 AM

Change 620821 merged by Marostegui:
[operations/puppet@production] site.pp: Specify the correct role for db1132

https://gerrit.wikimedia.org/r/620821

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL