Page MenuHomePhabricator

Upgrade m3 to Buster and MariaDB 10.4
Closed, ResolvedPublic

Description

m3 needs to be upgraded to Buster and MariaDB 10.4

These are the hosts:

  • db2134
  • db2078
  • db1128 (to be replaced by db1132)
  • db1132
  • db1117

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 4 2020, 8:35 AM
Marostegui triaged this task as Medium priority.Aug 4 2020, 8:35 AM
Marostegui moved this task from Triage to In progress on the DBA board.

Change 618238 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Reimage db2134 to Buster

https://gerrit.wikimedia.org/r/618238

Change 618238 merged by Marostegui:
[operations/puppet@production] mariadb: Reimage db2134 to Buster

https://gerrit.wikimedia.org/r/618238

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db2134.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202008040848_marostegui_23648.log.

Completed auto-reimage of hosts:

['db2134.codfw.wmnet']

and were ALL successful.

Marostegui updated the task description. (Show Details)Aug 4 2020, 9:30 AM

Change 618249 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2134: Enable notifications

https://gerrit.wikimedia.org/r/618249

Change 618249 merged by Marostegui:
[operations/puppet@production] db2134: Enable notifications

https://gerrit.wikimedia.org/r/618249

Marostegui added a subscriber: mmodell.

@mmodell would you be available on Thursday 13th at 05:00 AM UTC for failing over phabricator master? (I can do it earlier if it is easier for you)
Ideally, we should set phabricator into read only for it.
m1 and m2 failover took just a few seconds.

@Marostegui sure I can do that.

Thank you - I will block that maintenance window on the deployments page and send you a google calendar invite.

Oh, great. Thank you. I can do it myself for learning purposes, but if you are around to support it, just in case, it would be appreciated!

Change 618427 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Reimage db1132 to Buster

https://gerrit.wikimedia.org/r/618427

Change 618427 merged by Marostegui:
[operations/puppet@production] mariadb: Reimage db1132 to Buster

https://gerrit.wikimedia.org/r/618427

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['db1132.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202008050715_marostegui_6589.log.

Mentioned in SAL (#wikimedia-operations) [2020-08-05T07:49:15Z] <marostegui> Stop mysql on db1117:3323 (this will generate haproxy irc alerts) T259589

Completed auto-reimage of hosts:

['db1132.eqiad.wmnet']

and were ALL successful.

Marostegui updated the task description. (Show Details)Aug 5 2020, 8:32 AM

Change 618494 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1132: Set binlog format to ROW

https://gerrit.wikimedia.org/r/618494

Change 618494 merged by Marostegui:
[operations/puppet@production] db1132: Set binlog format to ROW

https://gerrit.wikimedia.org/r/618494

Change 618655 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1132: Enable notifications

https://gerrit.wikimedia.org/r/618655

Change 618655 merged by Marostegui:
[operations/puppet@production] db1132: Enable notifications

https://gerrit.wikimedia.org/r/618655

@mmodell Thursday is a non deployment day, so let's move this to Tuesday 18th if that's ok?
So that would be Tuesday 18th at 05:00 AM UTC?

@Marostegui: That works for me.

@Marostegui: That works for me.

Thanks - I have updated the calendar event.

mmodell moved this task from To Triage to Infrastructure on the Phabricator board.Aug 11 2020, 8:24 PM

Change 620648 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1016,20: Temporary test db1132

https://gerrit.wikimedia.org/r/620648

Change 620648 merged by Marostegui:
[operations/puppet@production] dbproxy1016,20: Temporary test db1132

https://gerrit.wikimedia.org/r/620648

Change 620664 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1016,dbproxy1020: Change m3 master

https://gerrit.wikimedia.org/r/620664

Change 620664 merged by Marostegui:
[operations/puppet@production] dbproxy1016,dbproxy1020: Change m3 master

https://gerrit.wikimedia.org/r/620664

Mentioned in SAL (#wikimedia-operations) [2020-08-18T05:00:11Z] <marostegui> Failover m3 (phabricator) database master from db1128 to db1132 - T259589

Test read-only off

Marostegui closed this task as Resolved.Aug 18 2020, 5:09 AM

This was done successfully. m3 fully runs Buster and MariaDB 10.4
db1128 will be moved to m5, and that will be tracked at T260324

Thanks @mmodell for helping out during the switchover!

Change 620821 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] site.pp: Specify the correct role for db1132

https://gerrit.wikimedia.org/r/620821

Change 620821 merged by Marostegui:
[operations/puppet@production] site.pp: Specify the correct role for db1132

https://gerrit.wikimedia.org/r/620821