Page MenuHomePhabricator

Failover s8 (wikidatawiki) db primary master db1071 to db1104 (read-only required)
Open, NormalPublic

Description

db1071 is an old host which is out of warranty and needs to be decommissioned.
Also, the normalization of wikidatawiki.wb_terms will require a master with more throughput and more disk space for the next wb_term table redesign phases (T221764)

We want to failover db1071 to db1104.

When: Tuesday 30th July, 05:00AM-05:30 AM UTC
Impact: Writes to wikidatawiki will be blocked. Reads will remain unaffected.

We do not expect to use the full 30 minutes window, and rather use around 3-4 minutes if everything goes as expected

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptTue, Jul 2, 8:14 AM
Marostegui triaged this task as Normal priority.Tue, Jul 2, 8:14 AM
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui updated the task description. (Show Details)Tue, Jul 2, 8:24 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-02T09:34:30Z] <marostegui> Upgrade mysql on 2080 db2081 db2083 - T227062

Mentioned in SAL (#wikimedia-operations) [2019-07-02T09:39:11Z] <marostegui> Upgrade db2094 (codfw sanitarium) T227062

Mentioned in SAL (#wikimedia-operations) [2019-07-02T12:47:28Z] <marostegui> Upgrade db2082 - T227062

All codfw is now running 10.1.39 (which is the version the new master will run) - will keep upgrading eqiad now.

Change 520673 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101

https://gerrit.wikimedia.org/r/520673

Change 520673 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101

https://gerrit.wikimedia.org/r/520673

Mentioned in SAL (#wikimedia-operations) [2019-07-04T05:16:29Z] <marostegui> Upgrade db1101 - T227062

Change 520732 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db1109 for upgrade

https://gerrit.wikimedia.org/r/520732

Change 520732 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db1109 for upgrade

https://gerrit.wikimedia.org/r/520732

Mentioned in SAL (#wikimedia-operations) [2019-07-04T11:59:20Z] <jynus> stop and upgrade db1109 T227062

Change 520739 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool db1109 after maintenance with low weight

https://gerrit.wikimedia.org/r/520739

Change 520739 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool db1109 after maintenance with low weight

https://gerrit.wikimedia.org/r/520739

Change 520830 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/520830

Change 520830 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104

https://gerrit.wikimedia.org/r/520830

Mentioned in SAL (#wikimedia-operations) [2019-07-05T05:23:23Z] <marostegui> Upgrade db1104 T227062

Change 520842 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1109: Convert it to candidate master

https://gerrit.wikimedia.org/r/520842

Change 520842 merged by Marostegui:
[operations/puppet@production] db1109: Convert it to candidate master

https://gerrit.wikimedia.org/r/520842

Change 521203 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/521203

Change 521203 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109

https://gerrit.wikimedia.org/r/521203

Mentioned in SAL (#wikimedia-operations) [2019-07-08T05:45:06Z] <marostegui> Restart MySQL on db1109 to pick up STATEMENT as binlog format - T227062

I have restarted db1109 to pickup STATEMENT as a binlog format. db1109 will be the candidate master once db1104 (current candidate master) gets promoted to master.

Change 524411 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1104 to s8 master

https://gerrit.wikimedia.org/r/524411

Change 524412 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Pool db1109 into API

https://gerrit.wikimedia.org/r/524412

Change 524412 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Pool db1109 into API

https://gerrit.wikimedia.org/r/524412