Page MenuHomePhabricator
Create Task
Maniphest T189107

DB meta task for next DC failover issues
Closed, ResolvedPublic
Actions

Description

This is just a meta task to group and track all the DB stuff that might need to happen during the next DC failover.
Not sorted in any way, just grouping them here for now.

Switchover dates:

Mediawiki: Wednesday, September 12th 2018: 14:00 UTC

Switchback:

Mediawiki: Wednesday, October 10th 2018: 14:00 UTC

DB planning: https://wikitech.wikimedia.org/wiki/Switch_Datacenter/planned_db_maintenance#2018_Switch_Datacenter

Details

SubjectRepoBranchLines +/-
mariadb: Fix DB configuration in preparation for dc switchoveroperations/mediawiki-configmaster+37 -22
mariadb: Set all codfw sections as read-only, codfw is still in rooperations/mediawiki-configmaster+7 -7
mariadb: Set s5 section in read-write, codfw should be still in rooperations/mediawiki-configmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 7 2018, 11:15 AM
Marostegui triaged this task as Medium priority.Mar 7 2018, 11:16 AM
Marostegui added a project: Epic.
Marostegui moved this task from Triage to Meta/Epic on the DBA board.
Marostegui added subtasks: T186503: Rebuild user_newtalk on db1052, T186188: Failover DB masters in row D, T189101: Deploy schema change for adding numeric primary key to wbqc_constraints table.
Marostegui added a subtask: T187089: Fix WMF schemas to not break when comment store goes WRITE_NEW.
Marostegui renamed this task from Meta task for next DC failover issues to DB meta task for next DC failover issues.Mar 16 2018, 7:56 AM
Marostegui added a subtask: Restricted Task.
Marostegui added a subtask: Restricted Task.
MoritzMuehlenhoff subscribed.Mar 16 2018, 8:18 AM
Marostegui added a subtask: Restricted Task.Mar 20 2018, 9:06 AM
Marostegui changed the status of subtask T187089: Fix WMF schemas to not break when comment store goes WRITE_NEW from Open to Stalled.Apr 10 2018, 5:19 AM
Marostegui closed subtask T186503: Rebuild user_newtalk on db1052 as Declined.Apr 10 2018, 5:54 AM
Marostegui changed the status of subtask T187089: Fix WMF schemas to not break when comment store goes WRITE_NEW from Stalled to Open.Apr 10 2018, 6:30 AM
Marostegui changed the status of subtask T187089: Fix WMF schemas to not break when comment store goes WRITE_NEW from Open to Stalled.Apr 17 2018, 3:47 PM
Marostegui added a subtask: T191316: Schema change to make archive.ar_rev_id NOT NULL.May 30 2018, 9:58 AM
Marostegui added a subtask: T192926: Schema change to drop archive.ar_text and archive.ar_flags.
Marostegui added a subtask: T89737: Make several mediawiki table fields unsigned ints on wmf databases.
Marostegui removed a subtask: T192926: Schema change to drop archive.ar_text and archive.ar_flags.Jun 6 2018, 6:04 AM
Marostegui removed a subtask: T191316: Schema change to make archive.ar_rev_id NOT NULL.
Malyacko subscribed.Jun 29 2018, 10:41 AM
Marostegui mentioned this in T89737: Make several mediawiki table fields unsigned ints on wmf databases.Jul 4 2018, 12:05 PM
Marostegui changed the status of subtask T89737: Make several mediawiki table fields unsigned ints on wmf databases from Open to Stalled.
Marostegui added a parent task: T199073: Perform a datacenter switchover (2018-19 Q1).Jul 9 2018, 8:34 AM
Marostegui added a subtask: T144010: Drop eu_touched in production.Jul 25 2018, 10:02 AM
Marostegui added a subtask: T100501: mysql user and group should be a system user/group.Jul 30 2018, 5:34 AM
Marostegui created subtask T201133: db1069 (x1 master) memory errors.Aug 3 2018, 5:22 AM
Marostegui changed the status of subtask T144010: Drop eu_touched in production from Open to Stalled.Aug 7 2018, 7:37 AM
Marostegui closed subtask T201133: db1069 (x1 master) memory errors as Resolved.Aug 7 2018, 1:06 PM
Marostegui changed the status of subtask T186188: Failover DB masters in row D from Open to Stalled.Aug 20 2018, 9:29 AM
Marostegui updated the task description. (Show Details)Aug 23 2018, 11:36 AM
gerritbot subscribed.Sep 4 2018, 8:57 AM

Change 457847 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Fix DB configuration in preparation for dc switchover

https://gerrit.wikimedia.org/r/457847

gerritbot added a project: Patch-For-Review.Sep 4 2018, 8:57 AM
jcrespo subscribed.Sep 4 2018, 10:47 AM

Missing partitions on codfw:

db2085:3311:enwiki:logging
db2088:3311:enwiki:logging

db2088:3312:bgwiktionary:revision
db2088:3312:bgwiktionary:logging
db2088:3312:eowiki:revision
db2088:3312:eowiki:logging
db2088:3312:idwiki:revision
db2088:3312:idwiki:logging

db2091:3312:bgwiktionary:revision
db2091:3312:bgwiktionary:logging
db2091:3312:eowiki:revision
db2091:3312:eowiki:logging
db2091:3312:idwiki:revision
db2091:3312:idwiki:logging

db2086:3317:frwiktionary:revision
db2086:3317:frwiktionary:logging
db2087:3317:frwiktionary:revision
db2087:3317:frwiktionary:logging

The only worrying are enwiki, the other may not be in eqiad either (I haven't checked).

Paladox subscribed.Sep 4 2018, 10:52 AM
Marostegui added a comment.Sep 4 2018, 11:10 AM

I have checked bgwitionary, eowiki, idwiki and frwiktionary and they do not exist on eqiad either.

Stashbot subscribed.Sep 4 2018, 11:11 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-04T11:11:50Z] <jynus> stopping replication and running partitioning on logging on db1085:3311 T189107

Marostegui added a comment.Sep 4 2018, 12:29 PM

I have checked that no codfw hosts have notifications disabled on puppet or on icinga itself.

Marostegui added a subscriber: • Banyek.Sep 4 2018, 1:46 PM
Stashbot added a comment.Sep 4 2018, 2:01 PM

Mentioned in SAL (#wikimedia-operations) [2018-09-04T14:01:59Z] <jynus> stopping replication and running partitioning on logging on db1088:3311 T189107

gerritbot added a comment.Sep 5 2018, 8:35 AM

Change 458125 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Set s5 section in read-write, codfw should be still in ro

https://gerrit.wikimedia.org/r/458125

gerritbot added a comment.Sep 5 2018, 8:39 AM

Change 458125 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Set s5 section in read-write, codfw should be still in ro

https://gerrit.wikimedia.org/r/458125

gerritbot added a comment.Sep 5 2018, 8:57 AM

Change 458128 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Set all codfw sections as read-only, codfw is still in ro

https://gerrit.wikimedia.org/r/458128

gerritbot added a comment.Sep 5 2018, 9:10 AM

Change 458128 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Set all codfw sections as read-only, codfw is still in ro

https://gerrit.wikimedia.org/r/458128

gerritbot added a comment.Sep 6 2018, 7:56 AM

Change 457847 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Fix DB configuration in preparation for dc switchover

https://gerrit.wikimedia.org/r/457847

Stashbot added a comment.Sep 6 2018, 8:06 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-06T08:06:16Z] <marostegui> Enable replication codfw -> eqiad on s5,s6,s2 - T189107

Stashbot added a comment.Sep 6 2018, 8:17 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-06T08:17:06Z] <marostegui> Enable replication codfw -> eqiad on s1,s3,s4 - T189107

Stashbot added a comment.Sep 6 2018, 8:25 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-06T08:25:07Z] <marostegui> Enable replication codfw -> eqiad on s7,s8,x1 - T189107

Stashbot added a comment.Sep 6 2018, 8:32 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-06T08:32:05Z] <marostegui> Enable replication codfw -> eqiad on es2,es3 - T189107

Marostegui added a comment.Sep 6 2018, 8:48 AM

Replication codfw -> eqiad has been enabled on s1-s8,x1,es2,es3

Pine updated the task description. (Show Details)Sep 6 2018, 11:24 PM
akosiaris added a parent task: T203776: Successfully switch backend traffic (MediaWiki, Swift, RESTBase, Parsoid and services) to be served from codfw.Sep 7 2018, 10:28 AM
akosiaris removed a parent task: T203776: Successfully switch backend traffic (MediaWiki, Swift, RESTBase, Parsoid and services) to be served from codfw.
akosiaris added a subtask: T203776: Successfully switch backend traffic (MediaWiki, Swift, RESTBase, Parsoid and services) to be served from codfw.
Stashbot added a comment.Sep 11 2018, 10:37 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-11T10:37:41Z] <marostegui> Disable GTID on all codfw masters (sX, x1, esX) (not in db2040 as it is not enabled there) T189107

Marostegui added a comment.Sep 11 2018, 10:43 AM

The following masters had GTID enabled - I have disabled it:

db2048
                   Using_Gtid: Slave_Pos
db2035
                   Using_Gtid: Slave_Pos
db2043
                   Using_Gtid: Slave_Pos
db2051
                   Using_Gtid: Slave_Pos
db2052
                   Using_Gtid: Slave_Pos
db2039
                   Using_Gtid: Slave_Pos
db2040
                   Using_Gtid: No
db2045
                   Using_Gtid: Slave_Pos
db2034
                   Using_Gtid: Slave_Pos
es2016
                   Using_Gtid: Slave_Pos
es2017
                   Using_Gtid: Slave_Pos

So we are now running without it:

db2048
                   Using_Gtid: No
db2035
                   Using_Gtid: No
db2043
                   Using_Gtid: No
db2051
                   Using_Gtid: No
db2052
                   Using_Gtid: No
db2039
                   Using_Gtid: No
db2040
                   Using_Gtid: No
db2045
                   Using_Gtid: No
db2034
                   Using_Gtid: No
es2016
                   Using_Gtid: No
es2017
                   Using_Gtid: No
akosiaris closed subtask T203776: Successfully switch backend traffic (MediaWiki, Swift, RESTBase, Parsoid and services) to be served from codfw as Resolved.Sep 12 2018, 4:49 PM
Stashbot added a comment.Sep 13 2018, 8:01 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-13T08:01:06Z] <marostegui> Disconnect replication eqiad -> codfw on s1-s8, x1, es2, es3 - T189107

Marostegui added a comment.Sep 13 2018, 8:06 AM

Replication has been disconnected from eqiad to codfw:

root@neodymium:/home/marostegui# for i in db2048 db2035 db2043 db2051 db2052 db2039 db2040 db2045 db2034 es2016 es2017; do mysql.py -h$i -e "show slave status\G" ; done
root@neodymium:/home/marostegui#
Marostegui changed the status of subtask T144010: Drop eu_touched in production from Stalled to Open.Sep 13 2018, 8:17 AM
Marostegui changed the status of subtask T89737: Make several mediawiki table fields unsigned ints on wmf databases from Stalled to Open.Sep 13 2018, 8:36 AM
Marostegui changed the status of subtask T187089: Fix WMF schemas to not break when comment store goes WRITE_NEW from Stalled to Open.
Marostegui closed subtask T144010: Drop eu_touched in production as Resolved.Sep 13 2018, 9:43 AM
Stashbot added a comment.Sep 13 2018, 9:44 AM

Mentioned in SAL (#wikimedia-operations) [2018-09-13T09:44:43Z] <marostegui> Enable GTID on eqiad masters - T189107

Marostegui added a comment.Sep 13 2018, 9:51 AM

GTID enabled on all eqiad masters but db1071 (s8) and db1068 (s4) as they are currently running a big alter.

Marostegui added a comment.EditedSep 13 2018, 9:55 AM

db1068 GTID enabled

Marostegui added a comment.Sep 13 2018, 12:49 PM

db1071 GTID enabled

Marostegui closed subtask T189101: Deploy schema change for adding numeric primary key to wbqc_constraints table as Resolved.Sep 14 2018, 5:27 AM
Marostegui closed subtask Restricted Task as Resolved.Sep 14 2018, 8:10 AM
Marostegui closed subtask T89737: Make several mediawiki table fields unsigned ints on wmf databases as Resolved.Sep 14 2018, 9:24 AM
• Banyek closed subtask T204127: Reclone db2054 and db2068 as Resolved.Sep 14 2018, 12:53 PM
jcrespo removed a subtask: T204302: db1062 management interface busy (no sessions allowed).Sep 14 2018, 1:32 PM
Marostegui closed subtask T187089: Fix WMF schemas to not break when comment store goes WRITE_NEW as Resolved.Sep 17 2018, 12:56 PM
Marostegui closed subtask T203548: Remove partitions from s7 masters (db1062 and db2040) for metawiki.pagelinks as Resolved.Sep 17 2018, 2:07 PM
jcrespo closed subtask T204311: Upgrade all core (mediawiki) database servers to mariadb 10.1 as Resolved.Sep 17 2018, 6:05 PM
• Banyek closed subtask T203565: Reclone db1114 (s1 api) from another API host as Resolved.Sep 19 2018, 10:27 AM
• Banyek reopened subtask T203565: Reclone db1114 (s1 api) from another API host as Open.Sep 20 2018, 1:50 PM
Marostegui removed subtasks: T203565: Reclone db1114 (s1 api) from another API host, T186188: Failover DB masters in row D.Sep 21 2018, 8:10 AM

T203565 and T186188 are no longer tasks that are blocked/depend on the DC failover - removing them as subtasks

Marostegui removed a subtask: T100501: mysql user and group should be a system user/group.Sep 21 2018, 8:11 AM
Marostegui added a comment.Sep 26 2018, 10:34 AM

An update on this.
We have pretty much completed all the tasks we had scheduled for the failover. We are now advancing on other tasks, to complete them faster.
What we have pending is: T148507 which is only blocked on one host, which will, presumably done during the network maintenance so we only have downtime once.

Regarding the other subtask: T184267 - we have upgraded all the "difficult" eqiad hosts, meaning, core masters. The rest of hosts will be done slowly but in a steady way :-)

Marostegui closed subtask Restricted Task as Resolved.Oct 4 2018, 4:09 PM
Marostegui closed this task as Resolved.Oct 11 2018, 5:21 AM
Marostegui claimed this task.
Marostegui added a subtask: T184805: Move some wikis to s5.

All the tasks we scheduled to do whilst eqiad was passive, were done!.
We also included T184805 on a last minute task, which was also done and only clean up is left. So I am going to close this meta task.

Thanks everyone for getting so much work done in eqiad!

Dzahn reopened subtask T201133: db1069 (x1 master) memory errors as Open.Oct 13 2018, 3:13 AM
Marostegui mentioned this in T204006: Execute the schema change for Partial Blocks.Oct 15 2018, 5:26 AM
Marostegui closed subtask T201133: db1069 (x1 master) memory errors as Resolved.Oct 17 2018, 5:10 AM
jcrespo closed subtask T184805: Move some wikis to s5 as Resolved.Oct 30 2018, 12:39 PM
MoritzMuehlenhoff closed subtask Restricted Task as Resolved.Jun 24 2019, 4:05 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL