Page MenuHomePhabricator
Create Task
Maniphest T119642

Create a Master-master topology between datacenters for easier failover (setup circular replication dallas -> eqiad for mysql databases)
Closed, ResolvedPublic
Actions

Description

Relatively easy task, literally "executing CHANGE MASTER on the 7 masters" with the following pending issues:

  • Master-Master breaks our monitoring "dbtree" (basically because it is stops being a tree and becomes a general graph)
  • We have to make sure that writes are not done my accident on the passive master, breaking production. Specifically, writing to codfw because "it is not production". It need some protection there.

Details

SubjectRepoBranchLines +/-
Fix other slaves not being examined when one was already visitedoperations/software/dbtreemaster+1 -1
Fix other slaves not being examined when one was already visitedoperations/software/tendrilmaster+1 -1
Avoid infinite loops when using circular replicationoperations/software/dbtreemaster+9 -4
Avoid infinite loops when using circular replicationoperations/software/tendrilmaster+10 -5
Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo created this task.Nov 25 2015, 6:17 PM
jcrespo raised the priority of this task from to Needs Triage.
jcrespo updated the task description. (Show Details)
jcrespo added projects: DBA, SRE.
jcrespo subscribed.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptNov 25 2015, 6:17 PM
jcrespo added a parent task: T119626: Eliminate SPOF at the main database infrastructure.Nov 25 2015, 6:17 PM
jcrespo added a parent task: T88445: MediaWiki active/active datacenter investigation and work (tracking).
fgiunchedi triaged this task as High priority.Dec 1 2015, 4:00 PM
fgiunchedi subscribed.
jcrespo mentioned this in T111654: Set up TLS for MariaDB replication.Dec 2 2015, 7:35 PM
jcrespo renamed this task from Create a Master-master topology between datacenters for easier failover to Create a Master-master topology between datacenters for easier failover (setup circular replication dallas -> eqiad for mysql databases).Feb 4 2016, 9:02 PM
jcrespo set Security to None.
jcrespo merged a task: T124698: Setup circular replication (dallas -> eqiad) for databases.
jcrespo added a project: codfw-rollout-Jan-Mar-2016.
jcrespo added a subscriber: faidon.
Restricted Application added a project: codfw-rollout. · View Herald TranscriptFeb 4 2016, 9:03 PM
jcrespo claimed this task.Feb 4 2016, 9:03 PM
jcrespo moved this task from Triage to Backlog on the DBA board.Feb 21 2016, 5:50 PM
jcrespo added a parent task: T124699: Change configuration to make codfw db masters as the masters of all datacenters.Mar 8 2016, 2:25 PM
gerritbot subscribed.Mar 9 2016, 10:13 AM

Change 276127 had a related patch set uploaded (by Jcrespo):
Avoid infinite loops when using circular replication

https://gerrit.wikimedia.org/r/276127

gerritbot added a project: Patch-For-Review.Mar 9 2016, 10:13 AM
gerritbot added a comment.Mar 9 2016, 10:22 AM

Change 276127 merged by Jcrespo:
Avoid infinite loops when using circular replication

https://gerrit.wikimedia.org/r/276127

gerritbot added a comment.Mar 9 2016, 10:56 AM

Change 276134 had a related patch set uploaded (by Jcrespo):
Avoid infinite loops when using circular replication

https://gerrit.wikimedia.org/r/276134

gerritbot added a comment.Mar 9 2016, 10:57 AM

Change 276134 merged by Jcrespo:
Avoid infinite loops when using circular replication

https://gerrit.wikimedia.org/r/276134

jcrespo added a comment.Mar 9 2016, 11:08 AM

Tendril and dbtree has been "fixed" (just applied a simple patch to avoid infinite loops when constructing the tree). We will need a proper replacement for ploting more complex graphs (returning links is currently not represented on the graph), but that is enough to unblock the circular replication between datacenter masters.

jcrespo moved this task from Backlog to In progress on the DBA board.Mar 9 2016, 11:08 AM
jcrespo moved this task from Backlog to In Progress on the codfw-rollout-Jan-Mar-2016 board.
jcrespo removed a project: Patch-For-Review.Mar 9 2016, 11:11 AM
gerritbot added a comment.Mar 9 2016, 11:35 AM

Change 276137 had a related patch set uploaded (by Jcrespo):
Fix other slaves not being examined when one was already visited

https://gerrit.wikimedia.org/r/276137

gerritbot added a project: Patch-For-Review.Mar 9 2016, 11:35 AM

Change 276137 merged by Jcrespo:
Fix other slaves not being examined when one was already visited

https://gerrit.wikimedia.org/r/276137

gerritbot added a comment.Mar 9 2016, 11:43 AM

Change 276140 had a related patch set uploaded (by Jcrespo):
Fix other slaves not being examined when one was already visited

https://gerrit.wikimedia.org/r/276140

gerritbot added a comment.Mar 9 2016, 11:43 AM

Change 276140 merged by Jcrespo:
Fix other slaves not being examined when one was already visited

https://gerrit.wikimedia.org/r/276140

jcrespo removed a project: Patch-For-Review.Mar 9 2016, 11:46 AM
jcrespo closed this task as Resolved.Mar 16 2016, 11:36 AM

All production machines that have cross datacenter replication (x1, s*, es2, es3) have been setup with circular replication.

Not setting up m* shards (misc), as they have a different setup (proxy) in front of it.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL