Page MenuHomePhabricator

Add support for multidc to auto_schema
Closed, ResolvedPublic

Description

Fixing the depooling is easy. The complicating part is deciding on how to run schema changes across the fleet.

A schema change can go in one of three ways in the secondary dc:

  • Status quo: Run it on master of secondary dc with replication enabled. This can be done possibly in some cases in the future as well or when a dc in depooled
  • Run on each replica and run it on master at the end. Similar to the most of schema changes we do on eqiad.
  • Run on all replicas. Do a switchover of master and run it on the depooled old master. This will be needed in schema changes that will take hours. Thankfully switchover on a read-only dc should be fairly straightforward. We might even be able to delegate it to auto_schema.

Event Timeline

Ladsgroup moved this task from Triage to In progress on the DBA board.

Change 820142 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/software@master] auto_schema: Start depooling codfw replicas

https://gerrit.wikimedia.org/r/820142

Change 820165 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/software@master] auto_schema: Change replica_set to be all replicas in all dcs

https://gerrit.wikimedia.org/r/820165

Change 820181 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/software@master] auto_schema: Add tests for replicas to pick

https://gerrit.wikimedia.org/r/820181

Change 820216 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/software@master] auto_schema: Treat master of all dcs like a master of active dc

https://gerrit.wikimedia.org/r/820216

Change 820142 merged by jenkins-bot:

[operations/software@master] auto_schema: Start depooling codfw replicas

https://gerrit.wikimedia.org/r/820142

Change 820165 merged by jenkins-bot:

[operations/software@master] auto_schema: Change replica_set to be all replicas in all dcs

https://gerrit.wikimedia.org/r/820165

Change 820181 merged by jenkins-bot:

[operations/software@master] auto_schema: Add tests for replicas to pick

https://gerrit.wikimedia.org/r/820181

Change 820216 merged by jenkins-bot:

[operations/software@master] auto_schema: Treat master of all dcs like a master of active dc

https://gerrit.wikimedia.org/r/820216