Page MenuHomePhabricator
Create Task
Maniphest T277070

Beta cluster master switchover to deployment-db07
Closed, ResolvedPublic
Actions

Description

We now have Buster/10.4 replicas: T276968: deployment-db05 needs replacing following disk corruption

After the new replicas have been working for a bit without problems, we should switch db07 to be the database master.

In T276968#6901184, @dduvall wrote:

Thanks for the help, everyone. I would still like to get off of db06 if possible at the end of this process since we have to finish the buster upgrade at some point anyhow. If we can get both db07 and db08 to reach the same point in the binlog from db06, can we simply:

  • STOP SLAVE on db07 and db08
  • FLUSH TABLES WITH READ LOCK on db07. double check SHOW SLAVE STATUS again to verify same position as db08
  • CHANGE MASTER on db08 to replicate from db07
  • START SLAVE on db08
  • configure mediawiki-config to use db07 as master (read load 0) and db08 as replica

Does that sound right?

In T276968#6901186, @dduvall wrote:

Forgot the UNLOCK TABLES on db07 :)

Details

ProjectBranchLines +/-Subject
operations/mediawiki-configmaster+2 -0betacluster: read only for db master switchover
operations/mediawiki-configmaster+5 -10betacluster: promote db07 as db master, decom db06
Customize query in gerrit

Related Objects
Search...

Event Timeline

taavi created this task.Mar 10 2021, 5:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 10 2021, 5:17 PM
taavi mentioned this in T277044: selenium-daily-beta(commons)-MediaWiki fails with `readonly: The wiki is currently in read-only mode.`.Mar 10 2021, 5:19 PM
dduvall added a project: Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)).Mar 10 2021, 5:19 PM
taavi added a parent task: T268628: Upgrade deployment-prep-db hosts to buster/MariaDB 10.4.Mar 10 2021, 5:19 PM
Urbanecm added a comment.Mar 10 2021, 5:37 PM

This is scheduled for Mar 11, 13:30 UTC. Beta will be read only during the switchover.

Urbanecm added a project: User-Urbanecm.Mar 10 2021, 5:38 PM
taavi added a project: User-Majavah.Mar 10 2021, 5:39 PM
RhinosF1 added a subscriber: RhinosF1.Mar 10 2021, 5:39 PM
zeljkofilipin added a project: User-zeljkofilipin.Mar 10 2021, 6:08 PM
zeljkofilipin moved this task from Backlog 🪒 to Waiting 🏹 on the User-zeljkofilipin board.
zeljkofilipin added a subscriber: zeljkofilipin.
gerritbot added a comment.Mar 11 2021, 12:01 PM

Change 670803 had a related patch set uploaded (by Majavah; owner: Majavah):
[operations/mediawiki-config@master] betacluster: promote db07 as database master

https://gerrit.wikimedia.org/r/670803

gerritbot added a project: Patch-For-Review.Mar 11 2021, 12:01 PM

Change 670804 had a related patch set uploaded (by Majavah; owner: Majavah):
[operations/mediawiki-config@master] betacluster: read only for db master switchover

https://gerrit.wikimedia.org/r/670804

Stashbot added a comment.Mar 11 2021, 1:32 PM

Mentioned in SAL (#wikimedia-releng) [2021-03-11T13:32:50Z] <Majavah> set deployment-db06 as read only T277070

Stashbot added a comment.Mar 11 2021, 1:34 PM

Mentioned in SAL (#wikimedia-releng) [2021-03-11T13:34:41Z] <Majavah> stop and reset slave on deployment-db07 T277070

Stashbot added a comment.Mar 11 2021, 1:37 PM

Mentioned in SAL (#wikimedia-releng) [2021-03-11T13:37:47Z] <Majavah> make deployment-db06 and deployment-db08 be replicas of deployment-db07 T277070

gerritbot added a comment.Mar 11 2021, 1:43 PM

Change 670803 merged by jenkins-bot:
[operations/mediawiki-config@master] betacluster: promote db07 as db master, decom db06

https://gerrit.wikimedia.org/r/670803

Stashbot added a comment.Mar 11 2021, 1:48 PM

Mentioned in SAL (#wikimedia-releng) [2021-03-11T13:48:01Z] <Majavah> stop mariadb to ensure reads have stopped on deployment-db06 T277070

Stashbot added a comment.Mar 11 2021, 1:48 PM

Mentioned in SAL (#wikimedia-releng) [2021-03-11T13:48:37Z] <Majavah> set deployment-db07 as r/w T277070

Stashbot added a comment.Mar 11 2021, 1:51 PM

Mentioned in SAL (#wikimedia-releng) [2021-03-11T13:51:06Z] <Majavah> shut down deployment-db06, now unused T277070

taavi closed this task as Resolved.Mar 11 2021, 1:52 PM
taavi claimed this task.

This was done with about 15 minutes of read-only time.

gerritbot added a comment.Mar 11 2021, 1:52 PM

Change 670804 abandoned by Majavah:
[operations/mediawiki-config@master] betacluster: read only for db master switchover

Reason:
not needed

https://gerrit.wikimedia.org/r/670804

Maintenance_bot removed a project: Patch-For-Review.Mar 12 2021, 9:46 AM
zeljkofilipin removed a subscriber: zeljkofilipin.Mar 15 2021, 2:49 PM
taavi mentioned this in T278007: Configure etcd/confd/conftool in beta/deployment-prep like production.Mar 20 2021, 1:13 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL