Right now we use repl.pl to move slaves around:
ie: when a master failover is needed, we use it to move all the slaves under the new master.
However, this script doesn't work when the master is unavailable.
It would be a good start to either refactor repl.pl or create a new script that could move slaves under a different host when the master is unavailable.
ie: master has crashed and we have to move all the slaves to replicate from the candidate master during an emergency.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | LSobanski | T156461 [META ticket] Automation for our DBs tracking task | |||
| Open | None | T196366 Implement (or refactor) a script to move slaves when the master is not available |
@jcrespo - I have been thinking about this ticket lately.
Given that switchover.py works so well already, do you think it would be doable to do a --emergency-slave-switch $new_master (or whatever option) to be able to move the slaves under a given host without checking the master?
This would allow us to do emergency failovers if a master isn't reachable - obviously this needs to be execute carefully, but during an emergency, it can simplify the process of having to execute the change master host to the preferred host.
A human should still check:
Sadly switchover.py wouldn't be reusable or helpful (the replication and other libraries may be) for an emergency- it has to start from 0. Switchover.py assumes all hosts are reachable and have very low lag, replication is working, etc. which won't be the case on a failover. A failover is a much harder case where every possibility of breakage has to be contemplated separately and some safe compromises have to be taken (e.g. what to do if we detect X amount of data has been lost).
Ah, I see!.
Yeah, I was thinking about a very primitive way to do it (for now), which would require human intervention to decide which is the most suitable host to be the new master and then the script to actually execute the batch of change master to master host.
Yeah, I understood you that -not a fully automated and autonomous script- but even that is not easy and still not reusable, as it would have to make it without using the master, and the requires arbitrary master changes that neither gtid nor WMFReplication.move() allow yet. We would need to implement binlog position matching first, and a way to detect replicas from a master down (tendril replacement "zarcillo" database?). All doable, but not immediate or reusable from existing code.
Maybe gtid will become usable at 10.4 ? https://jira.mariadb.org/browse/MDEV-12012?focusedCommentId=132462#comment-132462
Good point - with the master down there is not a canonical place to detect which hosts are hanging apart from tendril/zarcillo indeed.
With the great work done by @Ladsgroup at T281249: Create or modify an existing tool that quickly shows the db replication status in case of master failure I think we are a step closer to get this done.
Once we have that script, we could implement another one based on that one (rather than refactor db-switchover) which would take care of, once passed, the right candidate master, simply configure replication on all the other replicas.
The safety measure the script should be to disallow hosts that have the following items:
@Ladsgroup would you be ok working on this task?
@Ladsgroup: Removing task assignee as this open task has been assigned for more than two years - see the email sent to all task assignees on 2024-04-15.
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome! :)
If this task has been resolved in the meantime, or should not be worked on by anybody ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator. Thanks!
@Marostegui To get the list of direct replicas, something like this would work in cumin:
Which outputs something like this:
ladsgroup@cumin1002:~/ladsgroup/software2/dbtools$ python3 direct_replicas.py s2 direct replicas "db1156.eqiad.wmnet:3306" "db1182.eqiad.wmnet:3306" "db1188.eqiad.wmnet:3306" "db1197.eqiad.wmnet:3306" "db1222.eqiad.wmnet:3306" "db1225.eqiad.wmnet:3312" "db1229.eqiad.wmnet:3306" "db1233.eqiad.wmnet:3306" "db1239.eqiad.wmnet:3312" "db1246.eqiad.wmnet:3306" "db2207.codfw.wmnet:3306" "dbstore1007.eqiad.wmnet:3312"
(it only works from cumin)
We can make it find direct replicas for secondary dc too. The hardest part is to refactor db-switchover to take that list (in itself it's not hard, it's that it assumes in many places that the old master is reachable which is a good assumption for the main usecase). Maybe I should copy paste that into a new file and see what happens.
Keep in mind that you don't really need the list of replicas for the secondary, if you have that master, that is all you need. You don't really need to touch the secondary replicas. That is: all you need would be to reconfigure db2207 to replicate under the new primary master, their replicas don't need anything.
I agree, let's create db-emergency-switchover and work there without touching the current db-switchover for now.
ladsgroup opened https://gitlab.wikimedia.org/repos/sre/wmfmariadbpy/-/merge_requests/3
[WIP] Introduce emergency switchover