seen in this patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071570 db2223 collides with another db replication source: https://fault-tolerance.toolforge.org/map?cluster=db-masters
@Ladsgroup could you help me find a better suited candidate?
If you give me your train of thought I'll attach it to a task to create a cookbook "find.a.better.suited.candidate.master"
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| db2191: Make it s5 candidate master | operations/puppet | production | +3 -0 | |
| mariadb: add db2223 | operations/puppet | production | +4 -3 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | jcrespo | T376916 Upgrade backup hosts to Debian Bookworm 12.X | |||
| Resolved | jcrespo | T383902 Upgrade backup source or mediabackup database host os to Debian bookworm or decommission them | |||
| Unknown Object (Task) | |||||
| Resolved | • Marostegui | T373579 Productionize db22[21-40] | |||
| In Progress | FCeratto-WMF | T384810 MariaDB lifetime management system | |||
| Declined | FCeratto-WMF | T375460 mariadb: create a cookbook to convert a replica in primary candidate | |||
| Resolved | • Marostegui | T374951 find a viable replication source on codfw |
I'd say for now, pool it as a normal replica (remove the candidate master part from the puppet patch). Don't decommission the old one. Then let's go through pooled replicas of s5 and pick any that: 1- Isn't multiinstance 2- Isn't sharing a rack with any other master or candidate master 3- It's not sanitarium master 4- Is not about to be decommissioned. Once that's found. We need to change the binlog format to STATEMENT (make sure to set the binlog format both live and puppet). Then run flush logs. Then update tags and it's done. I'm not sure we can really automate this. We need to decide on it in every case. Fun.
Change #1083813 had a related patch set uploaded (by Arnaudb; author: Arnaudb):
[operations/puppet@production] mariadb: add db2223
Change #1083813 merged by Arnaudb:
[operations/puppet@production] mariadb: add db2223
db2223 has been pooled as a plain replica. db2123 being scheduled for removal, we'll have to pick a new candidate master for this section. I guess this could also be a good time for us to document and/or automate the decision process
Why not picking db2192 for instance?
The process is quite to actually maintain in place as we move servers logically a lot, so whatever state we leave it in now, it may not be the same one we'll have next week or even tomorrow if we have crashes, switchovers, decommisionings etc.
I think a good approach would be to start using Amir's tool to suggest single replicas (with no special features like RBR or hanging replicas) and see how that goes.
It could be a good candidate, I haven't had the time to unfold the process described in T374951#10168440 but I fully trust your choice, I'll swap it with db2123.
I think tooling to to help pick that kind of host could be useful as its mostly sieving down a set of nodes with another set of information. As far as I found, Amir's tool (just to be 100% sure: you're talking about this one: https://fault-tolerance.toolforge.org/map right?), was a good help, but we can automate this even further based on the information it provides. Otherwise, we should at least document on wikitech the set of criteria to be used to pick up a new candidate source.
Change #1109678 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2191: Make it s5 candidate master
Change #1109678 merged by Marostegui:
[operations/puppet@production] db2191: Make it s5 candidate master