Right now, on A6 and D3 we have three masters in each (https://fault-tolerance.toolforge.org/map?cluster=db-masters)
This is suboptimal.
For codfw, we have a couple racks with two masters:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Marostegui | T371361 A6 and D3 have 3 db masters each | |||
| Open | Ladsgroup | T371362 Get notified whenever we have 2 or more db masters per rack | |||
| Resolved | Marostegui | T371365 Switchover s6 master (db1173 -> db1201) | |||
| Resolved | Marostegui | T371368 Switchover s8 master (db1209 -> db1193) | |||
| Resolved | Marostegui | T371455 Switchover s3 master (db2205 -> db2209) | |||
| Resolved | Marostegui | T371462 Switchover s7 master (db2218 -> db2220) | |||
| Resolved | Marostegui | T372393 Switchover s3 master (db1223 -> db1189) |
Change #1058139 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1201: Make it s6 candidate
Change #1058139 merged by Marostegui:
[operations/puppet@production] db1201: Make it s6 candidate
Mentioned in SAL (#wikimedia-operations) [2024-07-30T12:08:06Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1201 T371361', diff saved to https://phabricator.wikimedia.org/P67064 and previous config saved to /var/cache/conftool/dbconfig/20240730-120805-root.json
Change #1058140 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1231: Remove it from candidate master
Change #1058140 merged by Marostegui:
[operations/puppet@production] db1231: Remove it from candidate master
Mentioned in SAL (#wikimedia-operations) [2024-07-30T12:15:01Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1231 T371361', diff saved to https://phabricator.wikimedia.org/P67066 and previous config saved to /var/cache/conftool/dbconfig/20240730-121500-root.json
Change #1058144 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1193: Make it candidate master for s8
Mentioned in SAL (#wikimedia-operations) [2024-07-30T12:22:44Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1193 T371361', diff saved to https://phabricator.wikimedia.org/P67068 and previous config saved to /var/cache/conftool/dbconfig/20240730-122243-root.json
Change #1058144 merged by Marostegui:
[operations/puppet@production] db1193: Make it candidate master for s8
Change #1058148 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1192: Remove from candidate master
Change #1058148 merged by Marostegui:
[operations/puppet@production] db1192: Remove from candidate master
I think A6 is fine now. s4 and m3 is fine. In any case, m3 will need to be switched for T356240
Change #1058291 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2209: Make it s3 candidate master
Change #1058291 merged by Marostegui:
[operations/puppet@production] db2209: Make it s3 candidate master
Mentioned in SAL (#wikimedia-operations) [2024-07-31T05:46:53Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2209 T371361', diff saved to https://phabricator.wikimedia.org/P67108 and previous config saved to /var/cache/conftool/dbconfig/20240731-054653-root.json
Mentioned in SAL (#wikimedia-operations) [2024-07-31T05:50:04Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Make db2127 vslow and remove it as candidate master T371361', diff saved to https://phabricator.wikimedia.org/P67109 and previous config saved to /var/cache/conftool/dbconfig/20240731-055004-marostegui.json
Change #1058292 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2127: No longer s3 master
Change #1058292 merged by Marostegui:
[operations/puppet@production] db2127: No longer s3 master
Change #1058568 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2220: Make it candidate master for s7
Change #1058568 merged by Marostegui:
[operations/puppet@production] db2220: Make it candidate master for s7
Mentioned in SAL (#wikimedia-operations) [2024-07-31T09:14:51Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2220 T371361', diff saved to https://phabricator.wikimedia.org/P67139 and previous config saved to /var/cache/conftool/dbconfig/20240731-091450-root.json
Mentioned in SAL (#wikimedia-operations) [2024-07-31T09:17:06Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Move db2121 to vslow T371361', diff saved to https://phabricator.wikimedia.org/P67140 and previous config saved to /var/cache/conftool/dbconfig/20240731-091706-root.json
Change #1058570 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2121: Remove from candidate master
Change #1058570 merged by Marostegui:
[operations/puppet@production] db2121: Remove from candidate master
@Ladsgroup I think we are in a better state now, I will work on s2 or s3 when I get back from holidays. I don't want to do so many switchovers before going off for a week :)
Change #1062379 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] mariadb: Change s3 candidate master
Change #1062379 merged by Marostegui:
[operations/puppet@production] mariadb: Change s3 candidate master