Page MenuHomePhabricator
Create Task
Maniphest T263227

Failover s6 master, db1093 to db1131
Closed, ResolvedPublic
Actions

Description

db1131 was moved to row C (T262901).

Let's failover db1093 (that also will be decommissioned next Q (T258361) ).
By doing this we'll balance a little bit more the amount of masters on eqiad's row D, as the current status is:

+----------+------+---------+
| instance | rack | section |
+----------+------+---------+
| db1081   | A2   | s4      |
| db1083   | B1   | s1      |
| db1086   | B3   | s7      |
| db1100   | C2   | s5      |
| db1122   | D6   | s2      |
| db1093   | D8   | s6      |
| db1109   | D8   | s8      |
| db1123   | D8   | s3      |
+----------+------+---------+
8 rows in set (0.002 sec)

By moving this host to row C and by moving s8's one (T239238) we would end up with:

2 masters in row A
3 masters in row B
2 master in row C
2 masters in row D

Details

ProjectBranchLines +/-Subject
operations/dnsmaster+1 -1wmnet: Update s6-master alias
operations/puppetproduction+4 -4mariadb: Promote db1131 to s6 master
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.Sep 18 2020, 5:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 18 2020, 5:57 AM
• Marostegui claimed this task.Sep 18 2020, 5:57 AM
• Marostegui triaged this task as Medium priority.
• Marostegui moved this task from Triage to Pending comment on the DBA board.
• Marostegui added a comment.EditedSep 24 2020, 11:31 AM

Steps and checklist:

Preparation

NEW master: db1131
OLD master: db1093

  • Check configuration differences between new and old master

pt-config-diff h=db1093.eqiad.wmnet,F=/root/.my.cnf h=db1131.eqiad.wmnet,F=/root/.my.cnf

  • Silence alerts on all hosts
  • Set NEW master with weight 0 s6

dbctl instance db1131 edit
dbctl config commit -m "Set db1131 with weight 0 T263227"

  • Topology changes, connect everything to db1131

db-switchover --timeout=15 --read-only-master --replicating-master --only-slave-move db1093.eqiad.wmnet db1131.eqiad.wmnet

Failover:

  • Start the failover

!log Starting s6 eqiad failover from db1093 to db1131 - T263227

  • run switchover script from cumin1001:

root@cumin1001:~/wmfmariadbpy/wmfmariadbpy# db-switchover --replicating-master --read-only-master db1093 db1131

  • Give weight to db1093 in s6 dbctl instance db1093 edit
  • Promote db1131 as new master

dbctl --scope eqiad section s6 set-master db1131 && dbctl config commit -m "Promote db1131 on s6 eqiad master "T263227"

  • Restart puppet on old and new masters (for heartbeat): db1131 and db1093

run-puppet-agent -e "switchover to db1131"

Clean up tasks:

  • change events for query killer:
events_coredb_master.sql on the new master db1131
events_coredb_slave.sql on the new slave db1093
dbctl instance db1131 set-candidate-master false
dbctl instance db1093 set-candidate-master true
dbctl config commit -m "Update candidate master status T263227"
  • Check tendril was updated
  • Check zarcillo was updated
  • Update/resolve phabricator ticket about failover
• Marostegui moved this task from Pending comment to In progress on the DBA board.Sep 28 2020, 8:38 AM
gerritbot added a comment.Sep 29 2020, 9:21 AM

Change 630773 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1131 to s6 master

https://gerrit.wikimedia.org/r/630773

gerritbot added a project: Patch-For-Review.Sep 29 2020, 9:21 AM
gerritbot added a comment.Sep 29 2020, 9:24 AM

Change 630774 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s6-master alias

https://gerrit.wikimedia.org/r/630774

Stashbot added a comment.Sep 30 2020, 7:18 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-30T07:18:42Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1131 with weight 0 T263227', diff saved to https://phabricator.wikimedia.org/P12851 and previous config saved to /var/cache/conftool/dbconfig/20200930-071841-marostegui.json

gerritbot added a comment.Sep 30 2020, 7:36 AM

Change 630773 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1131 to s6 master

https://gerrit.wikimedia.org/r/630773

Stashbot added a comment.Sep 30 2020, 7:41 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-30T07:41:37Z] <marostegui> Starting s6 eqiad failover from db1093 to db1131 - T263227

Stashbot added a comment.Sep 30 2020, 7:44 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-30T07:44:17Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1131 on s6 eqiad master T263227, also give weight to db1093 as new API host', diff saved to https://phabricator.wikimedia.org/P12852 and previous config saved to /var/cache/conftool/dbconfig/20200930-074417-marostegui.json

gerritbot added a comment.Sep 30 2020, 7:47 AM

Change 630774 merged by Marostegui:
[operations/dns@master] wmnet: Update s6-master alias

https://gerrit.wikimedia.org/r/630774

• Marostegui closed this task as Resolved.Sep 30 2020, 7:48 AM

This is all done - db1131 is the new s6 eqiad master

• Marostegui mentioned this in T186188: Failover DB masters in row D.Sep 30 2020, 7:51 AM
Maintenance_bot removed a project: Patch-For-Review.Sep 30 2020, 8:10 AM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL