Page MenuHomePhabricator

Switchover s6 master (db2114 -> db2129)
Closed, ResolvedPublic

Description

When: During a pre-defined DBA maintenance windows

Affected wikis:: https://noc.wikimedia.org/conf/highlight.php?file=dblists/s6.dblist

Checklist:

NEW primary: db2129
OLD primary: db2114

  • Check configuration differences between new and old primary:
sudo pt-config-diff --defaults-file /root/.my.cnf h=db2114.codfw.wmnet h=db2129.codfw.wmnet

Failover prep:

  • Silence alerts on all hosts:
sudo cookbook sre.hosts.downtime --hours 1 -r "Primary switchover s6 T327644" 'A:db-section-s6'
  • Set NEW primary with weight 0 (and depool it from API or vslow/dump groups if it is present).
sudo dbctl instance db2129 set-weight 0
sudo dbctl config commit -m "Set db2129 with weight 0 T327644"
  • Topology changes, move all replicas under NEW primary
sudo db-switchover --timeout=25 --replicating-master --read-only-master --only-slave-move db2114 db2129
  • Disable puppet on both nodes
sudo cumin 'db2114* or db2129*' 'disable-puppet "primary switchover T327644"'
  • Merge gerrit puppet change to promote NEW primary: FIXME

Failover:

  • Log the failover:
!log Starting s6 codfw failover from db2114 to db2129 - T327644
  • Switch primaries:
sudo db-switchover --replicating-master --read-only-master --skip-slave-move db2114 db2129
echo "===== db2114 (OLD)"; sudo db-mysql db2114 -e 'show slave status\G'
echo "===== db2129 (NEW)"; sudo db-mysql db2129 -e 'show slave status\G'
  • Promote NEW primary in dbctl
sudo dbctl --scope codfw section s6 set-master db2129
sudo dbctl config commit -m "Promote db2129 to s6 primary T327644"
  • Restart puppet on both hosts:
sudo cumin 'db2114* or db2129*' 'run-puppet-agent -e "primary switchover T327644"'

Clean up tasks:

  • Clean up heartbeat table(s).
sudo db-mysql db2129 heartbeat -e "delete from heartbeat where file like 'db2114%';"
  • change events for query killer:
events_coredb_master.sql on the new primary db2129
events_coredb_slave.sql on the new slave db2114
  • Update candidate primary dbctl and orchestrator notes
sudo dbctl instance db2114 set-candidate-master --section s6 true
sudo dbctl instance db2129 set-candidate-master --section s6 false
(dborch1001): sudo orchestrator-client -c untag -i db2129 --tag name=candidate
(dborch1001): sudo orchestrator-client -c tag -i db2114 --tag name=candidate
sudo db-mysql db1115 zarcillo -e "select * from masters where section = 's6';"
  • (If needed): Depool db2114 for maintenance.
sudo dbctl instance db2114 depool
sudo dbctl config commit -m "Depool db2114 T327644"
  • Change db2114 weight to mimic the previous weight db2129:
sudo dbctl instance db2114 edit
  • Apply outstanding schema changes to db2114 (if any)
  • Update/resolve this ticket.

Event Timeline

Change 882260 had a related patch set uploaded (by Gerrit maintenance bot; author: Gerrit maintenance bot):

[operations/puppet@production] mariadb: Promote db2129 to s6 master

https://gerrit.wikimedia.org/r/882260

Ladsgroup changed the task status from Open to In Progress.Mon, Jan 23, 10:53 AM
Ladsgroup claimed this task.
Ladsgroup triaged this task as Medium priority.
Ladsgroup updated the task description. (Show Details)
Ladsgroup changed the edit policy from "Custom Policy" to "All Users".
Ladsgroup moved this task from Triage to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2023-01-23T10:54:11Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s6 T327644

Mentioned in SAL (#wikimedia-operations) [2023-01-23T10:54:40Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s6 T327644

Mentioned in SAL (#wikimedia-operations) [2023-01-23T10:55:21Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Set db2129 with weight 0 T327644', diff saved to https://phabricator.wikimedia.org/P43231 and previous config saved to /var/cache/conftool/dbconfig/20230123-105520-ladsgroup.json

Mentioned in SAL (#wikimedia-operations) [2023-01-23T11:17:47Z] <Amir1> Starting s6 codfw failover from db2114 to db2129 - T327644

Mentioned in SAL (#wikimedia-operations) [2023-01-23T11:18:13Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Promote db2129 to s6 primary T327644', diff saved to https://phabricator.wikimedia.org/P43234 and previous config saved to /var/cache/conftool/dbconfig/20230123-111813-ladsgroup.json

Change 882260 merged by Ladsgroup:

[operations/puppet@production] mariadb: Promote db2129 to s6 master

https://gerrit.wikimedia.org/r/882260

Mentioned in SAL (#wikimedia-operations) [2023-01-23T11:21:35Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depool db2114 T327644', diff saved to https://phabricator.wikimedia.org/P43236 and previous config saved to /var/cache/conftool/dbconfig/20230123-112134-ladsgroup.json

Ladsgroup removed a project: Patch-For-Review.
Ladsgroup updated the task description. (Show Details)
Ladsgroup moved this task from In progress to Done on the DBA board.