Page MenuHomePhabricator

Switchover s4 master (db2140 -> db2179)
Closed, ResolvedPublic

Description

When: During a pre-defined DBA maintenance windows

Affected wikis:: https://noc.wikimedia.org/conf/highlight.php?file=dblists/s4.dblist

Checklist:

NEW primary: db2179
OLD primary: db2140

  • Check configuration differences between new and old primary:
sudo pt-config-diff --defaults-file /root/.my.cnf h=db2140.codfw.wmnet h=db2179.codfw.wmnet

Failover prep:

  • Silence alerts on all hosts:
sudo cookbook sre.hosts.downtime --hours 1 -r "Primary switchover s4 T367017" 'A:db-section-s4'
  • Set NEW primary with weight 0
sudo dbctl instance db2179 set-weight 0
sudo dbctl config commit -m "Set db2179 with weight 0 T367017"
  • Depool NEW from any specific group (API, vslow, dump) if present.
sudo dbctl instance db2179 edit
# If some changes were made:
sudo dbctl config commit -m "Remove db2179 from API/vslow/dump T367017"
  • Topology changes, move all replicas under NEW primary
sudo db-switchover --timeout=25 --replicating-master --read-only-master --only-slave-move db2140 db2179
  • Disable puppet on both nodes
sudo cumin 'db2140* or db2179*' 'disable-puppet "primary switchover T367017"'

Failover:

  • Log the failover:
!log Starting s4 codfw failover from db2140 to db2179 - T367017
  • Switch primaries:
sudo db-switchover --replicating-master --read-only-master --skip-slave-move db2140 db2179
echo "===== db2140 (OLD)"; sudo db-mysql db2140 -e 'show slave status\G'
echo "===== db2179 (NEW)"; sudo db-mysql db2179 -e 'show slave status\G'
  • Promote NEW primary in dbctl
sudo dbctl --scope codfw section s4 set-master db2179
sudo dbctl config commit -m "Promote db2179 to s4 primary T367017"
  • Restart puppet on both hosts:
sudo cumin 'db2140* or db2179*' 'run-puppet-agent -e "primary switchover T367017"'

Clean up tasks:

  • Clean up heartbeat table(s).
sudo db-mysql db2179 heartbeat -e "delete from heartbeat where file like 'db2140%';"
  • change events for query killer:
events_coredb_master.sql on the new primary db2179
events_coredb_slave.sql on the new slave db2140
  • Update candidate primary dbctl and orchestrator notes
sudo dbctl instance db2140 set-candidate-master --section s4 true
sudo dbctl instance db2179 set-candidate-master --section s4 false
(dborch1001): sudo orchestrator-client -c untag -i db2179 --tag name=candidate
(dborch1001): sudo orchestrator-client -c tag -i db2140 --tag name=candidate
sudo db-mysql db1215 zarcillo -e "select * from masters where section = 's4';"
  • (If needed): Depool db2140 for maintenance.
sudo dbctl instance db2140 depool
sudo dbctl config commit -m "Depool db2140 T367017"
  • Change db2140 weight to mimic the previous weight db2179:
sudo dbctl instance db2140 edit
  • Update/resolve this ticket.

Event Timeline

Change #1039604 had a related patch set uploaded (by Gerrit maintenance bot; author: Gerrit maintenance bot):

[operations/puppet@production] mariadb: Promote db2179 to s4 master

https://gerrit.wikimedia.org/r/1039604

Marostegui triaged this task as Medium priority.
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2024-06-10T06:18:42Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s4 T367017

Mentioned in SAL (#wikimedia-operations) [2024-06-10T06:18:49Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Set db2179 with weight 0 T367017', diff saved to https://phabricator.wikimedia.org/P64485 and previous config saved to /var/cache/conftool/dbconfig/20240610-061849-root.json

Mentioned in SAL (#wikimedia-operations) [2024-06-10T06:19:10Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s4 T367017

Mentioned in SAL (#wikimedia-operations) [2024-06-10T06:19:39Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Remove db2179 from API/vslow/dump T367017', diff saved to https://phabricator.wikimedia.org/P64486 and previous config saved to /var/cache/conftool/dbconfig/20240610-061939-root.json

Change #1039604 merged by Marostegui:

[operations/puppet@production] mariadb: Promote db2179 to s4 master

https://gerrit.wikimedia.org/r/1039604

Mentioned in SAL (#wikimedia-operations) [2024-06-10T06:38:13Z] <marostegui> Starting s4 codfw failover from db2140 to db2179 - T367017

Mentioned in SAL (#wikimedia-operations) [2024-06-10T06:38:31Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db2179 to s4 primary T367017', diff saved to https://phabricator.wikimedia.org/P64491 and previous config saved to /var/cache/conftool/dbconfig/20240610-063830-root.json

Mentioned in SAL (#wikimedia-operations) [2024-06-10T06:39:04Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2140 T367017', diff saved to https://phabricator.wikimedia.org/P64492 and previous config saved to /var/cache/conftool/dbconfig/20240610-063904-root.json