Page MenuHomePhabricator

Switchover s7 master (db2218 -> db2121)
Closed, ResolvedPublic

Description

When: During a pre-defined DBA maintenance windows

Affected wikis:: https://noc.wikimedia.org/conf/highlight.php?file=dblists/s7.dblist

Checklist:

NEW primary: db2121
OLD primary: db2218

  • Check configuration differences between new and old primary:
sudo pt-config-diff --defaults-file /root/.my.cnf h=db2218.codfw.wmnet h=db2121.codfw.wmnet

Failover prep:

  • Silence alerts on all hosts:
sudo cookbook sre.hosts.downtime --hours 1 -r "Primary switchover s7 T366875" 'A:db-section-s7'
  • Set NEW primary with weight 0
sudo dbctl instance db2121 set-weight 0
sudo dbctl config commit -m "Set db2121 with weight 0 T366875"
  • Depool NEW from any specific group (API, vslow, dump) if present.
sudo dbctl instance db2121 edit
# If some changes were made:
sudo dbctl config commit -m "Remove db2121 from API/vslow/dump T366875"
  • Topology changes, move all replicas under NEW primary
sudo db-switchover --timeout=25 --replicating-master --read-only-master --only-slave-move db2218 db2121
  • Disable puppet on both nodes
sudo cumin 'db2218* or db2121*' 'disable-puppet "primary switchover T366875"'

Failover:

  • Log the failover:
!log Starting s7 codfw failover from db2218 to db2121 - T366875
  • Switch primaries:
sudo db-switchover --replicating-master --read-only-master --skip-slave-move db2218 db2121
echo "===== db2218 (OLD)"; sudo db-mysql db2218 -e 'show slave status\G'
echo "===== db2121 (NEW)"; sudo db-mysql db2121 -e 'show slave status\G'
  • Promote NEW primary in dbctl
sudo dbctl --scope codfw section s7 set-master db2121
sudo dbctl config commit -m "Promote db2121 to s7 primary T366875"
  • Restart puppet on both hosts:
sudo cumin 'db2218* or db2121*' 'run-puppet-agent -e "primary switchover T366875"'

Clean up tasks:

  • Clean up heartbeat table(s).
sudo db-mysql db2121 heartbeat -e "delete from heartbeat where file like 'db2218%';"
  • change events for query killer:
events_coredb_master.sql on the new primary db2121
events_coredb_slave.sql on the new slave db2218
  • Update candidate primary dbctl and orchestrator notes
sudo dbctl instance db2218 set-candidate-master --section s7 true
sudo dbctl instance db2121 set-candidate-master --section s7 false
(dborch1001): sudo orchestrator-client -c untag -i db2121 --tag name=candidate
(dborch1001): sudo orchestrator-client -c tag -i db2218 --tag name=candidate
sudo db-mysql db1215 zarcillo -e "select * from masters where section = 's7';"
  • (If needed): Depool db2218 for maintenance.
sudo dbctl instance db2218 depool
sudo dbctl config commit -m "Depool db2218 T366875"
  • Change db2218 weight to mimic the previous weight db2121:
sudo dbctl instance db2218 edit
  • Update/resolve this ticket.

Event Timeline

Change #1039593 had a related patch set uploaded (by Gerrit maintenance bot; author: Gerrit maintenance bot):

[operations/puppet@production] mariadb: Promote db2121 to s7 master

https://gerrit.wikimedia.org/r/1039593

Marostegui triaged this task as Medium priority.
Marostegui moved this task from Triage to In progress on the DBA board.
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2024-06-10T04:36:40Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s7 T366875

Mentioned in SAL (#wikimedia-operations) [2024-06-10T04:36:49Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Set db2121 with weight 0 T366875', diff saved to https://phabricator.wikimedia.org/P64475 and previous config saved to /var/cache/conftool/dbconfig/20240610-043649-root.json

Old weight 300 and and belongs to api:

groups:
  api:
    pooled: true
    weight: 100

Mentioned in SAL (#wikimedia-operations) [2024-06-10T04:37:04Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s7 T366875

Mentioned in SAL (#wikimedia-operations) [2024-06-10T04:37:42Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Remove db2121 from API/vslow/dump T366875', diff saved to https://phabricator.wikimedia.org/P64476 and previous config saved to /var/cache/conftool/dbconfig/20240610-043741-root.json

Change #1039593 merged by Marostegui:

[operations/puppet@production] mariadb: Promote db2121 to s7 master

https://gerrit.wikimedia.org/r/1039593

Mentioned in SAL (#wikimedia-operations) [2024-06-10T05:06:16Z] <marostegui> Starting s7 codfw failover from db2218 to db2121 - T366875

Mentioned in SAL (#wikimedia-operations) [2024-06-10T05:06:38Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db2121 to s7 primary T366875', diff saved to https://phabricator.wikimedia.org/P64479 and previous config saved to /var/cache/conftool/dbconfig/20240610-050637-marostegui.json

Mentioned in SAL (#wikimedia-operations) [2024-06-10T05:07:40Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2218 T366875', diff saved to https://phabricator.wikimedia.org/P64480 and previous config saved to /var/cache/conftool/dbconfig/20240610-050738-root.json

Marostegui updated the task description. (Show Details)

This is completed