Page MenuHomePhabricator
Create Task
Maniphest T364541

Switchover s8 master (db1209 -> db1192)
Closed, ResolvedPublic
Actions

Description

When: During a pre-defined DBA maintenance windows

Prerequisites: https://wikitech.wikimedia.org/wiki/MariaDB/Primary_switchover

  • Team calendar invite

Affected wikis:: https://noc.wikimedia.org/conf/highlight.php?file=dblists/s8.dblist

Checklist:

NEW primary: db1192
OLD primary: db1209

  • Check configuration differences between new and old primary:
sudo pt-config-diff --defaults-file /root/.my.cnf h=db1209.eqiad.wmnet h=db1192.eqiad.wmnet

Failover prep:

  • Silence alerts on all hosts:
sudo cookbook sre.hosts.downtime --hours 1 -r "Primary switchover s8 T364541" 'A:db-section-s8'
  • Set NEW primary with weight 0
sudo dbctl instance db1192 set-weight 0
sudo dbctl config commit -m "Set db1192 with weight 0 T364541"
  • Depool NEW from any specific group (API, vslow, dump) if present.
sudo dbctl instance db1192 edit
# If some changes were made:
sudo dbctl config commit -m "Remove db1192 from API/vslow/dump T364541"
  • Topology changes, move all replicas under NEW primary
sudo db-switchover --timeout=25 --only-slave-move db1209 db1192
  • Disable puppet on both nodes
sudo cumin 'db1209* or db1192*' 'disable-puppet "primary switchover T364541"'

Failover:

  • Log the failover:
!log Starting s8 eqiad failover from db1209 to db1192 - T364541
  • Set section read-only:
sudo dbctl --scope eqiad section s8 ro "Maintenance until 06:15 UTC - T364541"
sudo dbctl --scope codfw section s8 ro "Maintenance until 06:15 UTC - T364541"
sudo dbctl config commit -m "Set s8 eqiad as read-only for maintenance - T364541"
  • Check s8 is indeed read-only
  • Switch primaries:
sudo db-switchover --skip-slave-move db1209 db1192
echo "===== db1209 (OLD)"; sudo db-mysql db1209 -e 'show slave status\G'
echo "===== db1192 (NEW)"; sudo db-mysql db1192 -e 'show slave status\G'
  • Promote NEW primary in dbctl, and remove read-only
sudo dbctl --scope eqiad section s8 set-master db1192
sudo dbctl --scope eqiad section s8 rw
sudo dbctl --scope codfw section s8 rw
sudo dbctl config commit -m "Promote db1192 to s8 primary and set section read-write T364541"
  • Restart puppet on both hosts:
sudo cumin 'db1209* or db1192*' 'run-puppet-agent -e "primary switchover T364541"'

Clean up tasks:

  • Clean up heartbeat table(s).
sudo db-mysql db1192 heartbeat -e "delete from heartbeat where file like 'db1209%';"
  • change events for query killer:
events_coredb_master.sql on the new primary db1192
events_coredb_slave.sql on the new slave db1209
sudo dbctl instance db1209 set-candidate-master --section s8 true
sudo dbctl instance db1192 set-candidate-master --section s8 false
(dborch1001): sudo orchestrator-client -c untag -i db1192 --tag name=candidate
(dborch1001): sudo orchestrator-client -c tag -i db1209 --tag name=candidate
sudo db-mysql db1215 zarcillo -e "select * from masters where section = 's8';"
  • (If needed): Depool db1209 for maintenance.
sudo dbctl instance db1209 depool
sudo dbctl config commit -m "Depool db1209 T364541"
  • Change db1209 weight to mimic the previous weight db1192:
sudo dbctl instance db1209 edit
  • Update/resolve this ticket.

Details

SubjectRepoBranchLines +/-
wmnet: Update s8-master aliasoperations/dnsmaster+1 -1
mariadb: Promote db1192 to masteroperations/puppetproduction+2 -2
mariadb: Promote db1192 to s8 masteroperations/puppetproduction+3 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

Maintenance_bot created this task.Thu, May 9, 12:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, May 9, 12:45 PM
Maintenance_bot updated the task description. (Show Details)Thu, May 9, 12:45 PM
gerritbot added a comment.Thu, May 9, 12:45 PM

Change #1028939 had a related patch set uploaded (by Gerrit maintenance bot; author: Gerrit maintenance bot):

[operations/puppet@production] mariadb: Promote db1192 to s8 master

https://gerrit.wikimedia.org/r/1028939

gerritbot added a project: Patch-For-Review.Thu, May 9, 12:45 PM

Change #1028940 had a related patch set uploaded (by Gerrit maintenance bot; author: Gerrit maintenance bot):

[operations/dns@master] wmnet: Update s8-master alias

https://gerrit.wikimedia.org/r/1028940

Marostegui claimed this task.Thu, May 9, 12:46 PM
Marostegui triaged this task as Medium priority.
Marostegui updated the task description. (Show Details)
Marostegui moved this task from Triage to Ready on the DBA board.
Marostegui added a parent task: T364299: Make rc_id a bigint.
Marostegui added a parent task: T356166: Drop cl_collation_ext index from categorylinks in production.
Marostegui added a parent task: T355609: Make cuc_id a bigint.
Marostegui added a parent task: T363792: Upgrade s8 to MariaDB 10.6.Tue, May 14, 6:31 AM
Marostegui updated the task description. (Show Details)Tue, May 14, 6:36 AM
Ladsgroup added a parent task: T352010: Gradually drop old pagelinks columns.Thu, May 16, 2:18 PM
gerritbot added a comment.Thu, May 23, 7:55 AM

Change #1035315 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Promote db1192 to master

https://gerrit.wikimedia.org/r/1035315

Marostegui updated the task description. (Show Details)Thu, May 23, 7:55 AM
Marostegui added a comment.Tue, May 28, 3:20 PM

I will do this on Thursday

Stashbot added a comment.Thu, May 30, 4:42 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-30T04:42:41Z] <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 33 hosts with reason: Primary switchover s8 T364541

Stashbot added a comment.Thu, May 30, 4:42 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-30T04:42:50Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Set db1192 with weight 0 T364541', diff saved to https://phabricator.wikimedia.org/P63628 and previous config saved to /var/cache/conftool/dbconfig/20240530-044249-root.json

Stashbot added a comment.Thu, May 30, 4:43 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-30T04:43:08Z] <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 33 hosts with reason: Primary switchover s8 T364541

Marostegui added a comment.Thu, May 30, 4:43 AM

Old weight: 300 in main, 100 in api

Stashbot added a comment.Thu, May 30, 4:43 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-30T04:43:28Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Remove db1192 from API/vslow/dump T364541', diff saved to https://phabricator.wikimedia.org/P63629 and previous config saved to /var/cache/conftool/dbconfig/20240530-044328-root.json

Marostegui updated the task description. (Show Details)Thu, May 30, 4:44 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Thu, May 30, 4:45 AM

Change #1035315 merged by Marostegui:

[operations/puppet@production] mariadb: Promote db1192 to master

https://gerrit.wikimedia.org/r/1035315

Marostegui updated the task description. (Show Details)Thu, May 30, 4:45 AM
Marostegui updated the task description. (Show Details)Thu, May 30, 5:09 AM
Stashbot added a comment.Thu, May 30, 5:09 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-30T05:09:44Z] <marostegui> Starting s8 eqiad failover from db1209 to db1192 - T364541

Stashbot added a comment.Thu, May 30, 5:10 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-30T05:10:12Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Set s8 eqiad as read-only for maintenance - T364541', diff saved to https://phabricator.wikimedia.org/P63630 and previous config saved to /var/cache/conftool/dbconfig/20240530-051012-marostegui.json

Stashbot added a comment.Thu, May 30, 5:10 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-30T05:10:32Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db1192 to s8 primary and set section read-write T364541', diff saved to https://phabricator.wikimedia.org/P63631 and previous config saved to /var/cache/conftool/dbconfig/20240530-051031-marostegui.json

Stashbot added a comment.Thu, May 30, 5:11 AM

Mentioned in SAL (#wikimedia-operations) [2024-05-30T05:11:32Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1209 T364541', diff saved to https://phabricator.wikimedia.org/P63632 and previous config saved to /var/cache/conftool/dbconfig/20240530-051132-root.json

gerritbot added a comment.Thu, May 30, 5:12 AM

Change #1028940 merged by Marostegui:

[operations/dns@master] wmnet: Update s8-master alias

https://gerrit.wikimedia.org/r/1028940

Marostegui closed this task as Resolved.Thu, May 30, 5:12 AM
Marostegui updated the task description. (Show Details)
Marostegui updated the task description. (Show Details)

This is done

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL