Page MenuHomePhabricator

Switchover s6 master (db1173 -> db1131)
Closed, ResolvedPublic

Description

When: Tuesday 14th at 06:00 AM UTC

  • Team calendar invite

Affected wikis:: https://noc.wikimedia.org/conf/highlight.php?file=dblists/s6.dblist

Checklist:

NEW primary: db1131
OLD primary: db1173

  • Check configuration differences between new and old primary:
sudo pt-config-diff --defaults-file /root/.my.cnf h=db1173.eqiad.wmnet h=db1131.eqiad.wmnet

Failover prep:

  • Silence alerts on all hosts:
sudo cookbook sre.hosts.downtime --hours 1 -r "Primary switchover s6 T300471" 'A:db-section-s6'
  • Set NEW primary with weight 0 (and depool it from API or vslow/dump groups if it is present).
sudo dbctl instance db1131 set-weight 0
sudo dbctl config commit -m "Set db1131 with weight 0 T300471"
  • Topology changes, move all replicas under NEW primary
sudo db-switchover --timeout=25 --only-slave-move db1173 db1131
  • Disable puppet on both nodes
sudo cumin 'db1173* or db1131*' 'disable-puppet "primary switchover T300471"'

Failover:

  • Log the failover:
!log Starting s6 eqiad failover from db1173 to db1131 - T300471
  • Set section read-only:
sudo dbctl --scope eqiad section s6 ro "Maintenance until 06:15 UTC - T300471"
sudo dbctl config commit -m "Set s6 eqiad as read-only for maintenance - T300471"
  • Check s6 is indeed read-only
  • Switch primaries:
sudo db-switchover --skip-slave-move db1173 db1131
echo "===== db1173 (OLD)"; sudo db-mysql db1173 -e 'show slave status\G'
echo "===== db1131 (NEW)"; sudo db-mysql db1131 -e 'show slave status\G'
  • Promote NEW primary in dbctl, and remove read-only
sudo dbctl --scope eqiad section s6 set-master db1131
sudo dbctl --scope eqiad section s6 rw
sudo dbctl config commit -m "Promote db1131 to s6 primary and set section read-write T300471"
  • Restart puppet on both hosts:
sudo cumin 'db1173* or db1131*' 'run-puppet-agent -e "primary switchover T300471"'

Clean up tasks:

  • Clean up heartbeat table(s): delete from heartbeat.heartbeat where server_id=171978805;
sudo db-mysql db1131 heartbeat -e "delete from heartbeat where file like 'db1173%';"
  • change events for query killer:
events_coredb_master.sql on the new primary db1131
events_coredb_slave.sql on the new slave db1173
sudo dbctl instance db1173 set-candidate-master --section s6 true
sudo dbctl instance db1131 set-candidate-master --section s6 false
(dborch1001): sudo orchestrator-client -c untag -i db1131 --tag name=candidate
(dborch1001): sudo orchestrator-client -c tag -i db1173 --tag name=candidate
sudo db-mysql db1115 zarcillo -e "select * from masters where section = 's6';"
  • (If needed): Depool db1173 for maintenance.
sudo dbctl instance db1173 depool
sudo dbctl config commit -m "Depool db1173 T300471"
  • Apply outstanding schema changes to db1173 (if any) - to be tracked on the schema changes tasks
  • Update/resolve this ticket.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Marostegui triaged this task as Medium priority.Jan 31 2022, 6:27 AM
Marostegui added a project: DBA.
Marostegui updated the task description. (Show Details)
Marostegui moved this task from Triage to Ready on the DBA board.

Change 764784 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/puppet@production] mariadb: Switch s7 primary db1173 -> db1131

https://gerrit.wikimedia.org/r/764784

Change 764786 had a related patch set uploaded (by Kormat; author: Kormat):

[operations/dns@master] wmnet: Update s6-master CNAME.

https://gerrit.wikimedia.org/r/764786

Marostegui updated the task description. (Show Details)
Marostegui updated Other Assignee, added: Kormat; removed: Marostegui.
Marostegui updated the task description. (Show Details)

I am doing this on Tuesday 14th at 06:00 AM UTC

Marostegui added a subscriber: Trizek-WMF.

@Trizek-WMF I am finally doing this on Tuesday 14th at 06:00 AM UTC

I updated Tech News, even if it is on a short notice. A two-weeks prior notice would be nicer for our users. :)

Per T303605: Stop announcing and scheduling primary database switchovers, we don't display banners anymore. But if the read-only time has chances to be be longer than a few minutes, I can drop a message at the wikis.

db1131 needs a kernel upgrade for: T310485

db1131 needs a kernel upgrade for: T310485

Done

Change 805121 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Promote db1131 to s6 master

https://gerrit.wikimedia.org/r/805121

Change 805137 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/dns@master] wmnet: Update s6-master CNAME

https://gerrit.wikimedia.org/r/805137

Marostegui updated the task description. (Show Details)
Marostegui updated Other Assignee, added: jcrespo; removed: Kormat.

Mentioned in SAL (#wikimedia-operations) [2022-06-14T04:52:00Z] <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on 23 hosts with reason: Primary switchover s6 T300471

Mentioned in SAL (#wikimedia-operations) [2022-06-14T04:52:15Z] <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 23 hosts with reason: Primary switchover s6 T300471

Mentioned in SAL (#wikimedia-operations) [2022-06-14T04:52:25Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1131 with weight 0 T300471', diff saved to https://phabricator.wikimedia.org/P29706 and previous config saved to /var/cache/conftool/dbconfig/20220614-045224-root.json

Original weight for db1131 was 400:

1--- eqiad/sectionLoads/s6 live
2+++ eqiad/sectionLoads/s6 generated
3@@ -6,7 +6,7 @@
4 "db1096:3316": 200,
5 "db1098:3316": 200,
6 "db1113:3316": 1,
7- "db1131": 400,
8+ "db1131": 0,
9 "db1165": 300,
10 "db1168": 400,
11 "db1180": 400

Change 805121 merged by Marostegui:

[operations/puppet@production] mariadb: Promote db1131 to s6 master

https://gerrit.wikimedia.org/r/805121

Change 805265 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1173: Disable notifications

https://gerrit.wikimedia.org/r/805265

Mentioned in SAL (#wikimedia-operations) [2022-06-14T06:01:41Z] <marostegui> Starting s6 eqiad failover from db1173 to db1131 - T300471

Mentioned in SAL (#wikimedia-operations) [2022-06-14T06:01:56Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s6 eqiad as read-only for maintenance - T300471', diff saved to https://phabricator.wikimedia.org/P29707 and previous config saved to /var/cache/conftool/dbconfig/20220614-060155-root.json

Mentioned in SAL (#wikimedia-operations) [2022-06-14T06:02:28Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1131 to s6 primary and set section read-write T300471', diff saved to https://phabricator.wikimedia.org/P29708 and previous config saved to /var/cache/conftool/dbconfig/20220614-060227-root.json

Change 805137 merged by Marostegui:

[operations/dns@master] wmnet: Update s6-master CNAME

https://gerrit.wikimedia.org/r/805137

Mentioned in SAL (#wikimedia-operations) [2022-06-14T06:06:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1173 T300471', diff saved to https://phabricator.wikimedia.org/P29709 and previous config saved to /var/cache/conftool/dbconfig/20220614-060608-root.json

This was done.
RO starts: 06:01:56
RO stops: 06:02:28

Total RO time: 32 seconds

Change 805265 merged by Marostegui:

[operations/puppet@production] db1173: Disable notifications

https://gerrit.wikimedia.org/r/805265