Page MenuHomePhabricator
Create Task
Maniphest T274336

Switchover s7 from db1086 to db1136
Closed, ResolvedPublic
Actions

Description

db1086, acting as s7 primary master is on the list of hosts that might have a BBU crash anytime (T258386), also we need to update the kernel (T273280)
We need to promote db1136 instead as a primary master.

When: 23rd March, 06:00 AM UTC

Checklist:

  • Double check db1136 has report_host enabled T271106
  • Create a task to communicate the chosen date and send an announcement to the community: T276899

NEW master: db1136
OLD master: db1086

  • Check configuration differences between new and old master

pt-config-diff h=db1086.eqiad.wmnet,F=/root/.my.cnf h=db1136.eqiad.wmnet,F=/root/.my.cnf

  • Silence alerts on all hosts
  • Set NEW master with weight 0 s7

dbctl instance db1136 edit
dbctl config commit -m "Set db1136 with weight 0 T274336"

  • Topology changes, connect everything to db1136

db-switchover --timeout=15 --only-slave-move db1086.eqiad.wmnet db1136.eqiad.wmnet

Failover:

  • Start the failover

!log Starting s7 eqiad failover from db1086 to db1136 - T274336

  • Read only on s7

dbctl --scope eqiad section s7 ro "Maintenance till 07:15M UTC " && dbctl config commit -m "Set s7 as read-only for maintenance T274336"

  • Check s7 is indeed on read only
  • run switchover script from cumin1001:

db-switchover --skip-slave-move db1086 db1136 ; echo db1086; mysql.py -hdb1086 -e "show slave status\G" ; echo db1136 ; mysql.py -hdb1136 -e "show slave status\G"

  • Promote db1136 as new master and remove read-only

dbctl --scope eqiad section s7 set-master db1136 && dbctl --scope eqiad section s7 rw && dbctl config commit -m "Promote db1136 to s7 master and remove read-only from s7 T274336"

  • Restart puppet on old and new masters (for heartbeat): db1136 and db1086

run-puppet-agent -e "switchover to db1136"

  • Give weight to db1086 in s7

dbctl instance db1086 edit

Clean up tasks:

  • change events for query killer:
events_coredb_master.sql on the new master db1136
events_coredb_slave.sql on the new slave db1086
dbctl instance db1136 set-candidate-master --section s7 false
dbctl instance db1086 set-candidate-master --section s7 true

Details

SubjectRepoBranchLines +/-
wmnet: Update s7-master cnameoperations/dnsmaster+1 -1
mariadb: Promote db1136 to s7 masteroperations/puppetproduction+4 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

Marostegui created this task.Feb 10 2021, 8:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 10 2021, 8:40 AM
Marostegui triaged this task as Medium priority.Feb 10 2021, 8:42 AM
Marostegui updated the task description. (Show Details)
Marostegui added parent tasks: Restricted Task, T266483: Enable report_host for mariadb, T258386: db1080-95 batch possibly suffering BBU issues.
Marostegui added a project: DBA.Feb 10 2021, 8:45 AM
Marostegui added a subscriber: MoritzMuehlenhoff.

Waiting for the new kernel to be released for Stretch @MoritzMuehlenhoff

Marostegui moved this task from Triage to Blocked on the DBA board.Feb 10 2021, 8:45 AM
Marostegui updated the task description. (Show Details)Mar 9 2021, 9:34 AM

I have scheduled this for 23rd March at 06:00 AM UTC

Marostegui moved this task from Blocked to Ready on the DBA board.Mar 9 2021, 9:35 AM
Marostegui updated the task description. (Show Details)Mar 9 2021, 9:41 AM
Marostegui mentioned this in T277007: Upgrade firmware on db1136.Mar 10 2021, 7:16 AM
Marostegui added a subtask: T276150: Schema change to make rc_id unsigned and rc_timestamp BINARY.Mar 11 2021, 7:29 AM
Marostegui added a subtask: T276156: Drop default of rc_timestamp.
Marostegui mentioned this in T276150: Schema change to make rc_id unsigned and rc_timestamp BINARY.Mar 11 2021, 9:25 AM
Marostegui removed a subtask: T276156: Drop default of rc_timestamp.
• Cmjohnson closed subtask T277007: Upgrade firmware on db1136 as Resolved.Mar 15 2021, 2:43 PM
Marostegui reopened subtask T277007: Upgrade firmware on db1136 as Open.Mar 15 2021, 2:50 PM
Marostegui closed subtask T277007: Upgrade firmware on db1136 as Resolved.Mar 16 2021, 6:44 AM
Marostegui updated the task description. (Show Details)Mar 16 2021, 10:17 AM
Marostegui updated the task description. (Show Details)
Marostegui claimed this task.Mar 16 2021, 10:25 AM
Marostegui moved this task from Ready to In progress on the DBA board.
Marostegui added a comment.Mar 16 2021, 10:31 AM

Reserved maintenance window on the Deployments' calendar

• Kormat subscribed.Mar 16 2021, 10:42 AM
Marostegui added a comment.Mar 16 2021, 11:06 AM
~# mysql.py -hdb1136 -e "select @@report_host"
+--------------------+
| @@report_host      |
+--------------------+
| db1136.eqiad.wmnet |
+--------------------+
gerritbot added a comment.Mar 18 2021, 7:36 AM

Change 673195 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1136 to s7 master

https://gerrit.wikimedia.org/r/673195

gerritbot added a project: Patch-For-Review.Mar 18 2021, 7:36 AM
Marostegui updated the task description. (Show Details)Mar 18 2021, 7:36 AM
Marostegui updated the task description. (Show Details)
gerritbot added a comment.Mar 18 2021, 7:38 AM

Change 673196 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s7-master cname

https://gerrit.wikimedia.org/r/673196

Marostegui updated the task description. (Show Details)Mar 18 2021, 7:40 AM
Stashbot added a comment.Mar 23 2021, 5:12 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-23T05:12:10Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set weight 0 to db1136 before failover T274336', diff saved to https://phabricator.wikimedia.org/P14992 and previous config saved to /var/cache/conftool/dbconfig/20210323-051210-marostegui.json

Stashbot added a comment.Mar 23 2021, 5:13 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-23T05:13:46Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1174 to api T274336', diff saved to https://phabricator.wikimedia.org/P14993 and previous config saved to /var/cache/conftool/dbconfig/20210323-051346-marostegui.json

Marostegui updated the task description. (Show Details)Mar 23 2021, 5:15 AM
Marostegui updated the task description. (Show Details)Mar 23 2021, 5:29 AM
gerritbot added a comment.Mar 23 2021, 5:34 AM

Change 673195 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1136 to s7 master

https://gerrit.wikimedia.org/r/673195

Marostegui updated the task description. (Show Details)Mar 23 2021, 5:36 AM
Marostegui updated the task description. (Show Details)Mar 23 2021, 5:59 AM
Stashbot added a comment.Mar 23 2021, 6:00 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-23T06:00:38Z] <marostegui> Starting s7 eqiad failover from db1086 to db1136 - T274336

Stashbot added a comment.Mar 23 2021, 6:01 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-23T06:01:05Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Set s7 as read-only for maintenance T274336', diff saved to https://phabricator.wikimedia.org/P14994 and previous config saved to /var/cache/conftool/dbconfig/20210323-060104-marostegui.json

Stashbot added a comment.Mar 23 2021, 6:02 AM

Mentioned in SAL (#wikimedia-operations) [2021-03-23T06:02:17Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1136 to s7 master and remove read-only from s7 T274336', diff saved to https://phabricator.wikimedia.org/P14995 and previous config saved to /var/cache/conftool/dbconfig/20210323-060216-marostegui.json

Marostegui updated the task description. (Show Details)Mar 23 2021, 6:04 AM
Marostegui updated the task description. (Show Details)Mar 23 2021, 6:07 AM
gerritbot added a comment.Mar 23 2021, 6:08 AM

Change 673196 merged by Marostegui:
[operations/dns@master] wmnet: Update s7-master cname

https://gerrit.wikimedia.org/r/673196

Marostegui updated the task description. (Show Details)Mar 23 2021, 6:09 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 23 2021, 6:10 AM
Marostegui updated the task description. (Show Details)Mar 23 2021, 6:13 AM
Marostegui added a comment.Mar 23 2021, 6:15 AM

This was done:
RO starts: 06:01:05
RO stops: 06:02:17

Total: read-only time: 1:12 minutes

Marostegui closed this task as Resolved.Mar 23 2021, 6:16 AM
Marostegui updated the task description. (Show Details)

Thanks everyone for the support!

Marostegui closed subtask T276899: Read only time window needed for s7 as Resolved.Mar 23 2021, 6:16 AM
Marostegui updated the task description. (Show Details)Mar 23 2021, 9:23 AM
Marostegui changed the status of subtask T276150: Schema change to make rc_id unsigned and rc_timestamp BINARY from Open to Stalled.Apr 13 2021, 8:05 AM
Marostegui changed the status of subtask T276150: Schema change to make rc_id unsigned and rc_timestamp BINARY from Stalled to Open.Jul 1 2021, 1:01 PM
Marostegui closed subtask T276150: Schema change to make rc_id unsigned and rc_timestamp BINARY as Resolved.Jul 7 2021, 12:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits