Page MenuHomePhabricator
Create Task
Maniphest T194870

Failover s2 primary master
Closed, ResolvedPublic
Actions

Description

s2 primary master db1054 has BBU issues (T194867) and it also needs to be decommissioned (T186320).

The candidate master is db1066.

This a checklist of what needs to be done before the failover

  • Move db1066 to a different rack T193847
  • Manually fail disk #6 on db1066 and get it replaced (T194955)
  • Finish s2 replicas reimage to stretch (only missing db2035 and db1074)
  • Pick a date for the failover: 13th June - 06:00AM UTC - 06:30AM UTC
  • Communicate liaisons to handle the read only time T195487
  • Prepare and do the actual failover

Details

ProjectBranchLines +/-Subject
operations/dnsmaster+1 -1wmnet: Update s2-master CNAME
operations/softwaremaster+1 -1s2.hosts: db1066 is now s2 primary master
operations/mediawiki-configmaster+2 -2db-eqiad.php: Promote db1066 to master and remove read only
operations/mediawiki-configmaster+1 -1db-eqiad.php: Set s2 as read only
operations/puppetproduction+5 -5mariadb: Promote db1066 to master
operations/mediawiki-configmaster+1 -1db-eqiad.php: Depool db1066
operations/mediawiki-configmaster+3 -3mariadb: Depool db1074 for maintenance
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui triaged this task as High priority.May 17 2018, 8:47 AM
• Marostegui created this task.
• Marostegui moved this task from Triage to Pending comment on the DBA board.
• Marostegui updated the task description. (Show Details)
• Marostegui added a subtask: T193847: Move db1066 to row A.May 17 2018, 9:00 AM
• Marostegui added a subtask: T194867: BBU issues on db1054 (s2 primary master).
• Marostegui updated the task description. (Show Details)May 17 2018, 3:09 PM
• Marostegui closed subtask T193847: Move db1066 to row A as Resolved.May 18 2018, 5:26 AM
Stashbot added a subscriber: Stashbot.May 18 2018, 1:59 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-18T13:59:56Z] <marostegui> Manually fail disk #6 on db1066 - T194870

• Marostegui updated the task description. (Show Details)May 18 2018, 3:45 PM
jcrespo updated the task description. (Show Details)May 18 2018, 6:08 PM
• Marostegui mentioned this in T194867: BBU issues on db1054 (s2 primary master).May 21 2018, 6:04 AM
• Marostegui closed subtask T194867: BBU issues on db1054 (s2 primary master) as Declined.
gerritbot added a subscriber: gerritbot.May 21 2018, 2:22 PM

Change 434339 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db1074 for maintenance

https://gerrit.wikimedia.org/r/434339

gerritbot added a project: Patch-For-Review.May 21 2018, 2:22 PM
gerritbot added a comment.May 21 2018, 2:35 PM

Change 434339 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db1074 for maintenance

https://gerrit.wikimedia.org/r/434339

jcrespo updated the task description. (Show Details)May 21 2018, 3:35 PM
• Marostegui mentioned this in T195487: Announce read-only time for wikis on s2 for 13th June 2018.May 24 2018, 1:50 PM
• Marostegui updated the task description. (Show Details)

We have agreed this will be done the 13th of June 2018 from 06:00AM UTC till 06:30AM UTC

• Marostegui updated the task description. (Show Details)May 24 2018, 1:52 PM
gerritbot added a comment.Jun 8 2018, 8:13 AM

Change 438200 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1066

https://gerrit.wikimedia.org/r/438200

gerritbot added a comment.Jun 8 2018, 8:17 AM

Change 438200 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1066

https://gerrit.wikimedia.org/r/438200

Stashbot added a comment.Jun 8 2018, 8:18 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T08:18:44Z] <marostegui> Stop MySQL and reboot db1066 for intel-microcode install - T194870

• Marostegui added a comment.Jun 8 2018, 3:42 PM

db1066 candidate master has been rebooted to pick up the intel-microcodes before the failover

• Marostegui mentioned this in T186320: Decommission db1051-db1060 (DBA tracking).Jun 8 2018, 3:45 PM
gerritbot added a comment.Jun 11 2018, 7:42 AM

Change 439530 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] mariadb: Promote db1066 to master

https://gerrit.wikimedia.org/r/439530

gerritbot added a comment.Jun 11 2018, 7:55 AM

Change 439531 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Set s2 as read only

https://gerrit.wikimedia.org/r/439531

gerritbot added a comment.Jun 11 2018, 8:00 AM

Change 439532 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Promote db1066 to master and remove read only

https://gerrit.wikimedia.org/r/439532

gerritbot added a comment.Jun 11 2018, 8:05 AM

Change 439533 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/dns@master] wmnet: Update s2-master CNAME

https://gerrit.wikimedia.org/r/439533

gerritbot added a comment.Jun 11 2018, 8:07 AM

Change 439534 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/software@master] s2.hosts: db1066 is now s2 primary master

https://gerrit.wikimedia.org/r/439534

• Marostegui added a comment.Jun 11 2018, 8:09 AM

The draft with the steps and the patches is now done.
@jcrespo please review them! Thanks

• Marostegui moved this task from Pending comment to In progress on the DBA board.Jun 11 2018, 8:44 AM
Stashbot added a comment.Jun 13 2018, 5:11 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-13T05:11:05Z] <marostegui> Starting topology changes in order to get ready for s2 failover - T194870

gerritbot added a comment.Jun 13 2018, 5:20 AM

Change 439530 merged by Marostegui:
[operations/puppet@production] mariadb: Promote db1066 to master

https://gerrit.wikimedia.org/r/439530

gerritbot added a comment.Jun 13 2018, 5:58 AM

Change 439531 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Set s2 as read only

https://gerrit.wikimedia.org/r/439531

Stashbot added a comment.Jun 13 2018, 6:00 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-13T06:00:16Z] <marostegui> Starting s2 failover from db1054 to db1066 - T194870

Stashbot added a comment.Jun 13 2018, 6:01 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-13T06:01:35Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Set s2 on read-only for primary db master maintnance - T194870 (duration: 01m 08s)

gerritbot added a comment.Jun 13 2018, 6:04 AM

Change 439532 merged by Marostegui:
[operations/mediawiki-config@master] db-eqiad.php: Promote db1066 to master and remove read only

https://gerrit.wikimedia.org/r/439532

Stashbot added a comment.Jun 13 2018, 6:05 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-13T06:05:31Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove read only from s2 - T194870 (duration: 00m 34s)

Stashbot added a comment.Jun 13 2018, 6:08 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-13T06:08:41Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove read only from s2 - T194870 (duration: 00m 33s)

Taiwania_Justo added a subscriber: Taiwania_Justo.Jun 13 2018, 6:10 AM
gerritbot added a comment.Jun 13 2018, 6:20 AM

Change 439534 merged by jenkins-bot:
[operations/software@master] s2.hosts: db1066 is now s2 primary master

https://gerrit.wikimedia.org/r/439534

gerritbot added a comment.Jun 13 2018, 6:20 AM

Change 439533 merged by Marostegui:
[operations/dns@master] wmnet: Update s2-master CNAME

https://gerrit.wikimedia.org/r/439533

jcrespo updated the task description. (Show Details)Jun 13 2018, 6:30 AM
• Marostegui added a comment.EditedJun 13 2018, 6:31 AM

This was completed.
read only time started at 06:01
read only time finished at 06:08
Total read only time was around 7 minutes

• Marostegui closed this task as Resolved.Jun 13 2018, 6:37 AM
• Marostegui claimed this task.
• Marostegui mentioned this in T197063: Decommission db1054.
• Marostegui mentioned this in T187089: Fix WMF schemas to not break when comment store goes WRITE_NEW.Jun 13 2018, 6:40 AM
• Marostegui closed subtask T195487: Announce read-only time for wikis on s2 for 13th June 2018 as Resolved.
• Vvjjkkii renamed this task from Failover s2 primary master to 9tcaaaaaaa.Jul 1 2018, 1:08 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed • Marostegui as the assignee of this task.
• Vvjjkkii reopened subtask T195487: Announce read-only time for wikis on s2 for 13th June 2018 as Open.
• Vvjjkkii reopened subtask T194867: BBU issues on db1054 (s2 primary master) as Open.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
• Vvjjkkii reopened subtask T193847: Move db1066 to row A as Open.Jul 1 2018, 1:13 AM
Wong128hk closed subtask T194867: BBU issues on db1054 (s2 primary master) as Declined.Jul 1 2018, 2:28 AM
• Marostegui closed subtask T193847: Move db1066 to row A as Resolved.Jul 1 2018, 7:33 AM
• Marostegui closed subtask T195487: Announce read-only time for wikis on s2 for 13th June 2018 as Resolved.Jul 1 2018, 6:57 PM
• Marostegui renamed this task from 9tcaaaaaaa to Failover s2 primary master.Jul 2 2018, 5:20 AM
• Marostegui closed this task as Resolved.
• Marostegui claimed this task.
• Marostegui lowered the priority of this task from High to Medium.
• Marostegui removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
• Marostegui updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL