Page MenuHomePhabricator
Create Task
Maniphest T197073

switchover es1014 to es1017
Closed, ResolvedPublic
Actions

Description

Change the active master of es3 shard from es1014 to es1017.

This will allow to:

  • Upgrade the es3 master to stretch/10.1
  • Finally change the socket location
  • Migrate the SPOF away from DC row B

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

• jcrespo created this task.Jun 13 2018, 8:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 13 2018, 8:59 AM
• jcrespo added a subtask: T197072: Physically move es1017 from D to C row.Jun 13 2018, 9:04 AM
• jcrespo added a parent task: T183585: Rack/cable/configure asw2-b-eqiad switch stack.
• Marostegui moved this task from Triage to Pending comment on the DBA board.Jun 13 2018, 9:18 AM
• Marostegui triaged this task as Medium priority.Jun 26 2018, 7:48 AM
• Vvjjkkii renamed this task from switchover es1014 to es1017 to 24aaaaaaaa.Jul 1 2018, 1:04 AM
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
• Marostegui renamed this task from 24aaaaaaaa to switchover es1014 to es1017.Jul 1 2018, 8:09 PM
• Marostegui lowered the priority of this task from High to Medium.
• Marostegui removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
• Marostegui updated the task description. (Show Details)
• Marostegui added a comment.Jul 3 2018, 2:11 PM

es1017 has been successfully moved to row C.
Once it has been working fine for a few days (not the first time we see old hardware failing after a few days), we should schedule the failover.

• jcrespo closed subtask T197072: Physically move es1017 from D to C row as Resolved.Jul 4 2018, 4:54 AM
• Marostegui added a comment.Jul 6 2018, 1:51 PM

We'd need to start thinking a date for this failover.

• jcrespo added a subtask: T199224: Test database master switchover script on codfw.Jul 10 2018, 1:35 PM
• Marostegui mentioned this in T183585: Rack/cable/configure asw2-b-eqiad switch stack.Jul 16 2018, 3:45 PM
• jcrespo added a comment.Jul 18 2018, 7:16 AM

What do you think of doing this next wednesday?

• Marostegui added a comment.Jul 18 2018, 7:18 AM

Good for me!

• jcrespo added a comment.Jul 18 2018, 9:05 AM

https://wikitech.wikimedia.org/w/index.php?title=Deployments&type=revision&diff=1797449&oldid=1797439

gerritbot subscribed.Jul 18 2018, 9:10 AM

Change 446551 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Allow reimage of es1019

https://gerrit.wikimedia.org/r/446551

gerritbot added a project: Patch-For-Review.Jul 18 2018, 9:10 AM
gerritbot added a comment.Jul 18 2018, 9:12 AM

Change 446553 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool es1018 for reimage

https://gerrit.wikimedia.org/r/446553

gerritbot added a comment.Jul 18 2018, 9:15 AM

Change 446551 merged by Jcrespo:
[operations/puppet@production] mariadb: Allow reimage of es1019

https://gerrit.wikimedia.org/r/446551

gerritbot added a comment.Jul 19 2018, 7:46 AM

Change 446553 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Depool es1019 for reimage

https://gerrit.wikimedia.org/r/446553

gerritbot added a comment.Jul 19 2018, 8:57 AM

Change 446752 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool es1019 with low load

https://gerrit.wikimedia.org/r/446752

gerritbot added a comment.Jul 19 2018, 8:59 AM

Change 446755 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool es1019 full after maintenance

https://gerrit.wikimedia.org/r/446755

gerritbot added a comment.Jul 19 2018, 9:03 AM

Change 446752 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool es1019 with low load

https://gerrit.wikimedia.org/r/446752

gerritbot added a comment.Jul 19 2018, 2:16 PM

Change 446827 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Productionize db1095 and db1102 into s1-test

https://gerrit.wikimedia.org/r/446827

gerritbot added a comment.Jul 19 2018, 2:39 PM

Change 446827 merged by Jcrespo:
[operations/puppet@production] mariadb: Productionize db1095 and db1102 into test-s1

https://gerrit.wikimedia.org/r/446827

gerritbot added a comment.Jul 19 2018, 2:51 PM

Change 446846 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db1099 for cloning and upgrade

https://gerrit.wikimedia.org/r/446846

gerritbot added a comment.Jul 19 2018, 2:58 PM

Change 446846 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Depool db1099 for cloning and upgrade

https://gerrit.wikimedia.org/r/446846

gerritbot added a comment.Jul 19 2018, 3:03 PM

Change 446849 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Fully repool db1099, including db1099:s8

https://gerrit.wikimedia.org/r/446849

gerritbot added a comment.Jul 19 2018, 3:36 PM

Change 446849 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Fully depool db1099, including db1099:s8

https://gerrit.wikimedia.org/r/446849

gerritbot added a comment.Jul 19 2018, 5:40 PM

Change 446875 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool db1099 (both instances) with low load

https://gerrit.wikimedia.org/r/446875

gerritbot added a comment.Jul 19 2018, 8:19 PM

Change 446875 merged by Jcrespo:
[operations/mediawiki-config@master] mariadb: Repool db1099 (both instances) with low load

https://gerrit.wikimedia.org/r/446875

gerritbot added a comment.Jul 19 2018, 8:21 PM

Change 446902 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool db1099 fully after warmup

https://gerrit.wikimedia.org/r/446902

gerritbot added a comment.Jul 20 2018, 7:00 AM

Change 446755 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool es1019 fully after maintenance

https://gerrit.wikimedia.org/r/446755

gerritbot added a comment.Jul 20 2018, 7:11 AM

Change 446902 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool db1099 fully after warmup

https://gerrit.wikimedia.org/r/446902

gerritbot added a comment.Jul 24 2018, 6:47 AM

Change 447584 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Promote es1017 as the master of es3-eqiad (instead of es1014)

https://gerrit.wikimedia.org/r/447584

gerritbot added a comment.Jul 24 2018, 6:56 AM

Change 447586 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Promote es1017 as the master of es3-eqiad

https://gerrit.wikimedia.org/r/447586

gerritbot added a comment.Jul 24 2018, 7:12 AM

Change 447587 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] Setup es1017 as the backend for the es3-eqiad master

https://gerrit.wikimedia.org/r/447587

• jcrespo closed subtask T199224: Test database master switchover script on codfw as Resolved.Jul 24 2018, 5:08 PM
gerritbot added a comment.Jul 25 2018, 5:52 AM

Change 447584 merged by Marostegui:
[operations/puppet@production] mariadb: Promote es1017 as the master of es3-eqiad (instead of es1014)

https://gerrit.wikimedia.org/r/447584

gerritbot added a comment.Jul 25 2018, 5:56 AM

Change 447586 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Promote es1017 as the master of es3-eqiad

https://gerrit.wikimedia.org/r/447586

• Marostegui added a comment.Jul 25 2018, 6:08 AM

Congratulations @jcrespo for doing the first automated failover with the new script!
The errors lasted:
First error at 06:01:27 and last error 06:02:05

Very impressive! :)

gerritbot added a comment.Jul 25 2018, 6:09 AM

Change 447587 merged by Jcrespo:
[operations/dns@master] Setup es1017 as the backend for the es3-eqiad master

https://gerrit.wikimedia.org/r/447587

Stashbot subscribed.Jul 25 2018, 6:33 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-25T06:33:54Z] <jynus> finished es1014 -> es1017 switch T197073

• jcrespo mentioned this in T200306: Improve database master switchover script.Jul 25 2018, 7:20 AM
• jcrespo closed this task as Resolved.Jul 25 2018, 7:31 AM
• jcrespo claimed this task.
• jcrespo moved this task from Pending comment to In progress on the DBA board.

This is now done, and while there are things pending to do related to es1014 maintenance, the main task T183585 is unblocked from all SPOF db hosts.

gerritbot added a comment.Aug 14 2018, 8:31 AM

Change 452637 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Failover db1066 (eqiad s2 master) to db1122

https://gerrit.wikimedia.org/r/452637

gerritbot added a comment.Aug 16 2018, 2:58 PM

Change 452637 abandoned by Jcrespo:
mariadb: Failover db1066 (eqiad s2 master) to db1122

Reason:
Not needed.

https://gerrit.wikimedia.org/r/452637

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits