Page MenuHomePhabricator
Create Task
Maniphest T202364

Switchover es2 master (es1011) to es1015
Closed, ResolvedPublic
Actions

Description

es2 master, es1011, is still running jessie/mariadb 10.1 and with a socket on the wrong location.

Switch es2 (cluster24) writes to es1015 on row C so it can be depooled and upgraded.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+2 -3install_server: Allow manual reimage of es1011
operations/mediawiki-configmaster+2 -2mariadb: Depool es1011 for reimage
operations/dnsmaster+1 -1mariadb: Point es2-master to es1015 after master switchover
operations/mediawiki-configmaster+3 -3mariadb: Promote es1015 to es2 master and repool es2 for writes
operations/mediawiki-configmaster+2 -2mariadb: Depool cluster24 (es2) from new writes
operations/puppetproduction+5 -5mariadb: Promote es1015 to be the new es2 master instead of es1011
operations/mediawiki-configmaster+3 -3mariadb: Depool es1015 for maintenance
Customize query in gerrit

Event Timeline

jcrespo created this task.Aug 21 2018, 8:54 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 21 2018, 8:54 AM
jcrespo claimed this task.Aug 21 2018, 8:54 AM
jcrespo moved this task from Triage to In progress on the DBA board.
gerritbot added a subscriber: gerritbot.Aug 21 2018, 8:54 AM

Change 454210 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool cluster24 (es2) from new writes

https://gerrit.wikimedia.org/r/454210

gerritbot added a project: Patch-For-Review.Aug 21 2018, 8:54 AM
gerritbot added a comment.Aug 21 2018, 9:04 AM

Change 454211 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Promote es1015 to es2 master and repool es2 for writes

https://gerrit.wikimedia.org/r/454211

gerritbot added a comment.Aug 21 2018, 9:18 AM

Change 454214 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Promote es1015 to be the new es2 master instead of es1011

https://gerrit.wikimedia.org/r/454214

gerritbot added a comment.Aug 21 2018, 9:49 AM

Change 454219 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/dns@master] mariadb: Point es2-master to es1015 after master switchover

https://gerrit.wikimedia.org/r/454219

gerritbot added a comment.Aug 21 2018, 11:46 AM

Change 454254 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool es1015 for maintenance

https://gerrit.wikimedia.org/r/454254

gerritbot added a comment.Aug 21 2018, 11:55 AM

Change 454254 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool es1015 for maintenance

https://gerrit.wikimedia.org/r/454254

Stashbot added a subscriber: Stashbot.Aug 22 2018, 5:53 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-22T05:53:28Z] <marostegui> Start topology changes for es2 failover - T202364

gerritbot added a comment.Aug 22 2018, 5:59 AM

Change 454214 merged by Jcrespo:
[operations/puppet@production] mariadb: Promote es1015 to be the new es2 master instead of es1011

https://gerrit.wikimedia.org/r/454214

gerritbot added a comment.Aug 22 2018, 6:02 AM

Change 454210 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool cluster24 (es2) from new writes

https://gerrit.wikimedia.org/r/454210

gerritbot added a comment.Aug 22 2018, 6:09 AM

Change 454211 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Promote es1015 to es2 master and repool es2 for writes

https://gerrit.wikimedia.org/r/454211

Stashbot added a comment.Aug 22 2018, 6:30 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-22T06:30:29Z] <jynus> finished es2 switchover T202364

gerritbot added a comment.Aug 22 2018, 6:31 AM

Change 454219 merged by Jcrespo:
[operations/dns@master] mariadb: Point es2-master to es1015 after master switchover

https://gerrit.wikimedia.org/r/454219

gerritbot added a comment.Aug 22 2018, 8:33 AM

Change 454485 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool es1011 for reimage

https://gerrit.wikimedia.org/r/454485

gerritbot added a comment.Aug 22 2018, 8:39 AM

Change 454487 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] install_server: Allow manual reimage of es1011

https://gerrit.wikimedia.org/r/454487

gerritbot added a comment.Aug 22 2018, 8:44 AM

Change 454485 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool es1011 for reimage

https://gerrit.wikimedia.org/r/454485

gerritbot added a comment.Aug 22 2018, 9:04 AM

Change 454487 merged by Jcrespo:
[operations/puppet@production] install_server: Allow manual reimage of es1011

https://gerrit.wikimedia.org/r/454487

ops-monitoring-bot added a subscriber: ops-monitoring-bot.Aug 22 2018, 9:22 AM

Script wmf-auto-reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['es1011.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201808220922_jynus_24242.log.

ops-monitoring-bot added a comment.Aug 22 2018, 9:41 AM

Completed auto-reimage of hosts:

['es1011.eqiad.wmnet']

and were ALL successful.

Stashbot added a comment.Aug 23 2018, 6:31 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-23T06:31:11Z] <marostegui> Enable semi-sync on es2 - T202364

Marostegui closed this task as Resolved.Aug 23 2018, 6:39 AM

All the clean up tasks are now done

I have enabled semi-sync on es2:

root@es1015.eqiad.wmnet[(none)]> SHOW GLOBAL STATUS like 'Rpl_semi_sync_master_clients';
+------------------------------+-------+
| Variable_name                | Value |
+------------------------------+-------+
| Rpl_semi_sync_master_clients | 2     |
+------------------------------+-------+
1 row in set (0.05 sec)

GTID was enabled yesterday

Also dropped the non used grants on es1011.
Nice work with the very smooth failover. Probably the failover with less stress we've ever done! :)

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL