Page MenuHomePhabricator
Create Task
Maniphest T189656

Switchover m2 master from db1020 to db1051
Closed, ResolvedPublic
Actions

Description

m2 databases:

root@neodymium:~$ mysql -BN -h db1020.eqiad.wmnet --skip-ssl -e "SHOW DATABASES"
frimpressions
heartbeat
iegreview
information_schema
mysql
otrs
performance_schema
reviewdb
scholarships

root@neodymium:~$ mysql -BN -h db1051.eqiad.wmnet -e "SHOW DATABASES"
frimpressions
heartbeat
iegreview
information_schema
mysql
otrs
performance_schema
reviewdb
scholarships

root@neodymium:~$ mysql -BN -h db2044.codfw.wmnet -e "SHOW DATABASES"
frimpressions
heartbeat
iegreview
information_schema
mysql
otrs
performance_schema
reviewdb
scholarships

Update: scheduled for Thu Mar 15, 2018 09:30 - 10:30 (CET)

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+3 -3prometheus-mysql-exporter: Reflect latest m2 changes, remove dbstore1001
operations/puppetproduction+7 -0dbproxy: Allow exim hosts to connect to misc proxies
operations/puppetproduction+1 -1dbproxy1002.yaml: Add missing port
operations/puppetproduction+2 -3mariadb: Switchover master from db1020 to db1051
operations/puppetproduction+8 -8dbproxy: Failover from db1020 to db1051
operations/puppetproduction+0 -1mariadb: Reenable notifications on db1051
operations/puppetproduction+2 -2dbproxy: Enable firewall on the active m1 and m2 proxies
operations/puppetproduction+38 -12dbproxy: Reenable firewall on passive m1 & m2 proxies with holes
Customize query in gerrit

Related Objects
Search...

Event Timeline

jcrespo triaged this task as Medium priority.Mar 14 2018, 9:48 AM
jcrespo created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 14 2018, 9:48 AM
jcrespo renamed this task from Switchover m2 master to a newer host to Switchover m2 master from db1020 to db1051.Mar 14 2018, 10:25 AM
jcrespo updated the task description. (Show Details)
jcrespo updated the task description. (Show Details)
gerritbot added a subscriber: gerritbot.Mar 14 2018, 11:02 AM

Change 419392 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Reenable firewall on proxies for m1 and m2 with holes

https://gerrit.wikimedia.org/r/419392

gerritbot added a project: Patch-For-Review.Mar 14 2018, 11:02 AM
gerritbot added a comment.Mar 14 2018, 3:06 PM

Change 419392 merged by Jcrespo:
[operations/puppet@production] dbproxy: Reenable firewall on passive m1 & m2 proxies with holes

https://gerrit.wikimedia.org/r/419392

• Marostegui moved this task from Triage to In progress on the DBA board.Mar 14 2018, 3:18 PM
• Marostegui updated the task description. (Show Details)
gerritbot added a comment.Mar 14 2018, 3:44 PM

Change 419456 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Enable firewall on the active m1 and m2 proxies

https://gerrit.wikimedia.org/r/419456

gerritbot added a comment.Mar 14 2018, 4:54 PM

Change 419456 merged by Jcrespo:
[operations/puppet@production] dbproxy: Enable firewall on the active m1 and m2 proxies

https://gerrit.wikimedia.org/r/419456

gerritbot added a comment.Mar 15 2018, 7:29 AM

Change 419669 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Failover from db1020 to db1051

https://gerrit.wikimedia.org/r/419669

gerritbot added a comment.Mar 15 2018, 7:33 AM

Change 419671 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Switchover master from db1020 to db1051

https://gerrit.wikimedia.org/r/419671

gerritbot added a comment.Mar 15 2018, 7:45 AM

Change 419672 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Reenable notifications on db1051

https://gerrit.wikimedia.org/r/419672

gerritbot added a comment.Mar 15 2018, 7:51 AM

Change 419672 merged by Jcrespo:
[operations/puppet@production] mariadb: Reenable notifications on db1051

https://gerrit.wikimedia.org/r/419672

gerritbot added a comment.Mar 15 2018, 8:21 AM

Change 419669 merged by Jcrespo:
[operations/puppet@production] dbproxy: Failover from db1020 to db1051

https://gerrit.wikimedia.org/r/419669

gerritbot added a comment.Mar 15 2018, 8:26 AM

Change 419671 merged by Jcrespo:
[operations/puppet@production] mariadb: Switchover master from db1020 to db1051

https://gerrit.wikimedia.org/r/419671

gerritbot added a comment.Mar 15 2018, 8:37 AM

Change 419680 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1002.yaml: Add missing port

https://gerrit.wikimedia.org/r/419680

gerritbot added a comment.Mar 15 2018, 8:40 AM

Change 419680 merged by Marostegui:
[operations/puppet@production] dbproxy1002.yaml: Add missing port

https://gerrit.wikimedia.org/r/419680

gerritbot added a comment.Mar 15 2018, 9:03 AM

Change 419685 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Allow exim hosts to connect to misc proxies

https://gerrit.wikimedia.org/r/419685

gerritbot added a comment.Mar 15 2018, 9:09 AM

Change 419685 merged by Jcrespo:
[operations/puppet@production] dbproxy: Allow exim hosts to connect to misc proxies

https://gerrit.wikimedia.org/r/419685

gerritbot added a comment.Mar 15 2018, 12:30 PM

Change 419727 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] prometheus-mysql-exporter: Reflect latest m2 changes, remove dbstore1001

https://gerrit.wikimedia.org/r/419727

jcrespo added a subtask: T189773: Decommission db1020.Mar 15 2018, 12:33 PM
jcrespo removed a subtask: T189773: Decommission db1020.Mar 15 2018, 12:35 PM
jcrespo added a parent task: T189773: Decommission db1020.
jcrespo mentioned this in T189773: Decommission db1020.
gerritbot added a comment.Mar 15 2018, 12:42 PM

Change 419727 merged by Jcrespo:
[operations/puppet@production] prometheus-mysql-exporter: Reflect latest m2 changes, remove dbstore1001

https://gerrit.wikimedia.org/r/419727

jcrespo closed this task as Resolved.Mar 15 2018, 4:05 PM
jcrespo claimed this task.

This is technically done, not without issues, but not a lot of real actionable once those are fixed. We can prepare an incident report if people wants to know more. Otherwise we will report the issues found as part of the regular meeting.

The followups is to check/backup/clone the old master and decomm it if no further problems are done T189773.

Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL