Page MenuHomePhabricator

Switchover m2 master from db1020 to db1051
Closed, ResolvedPublic

Description

m2 databases:

root@neodymium:~$ mysql -BN -h db1020.eqiad.wmnet --skip-ssl -e "SHOW DATABASES"
frimpressions
heartbeat
iegreview
information_schema
mysql
otrs
performance_schema
reviewdb
scholarships

root@neodymium:~$ mysql -BN -h db1051.eqiad.wmnet -e "SHOW DATABASES"
frimpressions
heartbeat
iegreview
information_schema
mysql
otrs
performance_schema
reviewdb
scholarships

root@neodymium:~$ mysql -BN -h db2044.codfw.wmnet -e "SHOW DATABASES"
frimpressions
heartbeat
iegreview
information_schema
mysql
otrs
performance_schema
reviewdb
scholarships

Update: scheduled for Thu Mar 15, 2018 09:30 - 10:30 (CET)

Event Timeline

jcrespo triaged this task as Normal priority.Mar 14 2018, 9:48 AM
jcrespo created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 14 2018, 9:48 AM
jcrespo renamed this task from Switchover m2 master to a newer host to Switchover m2 master from db1020 to db1051.Mar 14 2018, 10:25 AM
jcrespo updated the task description. (Show Details)
jcrespo updated the task description. (Show Details)

Change 419392 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Reenable firewall on proxies for m1 and m2 with holes

https://gerrit.wikimedia.org/r/419392

Change 419392 merged by Jcrespo:
[operations/puppet@production] dbproxy: Reenable firewall on passive m1 & m2 proxies with holes

https://gerrit.wikimedia.org/r/419392

Marostegui moved this task from Triage to In progress on the DBA board.Mar 14 2018, 3:18 PM
Marostegui updated the task description. (Show Details)

Change 419456 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Enable firewall on the active m1 and m2 proxies

https://gerrit.wikimedia.org/r/419456

Change 419456 merged by Jcrespo:
[operations/puppet@production] dbproxy: Enable firewall on the active m1 and m2 proxies

https://gerrit.wikimedia.org/r/419456

Change 419669 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Failover from db1020 to db1051

https://gerrit.wikimedia.org/r/419669

Change 419671 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Switchover master from db1020 to db1051

https://gerrit.wikimedia.org/r/419671

Change 419672 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Reenable notifications on db1051

https://gerrit.wikimedia.org/r/419672

Change 419672 merged by Jcrespo:
[operations/puppet@production] mariadb: Reenable notifications on db1051

https://gerrit.wikimedia.org/r/419672

Change 419669 merged by Jcrespo:
[operations/puppet@production] dbproxy: Failover from db1020 to db1051

https://gerrit.wikimedia.org/r/419669

Change 419671 merged by Jcrespo:
[operations/puppet@production] mariadb: Switchover master from db1020 to db1051

https://gerrit.wikimedia.org/r/419671

Change 419680 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1002.yaml: Add missing port

https://gerrit.wikimedia.org/r/419680

Change 419680 merged by Marostegui:
[operations/puppet@production] dbproxy1002.yaml: Add missing port

https://gerrit.wikimedia.org/r/419680

Change 419685 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] dbproxy: Allow exim hosts to connect to misc proxies

https://gerrit.wikimedia.org/r/419685

Change 419685 merged by Jcrespo:
[operations/puppet@production] dbproxy: Allow exim hosts to connect to misc proxies

https://gerrit.wikimedia.org/r/419685

Change 419727 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] prometheus-mysql-exporter: Reflect latest m2 changes, remove dbstore1001

https://gerrit.wikimedia.org/r/419727

Change 419727 merged by Jcrespo:
[operations/puppet@production] prometheus-mysql-exporter: Reflect latest m2 changes, remove dbstore1001

https://gerrit.wikimedia.org/r/419727

jcrespo closed this task as Resolved.Mar 15 2018, 4:05 PM
jcrespo claimed this task.

This is technically done, not without issues, but not a lot of real actionable once those are fixed. We can prepare an incident report if people wants to know more. Otherwise we will report the issues found as part of the regular meeting.

The followups is to check/backup/clone the old master and decomm it if no further problems are done T189773.