Page MenuHomePhabricator

dbproxy1002 down
Closed, ResolvedPublic

Description

godog has failovered its functionality to the actual master (db1020)

Event Timeline

Restricted Application added subscribers: Southparkfan, Aklapper. · View Herald Transcript
MariaDB MISC m2 localhost (none) > SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| bugzilla_testing   |
| frimpressions      |
| heartbeat          |
| iegreview          |
| information_schema |
| mysql              |
| otrs               |
| performance_schema |
| reviewdb           |
| scholarships       |
| test               |
| testblog           |
+--------------------+
12 rows in set (0.00 sec)
jcrespo triaged this task as Medium priority.Jul 21 2016, 4:10 PM
jcrespo moved this task from Triage to In progress on the DBA board.
jcrespo added subscribers: fgiunchedi, Joe.

dbproxy1002 seems to be back up again thanks to @fgiunchedi and @Joe. I will point the DNS back to the proxy again at an appropriate window.

I am checking times, according to logs (request numbers are too low) gerrit and OTRS were down between 12:43 and 12:58.

We need to revert https://gerrit.wikimedia.org/r/300254 once we check everything is working and have a window where it is not disruptive.

Change 300502 had a related patch set uploaded (by Jcrespo):
Revert m2-master back to the proxy (dbproxy1002)

https://gerrit.wikimedia.org/r/300502

Change 300502 merged by Jcrespo:
Revert m2-master back to the proxy (dbproxy1002)

https://gerrit.wikimedia.org/r/300502

Change 300505 had a related patch set uploaded (by Jcrespo):
Add dbproxy1006-10 to production as redundant instances of 1-5

https://gerrit.wikimedia.org/r/300505

Change 300505 merged by Jcrespo:
Add dbproxy1006-10 to production as redundant instances of 1-5

https://gerrit.wikimedia.org/r/300505

Change 300512 had a related patch set uploaded (by Jcrespo):
New m3 database grants for dbproxy1008

https://gerrit.wikimedia.org/r/300512

Change 300512 merged by Jcrespo:
New m3 database grants for dbproxy1008

https://gerrit.wikimedia.org/r/300512

Change 300517 had a related patch set uploaded (by Jcrespo):
Extra grants on m1 databases for dbproxy1006

https://gerrit.wikimedia.org/r/300517

Change 300517 merged by Jcrespo:
Extra grants on m1 databases for dbproxy1006

https://gerrit.wikimedia.org/r/300517

Mentioned in SAL [2016-07-22T10:47:31Z] <jynus> reimage dbproxy1007 T140983

Change 300520 had a related patch set uploaded (by Jcrespo):
Add extra grants on s2 for dbproxy1007

https://gerrit.wikimedia.org/r/300520

Change 300520 merged by Jcrespo:
Add extra grants on s2 for dbproxy1007

https://gerrit.wikimedia.org/r/300520

Mentioned in SAL [2016-07-22T11:12:12Z] <jynus> reimage dbproxy1009 T140983

Change 300521 had a related patch set uploaded (by Jcrespo):
Add grants on m4 databases for dbproxy1009

https://gerrit.wikimedia.org/r/300521

Change 300521 merged by Jcrespo:
Add grants on m4 databases for dbproxy1009

https://gerrit.wikimedia.org/r/300521

Change 300529 had a related patch set uploaded (by Jcrespo):
Add extra grants needed on m5 for the dbproxy1010

https://gerrit.wikimedia.org/r/300529

Change 300529 merged by Jcrespo:
Add extra grants needed on m5 for the dbproxy1010

https://gerrit.wikimedia.org/r/300529

No more issues from dbproxy1002, and we now have proxy redundancy.