Page MenuHomePhabricator
Create Task
Maniphest T140983

dbproxy1002 down
Closed, ResolvedPublic
Actions

Description

godog has failovered its functionality to the actual master (db1020)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add extra grants needed on m5 for the dbproxy1010operations/puppetproduction+1 -0
Add grants on m4 databases for dbproxy1009operations/puppetproduction+14 -2
Add extra grants on s2 for dbproxy1007operations/puppetproduction+78 -1
Extra grants on m1 databases for dbproxy1006operations/puppetproduction+42 -2
New m3 database grants for dbproxy1008operations/puppetproduction+50 -0
Add dbproxy1006-10 to production as redundant instances of 1-5operations/puppetproduction+11 -5
Revert m2-master back to the proxy (dbproxy1002)operations/dnsmaster+1 -1
Customize query in gerrit

Related Objects

Event Timeline

jcrespo created this task.Jul 21 2016, 1:13 PM
Restricted Application added a project: SRE. · View Herald TranscriptJul 21 2016, 1:13 PM
Restricted Application added subscribers: Southparkfan, Aklapper. · View Herald Transcript
jcrespo added a comment.Jul 21 2016, 1:14 PM
MariaDB MISC m2 localhost (none) > SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| bugzilla_testing   |
| frimpressions      |
| heartbeat          |
| iegreview          |
| information_schema |
| mysql              |
| otrs               |
| performance_schema |
| reviewdb           |
| scholarships       |
| test               |
| testblog           |
+--------------------+
12 rows in set (0.00 sec)
jcrespo triaged this task as Medium priority.Jul 21 2016, 4:10 PM
jcrespo moved this task from Triage to In progress on the DBA board.
jcrespo added subscribers: fgiunchedi, Joe.

dbproxy1002 seems to be back up again thanks to @fgiunchedi and @Joe. I will point the DNS back to the proxy again at an appropriate window.

jcrespo claimed this task.Jul 21 2016, 4:20 PM
jcrespo added a comment.Jul 21 2016, 5:39 PM

I am checking times, according to logs (request numbers are too low) gerrit and OTRS were down between 12:43 and 12:58.

jcrespo mentioned this in T125027: upgrade dbproxy1001/1002 to jessie .Jul 21 2016, 9:01 PM
jcrespo added a comment.Jul 21 2016, 9:04 PM

We need to revert https://gerrit.wikimedia.org/r/300254 once we check everything is working and have a window where it is not disruptive.

gerritbot subscribed.Jul 22 2016, 6:37 AM

Change 300502 had a related patch set uploaded (by Jcrespo):
Revert m2-master back to the proxy (dbproxy1002)

https://gerrit.wikimedia.org/r/300502

gerritbot added a project: Patch-For-Review.Jul 22 2016, 6:37 AM

Change 300502 merged by Jcrespo:
Revert m2-master back to the proxy (dbproxy1002)

https://gerrit.wikimedia.org/r/300502

gerritbot added a comment.Jul 22 2016, 7:18 AM

Change 300505 had a related patch set uploaded (by Jcrespo):
Add dbproxy1006-10 to production as redundant instances of 1-5

https://gerrit.wikimedia.org/r/300505

jcrespo mentioned this in rOPUPd341b3e9e9eb: Add dbproxy1006-10 to production as redundant instances of 1-5.Jul 22 2016, 7:21 AM
gerritbot added a comment.Jul 22 2016, 8:16 AM

Change 300505 merged by Jcrespo:
Add dbproxy1006-10 to production as redundant instances of 1-5

https://gerrit.wikimedia.org/r/300505

gerritbot added a comment.Jul 22 2016, 9:20 AM

Change 300512 had a related patch set uploaded (by Jcrespo):
New m3 database grants for dbproxy1008

https://gerrit.wikimedia.org/r/300512

jcrespo mentioned this in rOPUPb80eb2253113: New m3 database grants for dbproxy1008.Jul 22 2016, 9:26 AM
gerritbot added a comment.Jul 22 2016, 9:33 AM

Change 300512 merged by Jcrespo:
New m3 database grants for dbproxy1008

https://gerrit.wikimedia.org/r/300512

gerritbot added a comment.Jul 22 2016, 10:26 AM

Change 300517 had a related patch set uploaded (by Jcrespo):
Extra grants on m1 databases for dbproxy1006

https://gerrit.wikimedia.org/r/300517

jcrespo mentioned this in rOPUPdca17b55d416: Extra grants on m1 databases for dbproxy1006.Jul 22 2016, 10:29 AM
gerritbot added a comment.Jul 22 2016, 10:32 AM

Change 300517 merged by Jcrespo:
Extra grants on m1 databases for dbproxy1006

https://gerrit.wikimedia.org/r/300517

jcrespo mentioned this in rOPUP5720caa41bc8: Extra grants on m1 databases for dbproxy1006.Jul 22 2016, 10:34 AM
Stashbot subscribed.Jul 22 2016, 10:47 AM

Mentioned in SAL [2016-07-22T10:47:31Z] <jynus> reimage dbproxy1007 T140983

gerritbot added a comment.Jul 22 2016, 10:56 AM

Change 300520 had a related patch set uploaded (by Jcrespo):
Add extra grants on s2 for dbproxy1007

https://gerrit.wikimedia.org/r/300520

jcrespo mentioned this in rOPUP367ffb6ce774: Add extra grants on s2 for dbproxy1007.Jul 22 2016, 10:59 AM
gerritbot added a comment.Jul 22 2016, 10:59 AM

Change 300520 merged by Jcrespo:
Add extra grants on s2 for dbproxy1007

https://gerrit.wikimedia.org/r/300520

Stashbot added a comment.Jul 22 2016, 11:12 AM

Mentioned in SAL [2016-07-22T11:12:12Z] <jynus> reimage dbproxy1009 T140983

gerritbot added a comment.Jul 22 2016, 11:24 AM

Change 300521 had a related patch set uploaded (by Jcrespo):
Add grants on m4 databases for dbproxy1009

https://gerrit.wikimedia.org/r/300521

jcrespo mentioned this in rOPUPfc3625bf6349: Add grants on m4 databases for dbproxy1009.Jul 22 2016, 11:29 AM
gerritbot added a comment.Jul 22 2016, 11:37 AM

Change 300521 merged by Jcrespo:
Add grants on m4 databases for dbproxy1009

https://gerrit.wikimedia.org/r/300521

jcrespo mentioned this in rOPUP871b1040cc06: Add grants on m4 databases for dbproxy1009.Jul 22 2016, 11:39 AM
gerritbot added a comment.Jul 22 2016, 12:36 PM

Change 300529 had a related patch set uploaded (by Jcrespo):
Add extra grants needed on m5 for the dbproxy1010

https://gerrit.wikimedia.org/r/300529

gerritbot added a comment.Jul 22 2016, 12:39 PM

Change 300529 merged by Jcrespo:
Add extra grants needed on m5 for the dbproxy1010

https://gerrit.wikimedia.org/r/300529

jcrespo mentioned this in rOPUP2fbb74ebfdca: Add extra grants needed on m5 for the dbproxy1010.Jul 22 2016, 12:41 PM
jcrespo closed this task as Resolved.Jul 28 2016, 5:35 PM

No more issues from dbproxy1002, and we now have proxy redundancy.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits