Page MenuHomePhabricator
Create Task
Maniphest T322987

db2173 crashed and didn't alert
Closed, ResolvedPublic
Actions

Description

On Saturday 12th db2173 (sanitarium master) crashed:

19:42:35 <+icinga-wm> PROBLEM - Host db2173 is DOWN: PING CRITICAL - Packet loss = 100%
19:45:31 <+icinga-wm> PROBLEM - MariaDB Replica IO: s1 on db2094 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl@db2173.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 86400 message: Cant connect to MySQL server on db2173.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica

Apparently this didn't page anyone.
@fgiunchedi can you help us understand why we didn't get a page? Notifications are definitely enabled

Details

SubjectRepoBranchLines +/-
mariadb: Make codfw hosts ping when going downoperations/puppetproduction+4 -0
db2094: Disable notificationsoperations/puppetproduction+1 -0
db2173: Disable notificationsoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.Nov 14 2022, 6:06 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 14 2022, 6:06 AM
• Marostegui triaged this task as Medium priority.Nov 14 2022, 6:06 AM
• Marostegui moved this task from Triage to In progress on the DBA board.

I have manually depooled this host now

gerritbot added a comment.Nov 14 2022, 6:07 AM

Change 856199 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2173: Disable notifications

https://gerrit.wikimedia.org/r/856199

gerritbot added a project: Patch-For-Review.Nov 14 2022, 6:07 AM

Change 856199 merged by Marostegui:

[operations/puppet@production] db2173: Disable notifications

https://gerrit.wikimedia.org/r/856199

• Marostegui added a comment.Nov 14 2022, 6:12 AM
-------------------------------------------------------------------------------
Record:      6
Date/Time:   11/12/2022 17:40:28
Source:      system
Severity:    Critical
Description: CPU 1 MEM345 VTT PG voltage is outside of range.
-------------------------------------------------------------------------------
Record:      7
Date/Time:   11/12/2022 17:40:28
Source:      system
Severity:    Critical
Description: CPU 1 MEM345 VPP PG voltage is outside of range.
-------------------------------------------------------------------------------
Record:      8
Date/Time:   11/12/2022 17:41:47
Source:      system
Severity:    Critical
Description: The system board Pfault fail-safe voltage is outside of range.
-------------------------------------------------------------------------------
• Marostegui renamed this task from db2173 crashed to db2173 crashed and didn't alert.Nov 14 2022, 6:12 AM

Created T322988: db2173 HW errors for the HW on-site troubleshooting specific part

gerritbot added a comment.Nov 14 2022, 6:22 AM

Change 856375 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2094: Disable notifications

https://gerrit.wikimedia.org/r/856375

gerritbot added a comment.Nov 14 2022, 6:23 AM

Change 856375 merged by Marostegui:

[operations/puppet@production] db2094: Disable notifications

https://gerrit.wikimedia.org/r/856375

Maintenance_bot removed a project: Patch-For-Review.Nov 14 2022, 6:30 AM
RhinosF1 subscribed.Nov 14 2022, 7:57 AM
gerritbot added a comment.Nov 15 2022, 11:35 AM

Change 856939 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Make codfw hosts ping when going down

https://gerrit.wikimedia.org/r/856939

gerritbot added a project: Patch-For-Review.Nov 15 2022, 11:35 AM
• Marostegui added a subscriber: jbond.Nov 15 2022, 11:36 AM

So I think this host didn't alert because of this: https://gerrit.wikimedia.org/r/c/operations/puppet/+/736415

Per that commit message, sanitarium masters would only page in active DCs, which was a good approach as codfw wasn't active and we don't have any clouddb hosts hanging in codfw, but we should change that now.

I have put this up, but it really needs good reviews cause I am not sure if it will have the desired effect, I will get @jbond and @fgiunchedi to review: https://gerrit.wikimedia.org/r/c/operations/puppet/+/856939

gerritbot added a comment.Nov 15 2022, 11:56 AM

Change 856939 merged by Marostegui:

[operations/puppet@production] mariadb: Make codfw hosts ping when going down

https://gerrit.wikimedia.org/r/856939

• Marostegui added a parent task: T233684: Make primary DB masters page on HOST DOWN alert.Nov 15 2022, 12:07 PM
• Marostegui closed this task as Resolved.Nov 15 2022, 1:25 PM
• Marostegui claimed this task.
In T322987#8395656, @gerritbot wrote:

Change 856939 merged by Marostegui:

[operations/puppet@production] mariadb: Make codfw hosts ping when going down

https://gerrit.wikimedia.org/r/856939

I believe merging this patch has solved this. So closing for now.

• Marostegui added a comment.Nov 15 2022, 1:25 PM

The host itself needs to be fixed, that that has its own task: T322988

Maintenance_bot moved this task from In progress to Done on the DBA board.Nov 15 2022, 1:29 PM
Papaul closed subtask T322988: db2173 HW errors as Resolved.Nov 30 2022, 6:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL