Page MenuHomePhabricator
Create Task
Maniphest T309485

db2088 crashed
Closed, ResolvedPublic
Actions

Description

db2088 crashed on Saturday:

12:55:21 <+icinga-wm> PROBLEM - Host db2088 is DOWN: PING CRITICAL - Packet loss = 100%

Details

SubjectRepoBranchLines +/-
db2088: Disable notificationsoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects

Event Timeline

Marostegui created this task.May 30 2022, 5:30 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 30 2022, 5:30 AM
Marostegui triaged this task as Medium priority.May 30 2022, 5:31 AM
Marostegui moved this task from Triage to In progress on the DBA board.
gerritbot added a comment.May 30 2022, 5:32 AM

Change 801171 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2088: Disable notifications

https://gerrit.wikimedia.org/r/801171

gerritbot added a project: Patch-For-Review.May 30 2022, 5:32 AM

Change 801171 merged by Marostegui:

[operations/puppet@production] db2088: Disable notifications

https://gerrit.wikimedia.org/r/801171

Stashbot added a comment.May 30 2022, 5:35 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-30T05:35:00Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2088 (s1 and s2) T309485', diff saved to https://phabricator.wikimedia.org/P28913 and previous config saved to /var/cache/conftool/dbconfig/20220530-053459-marostegui.json

Marostegui assigned this task to Papaul.May 30 2022, 5:37 AM
Marostegui added a project: ops-codfw.
Marostegui added a subscriber: Papaul.

@Papaul db2088's mgmt interface is also unavailable so I cannot check the logs and/or if the host is up and the network failed.
Can you check on-site?

Thank you!

Maintenance_bot added a project: SRE.May 30 2022, 6:29 AM
Maintenance_bot removed a project: Patch-For-Review.
Papaul moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.May 31 2022, 3:18 PM
Papaul reassigned this task from Papaul to Marostegui.May 31 2022, 5:26 PM

I removed the power for 10 minutes, the server came backup. IDRAC log not showing any HW issues. I upgrade the BIOS and IDRAC on the node. The server is back up.

Marostegui added a comment.Jun 1 2022, 5:02 AM

Thanks Papaul. I can indeed access the host now.
MySQL seems to be fine.

I am going to repool this host once it catches up and close this. If it happens again, we can probably decommission it as it is scheduled for refresh and the replacement hardware has been ordered and will arrive in a few months. We'll probably not have a DC switchover before that anyways.

Stashbot added a comment.Jun 2 2022, 5:14 AM

Mentioned in SAL (#wikimedia-operations) [2022-06-02T05:14:52Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2088 (s1 and s2) T309485', diff saved to https://phabricator.wikimedia.org/P29327 and previous config saved to /var/cache/conftool/dbconfig/20220602-051451-marostegui.json

Marostegui closed this task as Resolved.Jun 2 2022, 5:15 AM
Marostegui reassigned this task from Marostegui to Papaul.

db2088 is back in sync with both s1 and s2 master. I have repooled it. Closing this for now. If it happens again we should probably just decommission it.
Thank you Papaul!

Maintenance_bot moved this task from In progress to Done on the DBA board.Jun 2 2022, 5:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL