Page MenuHomePhabricator
Create Task
Maniphest T145607

db1082 hardware check
Closed, ResolvedPublic
Actions

Description

Hello Chris,

db1082.eqiad.wmnet crashed yesterday and we had to power cycle it from the ILO.
So far we haven't been able to figure out why, apparently it had a kernel panic.

The RAID looks healthy at this point, but as @MoritzMuehlenhoff suggests - @Cmjohnson could you perform a hardware check to make sure the RAID controller and/or RAM is healthy?

This server is non pooled so it would be safe to power it off if needed. Just ping me if you need me for something.

Thanks
Manuel.

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• MarosteguiT145533 Investigate db1082 crash
 ResolvedjcrespoT145607 db1082 hardware check

Event Timeline

• Marostegui created this task.Sep 14 2016, 7:20 AM
• Marostegui removed Cmjohnson as the assignee of this task.Sep 14 2016, 7:34 AM
• Marostegui added a project: ops-eqiad.
• Marostegui updated the task description. (Show Details)
Restricted Application added a subscriber: Southparkfan. · View Herald TranscriptSep 14 2016, 7:34 AM
jcrespo renamed this task from Hardware check to db1082 hardware check.Sep 14 2016, 7:35 AM
jcrespo triaged this task as High priority.Sep 14 2016, 10:57 AM

I am going to put this high, because this block putting the server back into production, and that means it will lag for longer, so this is time sensitive.

• Marostegui added a comment.Sep 14 2016, 2:21 PM

Kernel has been upgraded to 4.4.0-2 and a full-upgrade has been performed as well.

• Marostegui added a comment.Sep 15 2016, 7:33 AM

Note:

Replication was started and it went well.
The host was powered off a bit after for a memtest

• Marostegui moved this task from Triage to In progress on the DBA board.Sep 15 2016, 7:59 AM
Cmjohnson added a comment.Sep 15 2016, 1:00 PM

performed a memtest, test came back with zero errors

jcrespo closed this task as Resolved.Sep 15 2016, 1:02 PM
jcrespo claimed this task.

:-( Let's repool, note it for the future, move on.

• Marostegui added a comment.Sep 15 2016, 1:28 PM

Thanks Chris - I will close this ticket and we will keep updating the upstream.

• Marostegui mentioned this in T258336: db1082 crashed.Jul 20 2020, 11:37 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL