Page MenuHomePhabricator
Create Task
Maniphest T322988

db2173 HW errors
Closed, ResolvedPublic
Actions

Description

db2173 crashed past Saturday with the following errors on its ILO:

-------------------------------------------------------------------------------
Record:      6
Date/Time:   11/12/2022 17:40:28
Source:      system
Severity:    Critical
Description: CPU 1 MEM345 VTT PG voltage is outside of range.
-------------------------------------------------------------------------------
Record:      7
Date/Time:   11/12/2022 17:40:28
Source:      system
Severity:    Critical
Description: CPU 1 MEM345 VPP PG voltage is outside of range.
-------------------------------------------------------------------------------
Record:      8
Date/Time:   11/12/2022 17:41:47
Source:      system
Severity:    Critical
Description: The system board Pfault fail-safe voltage is outside of range.
-------------------------------------------------------------------------------

Details

SubjectRepoBranchLines +/-
db2173: Enable notificationsoperations/puppetproduction+0 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.Nov 14 2022, 6:14 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 14 2022, 6:14 AM
• Marostegui triaged this task as Medium priority.Nov 14 2022, 6:14 AM
• Marostegui mentioned this in T322987: db2173 crashed and didn't alert.
• Marostegui added a comment.Nov 14 2022, 6:21 AM

I am trying to poweron the host but it is not working:

racadm>>serveraction powerup
Server power operation initiated successfully
racadm>>serveraction powerstatus
Server power status: OFF
Maintenance_bot added a project: SRE.Nov 14 2022, 6:29 AM
Papaul moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Nov 14 2022, 4:19 PM
• Marostegui added a project: DBA.Nov 15 2022, 1:25 PM
• Marostegui moved this task from Triage to Blocked on the DBA board.
Papaul added a comment.Nov 17 2022, 10:53 PM

I have a case open with Dell

Service Request: 1115331653
• Marostegui added a parent task: T321130: Add column cuc_private to cu_changes on wmf wikis.Nov 22 2022, 9:47 AM

Once this host is back we need to make sure we apply T321130: Add column cuc_private to cu_changes on wmf wikis (enwiki)

• Marostegui mentioned this in T321130: Add column cuc_private to cu_changes on wmf wikis.Nov 23 2022, 7:59 AM
Papaul closed this task as Resolved.Nov 30 2022, 6:42 PM

@Marostegui main board replaced. The server is back up running. Sorry it took this long to get this fix.

Thanks

• Marostegui mentioned this in T321126: Add column 'cul_actor' and index cul_actor_time to cu_log on wmf wikis.Dec 1 2022, 6:11 AM

Thank you Papaul!

gerritbot added a comment.Dec 5 2022, 6:12 AM

Change 864328 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2173: Enable notifications

https://gerrit.wikimedia.org/r/864328

gerritbot added a project: Patch-For-Review.Dec 5 2022, 6:12 AM
gerritbot added a comment.Dec 5 2022, 6:14 AM

Change 864328 merged by Marostegui:

[operations/puppet@production] db2173: Enable notifications

https://gerrit.wikimedia.org/r/864328

• Marostegui added a comment.Dec 5 2022, 6:14 AM

Host being repooled automatically.

Maintenance_bot removed a project: Patch-For-Review.Dec 5 2022, 6:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL