| ssingh | |
| Apr 23 2024, 12:26 AM |
| F51422232: Screenshot 2024-05-09 at 10.31.55 AM.png | |
| May 9 2024, 2:36 PM |
We got a page for this today evening:
PROBLEM Host db1246 #page - PING CRITICAL - Packet loss = 100%
I depooled it using dbctl. Unrelated to that, it came back up:
RECOVERY Host db1246 #page - PING WARNING - Packet loss = 90%, RTA = 162.36 ms
Given the above, I will still let it depooled overnight and let the team take over.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Jclark-ctr | T363119 db1246 crashed | |||
| Resolved | • Kormat | T364552 Fully format and reclone db1246 |
it seems to be a hardware issue as well:
------------------------------------------------------------------------------- Record: 51 Date/Time: 04/23/2024 00:19:19 Source: system Severity: Critical Description: The system board BP1 PG voltage is outside of range. ------------------------------------------------------------------------------- Record: 52 Date/Time: 04/23/2024 00:19:39 Source: system Severity: Ok Description: The system board BP1 PG voltage is within range. -------------------------------------------------------------------------------
@Jclark-ctr this is the third time this host crashes with the same exact HW error see: T361968 T359940 - hopefully Dell won't ask us again to upgrade firmware and BIOS and instead replace whatever piece of HW is broken because it seems pretty clear that there's something going on there.
@Marostegui "At the creation of ticket i requested to not repeat any troubleshooting steps the where not effective"
followed up with dell again they should be sending out parts shortly
Change #1026083 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] installserver: Allowing formatting db1246
Change #1026083 merged by Marostegui:
[operations/puppet@production] installserver: Allowing formatting db1246
I have merged the above patch. It is very likely that this host will have its filesystem corrupted once it is back up - this was the case in the previous two crashes with that same hardware error.
We'll need to reimage and reclone this host.
Change #1026259 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1246: Disable notifications
Change #1026259 merged by Marostegui:
[operations/puppet@production] db1246: Disable notifications
Friday dell agreed to replace Backplane and cables. shipped out Monday expected arrival Tuesday.
Replaced Backplane : cable that connects raid card<-> backplane / power control board. I did find a cable with a loose pin on the power control board (not replaced) but will be reaching out to Dell regarding it it has been reseated in connector and should be fine for the time being
You believe it is all good for us now to start getting this host back to production or you still want to test something else?
I believe we are good to reimage server OS looks corrupt. if you could just wait till tomorrow to put back in production while i wait for Dell to respond if they will send out new cable.
Absolutely - just close this task once your part is done and we will take it from there
@Marostegui you can put server back in rotation even though i uploaded multiple photos yesterday to Dell. They replied this morning requesting part number to send correct part
Thanks John, I will create a subtask for us to work on the formatting, reimage and recloning. Will leave this open until you've finished your side.
The replacement cable did just arrive yesterday. After multiple back and forth with dell Can we leave this open for 1 more week make sure error will not return. leave server running and I will reach out to you for downtime next week for replacement.
@ABran-WMF can you coordinate with @Jclark-ctr to schedule downtime for this host whenever he needs it?
Mentioned in SAL (#wikimedia-operations) [2024-06-05T12:49:19Z] <arnaudb@cumin1002> dbctl commit (dc=all): 'depool db1246 T363119', diff saved to https://phabricator.wikimedia.org/P64101 and previous config saved to /var/cache/conftool/dbconfig/20240605-124918-arnaudb.json
leaving the host depooled until tomorrow to see if it stays stable, will close the task upon repool.