We got paged twice on Sunday due to db1246 crashing and rebooting twice in quick succession. I've depooled it and downtimed it for 48 hours (not quite quickly enough to prevent the second page). It's a replica node in s2 per orchestrator.
I note in passing this node has crashed before (e.g. T374215).
Same problem as always:
------------------------------------------------------------------------------- Record: 16 Date/Time: 03/02/2025 20:45:55 Source: system Severity: Critical Description: The system board BP1 PG voltage is outside of range. ------------------------------------------------------------------------------- Record: 17 Date/Time: 03/02/2025 20:46:10 Source: system Severity: Ok Description: The system board BP1 PG voltage is within range. -------------------------------------------------------------------------------
Change #1124031 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1246: Update notes
Change #1124031 merged by Marostegui:
[operations/puppet@production] db1246: Update notes
Change #1124036 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1246: Disable notifications
Change #1124036 merged by Marostegui:
[operations/puppet@production] db1246: Disable notifications
db1246 has been cloned. I will repool it tomorrow. I am sure it will sooner or later crash again, but we need to see if it is again the same HW error.
Change #1124650 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1246: Enable notifications
Change #1124650 merged by Marostegui:
[operations/puppet@production] db1246: Enable notifications
Repooled
@wiki_willy I emailed Dell about this host (in the existing thread we have with them) but so far there's been no reply. Do you want to keep this ticket open and continue here or you'd rather handle this with them via mail?
Hi @Marostegui - thanks for checking. When I look back at previous email from Dell Support sent in November, MarcoAntonio says "we can temporarily archive the case, and if the issue reappears, you can open this case within 10days by contacting me via email or we can open a new case making reference to this case if any additional support is needed after 10 days, the record of the server is saved in the TAG history." So I have a feeling your email reply on Sunday didn't reopen the case because it was past 10 days.
@VRiley-WMF or @Jclark-ctr - can one of you create a new Dell Support ticket for this with the issue Manuel reported via the email thread "Dell Technologies service request:200579927, Case Summary: System Randomly crashes without hardware errors/ R650XS/ GSR58Y3/ PS-NBD, Asset: GSR58Y3 / [ thread::4CL034vWI5KwjTlOcx1WpD0:: ]" After that, I'm fine with either way, if we track it here in Phabricator or via email. Thanks, Willy
Thanks @wiki_willy - I thought the email was just a thread and not handled via some internal ticketing system.
Let's leave this open for now so we don't forget. If there's a way to CC me on that new case, that'd be appreciated, it is also fine if there's no way to do so.
Thank you!
Opened a new ticket with Dell.
206617456
currently speaking with them about this issue.
Looks like we'll be replacing the motherboard soon. Will update when we know a time it should be arriving
@VRiley-WMF the host is depooled and notifications are disabled. So you can change the mainboard anytime you want, whenever it arrives. I will leave it depooled.
Mentioned in SAL (#wikimedia-operations) [2025-03-11T12:56:02Z] <marostegui> Stop MariaDB on db1246 T387673
Mentioned in SAL (#wikimedia-operations) [2025-03-11T12:57:33Z] <marostegui> Poweroff db1246 T387673
Dell has had a delay in the part being delivered. They estimate that this should be in tomorrow. Postponed until tomorrow.
Change #1129610 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db1246: Enable notifications
Change #1129610 merged by Marostegui:
[operations/puppet@production] db1246: Enable notifications