db2209 (old s3 master) crashed and the HW logs showed nothing recent.
Can we take this opportunity to upgrade everything that needs upgrade before putting it back in production?
The host is currently up, so please ping me before rebooting.
Thanks
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| db2209: Disable notifications | Marostegui | operations/puppet | production | +2 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Restricted Task | |||||
| Open | Jhancock.wm | T431952 db2209 firmware upgrade after a crash |
Event Timeline
Comment Actions
Change #1309863 had a related patch set uploaded (by Marostegui; author: Marostegui):
[operations/puppet@production] db2209: Disable notifications
Comment Actions
Change #1309863 merged by Marostegui:
[operations/puppet@production] db2209: Disable notifications
Comment Actions
i think i pinged the wrong person @Marostegui but it's updated now. sorry about that.
Comment Actions
@Jhancock.wm this is what i got from idrac logs during the time of the crash:
-------------------------------------------------------------------------------- SeqNumber = 11914 Message ID = SYS1003 Category = Audit AgentID = DE Severity = Information Timestamp = 2026-07-12 15:06:09 Message = System CPU Resetting. FQDD = iDRAC.Embedded.1#HostPowerCtrl -------------------------------------------------------------------------------- SeqNumber = 11913 Message ID = SYS1000 Category = Audit AgentID = DE Severity = Information Timestamp = 2026-07-12 15:05:50 Message = System is turning on. FQDD = iDRAC.Embedded.1#HostPowerCtrl -------------------------------------------------------------------------------- SeqNumber = 11912 Message ID = SYS1001 Category = Audit AgentID = DE Severity = Information Timestamp = 2026-07-12 15:05:22 Message = System is turning off. FQDD = iDRAC.Embedded.1#HostPowerCtrl -------------------------------------------------------------------------------- SeqNumber = 11911 Message ID = SYS1003 Category = Audit AgentID = DE Severity = Information Timestamp = 2026-07-12 15:05:22 Message = System CPU Resetting. FQDD = iDRAC.Embedded.1#HostPowerCtrl -------------------------------------------------------------------------------- SeqNumber = 11910 Message ID = SYS1005 Category = Audit AgentID = iDRAC Severity = Information Timestamp = 2026-07-12 15:05:22 Message = The server power action is initiated because the host device initiated a power-down operation. Message Arg 1 = the host device initiated a power-down operation FQDD = iDRAC.Embedded.1 -------------------------------------------------------------------------------- SeqNumber = 11909 Message ID = SYS1003 Category = Audit AgentID = DE Severity = Information Timestamp = 2026-07-12 15:05:06 Message = System CPU Resetting. FQDD = iDRAC.Embedded.1#HostPowerCtrl -------------------------------------------------------------------------------- SeqNumber = 11908 Message ID = SYS1005 Category = Audit AgentID = iDRAC Severity = Information Timestamp = 2026-07-12 15:05:05 Message = The server power action is initiated because the management controller initiated a hard-reset operation. Message Arg 1 = the management controller initiated a hard-reset operation FQDD = iDRAC.Embedded.1 -------------------------------------------------------------------------------- SeqNumber = 11907 Message ID = RAC0703 Category = Audit AgentID = RACLOG Severity = Information Timestamp = 2026-07-12 15:05:04 Message = Requested system hardreset. FQDD = iDRAC.Embedded.1
Given this host is under warranty, can we contact dell about it and see what they tell us?
Thanks
Comment Actions
I looked into this and had a think. I don't think they're going to replace any parts cause it's the first time it's happened and there are no current errors. but I will submit this to the help desk and see if they can help us figure out what exactly caused the shut down. I'll update when they get back with the first response.
Comment Actions
they tried to tell me i did the TSR report wrong. so doing it again and sending it to them.
Comment Actions
scheduling a zoom meeting with Dell for some reason. suggests they have no idea why it's doing this.
Comment Actions
i gave the dell tech more info but still waiting for the results. will keep bugging them this week.
Comment Actions
Got something back. No findings on their part. He acknowledged that the idrac and bios were update but said to "update the other components" without going into specifics.
Here's the link to the recommended versions for almost everything: https://www.dell.com/support/kbdoc/en-us/000205512/target-revisions-and-runtime#XC_Series_Core_Nutanix
I'm gonna try to get a specific list from him cause I don't really find that helpful.