codfw S2 db2207 master failed and rebooted, probably due to hardware failure.
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| db2207: Disable notifications | Marostegui | operations/puppet | production | +2 -1 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Marostegui | T432398 codfw S2 db2207 master fail | |||
| Resolved | Marostegui | T432396 Switchover s2 master (db2207 -> db2204) |
Event Timeline
Comment Actions
Time line (UTC):
[19:49:21] <+icinga-wm> PROBLEM - Host db2207 #page is DOWN: PING CRITICAL - Packet loss = 100% [19:50:07] <marostegui> That's s2 master codfw [19:58:36] <@marostegui> btw the user impact is gone now [20:02:18] <@marostegui> switchover is done [20:02:22] <@marostegui> I am cleaning up things now
Comment Actions
HW logs
-------------------------------------------------------------------------------- SeqNumber = 11798 Message ID = SYS1003 Category = Audit AgentID = DE Severity = Information Timestamp = 2026-07-16 19:48:45 Message = System CPU Resetting. FQDD = iDRAC.Embedded.1#HostPowerCtrl -------------------------------------------------------------------------------- SeqNumber = 11797 Message ID = SYS1000 Category = Audit AgentID = DE Severity = Information Timestamp = 2026-07-16 19:48:27 Message = System is turning on. FQDD = iDRAC.Embedded.1#HostPowerCtrl -------------------------------------------------------------------------------- SeqNumber = 11796 Message ID = SYS1001 Category = Audit AgentID = DE Severity = Information Timestamp = 2026-07-16 19:47:59 Message = System is turning off. FQDD = iDRAC.Embedded.1#HostPowerCtrl -------------------------------------------------------------------------------- SeqNumber = 11795 Message ID = SYS1003 Category = Audit AgentID = DE Severity = Information Timestamp = 2026-07-16 19:47:59 Message = System CPU Resetting. FQDD = iDRAC.Embedded.1#HostPowerCtrl -------------------------------------------------------------------------------- SeqNumber = 11794 Message ID = SYS1005 Category = Audit AgentID = iDRAC Severity = Information Timestamp = 2026-07-16 19:47:59 Message = The server power action is initiated because the host device initiated a power-down operation. Message Arg 1 = the host device initiated a power-down operation FQDD = iDRAC.Embedded.1 -------------------------------------------------------------------------------- SeqNumber = 11794 Message ID = SYS1005 Category = Audit AgentID = iDRAC Severity = Information Timestamp = 2026-07-16 19:47:59 Message = The server power action is initiated because the host device initiated a power-down operation. Message Arg 1 = the host device initiated a power-down operation FQDD = iDRAC.Embedded.1 -------------------------------------------------------------------------------- SeqNumber = 11793 Message ID = SYS1003 Category = Audit AgentID = DE Severity = Information Timestamp = 2026-07-16 19:47:43 Message = System CPU Resetting. FQDD = iDRAC.Embedded.1#HostPowerCtrl -------------------------------------------------------------------------------- SeqNumber = 11792 Message ID = SYS1005 Category = Audit AgentID = iDRAC Severity = Information Timestamp = 2026-07-16 19:47:42 Message = The server power action is initiated because the management controller initiated a hard-reset operation. Message Arg 1 = the management controller initiated a hard-reset operation FQDD = iDRAC.Embedded.1 -------------------------------------------------------------------------------- SeqNumber = 11791 Message ID = RAC0703 Category = Audit AgentID = RACLOG Severity = Information Timestamp = 2026-07-16 19:47:40 Message = Requested system hardreset. FQDD = iDRAC.Embedded.1 --------------------------------------------------------------------------------
@Jhancock.wm I guess dell will want us to upgrade bios and firmware because of this crash?
We are seeing lots of these crashes lately across multiple Dell hosts unfortunately, which is causing user impact cc @wiki_willy
Comment Actions
idrac is updated. want conformation it's safe to reboot for bios update before i start. jic
Comment Actions
bios updated. gonna get a tsr report for this one and add it to the techinical support ticket of the one already in motion.
Comment Actions
@Jhancock.wm Thanks for the update. It looks like MariaDB is still stopped, any reason not to start it and see if it catches up with replication without any issues?
Comment Actions
I have started MariaDB and replication:
> SELECT ts, file, datacenter FROM heartbeat.heartbeat; +----------------------------+-------------------+------------+ | ts | file | datacenter | +----------------------------+-------------------+------------+ | 2026-07-17T06:04:03.000950 | db1162-bin.007512 | eqiad | | 2026-07-17T06:04:03.001880 | db2204-bin.003348 | codfw | +----------------------------+-------------------+------------+ 2 rows in set (0.001 sec)
The instance will remain depooled though:
% dbctl instance db2207 get
{
"db2207": {
"host_ip": "10.192.32.13",
"note": "",
"port": 3306,
"sections": {
"s2": {
"candidate_master": true,
"percentage": 100,
"pooled": false,
"weight": 300
}
}
},
"tags": "datacenter=codfw"
}Comment Actions
@Marostegui the host caught up with replication earlier this afternoon. It is still depooled for now.
Comment Actions
@Jhancock.wm I am going to repool this host for now to see how it behaves. My stress test didn't make it crash, but let's see if Dell ever replies.
Comment Actions
I am going to close this for now, as there's an email thread with Dell. We can reopen if needed.