Thanks for filing this!

There have been several reports of mw2448 going offline this year: T345597, T334429 + T334420 (probably the same event)

Mentioned in SAL (#wikimedia-operations) [2023-12-18T23:40:58Z] <taavi> conftool codfw/appserver/nginx/mw2448.codfw.wmnet: pooled changed yes => inactive # T353679, not sure why it was not logged automatically

I've set this host as pooled=inactive in conftool. Tagging serviceops since this is their host and ops-codfw directly given the history of this host.

Thanks @taavi for setting this host to inactive. The CPU 1 machine check error was also logged one more time,

-------------------------------------------------------------------------------
Record:      9
Date/Time:   12/18/2023 20:52:16
Source:      system
Severity:    Critical
Description: CPU 1 machine check error detected.
-------------------------------------------------------------------------------

surrounded by a lot of An OEM diagnostic event occurred. as well as 1 instance of

-------------------------------------------------------------------------------
Record:      8
Date/Time:   12/18/2023 20:52:16
Source:      system
Severity:    Ok
Description: A problem was detected related to the previous server boot.
-------------------------------------------------------------------------------

Side note: I love how the detection of a problem has a severity of OK.

RAC log also reports these 2 interesting entries. (ractime is correct, so timestamps should be trusteable)

--------------------------------------------------------------------------------
SeqNumber       = 6926
Message ID      = SYS1005
Category        = Audit
AgentID         = iDRAC
Severity        = Information
Timestamp       = 2023-12-18 19:49:41
Message         = The server power action is initiated because the host device initiated a warm-reset operation.
Message Arg   1 = the host device initiated a warm-reset operation
FQDD            = iDRAC.Embedded.1
--------------------------------------------------------------------------------
SeqNumber       = 6925
Message ID      = SYS1003
Category        = Audit
AgentID         = DE
Severity        = Information
Timestamp       = 2023-12-18 19:48:55
Message         = System CPU Resetting.
FQDD            = iDRAC.Embedded.1#HostPowerCtrl
--------------------------------------------------------------------------------

@Jhancock.wm, @Papaul, I 've refrained from taking any action like rebooting the server so you can debug it further if you wish. I should note that this is the 3rd ticket for this host in 8 months and apparently the 4th occurrence (I see we had 2 events in 1 ticket).

Should the icinga alert for host down and related service alerts in icinga and alertmanager be silenced given it's known and there is a task?

Icinga downtime and Alertmanager silence (ID=c526ca54-768b-461b-9bc7-1666a80b4153) set by cgoubert@cumin2002 for 7 days, 0:00:00 on 1 host(s) and their services with reason: hw failure

mw2448.codfw.wmnet

Icinga downtime and Alertmanager silence (ID=caa5cb6a-6361-4151-9946-cb47394c0eec) set by cgoubert@cumin2002 for 14 days, 0:00:00 on 1 host(s) and their services with reason: hw failure

mw2448.codfw.wmnet

I've put in a dispatch request with Dell to replace the CPU. I have power cycled the server and replaced the CMOS battery. which fixed one of the errors. will update when part arrives.

@Jhancock.wm the easiest and safest way to reconfigure a BIOS is to run the sre.hosts.provision cookbook like it was a new host just with some options to skip unnecessary steps like --no-dhcp, --no-switch, --no-users. See the --help for more details.

thank you. I'll remember that one.

Mentioned in SAL (#wikimedia-operations) [2023-12-23T15:40:26Z] <taavi> fix date-time on mw2448 (which thought it is the year 2098) by manually setting it once and then restarting systemd-timesyncd.service after bios was reset in T353679

The machine isn't pooled yet into traffic. There is an alert for frequent changes due to puppet run. Indeed the following happens at every puppet run

Notice: /Stage[main]/Cpufrequtils/Exec[cpufrequtils_reload]/returns: executed successfully (corrective)

cpufreq-info, the binary that is executed as the check triggering that reload spews out the following 48 times (1 per CPU)

analyzing CPU 0:
  no or unknown cpufreq driver is active on this CPU
  maximum transition latency: 4294.55 ms.

I 've run the sre.hosts.provision cookbook as instructed by @Volans and indeed it applied

[snip]
Updated value for attribute BIOS.Setup.1-1 -> ProcPwrPerf (marked Set On Import to True): SysDbpm => OsDbpm 
Updated value for attribute BIOS.Setup.1-1 -> SysProfile: PerfPerWattOptimizedDapc => PerfPerWattOptimizedOs 
Updated value for attribute BIOS.Setup.1-1 -> UsbPorts: AllOn => OnlyBackPortsOn
Updated value for attribute iDRAC.Embedded.1 -> WebServer.1#HostHeaderCheck: Disabled => Enabled

A reboot later (which was probably not warranted, I just did it anyway) I see everything ok.

I ran scap pull to synchronize the deployment of mediawiki with other mw hosts since the host was as as inactive and was getting the latest and I am pooling all services on mw2448.