hw troubleshooting: CPU1 thermal fault for wdqs1015.eqiad.wmnet
Closed, ResolvedPublicRequest
Actions

Assigned To

Authored By

	RKemper
	Mon, Jun 1, 8:36 PM

Project Tags

Referenced Files

None

Subscribers

RKemper

wiki_willy

Description

Template

FQDN: wdqs1015.eqiad.wmnet
Depooled: Yes
Netbox Status: Failed (https://netbox.wikimedia.org/dcim/devices/4253/)
Urgency: High-Medium (wdqs-main is less robust to host failures given it routinely falls under excessive load)

Issue

wdqs1015 thermal-tripped; SEL/LC log shows an escalating CPU 1 (CPU.Socket.1) thermal fault:

2026-05-28 02:31 — CPU 1 over-temp + throttle, followed by machine-check error (self-recovered)
2026-05-31 09:52 — CPU 1 machine-check error (reboot)
2026-05-31 10:22 — CPU 1 thermal trip (over-temperature) -> host powered off, down since

System Board Inlet Temp is normal (25C), so this is likely localized to CPU 1's cooling, not rack airflow.
Please inspect CPU 1 heatsink/fan seating, thermal paste, and fan health; reseat/replace as needed.
(Host is already downtimed and depooled)

Related Objects

Mentioned In: T428582: decommission wdqs1015.eqiad.wmnet
Mentioned Here: T428582: decommission wdqs1015.eqiad.wmnet
T423314: Q4:rack/setup/install dse-k8s-wdqs100[1-3] (formerly wdqs103[6-8])

Event Timeline

RKemper created this task.Mon, Jun 1, 8:36 PM

Mentioned in SAL (#wikimedia-operations) [2026-06-01T20:37:40Z] <ryankemper@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wdqs1015.eqiad.wmnet with reason: T427852 hw failure

Maintenance_bot added a project: SRE.Mon, Jun 1, 9:29 PM

Jclark-ctr moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Mon, Jun 1, 11:22 PM

This server is out of warranty @RKemper. but I am looking at it right now

@RKemper @wiki_willy I have gone through all decommissioned servers and do not have a matching Intel(R) Xeon(R) Silver 4215 CPU @ 2.50GHz available to replace CPU1.

I also reviewed all active EOL servers and found only three servers with the same CPU. Those servers were scheduled for decommissioning in December 2024.

wdqs1011 Active refreshed in T376670 in Dec '24
wdqs1012 Active refreshed in T376670 in Dec '24
wdqs1013 Active refreshed in T376670 in Dec '24

I did attempt the firmware updates, but after rebooting, the server became unresponsive and will not boot.

At this point, I would need a compatible CPU to test whether the issue is with the motherboard or the CPU itself.

Discussed with @RKemper via IRC. He mentioned that we should decommission this one if the replacement is already here T423314 and is racked and cabled, pending a Puppet fix to image the server.

Jclark-ctr added a project: Data-Platform-SRE.Mon, Jun 8, 3:30 PM