cp3037 is unreachable via both production and management network interfaces.
production interface appears to be physically up as reported by the switch, but is not responding to ICMP or SSH.
management interface is currently replying to ICMP and 3way handshake on 22/tcp but we are unable to get a ssh session.
a manual power drain is required to recover access to the server.
Mentioned in SAL (#wikimedia-operations) [2018-06-12T09:41:55Z] <vgutierrez> cp3037 has been depooled due to unknown hardware issues T196974
The host and its management interface are back online after a power drain performed by remote hands.
It seems like we're looking at a thermal issue, here are kernel logs at the time of the crash:
Jun 12 06:26:51 cp3037 kernel: [8169606.570606] CPU13: Package temperature above threshold, cpu clock throttled (total events = 1) [...] Jun 12 07:11:51 cp3037 kernel: [8172306.980868] CPU13: Package temperature above threshold, cpu clock throttled (total events = 181920) Jun 12 07:11:51 cp3037 kernel: [8172306.991163] CPU13: Package temperature/speed normal
And then silence.
Mentioned in SAL (#wikimedia-operations) [2018-07-02T11:28:28Z] <ema> repool cp3037 T196974