| • ema | |
| May 6 2019, 2:47 PM |
| F28953344: image.png | |
| May 7 2019, 12:10 PM |
| F28951427: Screenshot from 2019-05-07 10-42-11.png | |
| May 7 2019, 8:42 AM |
cp1083 suddenly crashed today at 14:37. Nothing in console. It came back online fine after a power cycle. We should check and see if it has any hardware issues.
SEL was apparently cleared today too, though it's not clear why and by whom.
/admin1-> racadm getsel Record: 1 Date/Time: 06/06/2018 13:30:25 Source: system Severity: Ok Description: Log cleared. ------------------------------------------------------------------------------- /admin1->
Mentioned in SAL (#wikimedia-operations) [2019-05-07T08:39:28Z] <ema> repool cp1083 T222620
Interestingly, there was a memory usage spike right before the host crashed.
I think that is just a strange monitoring artifact. If you zoom in more closely you see the 'cached' timeseries get duplicated for some reason, while the values of each individual timeseries stay the same.
Interestingly, there was a memory usage spike right before the host crashed.
I think that is just a strange monitoring artifact. If you zoom in more closely you see the 'cached' timeseries get duplicated for some reason, while the values of each individual timeseries stay the same.
I agree on the monitoring artifact, it looks like due to our use of old/new names for node_exporter metrics new or old expressions via recording rules. My guess is that the host went offline before prometheus could evaluate both the "real" metric from the host and the recording rules one.
It's been up for ~15 days now without incident, but depooled for frontend traffic. Re-pooling it today to see if we can get a recurrence or not.
