Today some services running on cloudmetrics1002 failed. When tried to access the server by either ssh or the serial console, it failed. We had to force-reboot the system.
There is no information on logs about what happened. A hardware issue is suspected.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | RobH | T161750 eqiad: (1) hardware access request for dedicated labmon1002 | |||
| Unknown Object (Task) | |||||
| Resolved | • chasemp | T165784 rack/setup/install labmon1002 | |||
| Duplicate | None | T275605 cloudmetrics1002: mysterious issue |
Looking into some graphs, it seems that at some point, all io to the disks stops, and the CPU starts running, bringing the load higher and higher:
CPU:
https://grafana.wikimedia.org/dashboard/snapshot/iCua3QJKbTtDuA1jdYl1zhFbmKQxzYwr?orgId=1
Disks io:
https://grafana.wikimedia.org/dashboard/snapshot/jVS71tjV2O76jcWylJofUVONucW5bG8D
Load:
https://grafana.wikimedia.org/dashboard/snapshot/NtjdLQQbDH1ZwLW1HAWg1Q0eCL4r9WB2
This happened again just now. I forced a reboot via mgmt and it seems to be back. The perf graphs show the same shape as they did on Monday.
Mentioned in SAL (#wikimedia-cloud) [2021-04-29T15:11:52Z] <dcaro> hard rebooting cloudmetrics1002, got hung again (T275605)
Mentioned in SAL (#wikimedia-operations) [2021-04-30T15:25:22Z] <bstorm> hard rebooting cloudmetrics1002 T275605
Icinga downtime set by dcaro@cumin1001 for 2:00:00 1 host(s) and their services with reason: Flaky host
cloudmetrics1002.eqiad.wmnet
Icinga downtime set by dcaro@cumin1001 for 2 days, 0:00:00 1 host(s) and their services with reason: Flaky host
cloudmetrics1002.eqiad.wmnet
Mentioned in SAL (#wikimedia-cloud) [2021-05-04T13:19:46Z] <dcaro> rebooting cloudmetrics1002, got stuck again (T275605)
I am looking at updating firmware first will have to download latest version of spp off site. with more stable internet
Mentioned in SAL (#wikimedia-operations) [2021-05-09T10:52:40Z] <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 180 days, 0:00:00 on cloudmetrics1002.eqiad.wmnet with reason: T275605
Mentioned in SAL (#wikimedia-operations) [2021-05-09T10:52:50Z] <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 180 days, 0:00:00 on cloudmetrics1002.eqiad.wmnet with reason: T275605
Mentioned in SAL (#wikimedia-cloud) [2021-05-09T10:53:06Z] <arturo> icinga-downtime cloudmetrics1002 for 3 months (T275605)
Change 687585 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/dns@master] grafana-labs: point to cloudmetrics1001
Change 687585 merged by Arturo Borrero Gonzalez:
[operations/dns@master] grafana-labs: point to cloudmetrics1001
Change 688213 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/dns@master] prometheus-labmon: point to cloudmetrics1001
Change 688213 merged by Arturo Borrero Gonzalez:
[operations/dns@master] prometheus-labmon: point to cloudmetrics1001
@Jclark-ctr @wiki_willy were we able to upgrade this firmware and see if that helped?