We've seen a few calls to prometheus-gateway.discovery.wmnet failing because of timeouts. The host itself looks offline or unreachable.
curl: (28) Failed to connect to prometheus-pushgateway.discovery.wmnet port 80: Connection timed out
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | colewhite | T362989 Requests to prometheus pushgateway are timing out | |||
| Resolved | Request | VRiley-WMF | T362990 hw troubleshooting: memory DIMM_B3 multi-bit memory errors for prometheus1005 |
Event Timeline
Comment Actions
------------------------------------------------------------------------------- Record: 10 Date/Time: 04/19/2024 08:01:45 Source: system Severity: Critical Description: The system memory has faced an uncorrectable multi-bit memory errors in the non-execution path of a memory device at the location DIMM_B3. ------------------------------------------------------------------------------- ... lots of self heal operations ... ------------------------------------------------------------------------------- Record: 492 Date/Time: 04/19/2024 08:08:20 Source: system Severity: Critical Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_B3. -------------------------------------------------------------------------------
Seems like a bad stick of memory.
Comment Actions
Change #1022027 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/dns@master] promote prometheus1006 as pushgateway primary
Comment Actions
Change #1022028 had a related patch set uploaded (by Cwhite; author: Cwhite):
[operations/puppet@production] prometheus: promote prometheus1006 to pushgateway duty
Comment Actions
Change #1022028 merged by Cwhite:
[operations/puppet@production] prometheus: promote prometheus1006 to pushgateway duty
Comment Actions
Change #1022027 merged by Cwhite:
[operations/dns@master] promote prometheus1006 as pushgateway primary