Page MenuHomePhabricator

Monitor prometheus exporters "up" status
Closed, ResolvedPublic

Description

We should monitor the "upness" (according to the up metric from Prometheus) of various prometheus exporters we have deployed now. The metric is exported automatically by Prometheus and set to 0 whenever Prometheus is unable to scrape metrics from the given exporter.

In addition to the up metrics, exporters often export the status of the underlying daemon as <daemon>_up and we should monitor that too.

Dashboard linked to the alerts: https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets

Details

Related Gerrit Patches:
operations/puppet : productionprometheus: alert on exporter's 'up' metrics
operations/puppet : productionprometheus: alert on low job availability

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 19 2018, 11:23 AM
fgiunchedi moved this task from Backlog to Up next on the User-fgiunchedi board.Mar 28 2018, 3:49 PM
fgiunchedi moved this task from Up next to Backlog on the User-fgiunchedi board.Oct 9 2019, 11:31 PM
fgiunchedi moved this task from Backlog to Up next on the User-fgiunchedi board.Nov 22 2019, 12:13 PM

Change 552521 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: alert on low job availability

https://gerrit.wikimedia.org/r/552521

Change 552521 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: alert on low job availability

https://gerrit.wikimedia.org/r/552521

fgiunchedi moved this task from Up next to Doing on the User-fgiunchedi board.Nov 26 2019, 3:42 PM

Change 553335 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: alert on exporter's 'up' metrics

https://gerrit.wikimedia.org/r/553335

fgiunchedi updated the task description. (Show Details)Nov 27 2019, 5:39 PM

Change 553335 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: alert on exporter's 'up' metrics

https://gerrit.wikimedia.org/r/553335

fgiunchedi closed this task as Resolved.Nov 28 2019, 10:25 AM
fgiunchedi claimed this task.

All deployed now, boldly resolving