(part of https://wikitech.wikimedia.org/wiki/Incident_documentation/20190425-prometheus)
It's known that prometheus restarts can cause strange monitoring artifacts. A one-time Icinga notification on IRC might be nice. Maybe we just look for process uptime < 10 minutes?
Checking process uptime sounds good to me, if I understood correctly (the one-time icinga notifcation) the alert would self-recover once uptime is no longer in breach, is that correct?
I can think of at least two ways for implementation, namely by checking activation time of systemd unit(s) or checking process_start_time_seconds prometheus metric from the prometheus server itself.
Change 508011 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] prometheus: one-shot alert on restarts
Change 508011 merged by CDanis:
[operations/puppet@production] prometheus: one-shot alert on restarts
Change 508356 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] prometheus uptime alert: fix query
Change 508356 merged by CDanis:
[operations/puppet@production] prometheus uptime alert: fix query
We now have IRC alerting based on scraping each prometheus for its process_start_time_seconds metric.