Set up grafana alerting for services
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• GWicke
	Apr 12 2017, 3:00 AM

Description

Grafana now has alerting capabilities, which are significantly more user friendly than defining icinga alerts on graphite data via puppet. However, per T153167 email notifications in grafana are not operational, and the only way to use grafana alerts is via a per-dashboard indirection in nagios.

In a first step, we would like to be alerted for the following dashboards:

restbase
api-summary
services-alerts
eventbus

For reference, the original commit to set up alerting for performance is this one:

https://github.com/wikimedia/puppet/commit/401973ab7f79fd4567749fe074ccce1d47446581

The alerts should hit 'team-services' in nagios.

Details

	Subject	Repo	Branch	Lines +/-
	Set up grafana dashboard monitoring for services	operations/puppet	production	+21 -0

Customize query in gerrit

Related Objects

Mentioned In: T167830: Extend icinga check to catch 500 errors like those of the 20170613 incident
T123854: Set up action API latency / error rate metrics & alerts
Mentioned Here: T167830: Extend icinga check to catch 500 errors like those of the 20170613 incident
T153167: Configure Grafana for sending e-mail

Event Timeline

• GWicke created this task.Apr 12 2017, 3:00 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 12 2017, 3:00 AM

• GWicke updated the task description. (Show Details)Apr 12 2017, 3:00 AM

• GWicke updated the task description. (Show Details)Apr 12 2017, 3:05 AM

• GWicke updated the task description. (Show Details)Apr 12 2017, 3:22 AM

fgiunchedi triaged this task as Medium priority.Apr 12 2017, 7:58 AM

• GWicke mentioned this in T123854: Set up action API latency / error rate metrics & alerts.Apr 13 2017, 2:23 AM

Halfak mentioned this in T167830: Extend icinga check to catch 500 errors like those of the 20170613 incident.Jun 15 2017, 6:17 PM

@Halfak pointed out that using this to check the retry topic rate for ORES would help them

Our task: T167830: Extend icinga check to catch 500 errors like those of the 20170613 incident

Essentially intermittent 500s were not getting caught by the icinga check quickly enough so we'd like to catch it via ChangeProp since it hits ORES really fast :)

@Pchelolo, @Halfak: If you want to go the grafana route for ease of modification, I would recommend to set up a separate dashboard for ORES retries. You can then make that dashboard alert the right ORES group (only one group per dashboard), and tweak / add alerts as needed in that dashboard.

• GWicke edited projects, added Services (next); removed Services (blocked).Jun 27 2017, 2:25 PM

Change 362567 had a related patch set uploaded (by GWicke; owner: GWicke):
[operations/puppet@production] Set up grafana dashboard monitoring for services

https://gerrit.wikimedia.org/r/362567

gerritbot added a project: Patch-For-Review.Jun 30 2017, 9:39 PM

• GWicke edited projects, added Services (blocked); removed Services (next).Jun 30 2017, 9:46 PM

Change 362567 merged by Alexandros Kosiaris:
[operations/puppet@production] Set up grafana dashboard monitoring for services

https://gerrit.wikimedia.org/r/362567