Page MenuHomePhabricator

Set up grafana alerts for JobQueue-EventBus
Closed, ResolvedPublic


The JobQueue-EventBus dashboard is quite useful and we could benefit from setting up alerts on it, for example, if a certain job processing rate suddenly drops to zero, or the backlog grows above a certain level.

Event Timeline

Pchelolo triaged this task as Normal priority.Mar 6 2018, 6:28 PM
Pchelolo created this task.
Restricted Application added a project: Analytics. · View Herald TranscriptMar 6 2018, 6:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 416740 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[operations/puppet@production] Enable grafana alerts for jobqueue-eventbus dashboard.

fdans moved this task from Incoming to Radar on the Analytics board.Mar 8 2018, 6:21 PM

Change 416740 merged by Filippo Giunchedi:
[operations/puppet@production] Enable grafana alerts for jobqueue-eventbus dashboard.

Pchelolo closed this task as Resolved.Mar 22 2018, 7:34 PM
Pchelolo edited projects, added Services (done); removed Patch-For-Review, Services (doing).

We now have a couple of alerts - it should fire when the backlog is too high or when a rule suddenly stops processing messages. Let's see how they will work, but there's nothing to do here on this task anymore. Resolving.