Page MenuHomePhabricator

alert when tiles are not re-generated on the maps cluster
Closed, ResolvedPublic

Description

We discovered that T175123 prevents tiles from being re-generated when updates are received from OSM. This can be seen on grafana. Putting in place an alert on that graph should be possible.

Note: at the moment, tile generation is aggregated for all clusters, which makes alerting on it not all that great. It is probably possible to prefix metrics with the cluster name, but some investigation is needed.

Note: since tile generation is done at relatively low frequency (daily at the moment), this needs to be taken into account to tune the alerting. Response time on those alerts is also fairly un-demanding: not generating tiles for a few days does not cause any major issue.

Event Timeline

debt triaged this task as High priority.Sep 12 2017, 7:06 PM
debt moved this task from Backlog to To-do on the Maps-Sprint board.

Change 410136 had a related patch set uploaded (by Gehel; owner: Gehel):
[operations/puppet@production] maps: icinga alert if tiles are not being generated

https://gerrit.wikimedia.org/r/410136

Change 410136 merged by Gehel:
[operations/puppet@production] maps: icinga alert if tiles are not being generated

https://gerrit.wikimedia.org/r/410136

Gehel claimed this task.

Alerting on tiles is deployed and the test is green.