Page MenuHomePhabricator

Request to enable critical alert notifications for maps-warper instance on shinken.wmflabs.org
Closed, ResolvedPublic

Description

Hello,

I'd like to receive critical notifications of low disk space, high load etc on the maps-warper instance via shinken.wmflabs.org

In the past the instance has run out of disk space so we'd like to be sure that it would let us know before that happens! I've cleaned out unused kernels and will move the db data files to another filesystem so this is now less probable, but would be wise to have this in place for the future together with other variables.

I've been keeping a manual eye on it so far via https://tools.wmflabs.org/nagf and http://graphite.wmflabs.org/

Generally, it would be good to get the default set thresholds established and I imagine I can customize these.

Event Timeline

Chippyy assigned this task to yuvipanda.
Chippyy raised the priority of this task from to Medium.
Chippyy updated the task description. (Show Details)
Chippyy added projects: Maps, Cloud-Services.
Chippyy added subscribers: Chippyy, Susannaanas.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I've set up http://graphite.wmflabs.org/dashboard/#maps_warper1 which may be the kind of persistent monitoring we are after - so but now we'd just need some alerts if possible.

Gehel claimed this task.
Gehel added a subscriber: Gehel.

I'm closing this as it looks abandoned. Feel free to re-open if there is still work needed on this.