icinga notification if elevated writing to badpass.log
Open, NormalPublic

Description

It'd be useful to know if there's more writing than normal to this log... Similar for general logs/errors

Reedy created this task.Nov 8 2016, 9:56 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 8 2016, 9:56 PM
Tgr added a subscriber: bd808.Nov 17 2016, 5:17 AM

@bd808 pointed to the Kibana watcher plugin: https://github.com/elasticfence/kaae

bd808 added a comment.Nov 17 2016, 4:44 PM

@bd808 pointed to the Kibana watcher plugin: https://github.com/elasticfence/kaae

If we wanted to try this plugin out, I think we would want to setup a new kibana instance somewhere. The current logstash.wikimedia.org kibana is actually 3 backend servers running behind an LB. This raises 2 problems: you don't know which of the 3 you are getting round-robbined to; if they share state via the elasticsearch cluster (which looks like how things are stored) then you would potentially get 3 alerts for each watch that fired.

We could (also) export the number of lines written to badpass to graphite and setup an icinga alert. The metric would be public though and so will the alert, I don't think it would be particularly troubling.

bd808 added a comment.Nov 17 2016, 6:00 PM

We could (also) export the number of lines written to badpass to graphite and setup an icinga alert. The metric would be public though and so will the alert, I don't think it would be particularly troubling.

https://graphite.wikimedia.org/render?from=-2hours&until=now&width=400&height=250&target=logstash.rate.mediawiki.badpass.INFO.count&_uniq=0.9259289370548927&title=logstash.rate.mediawiki.badpass.INFO.count

@bd808 looks good! I guess the regular icinga/graphite check could be used in this case then

fgiunchedi triaged this task as Normal priority.Nov 30 2016, 2:10 AM