(Came out of https://wikitech.wikimedia.org/wiki/Incident_documentation/20140517-bits )

Discussion:

Timo: In retrospect we saw that we had data in logstash clearly indicating a massive increase in read failures from this memcached instance (basicaly from < 1% to nearly 100%). This could and should be monitored by icinga and reported to ops automatically. This would've helped us catch it much earlier.

Chat from irc on 2014-07-09:
[10:16] <Krinkle> For something that is logged in logstash (e.g. memcached errors). What is the strategy you'd typically take to monitor it icinga? Is there a step in between or would you actually have icinga use logstash?
[10:16] <Krinkle> I think the latter should be possible for more complex queries or aggregated data. Though I reckon in case of memcached there's probably a more direct approach possible.
[10:18] <bd808> Good question. Logstash by itself can do point-in-time monitoring, but it really has no useful way to alert on trends itself.
[10:19] <Krinkle> I think most critical thigns should probably be polled by icinga directly. But more on multiple ocasions have I used logstash to quite easily pinpoint where an error came from. And it'd be useful to have those trends also result in pings to ops (perhaps not as critical via text but at least an irc ping would be useful).
[10:20] <bd808> One way™ to do it would be to graph trends in graphite driven by counts made by logstash and alert with icinga when the trend does something.
[10:20] <Krinkle> Right now logstash is mostly polling and digging manually, after the fact. That's immensly useful and it's good at that. But I think it has more potential.
[10:20] <Krinkle> Ah, I see. So it'd go to graphite after logstash. Interesting.
[10:20] <bd808> We aren't doing it now, but logstash can feed graphite in a statsd fashion
[10:20] <Krinkle> Right.
[10:21] <Krinkle> For some reason I thought they might also be able to feed graphite from the source that feeds logstash.
[10:21] <Krinkle> guess that's still possible, unless the source is distributed (or if the query is more advanced). In which case using logstash in between makes sense
[10:21] < Krinkle> (or if the query is more advanced)
[10:22] bd808 nods
[10:22] <Krinkle> cool

Version: wmf-deployment
Severity: normal