Maniphest T199479

Add alerts for Logstash rates in production
Closed, ResolvedPublic
Actions

Description

Scap monitors Logstash error rates from canary servers during a deployment. However, problems are not always triggered by a deployment. They may be triggered by an external factor, or a cron job, or may only reveal themselves after a certain cache is purged or expired, etc.

As such, we should have an Icinga alert (Based on Graphite, Prometheus or Grafana?) that triggers when the WARNING or ERROR rate of mediawiki logs increases above a certain threshold for a prolonged period of time.

This would be similar for the alerts that we have already for MediaWiki exceptions.

This is actionable from https://wikitech.wikimedia.org/wiki/Incident_documentation/20180710-MediaWiki.

Related Objects

Mentioned In: T117470: Ensure CAS errors aren't cause problems / trending up

Event Timeline

Krinkle created this task.Jul 12 2018, 9:37 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 12 2018, 9:37 PM

Krinkle mentioned this in T117470: Ensure CAS errors aren't cause problems / trending up.Jul 14 2018, 4:15 AM

tstarling moved this task from Inbox to Watching on the Core-Platform-Team-Old board.Jul 17 2018, 1:53 AM

Adding Operations, but not sure where this task should actually go

herron triaged this task as Medium priority.Jul 17 2018, 3:28 PM

herron added a subscriber: fgiunchedi.

Krinkle moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.Jul 18 2018, 7:20 PM

Looks like logstash rates are in graphite ATM, so either a grafana alert and its counterpart in puppet or a graphite alert. Both would work fine I think, the grafana alert has the advantage of tuning thresholds is self-service (i.e. no puppet merge required)

CCicalese_WMF edited projects, added Platform Team Legacy (Watching / External); removed Core-Platform-Team-Old.Oct 1 2018, 3:28 PM

Phabricator_maintenance moved this task from Backlog to Acknowledged on the Operations board.Jan 26 2019, 10:10 PM

Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM

We have icinga alerts for mediawiki errors rates nowadays, based on Prometheus metrics (via logstash -> statsd -> prometheus)

e.g.

12:49 -icinga-wm:#wikimedia-operations- PROBLEM - MediaWiki exceptions and fatals per 
          minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter 
          level=ERROR site=eqiad 
          https://wikitech.wikimedia.org/wiki/Application_servers 
https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops

Add alerts for Logstash rates in productionClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Add alerts for Logstash rates in production
Closed, ResolvedPublic
Actions