@jcrespo said on #wikimedia-operations "<jynus> robh: see eqsin mails" when talking about this issue, see P8748.

I never talked about this issue, and had not idea why @Urbanecm thoughout I was talking about this while I was having a private conversation with other person.

I'm sorry, I thought so because said conversation directly followed the report from @Daimona.

Huh, when I first reported I thought someone already knew about this. Anyway. Looking at this I see there has been a spike around 12:20-30, which is right before data stopped to flow. It's also interesting to note that, apparently, there hasn't been any drop in the amount of data, which was in fact replaced by these errors. So something is wrong with Elastic (?).

In T228089#5334366, @Urbanecm wrote:

@jcrespo said on #wikimedia-operations "<jynus> robh: see eqsin mails" when talking about this issue, see P8748.

In T228089#5334379, @jcrespo wrote:

I never talked about this issue, and had not idea why @Urbanecm thoughout I was talking about this while I was having a private conversation with other person.

In T228089#5334398, @Urbanecm wrote:

I'm sorry, I thought so because said conversation directly followed the report from @Daimona.

For the record, @jcrespo was replying to RobH about an unrelated topic in response to messages from the wikibugs and icinga-wm bots.

Meanwhile, back on topic. Some graphs that have been mentioned in the IRC conversation about this.

Dashboard: kafka-consumer-lag

Messages appear to be arriving fine into the Kafka layer, but not consumed properly (by Logstash/Elasticsarch?).

Dashboard: logstash

We decided to drop logs from cpjobqueue and changeprop at the logstash layer with the following config:

89-filter_drop_cpjobque_changeprop.conf:

filter {
  if [type] == "cpjobqueue" {
    drop {}
  }
  if [type] == "changeprop" {
    drop {}
  }
  if [_type] == "cpjobqueue" {
    drop {}
  }
  if [_type] == "changeprop" {
    drop {}
  }
}

Related to T150106: Type collisions in log events causing indexing failures in ELK Elasticsearch if the root problem is type collisions in the Elasticsearch index

Once the backlog is processed, https://grafana.wikimedia.org/d/000000102/production-logging?refresh=5m&panelId=8&fullscreen&orgId=1 This can be lowered to high, but something should be put in place to prevent another logs outage, even if it is a rough way, such as an alert to identify it and a runbook to drop a source of logs like above.

I've started an incident document at https://wikitech.wikimedia.org/wiki/Incident_documentation/20190715-logstash and would appreciate more contributors.

I also wanted to reference T150106: Type collisions in log events causing indexing failures in ELK Elasticsearch here

Mentioned in SAL (#wikimedia-operations) [2019-07-16T00:03:42Z] <shdubsh> restart logstash to revert mitigations - T228089