Page MenuHomePhabricator

Logstash elasticsearch mapping does not allow err.code to be a string
Closed, ResolvedPublic

Description

The err.code field is in most cases an int, but not always. We had a case where a ton of err.code=EADDRNOTAVAIL were sent to logstash. At the moment, we let elasticsearch autodetect mappings, so it creates err.code as an int field. This leads to errors, lost log messages and log overflow.

We will address this issue during or after the upgrade of logstash to elasticsearch 2.3.

Event Timeline

Restricted Application added subscribers: Zppix, Aklapper. · View Herald Transcript

Mentioned in SAL [2016-06-13T08:51:44Z] <moritzm> removed /var/log/logstash/logstash.log.1 on logstash1001, depleted disk space on the root partition, fallout of T137400

debt triaged this task as Medium priority.Jun 14 2016, 10:21 PM
debt moved this task from needs triage to Up Next on the Discovery-Search board.

Not sure the best solution here. We could force err.code to a string from logstash. This is wasteful of memory/disk but it's probably not the end of the world.

Alternatively we could find and fix the incorrect err.code, have it send err.string or some such. That may just be playing a game of whack-a-mole though.

debt subscribed.

everything is a string now - it's in production.