HomePhabricator

logstash: Update default mappings for Elasticsearch 2.x
d5b95160faefUnpublished

Authored by bd808 on Jul 11 2016, 3:47 PM.

Unpublished Commit · Learn More

Not On Permanent Ref: This commit is not an ancestor of any permanent ref.
This commit no longer exists in the repository. It may have been part of a branch which was deleted.This commit has been deleted in the repository: it is no longer reachable from any branch, tag, or ref.

Description

logstash: Update default mappings for Elasticsearch 2.x

Update the default mapping template to coerce all non-string primitive
value fields without explicit mappings to strings.

Starting with Elasticsearch 2.x, fields with the same name, in the same
index, in different types, must have the same mapping. This is
problematic for our Logstash traffic where different applications may
(and do!) use common names like "code" and "status" as structured log
data with differing content. We have a "normalize_fields" filter that
has been used to try and clean up these differences, but that is
a fragile approach that could be broken at any time by a new application
or new event type for an existing application that has conflicts with
anything else in our logging environment.

This mapping will still have issues if one log source sends an
array/object for the same field name that anther uses to record
a primitive value. Those will have to be sorted out either in Logstash
filters or the origin applications themselves by renaming one of the log
event fields to avoid the collision.

Bug: T136001
Change-Id: I638d88e1d874fdb8be211bd74a1e36998d42dc09

Details

Committed
GehelJul 13 2016, 3:25 PM
ChangeId
I638d88e1d874fdb8be211bd74a1e36998d42dc09

Commit No Longer Exists

This commit no longer exists in the repository.