HomePhabricator

logstash: Update logstash for sending to es 2.x

Description

logstash: Update logstash for sending to es 2.x

Adjust index mapping template to be 2.x compatible:

  • Remove path: full from geoip. This is now unsupported. full was already the default, and is what new versions of es always do so no other changes required.
  • Remove 'index_name: tag' from tags. This was aliasing the tag field to tags. It is no longer supported in 2.x. Any dashboards that query tag must query tags instead now.

Post process logs to make them 2.x compatible:

  • converts dot's in the properties into underscores. 2.x does not allow dots in properties, they are used as separators.
  • Normalizes the pid and line fields into int. Some of the log types for nodejs services auto-created these as strings.
  • Drops the extra timestamp field. @timestamp is definitive, and this timestamp field is seen sometimes as a date, sometimes as a string in the auto generated mappings depending on the content of the field the first time it was seen.

Change-Id: I46d177ce1218eee6f86fa9468b917dc54b3d55da
Bug: T138335

Details

Provenance
EBernhardsonAuthored on Jun 22 2016, 7:48 PM
GehelCommitted on Jun 27 2016, 5:14 PM
Parents
rOPUPd5aa541795fe: site: add prometheus[12]00[12]
Branches
Unknown
Tags
Unknown
ChangeId
I46d177ce1218eee6f86fa9468b917dc54b3d55da