Page MenuHomePhabricator

Logstash syslog input grok parse failure on some network devices lines
Open, Needs TriagePublic

Description

While investigating T213899: Migrate at least 3 existing Logstash inputs and associated producers to the new Kafka-logging pipeline, and remove the associated non-Kafka Logstash inputs I ran into some syslog lines that had a _grokparsefailure_sysloginput tag, e.g.:

{
  "_index": "logstash-syslog-2019.02.25",
  "_type": "syslog",
  "_id": "AWkk2uwsZ9NltGfJqQ1p",
  "_version": 1,
  "_score": null,
  "_source": {
    "severity": 0,
    "level": "EMERGENCY",
    "message": "<158>Feb 25 13:32:00 /usr/sbin/cron[86116]: (root) CMD (newsyslog)",
    "type": "syslog",
    "priority": 0,
    "normalized_message": "<158>Feb 25 13:32:00 /usr/sbin/cron[86116]: (root) CMD (newsyslog)",
    "tags": [
      "input-syslog-10514",
      "_grokparsefailure_sysloginput",
      "syslog",
      "es",
      "normalized_message_untrimmed"
    ],
    "@timestamp": "2019-02-25T13:32:00.716Z",
    "@version": "1",
    "host": "%{logsource}",
    "facility": 0,
    "severity_label": "Emergency",
    "facility_label": "kernel"
  },
  "fields": {
    "@timestamp": [
      1551101520716
    ]
  },
  "sort": [
    1551101520716
  ]
}

And on the wire I think these messages look something like this:

<158>Feb 25 13:45:00 asw2-a5-eqiad /usr/sbin/cron[12760]: %-: (root) CMD (newsyslog)

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 25 2019, 1:51 PM