While investigating T213899: Migrate at least 3 existing Logstash inputs and associated producers to the new Kafka-logging pipeline, and remove the associated non-Kafka Logstash inputs I ran into some syslog lines that had a _grokparsefailure_sysloginput tag, e.g.:
{ "_index": "logstash-syslog-2019.02.25", "_type": "syslog", "_id": "AWkk2uwsZ9NltGfJqQ1p", "_version": 1, "_score": null, "_source": { "severity": 0, "level": "EMERGENCY", "message": "<158>Feb 25 13:32:00 /usr/sbin/cron[86116]: (root) CMD (newsyslog)", "type": "syslog", "priority": 0, "normalized_message": "<158>Feb 25 13:32:00 /usr/sbin/cron[86116]: (root) CMD (newsyslog)", "tags": [ "input-syslog-10514", "_grokparsefailure_sysloginput", "syslog", "es", "normalized_message_untrimmed" ], "@timestamp": "2019-02-25T13:32:00.716Z", "@version": "1", "host": "%{logsource}", "facility": 0, "severity_label": "Emergency", "facility_label": "kernel" }, "fields": { "@timestamp": [ 1551101520716 ] }, "sort": [ 1551101520716 ] }
And on the wire I think these messages look something like this:
<158>Feb 25 13:45:00 asw2-a5-eqiad /usr/sbin/cron[12760]: %-: (root) CMD (newsyslog)