Page MenuHomePhabricator

Eliminate field collisions between syslog_cee and ECS-formatted logs
Open, Needs TriagePublic

Description

While implementing T257861, it was discovered that the host.name field was not being populated because tcpircbot does not gather that information. This issue has been worked around elsewhere for other producers.

The syslog_cee template injects timestamp, logsource, host, program, severity, and facility fields as string values.

For ECS-formatted logs:

  1. the host field is an object.
  2. timestamp often collides with the software-generated timestamp.
  3. logsource, severity, facility are not ECS fields.

These conflicts have existed since we enabled @cee logs in rsyslog and became problematic at the adoption of ECS.

One possible solution is to namespace the rsyslog fields in the template. To preserve dashboards, this would require us to move the fields into the right place in logstash.

Questions to answer:

  1. What fields are auto-added by the input plugin? The answer may well be "none" in kafka.
  2. Any other fields added by rsyslog?

Event Timeline

Change 824314 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add support for rsyslog-namespaced fields

https://gerrit.wikimedia.org/r/824314

Change 824315 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] rsyslog: add rsyslog-namespaced fields to syslog_cee

https://gerrit.wikimedia.org/r/824315

Change 824316 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: use rsyslog-namespaced fields

https://gerrit.wikimedia.org/r/824316

Change 824314 merged by Cwhite:

[operations/puppet@production] logstash: add support for rsyslog-namespaced fields

https://gerrit.wikimedia.org/r/824314

Change 824315 merged by Cwhite:

[operations/puppet@production] rsyslog: add rsyslog-namespaced fields to syslog_cee

https://gerrit.wikimedia.org/r/824315

Change 824316 merged by Cwhite:

[operations/puppet@production] logstash: use rsyslog-namespaced fields

https://gerrit.wikimedia.org/r/824316