Page MenuHomePhabricator

Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline
Open, MediumPublic

Description

Elasticsearch is currently using the now deprecated logstash gelf input to ship logs to logstash.

Creating this task to track migration of Elasticsearch logging to the Kafka-logging pipeline.

Details

Related Gerrit Patches:

Event Timeline

herron created this task.Jun 5 2019, 5:33 PM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptJun 5 2019, 5:33 PM
debt moved this task from needs triage to Ops / SRE on the Discovery-Search board.Jun 6 2019, 6:04 PM
ArielGlenn triaged this task as Medium priority.Jun 14 2019, 7:13 AM

Change 531922 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: ship logs to syslog

https://gerrit.wikimedia.org/r/531922

Change 533928 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: add syslog logging option

https://gerrit.wikimedia.org/r/533928

rsyslog Json requires the @cee token which must be provided according to standard via profile::rsyslog::udp_localhost_compat. Let's use profile::rsyslog::udp_json_logback_compat instead as it permits parsing of json from log4j without the token.

Change 534399 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: switch elasticsearch logging to syslog

https://gerrit.wikimedia.org/r/534399

Change 531922 abandoned by Mathew.onipe:
elasticsearch: ship logs to local syslog server

Reason:
moved: https://gerrit.wikimedia.org/r/c/operations/puppet/ /534399

https://gerrit.wikimedia.org/r/531922

Change 533928 merged by Gehel:
[operations/puppet@production] elasticsearch: add syslog logging option

https://gerrit.wikimedia.org/r/533928

Change 534399 merged by Gehel:
[operations/puppet@production] elasticsearch: switch relforge to new logging pipeline

https://gerrit.wikimedia.org/r/534399

JsonLayout requires other dependencies for log4j. This include jackson databind. See https://logging.apache.org/log4j/2.x/runtime-dependencies.html.
Two options:

  1. Rebuild log4j with this dependencies
  2. Fall back to shipping logs with PatternLayout.

Change 535157 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: add dependencies for JsonLayout

https://gerrit.wikimedia.org/r/535157

Change 535158 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: switch relforge to new logging pipeline

https://gerrit.wikimedia.org/r/535158

We should talk to elastic to see how we can move this forward.
Currently, we require jackson-databind 2.8.11 and jackson-annotation 2.8.11 for JsonLayout to work when using SyslogAppender. Version 2.8.6 is provided by debian for this packages. We should use the correct version to make sure everything work as expected.