Page MenuHomePhabricator

Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline
Open, MediumPublic

Description

Elasticsearch is currently using the now deprecated logstash gelf input to ship logs to logstash.

Creating this task to track migration of Elasticsearch logging to the Kafka-logging pipeline.

Event Timeline

herron created this task.Jun 5 2019, 5:33 PM
Restricted Application added a project: Discovery-Search. · View Herald TranscriptJun 5 2019, 5:33 PM
debt moved this task from needs triage to Ops / SRE on the Discovery-Search board.Jun 6 2019, 6:04 PM
ArielGlenn triaged this task as Medium priority.Jun 14 2019, 7:13 AM

Change 531922 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: ship logs to syslog

https://gerrit.wikimedia.org/r/531922

Change 533928 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: add syslog logging option

https://gerrit.wikimedia.org/r/533928

rsyslog Json requires the @cee token which must be provided according to standard via profile::rsyslog::udp_localhost_compat. Let's use profile::rsyslog::udp_json_logback_compat instead as it permits parsing of json from log4j without the token.

Change 534399 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: switch elasticsearch logging to syslog

https://gerrit.wikimedia.org/r/534399

Change 531922 abandoned by Mathew.onipe:
elasticsearch: ship logs to local syslog server

Reason:
moved: https://gerrit.wikimedia.org/r/c/operations/puppet/ /534399

https://gerrit.wikimedia.org/r/531922

Change 533928 merged by Gehel:
[operations/puppet@production] elasticsearch: add syslog logging option

https://gerrit.wikimedia.org/r/533928

Change 534399 merged by Gehel:
[operations/puppet@production] elasticsearch: switch relforge to new logging pipeline

https://gerrit.wikimedia.org/r/534399

JsonLayout requires other dependencies for log4j. This include jackson databind. See https://logging.apache.org/log4j/2.x/runtime-dependencies.html.
Two options:

  1. Rebuild log4j with this dependencies
  2. Fall back to shipping logs with PatternLayout.

Change 535157 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: add dependencies for JsonLayout

https://gerrit.wikimedia.org/r/535157

Change 535158 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: switch relforge to new logging pipeline

https://gerrit.wikimedia.org/r/535158

We should talk to elastic to see how we can move this forward.
Currently, we require jackson-databind 2.8.11 and jackson-annotation 2.8.11 for JsonLayout to work when using SyslogAppender. Version 2.8.6 is provided by debian for this packages. We should use the correct version to make sure everything work as expected.

Change 570374 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] elasticsearch: cirrus logs to logging pipeline

https://gerrit.wikimedia.org/r/570374

Change 570374 abandoned by Filippo Giunchedi:
elasticsearch: cirrus logs to logging pipeline

Reason:
Abandoning this for now, after a chat with gehel. ES7 comes with native json logging, thus likely not worth the effort of attempting json logging for ES6.

https://gerrit.wikimedia.org/r/570374

Status update: out of the box json logging support has been introduced in elasticsearch 7 (https://github.com/elastic/elasticsearch/issues/8786). Whereas for previous versions we'd need to bring in jackson-databind, which comes with its own set of challenges (e.g. https://github.com/elastic/elasticsearch/issues/22103). Thus I'm of the opinion that waiting for the elasticsearch 7 upgrade on cirrus/relforge/cloudelastic will be easier.

fgiunchedi moved this task from Inbox to Backlog on the observability board.Apr 6 2020, 12:36 PM
Gehel removed Mathew.onipe as the assignee of this task.Apr 20 2020, 5:38 PM
Gehel added a subscriber: Mathew.onipe.