Change 531922 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: ship logs to syslog

https://gerrit.wikimedia.org/r/531922

Change 533928 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: add syslog logging option

https://gerrit.wikimedia.org/r/533928

rsyslog Json requires the @cee token which must be provided according to standard via profile::rsyslog::udp_localhost_compat. Let's use profile::rsyslog::udp_json_logback_compat instead as it permits parsing of json from log4j without the token.

Change 534399 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: switch elasticsearch logging to syslog

https://gerrit.wikimedia.org/r/534399

Change 531922 abandoned by Mathew.onipe:
elasticsearch: ship logs to local syslog server

Reason:
moved: https://gerrit.wikimedia.org/r/c/operations/puppet/ /534399

https://gerrit.wikimedia.org/r/531922

Change 533928 merged by Gehel:
[operations/puppet@production] elasticsearch: add syslog logging option

https://gerrit.wikimedia.org/r/533928

Change 534399 merged by Gehel:
[operations/puppet@production] elasticsearch: switch relforge to new logging pipeline

https://gerrit.wikimedia.org/r/534399

JsonLayout requires other dependencies for log4j. This include jackson databind. See https://logging.apache.org/log4j/2.x/runtime-dependencies.html.
Two options:

1. Rebuild log4j with this dependencies
2. Fall back to shipping logs with PatternLayout.

Change 535157 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: add dependencies for JsonLayout

https://gerrit.wikimedia.org/r/535157

Change 535158 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] elasticsearch: switch relforge to new logging pipeline

https://gerrit.wikimedia.org/r/535158

We should talk to elastic to see how we can move this forward.
Currently, we require jackson-databind 2.8.11 and jackson-annotation 2.8.11 for JsonLayout to work when using SyslogAppender. Version 2.8.6 is provided by debian for this packages. We should use the correct version to make sure everything work as expected.

Change 570374 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] elasticsearch: cirrus logs to logging pipeline

https://gerrit.wikimedia.org/r/570374

Change 570374 abandoned by Filippo Giunchedi:
elasticsearch: cirrus logs to logging pipeline

Reason:
Abandoning this for now, after a chat with gehel. ES7 comes with native json logging, thus likely not worth the effort of attempting json logging for ES6.

https://gerrit.wikimedia.org/r/570374

Status update: out of the box json logging support has been introduced in elasticsearch 7 (https://github.com/elastic/elasticsearch/issues/8786). Whereas for previous versions we'd need to bring in jackson-databind, which comes with its own set of challenges (e.g. https://github.com/elastic/elasticsearch/issues/22103). Thus I'm of the opinion that waiting for the elasticsearch 7 upgrade on cirrus/relforge/cloudelastic will be easier.

With the elastic SSPL changes that happened this year (T272111 T272238 etc.) is ES7 still a part of the roadmap for moving these logs away from gelf?

These logs have been migrated to kafka-logging with the deployment of gelf_relay in T288620. Resolving!