Page MenuHomePhabricator

Migrate at least 3 existing Logstash inputs and associated producers to the new Kafka-logging pipeline, and remove the associated non-Kafka Logstash inputs
Closed, ResolvedPublic

Description

Deprecate the following Logstash inputs:

  • log4j
  • json_lines
  • syslog (excluding network devices)
  • logback

Auditing current input traffic based on tags field (last 24h) https://logstash.wikimedia.org/goto/feb0db9d38340a81d2f2d430104b8f12
Dashboard breaking down kafka and non-kafka inputs https://logstash.wikimedia.org/goto/7900d9a70d24467c2a1f4545651a0b16

Selection of inputs and their producers:

nameproducers
input-log4j-4560gerrit
input-udp2log-mediawiki-8324scap
input-tcp-json_lines-11514webrequest (5xx), mjolnir, striker
input-syslog-10514network devices, apache2, hhvm, php(-fpm), varnish, varnishkafka, varnishstatsd, varnishmtail-backend, prometheus-varnish-exporter, varnishreqstats
input-udp-logback-11514wdqs-updater, wdqs-blazegraph, wdqs-categories, ores, varnishospital, varnishslowlog, varnishtlsinspector, cassandra, thumbor, debmonitor, netbox

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+11 -8
operations/puppetproduction+0 -21
operations/puppetproduction+1 -0
operations/puppetproduction+8 -1
operations/puppetproduction+9 -1
operations/puppetproduction+48 -0
operations/puppetproduction+2 -0
operations/puppetproduction+2 -0
operations/puppetproduction+1 -1
operations/puppetproduction+5 -2
operations/puppetproduction+14 -3
operations/puppetproduction+1 -0
operations/puppetproduction+0 -1
operations/puppetproduction+2 -1
operations/puppetproduction+17 -17
operations/puppetproduction+1 -1
operations/puppetproduction+81 -0
operations/puppetproduction+4 -0
operations/puppetproduction+0 -81
operations/puppetproduction+5 -25
operations/puppetproduction+0 -23
operations/puppetproduction+3 -3
Show related patches Customize query in gerrit

Event Timeline

fgiunchedi created this task.

I've ran an audit on producers that sent logs through the least three inputs used over the last 24h (sorted by increasing volume by input)

nameproducers
input-log4j-4560gerrit
input-udp2log-mediawiki-8324scap
input-tcp-json_lines-11514webrequest (5xx status codes), mjolnir, striker
input-syslog-10514network devices, apache2, hhvm, php(-fpm), varnish, varnishkafka, varnishstatsd, varnishmtail-backend, prometheus-varnish-exporter, varnishreqstats
input-udp-logback-11514wdqs-updater, wdqs-blazegraph, wdqs-categories, ores, varnishospital, varnishslowlog, varnishtlsinspector, cassandra, thumbor

Change 489006 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add input identifier tags to kafka logstash inputs

https://gerrit.wikimedia.org/r/489006

Change 489006 merged by Herron:
[operations/puppet@production] logstash: add input identifier tags to kafka logstash inputs

https://gerrit.wikimedia.org/r/489006

I've ran an audit on producers that sent logs through the least three inputs used over the last 24h (sorted by increasing volume by input)

On a related note, here is a logstash dashboard to help break down input volumes by input plugin with log type as a "sub-bucket"

https://logstash.wikimedia.org/goto/7900d9a70d24467c2a1f4545651a0b16

The left includes all inputs, and the right excludes kafka inputs. It should be useful to watch over time as we migrate.

Change 492633 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] gerrit: stop sending logs directly to logstash

https://gerrit.wikimedia.org/r/492633

Change 492640 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: remove log4j input

https://gerrit.wikimedia.org/r/492640

Change 492676 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] [WIP] move varnish logging to logging pipeline

https://gerrit.wikimedia.org/r/492676

Change 492692 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: disable syslog-tls input, unused

https://gerrit.wikimedia.org/r/492692

Change 492692 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: disable syslog-tls input, unused

https://gerrit.wikimedia.org/r/492692

Change 492633 merged by Filippo Giunchedi:
[operations/puppet@production] gerrit: stop sending logs directly to logstash

https://gerrit.wikimedia.org/r/492633

Change 492640 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: remove log4j input

https://gerrit.wikimedia.org/r/492640

Change 493243 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] WIP move webrequest-5xx to logging pipeline

https://gerrit.wikimedia.org/r/493243

Change 495962 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: migrate mediawiki syslogs to logging pipeline

https://gerrit.wikimedia.org/r/495962

Change 495980 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: add netdev_kafka_relay compatability endpoint

https://gerrit.wikimedia.org/r/495980

Change 496021 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add tcp json_lines localhost compatability endpoint

https://gerrit.wikimedia.org/r/496021

Change 496022 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add udp json logback localhost compatability endpoint

https://gerrit.wikimedia.org/r/496022

Change 495962 merged by Herron:
[operations/puppet@production] logstash: send mediawiki syslogs to logging pipeline

https://gerrit.wikimedia.org/r/495962

Change 496022 merged by Herron:
[operations/puppet@production] logstash: add udp json logback localhost compatibility endpoint

https://gerrit.wikimedia.org/r/496022

Change 496806 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: update syslog_json template with format jsonfr

https://gerrit.wikimedia.org/r/496806

Change 496871 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: remove format=json from msg field in syslog_json template

https://gerrit.wikimedia.org/r/496871

Change 496871 abandoned by Herron:
rsyslog: remove format=json from msg field in syslog_json template

Reason:
in favor of Ib228fdc7c856f68590261b5c410594a60cde6fe6

https://gerrit.wikimedia.org/r/496871

Change 496806 merged by Herron:
[operations/puppet@production] rsyslog: update syslog_json template with format jsonf

https://gerrit.wikimedia.org/r/496806

Change 497570 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: move mediawiki syslogs to logging pipeline

https://gerrit.wikimedia.org/r/497570

Change 497587 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: add omkafka load statement to udp_json_logback config

https://gerrit.wikimedia.org/r/497587

Change 497587 merged by Herron:
[operations/puppet@production] rsyslog: add omkafka load statement to udp_json_logback config

https://gerrit.wikimedia.org/r/497587

Change 497614 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] ores: ship to logstash via the kafka logging pipeline

https://gerrit.wikimedia.org/r/497614

Change 497570 merged by Herron:
[operations/puppet@production] logstash: move mediawiki syslogs to logging pipeline

https://gerrit.wikimedia.org/r/497570

Change 498467 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: send varnish syslogs via kafka logging pipeline

https://gerrit.wikimedia.org/r/498467

Change 499829 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] role: add kafka_shipper to elasticsearch::cirrus role

https://gerrit.wikimedia.org/r/499829

Change 499829 merged by Cwhite:
[operations/puppet@production] role: add kafka_shipper to elasticsearch::cirrus role

https://gerrit.wikimedia.org/r/499829

Change 500099 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: do not mutate level for mjolnir

https://gerrit.wikimedia.org/r/500099

Change 497614 merged by Herron:
[operations/puppet@production] ores: ship to logstash via the kafka logging pipeline

https://gerrit.wikimedia.org/r/497614

Change 502527 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] ores: ship to logstash via the kafka logging pipeline

https://gerrit.wikimedia.org/r/502527

Change 493243 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logging: move webrequest-5xx to logging pipeline

https://gerrit.wikimedia.org/r/493243

Change 493243 merged by Filippo Giunchedi:
[operations/puppet@production] logging: move webrequest-5xx to logging pipeline

https://gerrit.wikimedia.org/r/493243

Change 502985 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] webrequest: set logger max message size

https://gerrit.wikimedia.org/r/502985

Change 502985 merged by Filippo Giunchedi:
[operations/puppet@production] webrequest: set logger max message size

https://gerrit.wikimedia.org/r/502985

Change 500099 merged by Cwhite:
[operations/puppet@production] profile: do not mutate level for mjolnir

https://gerrit.wikimedia.org/r/500099

Change 504074 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: do not mutate level for mjolnir

https://gerrit.wikimedia.org/r/504074

Change 504074 merged by Cwhite:
[operations/puppet@production] profile: do not mutate level for mjolnir

https://gerrit.wikimedia.org/r/504074

Change 495980 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: add netdev_kafka_relay compatability endpoint

https://gerrit.wikimedia.org/r/495980

Change 496021 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add tcp json_lines localhost compatability endpoint

https://gerrit.wikimedia.org/r/496021

Change 565573 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: turn down logstash tcp json_lines endpoint

https://gerrit.wikimedia.org/r/565573

Change 569564 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] wip: cassandra logs to logging pipeline

https://gerrit.wikimedia.org/r/569564

Change 569570 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] cassandra: use wmflib::secret for binary files

https://gerrit.wikimedia.org/r/569570

fgiunchedi claimed this task.

I'm resolving this task since all the followup is already tracked in parent T227080: Deprecate all non-Kafka logstash inputs

Change 492676 abandoned by Filippo Giunchedi:
[operations/puppet@production] [WIP] move varnish logging to logging pipeline

Reason:
Superceded by Ic56b29517

https://gerrit.wikimedia.org/r/492676