Page MenuHomePhabricator

Migrate at least 3 existing Logstash inputs and associated producers to the new Kafka-logging pipeline, and remove the associated non-Kafka Logstash inputs
Open, MediumPublic

Description

Deprecate the following Logstash inputs:

  • log4j
  • json_lines
  • syslog (excluding network devices)
  • logback

Auditing current input traffic based on tags field (last 24h) https://logstash.wikimedia.org/goto/feb0db9d38340a81d2f2d430104b8f12
Dashboard breaking down kafka and non-kafka inputs https://logstash.wikimedia.org/goto/7900d9a70d24467c2a1f4545651a0b16

Selection of inputs and their producers:

nameproducers
input-log4j-4560gerrit
input-udp2log-mediawiki-8324scap
input-tcp-json_lines-11514webrequest (5xx), mjolnir, striker
input-syslog-10514network devices, apache2, hhvm, php(-fpm), varnish, varnishkafka, varnishstatsd, varnishmtail-backend, prometheus-varnish-exporter, varnishreqstats
input-udp-logback-11514wdqs-updater, wdqs-blazegraph, wdqs-categories, ores, varnishospital, varnishslowlog, varnishtlsinspector, cassandra, thumbor

Details

Related Gerrit Patches:
operations/puppet : productionores: ship to logstash via the kafka logging pipeline
operations/puppet : productionores: ship to logstash via the kafka logging pipeline
operations/puppet : productionlogstash: add tcp json_lines localhost compatability endpoint
operations/puppet : productionprofile: do not mutate level for mjolnir
operations/puppet : productionprofile: do not mutate level for mjolnir
operations/puppet : productionwebrequest: set logger max message size
operations/puppet : productionlogging: move webrequest-5xx to logging pipeline
operations/puppet : productionlogstash: send varnish syslogs via kafka logging pipeline
operations/puppet : productionrole: add kafka_shipper to elasticsearch::cirrus role
operations/puppet : productionlogstash: move mediawiki syslogs to logging pipeline
operations/puppet : productionrsyslog: add omkafka load statement to udp_json_logback config
operations/puppet : productionrsyslog: update syslog_json template with format jsonf
operations/puppet : productionrsyslog: remove format=json from msg field in syslog_json template
operations/puppet : productionlogstash: add udp json logback localhost compatibility endpoint
operations/puppet : productionlogstash: send mediawiki syslogs to logging pipeline
operations/puppet : production[WIP] move varnish logging to logging pipeline
operations/puppet : productionlogstash: remove log4j input
operations/puppet : productiongerrit: stop sending logs directly to logstash
operations/puppet : productionlogstash: disable syslog-tls input, unused
operations/puppet : productionlogstash: add input identifier tags to kafka logstash inputs

Event Timeline

fgiunchedi triaged this task as Medium priority.Jan 16 2019, 11:13 AM
fgiunchedi created this task.
herron moved this task from Backlog to Working on on the User-herron board.Jan 22 2019, 9:23 PM
fgiunchedi updated the task description. (Show Details)Feb 7 2019, 10:53 AM
fgiunchedi added a comment.EditedFeb 7 2019, 3:53 PM

I've ran an audit on producers that sent logs through the least three inputs used over the last 24h (sorted by increasing volume by input)

nameproducers
input-log4j-4560gerrit
input-udp2log-mediawiki-8324scap
input-tcp-json_lines-11514webrequest (5xx status codes), mjolnir, striker
input-syslog-10514network devices, apache2, hhvm, php(-fpm), varnish, varnishkafka, varnishstatsd, varnishmtail-backend, prometheus-varnish-exporter, varnishreqstats
input-udp-logback-11514wdqs-updater, wdqs-blazegraph, wdqs-categories, ores, varnishospital, varnishslowlog, varnishtlsinspector, cassandra, thumbor

Change 489006 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add input identifier tags to kafka logstash inputs

https://gerrit.wikimedia.org/r/489006

Change 489006 merged by Herron:
[operations/puppet@production] logstash: add input identifier tags to kafka logstash inputs

https://gerrit.wikimedia.org/r/489006

herron added a comment.Feb 7 2019, 9:06 PM

I've ran an audit on producers that sent logs through the least three inputs used over the last 24h (sorted by increasing volume by input)

On a related note, here is a logstash dashboard to help break down input volumes by input plugin with log type as a "sub-bucket"

https://logstash.wikimedia.org/goto/7900d9a70d24467c2a1f4545651a0b16

The left includes all inputs, and the right excludes kafka inputs. It should be useful to watch over time as we migrate.

Change 492633 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] gerrit: stop sending logs directly to logstash

https://gerrit.wikimedia.org/r/492633

Change 492640 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: remove log4j input

https://gerrit.wikimedia.org/r/492640

Change 492676 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] [WIP] move varnish logging to logging pipeline

https://gerrit.wikimedia.org/r/492676

Change 492692 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: disable syslog-tls input, unused

https://gerrit.wikimedia.org/r/492692

Change 492692 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: disable syslog-tls input, unused

https://gerrit.wikimedia.org/r/492692

Change 492633 merged by Filippo Giunchedi:
[operations/puppet@production] gerrit: stop sending logs directly to logstash

https://gerrit.wikimedia.org/r/492633

fgiunchedi updated the task description. (Show Details)Feb 26 2019, 11:23 AM

Change 492640 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: remove log4j input

https://gerrit.wikimedia.org/r/492640

fgiunchedi updated the task description. (Show Details)Feb 27 2019, 11:41 AM

Change 493243 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] WIP move webrequest-5xx to logging pipeline

https://gerrit.wikimedia.org/r/493243

herron updated the task description. (Show Details)Mar 4 2019, 3:00 PM
herron updated the task description. (Show Details)Mar 12 2019, 5:49 PM

Change 495962 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: migrate mediawiki syslogs to logging pipeline

https://gerrit.wikimedia.org/r/495962

Change 495980 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: add netdev_kafka_relay compatability endpoint

https://gerrit.wikimedia.org/r/495980

Change 496021 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add tcp json_lines localhost compatability endpoint

https://gerrit.wikimedia.org/r/496021

Change 496022 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add udp json logback localhost compatability endpoint

https://gerrit.wikimedia.org/r/496022

Change 495962 merged by Herron:
[operations/puppet@production] logstash: send mediawiki syslogs to logging pipeline

https://gerrit.wikimedia.org/r/495962

Change 496022 merged by Herron:
[operations/puppet@production] logstash: add udp json logback localhost compatibility endpoint

https://gerrit.wikimedia.org/r/496022

Change 496806 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: update syslog_json template with format jsonfr

https://gerrit.wikimedia.org/r/496806

herron updated the task description. (Show Details)Mar 15 2019, 5:49 PM

Change 496871 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: remove format=json from msg field in syslog_json template

https://gerrit.wikimedia.org/r/496871

Change 496871 abandoned by Herron:
rsyslog: remove format=json from msg field in syslog_json template

Reason:
in favor of Ib228fdc7c856f68590261b5c410594a60cde6fe6

https://gerrit.wikimedia.org/r/496871

Change 496806 merged by Herron:
[operations/puppet@production] rsyslog: update syslog_json template with format jsonf

https://gerrit.wikimedia.org/r/496806

Change 497570 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: move mediawiki syslogs to logging pipeline

https://gerrit.wikimedia.org/r/497570

Change 497587 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: add omkafka load statement to udp_json_logback config

https://gerrit.wikimedia.org/r/497587

Change 497587 merged by Herron:
[operations/puppet@production] rsyslog: add omkafka load statement to udp_json_logback config

https://gerrit.wikimedia.org/r/497587

Change 497614 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] ores: ship to logstash via the kafka logging pipeline

https://gerrit.wikimedia.org/r/497614

Change 497570 merged by Herron:
[operations/puppet@production] logstash: move mediawiki syslogs to logging pipeline

https://gerrit.wikimedia.org/r/497570

herron updated the task description. (Show Details)Mar 20 2019, 6:46 PM

Change 498467 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: send varnish syslogs via kafka logging pipeline

https://gerrit.wikimedia.org/r/498467

Change 499829 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] role: add kafka_shipper to elasticsearch::cirrus role

https://gerrit.wikimedia.org/r/499829

Change 499829 merged by Cwhite:
[operations/puppet@production] role: add kafka_shipper to elasticsearch::cirrus role

https://gerrit.wikimedia.org/r/499829

Change 500099 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: do not mutate level for mjolnir

https://gerrit.wikimedia.org/r/500099

Change 497614 merged by Herron:
[operations/puppet@production] ores: ship to logstash via the kafka logging pipeline

https://gerrit.wikimedia.org/r/497614

Change 502527 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] ores: ship to logstash via the kafka logging pipeline

https://gerrit.wikimedia.org/r/502527

Change 493243 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logging: move webrequest-5xx to logging pipeline

https://gerrit.wikimedia.org/r/493243

fgiunchedi updated the task description. (Show Details)Apr 10 2019, 1:28 PM
fgiunchedi updated the task description. (Show Details)

Change 493243 merged by Filippo Giunchedi:
[operations/puppet@production] logging: move webrequest-5xx to logging pipeline

https://gerrit.wikimedia.org/r/493243

Change 502985 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] webrequest: set logger max message size

https://gerrit.wikimedia.org/r/502985

Change 502985 merged by Filippo Giunchedi:
[operations/puppet@production] webrequest: set logger max message size

https://gerrit.wikimedia.org/r/502985

fgiunchedi updated the task description. (Show Details)Apr 11 2019, 12:41 PM

Change 500099 merged by Cwhite:
[operations/puppet@production] profile: do not mutate level for mjolnir

https://gerrit.wikimedia.org/r/500099

Change 504074 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: do not mutate level for mjolnir

https://gerrit.wikimedia.org/r/504074

Change 504074 merged by Cwhite:
[operations/puppet@production] profile: do not mutate level for mjolnir

https://gerrit.wikimedia.org/r/504074

Change 495980 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: add netdev_kafka_relay compatability endpoint

https://gerrit.wikimedia.org/r/495980

Change 496021 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add tcp json_lines localhost compatability endpoint

https://gerrit.wikimedia.org/r/496021