Page MenuHomePhabricator
Create Task
Maniphest T213899

Migrate at least 3 existing Logstash inputs and associated producers to the new Kafka-logging pipeline, and remove the associated non-Kafka Logstash inputs
Closed, ResolvedPublic
Actions

Description

Deprecate the following Logstash inputs:

  • log4j
  • json_lines
  • syslog (excluding network devices)
  • logback

Auditing current input traffic based on tags field (last 24h) https://logstash.wikimedia.org/goto/feb0db9d38340a81d2f2d430104b8f12
Dashboard breaking down kafka and non-kafka inputs https://logstash.wikimedia.org/goto/7900d9a70d24467c2a1f4545651a0b16

Selection of inputs and their producers:

nameproducers
input-log4j-4560gerrit
input-udp2log-mediawiki-8324scap
input-tcp-json_lines-11514webrequest (5xx), mjolnir, striker
input-syslog-10514network devices, apache2, hhvm, php(-fpm), varnish, varnishkafka, varnishstatsd, varnishmtail-backend, prometheus-varnish-exporter, varnishreqstats
input-udp-logback-11514wdqs-updater, wdqs-blazegraph, wdqs-categories, ores, varnishospital, varnishslowlog, varnishtlsinspector, cassandra, thumbor, debmonitor, netbox

Details

SubjectRepoBranchLines +/-
logstash: send varnish syslogs via kafka logging pipelineoperations/puppetproduction+14 -3
logstash: add tcp json_lines localhost compatability endpointoperations/puppetproduction+48 -0
hieradata: turn down logstash tcp json_lines endpointoperations/puppetproduction+0 -21
[WIP] move varnish logging to logging pipelineoperations/puppetproduction+11 -8
hieradata: use logging pipeline for ores uwsgioperations/puppetproduction+1 -0
ores: ship to logstash via the kafka logging pipelineoperations/puppetproduction+8 -1
ores: ship to logstash via the kafka logging pipelineoperations/puppetproduction+9 -1
profile: do not mutate level for mjolniroperations/puppetproduction+2 -0
profile: do not mutate level for mjolniroperations/puppetproduction+2 -0
webrequest: set logger max message sizeoperations/puppetproduction+1 -1
logging: move webrequest-5xx to logging pipelineoperations/puppetproduction+5 -2
role: add kafka_shipper to elasticsearch::cirrus roleoperations/puppetproduction+1 -0
logstash: move mediawiki syslogs to logging pipelineoperations/puppetproduction+0 -1
rsyslog: add omkafka load statement to udp_json_logback configoperations/puppetproduction+2 -1
rsyslog: update syslog_json template with format jsonfoperations/puppetproduction+17 -17
rsyslog: remove format=json from msg field in syslog_json templateoperations/puppetproduction+1 -1
logstash: add udp json logback localhost compatibility endpointoperations/puppetproduction+81 -0
logstash: send mediawiki syslogs to logging pipelineoperations/puppetproduction+4 -0
logstash: remove log4j inputoperations/puppetproduction+0 -81
gerrit: stop sending logs directly to logstashoperations/puppetproduction+5 -25
logstash: disable syslog-tls input, unusedoperations/puppetproduction+0 -23
logstash: add input identifier tags to kafka logstash inputsoperations/puppetproduction+3 -3
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi triaged this task as Medium priority.Jan 16 2019, 11:13 AM
fgiunchedi created this task.
herron moved this task from Backlog to Working on on the User-herron board.Jan 22 2019, 9:23 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 11:05 PM
fgiunchedi updated the task description. (Show Details)Feb 7 2019, 10:53 AM
fgiunchedi added a comment.EditedFeb 7 2019, 3:53 PM

I've ran an audit on producers that sent logs through the least three inputs used over the last 24h (sorted by increasing volume by input)

nameproducers
input-log4j-4560gerrit
input-udp2log-mediawiki-8324scap
input-tcp-json_lines-11514webrequest (5xx status codes), mjolnir, striker
input-syslog-10514network devices, apache2, hhvm, php(-fpm), varnish, varnishkafka, varnishstatsd, varnishmtail-backend, prometheus-varnish-exporter, varnishreqstats
input-udp-logback-11514wdqs-updater, wdqs-blazegraph, wdqs-categories, ores, varnishospital, varnishslowlog, varnishtlsinspector, cassandra, thumbor
gerritbot subscribed.Feb 7 2019, 7:57 PM

Change 489006 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add input identifier tags to kafka logstash inputs

https://gerrit.wikimedia.org/r/489006

gerritbot added a project: Patch-For-Review.Feb 7 2019, 7:57 PM
gerritbot added a comment.Feb 7 2019, 8:02 PM

Change 489006 merged by Herron:
[operations/puppet@production] logstash: add input identifier tags to kafka logstash inputs

https://gerrit.wikimedia.org/r/489006

herron added a comment.Feb 7 2019, 9:06 PM
In T213899#4935098, @fgiunchedi wrote:

I've ran an audit on producers that sent logs through the least three inputs used over the last 24h (sorted by increasing volume by input)

On a related note, here is a logstash dashboard to help break down input volumes by input plugin with log type as a "sub-bucket"

https://logstash.wikimedia.org/goto/7900d9a70d24467c2a1f4545651a0b16

The left includes all inputs, and the right excludes kafka inputs. It should be useful to watch over time as we migrate.

gerritbot added a comment.Feb 25 2019, 10:07 AM

Change 492633 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] gerrit: stop sending logs directly to logstash

https://gerrit.wikimedia.org/r/492633

gerritbot added a comment.Feb 25 2019, 10:52 AM

Change 492640 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: remove log4j input

https://gerrit.wikimedia.org/r/492640

fgiunchedi mentioned this in T217032: Logstash syslog input grok parse failure on some network devices lines.Feb 25 2019, 1:51 PM
gerritbot added a comment.Feb 25 2019, 2:17 PM

Change 492676 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] [WIP] move varnish logging to logging pipeline

https://gerrit.wikimedia.org/r/492676

gerritbot added a comment.Feb 25 2019, 3:27 PM

Change 492692 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: disable syslog-tls input, unused

https://gerrit.wikimedia.org/r/492692

gerritbot added a comment.Feb 25 2019, 3:46 PM

Change 492692 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: disable syslog-tls input, unused

https://gerrit.wikimedia.org/r/492692

gerritbot added a comment.Feb 26 2019, 10:10 AM

Change 492633 merged by Filippo Giunchedi:
[operations/puppet@production] gerrit: stop sending logs directly to logstash

https://gerrit.wikimedia.org/r/492633

Stashbot subscribed.Feb 26 2019, 10:12 AM

Mentioned in SAL (#wikimedia-operations) [2019-02-26T10:12:48Z] <godog> bounce gerrit on gerrit2001 and cobalt after https://gerrit.wikimedia.org/r/c/operations/puppet/+/492633 - T213899

fgiunchedi updated the task description. (Show Details)Feb 26 2019, 11:23 AM
gerritbot added a comment.Feb 27 2019, 10:57 AM

Change 492640 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: remove log4j input

https://gerrit.wikimedia.org/r/492640

fgiunchedi updated the task description. (Show Details)Feb 27 2019, 11:41 AM
gerritbot added a comment.Feb 27 2019, 3:18 PM

Change 493243 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] WIP move webrequest-5xx to logging pipeline

https://gerrit.wikimedia.org/r/493243

herron updated the task description. (Show Details)Mar 4 2019, 3:00 PM
bd808 added a subtask: T217932: Change log routing to ELK cluster to use rsyslog->kafka rather than talking directly to the ELK cluster.Mar 8 2019, 11:49 PM
herron updated the task description. (Show Details)Mar 12 2019, 5:49 PM
gerritbot added a comment.Mar 12 2019, 6:45 PM

Change 495962 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: migrate mediawiki syslogs to logging pipeline

https://gerrit.wikimedia.org/r/495962

gerritbot added a comment.Mar 12 2019, 7:17 PM

Change 495980 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: add netdev_kafka_relay compatability endpoint

https://gerrit.wikimedia.org/r/495980

gerritbot added a comment.Mar 12 2019, 9:18 PM

Change 496021 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add tcp json_lines localhost compatability endpoint

https://gerrit.wikimedia.org/r/496021

gerritbot added a comment.Mar 12 2019, 9:18 PM

Change 496022 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add udp json logback localhost compatability endpoint

https://gerrit.wikimedia.org/r/496022

gerritbot added a comment.Mar 14 2019, 8:43 PM

Change 495962 merged by Herron:
[operations/puppet@production] logstash: send mediawiki syslogs to logging pipeline

https://gerrit.wikimedia.org/r/495962

gerritbot added a comment.Mar 14 2019, 9:28 PM

Change 496022 merged by Herron:
[operations/puppet@production] logstash: add udp json logback localhost compatibility endpoint

https://gerrit.wikimedia.org/r/496022

gerritbot added a comment.Mar 15 2019, 3:34 PM

Change 496806 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: update syslog_json template with format jsonfr

https://gerrit.wikimedia.org/r/496806

herron updated the task description. (Show Details)Mar 15 2019, 5:49 PM
gerritbot added a comment.Mar 15 2019, 7:36 PM

Change 496871 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: remove format=json from msg field in syslog_json template

https://gerrit.wikimedia.org/r/496871

gerritbot added a comment.Mar 15 2019, 9:20 PM

Change 496871 abandoned by Herron:
rsyslog: remove format=json from msg field in syslog_json template

Reason:
in favor of Ib228fdc7c856f68590261b5c410594a60cde6fe6

https://gerrit.wikimedia.org/r/496871

gerritbot added a comment.Mar 18 2019, 3:10 PM

Change 496806 merged by Herron:
[operations/puppet@production] rsyslog: update syslog_json template with format jsonf

https://gerrit.wikimedia.org/r/496806

gerritbot added a comment.Mar 19 2019, 5:37 PM

Change 497570 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: move mediawiki syslogs to logging pipeline

https://gerrit.wikimedia.org/r/497570

gerritbot added a comment.Mar 19 2019, 7:23 PM

Change 497587 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: add omkafka load statement to udp_json_logback config

https://gerrit.wikimedia.org/r/497587

gerritbot added a comment.Mar 19 2019, 8:01 PM

Change 497587 merged by Herron:
[operations/puppet@production] rsyslog: add omkafka load statement to udp_json_logback config

https://gerrit.wikimedia.org/r/497587

gerritbot added a comment.Mar 19 2019, 9:05 PM

Change 497614 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] ores: ship to logstash via the kafka logging pipeline

https://gerrit.wikimedia.org/r/497614

gerritbot added a comment.Mar 20 2019, 5:59 PM

Change 497570 merged by Herron:
[operations/puppet@production] logstash: move mediawiki syslogs to logging pipeline

https://gerrit.wikimedia.org/r/497570

herron updated the task description. (Show Details)Mar 20 2019, 6:46 PM
gerritbot added a comment.Mar 22 2019, 8:02 PM

Change 498467 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: send varnish syslogs via kafka logging pipeline

https://gerrit.wikimedia.org/r/498467

gerritbot added a comment.Mar 28 2019, 5:35 PM

Change 499829 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] role: add kafka_shipper to elasticsearch::cirrus role

https://gerrit.wikimedia.org/r/499829

gerritbot added a comment.Mar 28 2019, 5:41 PM

Change 499829 merged by Cwhite:
[operations/puppet@production] role: add kafka_shipper to elasticsearch::cirrus role

https://gerrit.wikimedia.org/r/499829

bd808 closed subtask T217932: Change log routing to ELK cluster to use rsyslog->kafka rather than talking directly to the ELK cluster as Resolved.Mar 28 2019, 10:23 PM
gerritbot added a comment.Mar 29 2019, 7:04 PM

Change 500099 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: do not mutate level for mjolnir

https://gerrit.wikimedia.org/r/500099

debt closed subtask T218833: Migrate mjolnir to stdout/syslog/cee logging output as Resolved.Apr 5 2019, 10:30 PM
gerritbot added a comment.Apr 8 2019, 3:22 PM

Change 497614 merged by Herron:
[operations/puppet@production] ores: ship to logstash via the kafka logging pipeline

https://gerrit.wikimedia.org/r/497614

gerritbot added a comment.Apr 9 2019, 4:03 PM

Change 502527 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] ores: ship to logstash via the kafka logging pipeline

https://gerrit.wikimedia.org/r/502527

gerritbot added a comment.Apr 10 2019, 1:22 PM

Change 493243 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logging: move webrequest-5xx to logging pipeline

https://gerrit.wikimedia.org/r/493243

fgiunchedi updated the task description. (Show Details)Apr 10 2019, 1:28 PM
fgiunchedi updated the task description. (Show Details)
gerritbot added a comment.Apr 11 2019, 9:23 AM

Change 493243 merged by Filippo Giunchedi:
[operations/puppet@production] logging: move webrequest-5xx to logging pipeline

https://gerrit.wikimedia.org/r/493243

gerritbot added a comment.Apr 11 2019, 12:32 PM

Change 502985 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] webrequest: set logger max message size

https://gerrit.wikimedia.org/r/502985

gerritbot added a comment.Apr 11 2019, 12:34 PM

Change 502985 merged by Filippo Giunchedi:
[operations/puppet@production] webrequest: set logger max message size

https://gerrit.wikimedia.org/r/502985

fgiunchedi updated the task description. (Show Details)Apr 11 2019, 12:41 PM
gerritbot added a comment.Apr 15 2019, 5:00 PM

Change 500099 merged by Cwhite:
[operations/puppet@production] profile: do not mutate level for mjolnir

https://gerrit.wikimedia.org/r/500099

gerritbot added a comment.Apr 15 2019, 6:41 PM

Change 504074 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: do not mutate level for mjolnir

https://gerrit.wikimedia.org/r/504074

gerritbot added a comment.Apr 15 2019, 6:53 PM

Change 504074 merged by Cwhite:
[operations/puppet@production] profile: do not mutate level for mjolnir

https://gerrit.wikimedia.org/r/504074

gerritbot added a comment.May 2 2019, 3:40 PM

Change 495980 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: add netdev_kafka_relay compatability endpoint

https://gerrit.wikimedia.org/r/495980

gerritbot added a comment.May 2 2019, 4:32 PM

Change 496021 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add tcp json_lines localhost compatability endpoint

https://gerrit.wikimedia.org/r/496021

fgiunchedi edited parent tasks, added: T227080: Deprecate all non-Kafka logstash inputs; removed: T213157: Increase utilization of application logging pipeline (FY2018-2019 Q3 TEC6).Jul 2 2019, 12:51 PM
fgiunchedi added a project: observability.Aug 19 2019, 2:30 PM
hashar mentioned this in T233134: logstash-beta.wmflabs.org does not receive any mediawiki events.Sep 17 2019, 5:38 PM
Paladox subscribed.Sep 19 2019, 5:40 PM
fgiunchedi updated the task description. (Show Details)Jan 17 2020, 1:33 PM
fgiunchedi updated the task description. (Show Details)Jan 17 2020, 1:40 PM
fgiunchedi updated the task description. (Show Details)Jan 17 2020, 1:46 PM
fgiunchedi updated the task description. (Show Details)Jan 17 2020, 1:54 PM
gerritbot added a comment.Jan 17 2020, 1:59 PM

Change 565573 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: turn down logstash tcp json_lines endpoint

https://gerrit.wikimedia.org/r/565573

fgiunchedi updated the task description. (Show Details)Jan 17 2020, 2:00 PM
fgiunchedi updated the task description. (Show Details)Jan 21 2020, 11:40 AM
gerritbot added a comment.Feb 3 2020, 3:22 PM

Change 569564 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] wip: cassandra logs to logging pipeline

https://gerrit.wikimedia.org/r/569564

fgiunchedi updated the task description. (Show Details)Feb 3 2020, 3:33 PM
gerritbot added a comment.Feb 3 2020, 3:50 PM

Change 569570 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] cassandra: use wmflib::secret for binary files

https://gerrit.wikimedia.org/r/569570

fgiunchedi updated the task description. (Show Details)Feb 11 2020, 9:22 AM
fgiunchedi updated the task description. (Show Details)Feb 18 2020, 2:03 PM
fgiunchedi closed this task as Resolved.Feb 18 2020, 2:05 PM
fgiunchedi claimed this task.

I'm resolving this task since all the followup is already tracked in parent T227080: Deprecate all non-Kafka logstash inputs

gerritbot added a comment.Sep 22 2020, 10:50 AM

Change 492676 abandoned by Filippo Giunchedi:
[operations/puppet@production] [WIP] move varnish logging to logging pipeline

Reason:
Superceded by Ic56b29517

https://gerrit.wikimedia.org/r/492676

gerritbot added a comment.Aug 9 2021, 12:14 PM

Change 565573 abandoned by Filippo Giunchedi:

[operations/puppet@production] hieradata: turn down logstash tcp json_lines endpoint

Reason:

Obsolete

https://gerrit.wikimedia.org/r/565573

Maintenance_bot added a project: SRE Observability.Aug 9 2021, 12:45 PM
gerritbot added a comment.Apr 2 2024, 2:09 PM

Change #496021 abandoned by Herron:

[operations/puppet@production] logstash: add tcp json_lines localhost compatability endpoint

Reason:

spring cleaning -- stale patch

https://gerrit.wikimedia.org/r/496021

gerritbot added a comment.Apr 2 2024, 2:09 PM

Change #498467 abandoned by Herron:

[operations/puppet@production] logstash: send varnish syslogs via kafka logging pipeline

Reason:

spring cleaning -- stale patch

https://gerrit.wikimedia.org/r/498467

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits