Page MenuHomePhabricator
Create Task
Maniphest T205852

Onboard at least 10 new non-sensitive log producers to the logging pipeline
Closed, ResolvedPublic
Actions

Description

By non-sensitive log producers in this context we mean applications/services that currently are not present in Logstash but should. Additionally their logs shouldn't contain anything considered sensitive to non-root users (e.g. passwords or other secrets).
A first audit can be found at https://phabricator.wikimedia.org/T198756#4594061 for applications that have logs on disk and https://phabricator.wikimedia.org/T198756#4430601 for applications logging to the central syslog hosts already.

  • Ship peopleweb apache2 error logs to ELK T209860
  • Get phabricator apache error logs into logstash T141895
  • Get icinga alerts into logstash T7
  • Jenkins daemon logs T143733
  • Gerrit logs T141324
  • PuppetDB logs T210458
  • Prometheus service logs T210455
  • Swift logs T63780 (postponed until logstash storage expansion completes)
  • postgres
  • logstash
  • grafana T210846
  • kafka T63788
  • zookeeper T63789
  • etherpad

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
logstash: ship zookeeper logs to ELKoperations/puppetproduction+32 -19
logstash: ship kafka server logs to ELKoperations/puppetproduction+11 -0
logstash: ship etherpad logs to ELKoperations/puppetproduction+9 -0
logstash: ship logstash server logs to ELKoperations/puppetproduction+10 -0
grafana: ship grafana-server syslogs to ELKoperations/puppetproduction+3 -0
logstash: update gerrit multiline message start regexoperations/puppetproduction+1 -1
logstash: add mutate for escaped tabs in rsyslog multiline eventsoperations/puppetproduction+3 -1
kafka_shipper: set format=json on message field in rsyslog templateoperations/puppetproduction+1 -1
rsyslog: input::file add multiline handling & ship gerrit logs to ELKoperations/puppetproduction+45 -0
rsyslog: input::file add multiline handling & ship gerrit logs to ELKoperations/puppetproduction+45 -0
logstash: rename 'severity' syslog field if presentoperations/puppetproduction+18 -0
logstash: add 'level' normalization rulesoperations/puppetproduction+4 -1
rsyslog: output logs tagged 'input-file-apache2-error' to kafkaoperations/puppetproduction+1 -0
logstash: add type "apache2-error" and use logstash core patternsoperations/puppetproduction+40 -0
rsyslog::input::file add rsyslog imfile wrapper for file ingestionoperations/puppetproduction+39 -0
kafka_shipper: add apache2 to lookup table with kafka outputoperations/puppetproduction+1 -1
kafka_shipper: use mmrm1stspace to remove leading space in msg fieldoperations/puppetproduction+7 -0
peopleweb: include rsyslog kafka_shipperoperations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
fgiunchedi moved this task from Backlog to In Dev/Progress on the Wikimedia-Logstash board.Oct 15 2018, 2:25 PM
fgiunchedi moved this task from In Dev/Progress to Up next on the Wikimedia-Logstash board.Oct 15 2018, 2:34 PM
gerritbot subscribed.Nov 15 2018, 9:09 PM

Change 473847 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] peopleweb: include rsyslog kafka_shipper

https://gerrit.wikimedia.org/r/473847

gerritbot added a project: Patch-For-Review.Nov 15 2018, 9:09 PM
herron subscribed.Nov 15 2018, 9:21 PM
gerritbot added a comment.Nov 16 2018, 3:01 AM

Change 473847 merged by Herron:
[operations/puppet@production] peopleweb: include rsyslog kafka_shipper

https://gerrit.wikimedia.org/r/473847

gerritbot added a comment.Nov 16 2018, 7:35 PM

Change 474317 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kafka_shipper: use mmrm1stspace to remove leading space in msg field

https://gerrit.wikimedia.org/r/474317

gerritbot added a comment.Nov 16 2018, 7:53 PM

Change 474320 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kafka_shipper: add apache2 to lookup table with kafka output

https://gerrit.wikimedia.org/r/474320

gerritbot added a comment.Nov 17 2018, 1:06 AM

Change 474317 merged by Herron:
[operations/puppet@production] kafka_shipper: use mmrm1stspace to remove leading space in msg field

https://gerrit.wikimedia.org/r/474317

gerritbot added a comment.Nov 17 2018, 1:17 AM

Change 474320 merged by Herron:
[operations/puppet@production] kafka_shipper: add apache2 to lookup table with kafka output

https://gerrit.wikimedia.org/r/474320

herron mentioned this in T206454: Setup Kafka cluster, producers and consumers for logging pipeline.Nov 19 2018, 3:19 PM
herron added a project: User-herron.Nov 19 2018, 4:27 PM
herron added a subtask: T7: Get icinga alerts into logstash.Nov 19 2018, 4:29 PM
herron added a subtask: T141895: Get phabricator error logs into logstash.
herron added a subtask: T209860: Ship peopleweb apache2 error logs to ELK.Nov 19 2018, 4:37 PM
herron updated the task description. (Show Details)Nov 19 2018, 4:47 PM
gerritbot added a comment.Nov 19 2018, 7:10 PM

Change 474760 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog::input::file add rsyslog imfile wrapper for file ingestion

https://gerrit.wikimedia.org/r/474760

gerritbot added a comment.Nov 19 2018, 10:47 PM

Change 474813 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add type "apache-error" and use logstash core patterns

https://gerrit.wikimedia.org/r/474813

gerritbot added a comment.Nov 20 2018, 5:32 PM

Change 474760 merged by Herron:
[operations/puppet@production] rsyslog::input::file add rsyslog imfile wrapper for file ingestion

https://gerrit.wikimedia.org/r/474760

gerritbot added a comment.Nov 20 2018, 7:28 PM

Change 474813 merged by Herron:
[operations/puppet@production] logstash: add type "apache2-error" and use logstash core patterns

https://gerrit.wikimedia.org/r/474813

gerritbot added a comment.Nov 20 2018, 7:50 PM

Change 474974 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: output logs tagged 'input-file-apache2-error' to kafka

https://gerrit.wikimedia.org/r/474974

gerritbot added a comment.Nov 20 2018, 7:53 PM

Change 474974 merged by Herron:
[operations/puppet@production] rsyslog: output logs tagged 'input-file-apache2-error' to kafka

https://gerrit.wikimedia.org/r/474974

herron added a subtask: T143733: Send Jenkins daemon logs to logstash.Nov 20 2018, 8:54 PM
herron updated the task description. (Show Details)
herron updated the task description. (Show Details)Nov 20 2018, 9:22 PM
gerritbot added a comment.Nov 21 2018, 3:14 PM

Change 475104 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: rename 'severity' syslog field if present

https://gerrit.wikimedia.org/r/475104

gerritbot added a comment.Nov 21 2018, 4:16 PM

Change 475110 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: add 'level' normalization rules

https://gerrit.wikimedia.org/r/475110

gerritbot added a comment.Nov 22 2018, 8:38 AM

Change 475110 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: add 'level' normalization rules

https://gerrit.wikimedia.org/r/475110

gerritbot added a comment.Nov 22 2018, 8:58 AM

Change 475104 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: rename 'severity' syslog field if present

https://gerrit.wikimedia.org/r/475104

herron closed subtask T209860: Ship peopleweb apache2 error logs to ELK as Resolved.Nov 26 2018, 5:54 PM
herron updated the task description. (Show Details)Nov 26 2018, 7:21 PM
herron closed subtask T7: Get icinga alerts into logstash as Resolved.Nov 26 2018, 7:34 PM
herron closed subtask T141895: Get phabricator error logs into logstash as Resolved.Nov 26 2018, 8:05 PM
herron updated the task description. (Show Details)
herron mentioned this in T141324: Look into shoving gerrit logs into logstash.Nov 26 2018, 9:14 PM
herron updated the task description. (Show Details)Nov 26 2018, 9:23 PM
herron added a subtask: T210455: Ship prometheus logs to ELK.Nov 26 2018, 9:47 PM
herron updated the task description. (Show Details)
herron updated the task description. (Show Details)Nov 26 2018, 10:03 PM
herron added a subtask: T210458: Ship PuppetDB logs to ELK .
herron updated the task description. (Show Details)Nov 26 2018, 10:39 PM
herron updated the task description. (Show Details)
herron added a subtask: T63780: Add swift logs to logstash.
herron updated the task description. (Show Details)Nov 28 2018, 4:00 PM
herron updated the task description. (Show Details)Nov 28 2018, 9:14 PM
herron closed subtask T210455: Ship prometheus logs to ELK as Resolved.
herron added a subtask: T141324: Look into shoving gerrit logs into logstash.Nov 29 2018, 4:49 PM
gerritbot added a comment.Nov 29 2018, 4:50 PM

Change 475840 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog:input:file add multiline handling and ship gerrit logs to ELK

https://gerrit.wikimedia.org/r/475840

gerritbot added a comment.Nov 29 2018, 5:58 PM

Change 475840 merged by Herron:
[operations/puppet@production] rsyslog: input::file add multiline handling & ship gerrit logs to ELK

https://gerrit.wikimedia.org/r/475840

gerritbot added a comment.Nov 29 2018, 6:26 PM

Change 476592 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: input::file add multiline handling & ship gerrit logs to ELK

https://gerrit.wikimedia.org/r/476592

gerritbot added a comment.Nov 30 2018, 12:32 AM

Change 476592 merged by Herron:
[operations/puppet@production] rsyslog: input::file add multiline handling & ship gerrit logs to ELK

https://gerrit.wikimedia.org/r/476592

gerritbot added a comment.Nov 30 2018, 4:51 AM

Change 476803 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kafka_shipper: set format=json on message field in rsyslog template

https://gerrit.wikimedia.org/r/476803

gerritbot added a comment.Nov 30 2018, 4:53 AM

Change 476803 merged by Herron:
[operations/puppet@production] kafka_shipper: set format=json on message field in rsyslog template

https://gerrit.wikimedia.org/r/476803

gerritbot added a comment.Nov 30 2018, 5:00 AM

Change 476804 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add mutate for escaped tabs in rsyslog multiline events

https://gerrit.wikimedia.org/r/476804

gerritbot added a comment.Nov 30 2018, 5:03 AM

Change 476804 merged by Herron:
[operations/puppet@production] logstash: add mutate for escaped tabs in rsyslog multiline events

https://gerrit.wikimedia.org/r/476804

gerritbot added a comment.Nov 30 2018, 5:17 AM

Change 476805 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: update gerrit multiline message start regex

https://gerrit.wikimedia.org/r/476805

gerritbot added a comment.Nov 30 2018, 3:10 PM

Change 476805 merged by Herron:
[operations/puppet@production] logstash: update gerrit multiline message start regex

https://gerrit.wikimedia.org/r/476805

herron added a subtask: T210846: Ship Grafana server logs to ELK.Nov 30 2018, 3:26 PM
herron updated the task description. (Show Details)
gerritbot added a comment.Nov 30 2018, 3:27 PM

Change 476875 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] grafana: ship grafana-server syslogs to ELK

https://gerrit.wikimedia.org/r/476875

gerritbot added a comment.Nov 30 2018, 3:40 PM

Change 476875 merged by Herron:
[operations/puppet@production] grafana: ship grafana-server syslogs to ELK

https://gerrit.wikimedia.org/r/476875

herron closed subtask T210846: Ship Grafana server logs to ELK as Resolved.Nov 30 2018, 3:49 PM
herron updated the task description. (Show Details)
herron updated the task description. (Show Details)
gerritbot added a comment.Nov 30 2018, 4:26 PM

Change 476890 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: ship logstash server logs to ELK

https://gerrit.wikimedia.org/r/476890

gerritbot added a comment.Nov 30 2018, 4:43 PM

Change 476890 merged by Herron:
[operations/puppet@production] logstash: ship logstash server logs to ELK

https://gerrit.wikimedia.org/r/476890

herron updated the task description. (Show Details)Nov 30 2018, 5:01 PM
herron updated the task description. (Show Details)Nov 30 2018, 9:30 PM
gerritbot added a comment.Nov 30 2018, 9:36 PM

Change 476926 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: ship etherpad logs to ELK

https://gerrit.wikimedia.org/r/476926

gerritbot added a comment.Nov 30 2018, 9:39 PM

Change 476926 merged by Herron:
[operations/puppet@production] logstash: ship etherpad logs to ELK

https://gerrit.wikimedia.org/r/476926

herron updated the task description. (Show Details)Nov 30 2018, 9:43 PM
herron added a subtask: T63789: Add Zookeeper log files to logstash.Nov 30 2018, 10:15 PM
herron added a subtask: T63788: Add kafka log files to logstash.
herron updated the task description. (Show Details)
gerritbot added a comment.Nov 30 2018, 10:18 PM

Change 476977 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: ship zookeeper logs to ELK

https://gerrit.wikimedia.org/r/476977

gerritbot added a comment.Nov 30 2018, 10:36 PM

Change 476982 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: ship kafka server logs to ELK

https://gerrit.wikimedia.org/r/476982

herron closed subtask T210458: Ship PuppetDB logs to ELK as Resolved.Dec 4 2018, 3:12 PM
Paladox closed subtask T141324: Look into shoving gerrit logs into logstash as Resolved.Dec 4 2018, 6:42 PM
gerritbot added a comment.Dec 6 2018, 8:28 PM

Change 476982 merged by Herron:
[operations/puppet@production] logstash: ship kafka server logs to ELK

https://gerrit.wikimedia.org/r/476982

hashar reopened subtask T141324: Look into shoving gerrit logs into logstash as Open.Dec 12 2018, 8:52 AM
fgiunchedi closed this task as Resolved.Jan 16 2019, 11:18 AM
fgiunchedi claimed this task.

This was completed, there will be more followup while deprecating more logstash inputs on T213157

Paladox closed subtask T141324: Look into shoving gerrit logs into logstash as Resolved.Feb 12 2019, 5:27 PM
Paladox reopened subtask T141324: Look into shoving gerrit logs into logstash as Open.
fgiunchedi added a project: observability.Aug 19 2019, 2:31 PM
hashar closed subtask T141324: Look into shoving gerrit logs into logstash as Resolved.Feb 12 2021, 4:02 PM
gerritbot added a comment.Apr 2 2024, 2:02 PM

Change #476977 abandoned by Herron:

[operations/puppet@production] logstash: ship zookeeper logs to ELK

Reason:

spring cleaning -- stale patch

https://gerrit.wikimedia.org/r/476977

Maintenance_bot removed a project: Patch-For-Review.Apr 2 2024, 2:31 PM
hashar closed subtask T143733: Send Jenkins daemon logs to logstash as Declined.Oct 24 2024, 1:34 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits