Page MenuHomePhabricator

Onboard at least 10 new non-sensitive log producers to the logging pipeline
Closed, ResolvedPublic

Description

By non-sensitive log producers in this context we mean applications/services that currently are not present in Logstash but should. Additionally their logs shouldn't contain anything considered sensitive to non-root users (e.g. passwords or other secrets).
A first audit can be found at https://phabricator.wikimedia.org/T198756#4594061 for applications that have logs on disk and https://phabricator.wikimedia.org/T198756#4430601 for applications logging to the central syslog hosts already.

  • Ship peopleweb apache2 error logs to ELK T209860
  • Get phabricator apache error logs into logstash T141895
  • Get icinga alerts into logstash T7
  • Jenkins daemon logs T143733
  • Gerrit logs T141324
  • PuppetDB logs T210458
  • Prometheus service logs T210455
  • Swift logs T63780 (postponed until logstash storage expansion completes)
  • postgres
  • logstash
  • grafana T210846
  • kafka T63788
  • zookeeper T63789
  • etherpad

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
herron triaged this task as Normal priority.Oct 2 2018, 5:25 PM

Change 473847 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] peopleweb: include rsyslog kafka_shipper

https://gerrit.wikimedia.org/r/473847

herron added a subscriber: herron.Nov 15 2018, 9:21 PM

Change 473847 merged by Herron:
[operations/puppet@production] peopleweb: include rsyslog kafka_shipper

https://gerrit.wikimedia.org/r/473847

Change 474317 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kafka_shipper: use mmrm1stspace to remove leading space in msg field

https://gerrit.wikimedia.org/r/474317

Change 474320 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kafka_shipper: add apache2 to lookup table with kafka output

https://gerrit.wikimedia.org/r/474320

Change 474317 merged by Herron:
[operations/puppet@production] kafka_shipper: use mmrm1stspace to remove leading space in msg field

https://gerrit.wikimedia.org/r/474317

Change 474320 merged by Herron:
[operations/puppet@production] kafka_shipper: add apache2 to lookup table with kafka output

https://gerrit.wikimedia.org/r/474320

herron updated the task description. (Show Details)Nov 19 2018, 4:47 PM

Change 474760 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog::input::file add rsyslog imfile wrapper for file ingestion

https://gerrit.wikimedia.org/r/474760

Change 474813 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add type "apache-error" and use logstash core patterns

https://gerrit.wikimedia.org/r/474813

Change 474760 merged by Herron:
[operations/puppet@production] rsyslog::input::file add rsyslog imfile wrapper for file ingestion

https://gerrit.wikimedia.org/r/474760

Change 474813 merged by Herron:
[operations/puppet@production] logstash: add type "apache2-error" and use logstash core patterns

https://gerrit.wikimedia.org/r/474813

Change 474974 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: output logs tagged 'input-file-apache2-error' to kafka

https://gerrit.wikimedia.org/r/474974

Change 474974 merged by Herron:
[operations/puppet@production] rsyslog: output logs tagged 'input-file-apache2-error' to kafka

https://gerrit.wikimedia.org/r/474974

herron updated the task description. (Show Details)Nov 20 2018, 9:22 PM

Change 475104 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: rename 'severity' syslog field if present

https://gerrit.wikimedia.org/r/475104

Change 475110 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] logstash: add 'level' normalization rules

https://gerrit.wikimedia.org/r/475110

Change 475110 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: add 'level' normalization rules

https://gerrit.wikimedia.org/r/475110

Change 475104 merged by Filippo Giunchedi:
[operations/puppet@production] logstash: rename 'severity' syslog field if present

https://gerrit.wikimedia.org/r/475104

herron updated the task description. (Show Details)Nov 26 2018, 7:21 PM
herron updated the task description. (Show Details)Nov 26 2018, 9:23 PM
herron updated the task description. (Show Details)Nov 26 2018, 10:39 PM
herron updated the task description. (Show Details)
herron updated the task description. (Show Details)Nov 28 2018, 4:00 PM
herron updated the task description. (Show Details)Nov 28 2018, 9:14 PM
herron closed subtask T210455: Ship prometheus logs to ELK as Resolved.

Change 475840 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog:input:file add multiline handling and ship gerrit logs to ELK

https://gerrit.wikimedia.org/r/475840

Change 475840 merged by Herron:
[operations/puppet@production] rsyslog: input::file add multiline handling & ship gerrit logs to ELK

https://gerrit.wikimedia.org/r/475840

Change 476592 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: input::file add multiline handling & ship gerrit logs to ELK

https://gerrit.wikimedia.org/r/476592

Change 476592 merged by Herron:
[operations/puppet@production] rsyslog: input::file add multiline handling & ship gerrit logs to ELK

https://gerrit.wikimedia.org/r/476592

Change 476803 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] kafka_shipper: set format=json on message field in rsyslog template

https://gerrit.wikimedia.org/r/476803

Change 476803 merged by Herron:
[operations/puppet@production] kafka_shipper: set format=json on message field in rsyslog template

https://gerrit.wikimedia.org/r/476803

Change 476804 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add mutate for escaped tabs in rsyslog multiline events

https://gerrit.wikimedia.org/r/476804

Change 476804 merged by Herron:
[operations/puppet@production] logstash: add mutate for escaped tabs in rsyslog multiline events

https://gerrit.wikimedia.org/r/476804

Change 476805 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: update gerrit multiline message start regex

https://gerrit.wikimedia.org/r/476805

Change 476805 merged by Herron:
[operations/puppet@production] logstash: update gerrit multiline message start regex

https://gerrit.wikimedia.org/r/476805

Change 476875 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] grafana: ship grafana-server syslogs to ELK

https://gerrit.wikimedia.org/r/476875

Change 476875 merged by Herron:
[operations/puppet@production] grafana: ship grafana-server syslogs to ELK

https://gerrit.wikimedia.org/r/476875

herron updated the task description. (Show Details)
herron updated the task description. (Show Details)

Change 476890 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: ship logstash server logs to ELK

https://gerrit.wikimedia.org/r/476890

Change 476890 merged by Herron:
[operations/puppet@production] logstash: ship logstash server logs to ELK

https://gerrit.wikimedia.org/r/476890

herron updated the task description. (Show Details)Nov 30 2018, 5:01 PM
herron updated the task description. (Show Details)Nov 30 2018, 9:30 PM

Change 476926 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: ship etherpad logs to ELK

https://gerrit.wikimedia.org/r/476926

Change 476926 merged by Herron:
[operations/puppet@production] logstash: ship etherpad logs to ELK

https://gerrit.wikimedia.org/r/476926

herron updated the task description. (Show Details)Nov 30 2018, 9:43 PM

Change 476977 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: ship zookeeper logs to ELK

https://gerrit.wikimedia.org/r/476977

Change 476982 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: ship kafka server logs to ELK

https://gerrit.wikimedia.org/r/476982

Change 476982 merged by Herron:
[operations/puppet@production] logstash: ship kafka server logs to ELK

https://gerrit.wikimedia.org/r/476982

fgiunchedi closed this task as Resolved.Jan 16 2019, 11:18 AM
fgiunchedi claimed this task.

This was completed, there will be more followup while deprecating more logstash inputs on T213157