Page MenuHomePhabricator

Setup rsyslog to be able to produce logs to Kafka
Closed, ResolvedPublic

Description

This task tracks adding Kafka producer capabilities to rsyslog, by way of omkafka. I did some exploratory/prototyping work in T198757: Investigate log shipping methods and standardize on them (logstash) and seems to be generally working.

Non-exhaustive list of things to do:

  • Get rsyslog 8.38 (from stretch-backports) rebuilt for jessie-backports and uploaded internally, this is mostly done in https://phabricator.wikimedia.org/T198757#4581552. Part of the reason is that starting with rsyslog 8.28 the omkafka module has received significant work, including resending failed messages: https://github.com/rsyslog/rsyslog/issues/1052
  • Upgrade to rsyslog 8.38 across the fleet (jessie/stretch)
  • Select one of the forks of https://github.com/soundcloud/rsyslog_exporter (or that repo itself) to get some visibility into rsyslog itself and configure rsyslog to use it
  • Rollout rsyslog_exporter to the fleet and instruct Prometheus to fetch its metrics
  • Come up with a rsyslog configuration that includes omkafka and a way to opt-in new and existing applications

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -1
operations/puppetproduction+22 -1
operations/puppetproduction+20 -0
operations/puppetproduction+1 -3
operations/puppetproduction+2 -0
operations/puppetproduction+2 -2
operations/puppetproduction+12 -0
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+112 -0
operations/puppetproduction+340 -317
operations/debs/prometheus-rsyslog-exportermaster+128 -0
operations/debs/rsyslogdebian/stretch-wikimedia+20 -0
operations/debs/rsyslogdebian/jessie-wikimedia+15 -0
operations/debs/rsyslogdebian/jessie-wikimedia+2 -1
operations/debs/rsyslogdebian/jessie-wikimedia+3 -152
operations/debs/rsyslogdebian/jessie-wikimedia+9 -5
Show related patches Customize query in gerrit

Event Timeline

fgiunchedi triaged this task as Medium priority.Oct 10 2018, 2:52 PM
fgiunchedi created this task.

Change 468600 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/debs/rsyslog@debian/jessie-wikimedia] Rebuild for jessie-wikimedia

https://gerrit.wikimedia.org/r/468600

Change 468601 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/debs/rsyslog@debian/jessie-wikimedia] Drop mongodb/relp/czmq integrations, not used at WMF and missing/old from jessie(-backports)

https://gerrit.wikimedia.org/r/468601

Change 468602 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/debs/rsyslog@debian/jessie-wikimedia] Build-depend on newer librdkafka 0.11

https://gerrit.wikimedia.org/r/468602

Change 468603 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/debs/rsyslog@debian/jessie-wikimedia] Enable mmkubernetes (build depends on libcurl and liblognorm)

https://gerrit.wikimedia.org/r/468603

fgiunchedi updated the task description. (Show Details)Oct 22 2018, 10:52 AM

Change 468965 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/debs/rsyslog@debian/stretch-wikimedia] Enable mmkubernetes (build depends on libcurl and liblognorm) and build rsyslog-kubernetes

https://gerrit.wikimedia.org/r/468965

Change 468978 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/debs/prometheus-rsyslog-exporter@master] debian: add packaging

https://gerrit.wikimedia.org/r/468978

Change 468603 merged by Filippo Giunchedi:
[operations/debs/rsyslog@debian/jessie-wikimedia] Enable mmkubernetes (build depends on libcurl and liblognorm)

https://gerrit.wikimedia.org/r/468603

Change 468602 merged by Filippo Giunchedi:
[operations/debs/rsyslog@debian/jessie-wikimedia] Build-depend on newer librdkafka 0.11

https://gerrit.wikimedia.org/r/468602

Change 468601 merged by Filippo Giunchedi:
[operations/debs/rsyslog@debian/jessie-wikimedia] Drop mongodb/relp/czmq integrations, not used at WMF and missing/old from jessie(-backports)

https://gerrit.wikimedia.org/r/468601

Change 468600 merged by Filippo Giunchedi:
[operations/debs/rsyslog@debian/jessie-wikimedia] Rebuild for jessie-wikimedia

https://gerrit.wikimedia.org/r/468600

Change 468965 merged by Filippo Giunchedi:
[operations/debs/rsyslog@debian/stretch-wikimedia] Enable mmkubernetes (build depends on libcurl and liblognorm) and build rsyslog-kubernetes

https://gerrit.wikimedia.org/r/468965

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Oct 24 2018, 3:09 PM

Change 468978 merged by Filippo Giunchedi:
[operations/debs/prometheus-rsyslog-exporter@master] debian: add packaging

https://gerrit.wikimedia.org/r/468978

Change 470452 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] role::logstash::collector: migrate to profile::logstash::collector

https://gerrit.wikimedia.org/r/470452

herron added a subscriber: herron.Oct 29 2018, 7:04 PM

Change 470454 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add generic kafka input config

https://gerrit.wikimedia.org/r/470454

Mentioned in SAL (#wikimedia-operations) [2018-10-30T14:56:38Z] <godog> gradually upgrade rsyslog to 8.38 on jessie hosts - T206633

Change 470452 merged by Herron:
[operations/puppet@production] role::logstash::collector: migrate to profile::logstash::collector

https://gerrit.wikimedia.org/r/470452

Change 471114 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] WIP: profile::rsyslog::logstash_shipper: generalized logstash shipper

https://gerrit.wikimedia.org/r/471114

fgiunchedi updated the task description. (Show Details)Nov 5 2018, 3:16 PM

Change 471114 merged by Herron:
[operations/puppet@production] profile::rsyslog::kafka_shipper: generalized log pipeline/elk shipper

https://gerrit.wikimedia.org/r/471114

herron updated the task description. (Show Details)Nov 12 2018, 8:53 PM

Change 473170 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: rollout syslog_exporter in esams

https://gerrit.wikimedia.org/r/473170

Change 473171 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: rollout syslog_exporter in eqsin

https://gerrit.wikimedia.org/r/473171

Change 473172 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: rollout syslog_exporter in codfw

https://gerrit.wikimedia.org/r/473172

Change 473173 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] hieradata: rollout syslog_exporter in eqiad

https://gerrit.wikimedia.org/r/473173

Change 473171 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: rollout syslog_exporter in eqsin

https://gerrit.wikimedia.org/r/473171

Change 473170 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: rollout syslog_exporter in esams

https://gerrit.wikimedia.org/r/473170

Change 473207 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: add rsyslog_exporter rules

https://gerrit.wikimedia.org/r/473207

Change 473207 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: add rsyslog_exporter rules

https://gerrit.wikimedia.org/r/473207

Change 473172 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: rollout syslog_exporter in codfw

https://gerrit.wikimedia.org/r/473172

Change 473173 merged by Filippo Giunchedi:
[operations/puppet@production] hieradata: rollout syslog_exporter in eqiad

https://gerrit.wikimedia.org/r/473173

Change 473501 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] prometheus: aggregate rsyslog_queue_full rate

https://gerrit.wikimedia.org/r/473501

Change 473501 merged by Filippo Giunchedi:
[operations/puppet@production] prometheus: aggregate rsyslog_queue_full rate

https://gerrit.wikimedia.org/r/473501

Change 470454 merged by Herron:
[operations/puppet@production] logstash: add rsyslog-shipper kafka input config

https://gerrit.wikimedia.org/r/470454

Change 473607 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: add rsyslog-shipper kafka input config

https://gerrit.wikimedia.org/r/473607

Change 473607 merged by Herron:
[operations/puppet@production] logstash: add rsyslog-shipper kafka input config

https://gerrit.wikimedia.org/r/473607

fgiunchedi closed this task as Resolved.Dec 4 2018, 1:06 PM
fgiunchedi claimed this task.
fgiunchedi updated the task description. (Show Details)

This is completed!

Change 478045 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] rsyslog: increase omkafka timeout to 10s

https://gerrit.wikimedia.org/r/478045

Change 478045 merged by Herron:
[operations/puppet@production] rsyslog: increase omkafka timeout to 10s

https://gerrit.wikimedia.org/r/478045