This task tracks adding Kafka producer capabilities to rsyslog, by way of omkafka. I did some exploratory/prototyping work in T198757: Investigate log shipping methods and standardize on them (logstash) and seems to be generally working.
Non-exhaustive list of things to do:
- Get rsyslog 8.38 (from stretch-backports) rebuilt for jessie-backports and uploaded internally, this is mostly done in https://phabricator.wikimedia.org/T198757#4581552. Part of the reason is that starting with rsyslog 8.28 the omkafka module has received significant work, including resending failed messages: https://github.com/rsyslog/rsyslog/issues/1052
- Upgrade to rsyslog 8.38 across the fleet (jessie/stretch)
- Select one of the forks of https://github.com/soundcloud/rsyslog_exporter (or that repo itself) to get some visibility into rsyslog itself and configure rsyslog to use it
- Rollout rsyslog_exporter to the fleet and instruct Prometheus to fetch its metrics
- Come up with a rsyslog configuration that includes omkafka and a way to opt-in new and existing applications