This task tracks adding Kafka producer capabilities to rsyslog, by way of `omkafka`. I did some exploratory/prototyping work in {T198757} and seems to be generally working.
Non-exhaustive list of things to do:
[x] Get rsyslog 8.38 (from `stretch-backports`) rebuilt for `jessie-backports` and uploaded internally, this is mostly done in https://phabricator.wikimedia.org/T198757#4581552. Part of the reason is that starting with rsyslog 8.28 the `omkafka` module has received significant work, including resending failed messages: https://github.com/rsyslog/rsyslog/issues/1052
[x] Upgrade to rsyslog 8.38 across the fleet (jessie/stretch)
[x] Select one of the forks of https://github.com/soundcloud/rsyslog_exporter (or that repo itself) to get some visibility into rsyslog itself and configure rsyslog to use it
[] Rollout `rsyslog_exporter` to the fleet and instruct Prometheus to fetch its metrics
[] Come up with a rsyslog configuration that includes `omkafka` and a way to opt-in new and existing applications