The intention is that ulogd logs from all servers will be sent to kafaka as such it would seem to make senses to move ::profile::rsyslog::kafka_shipper to the ::standard class

Indeed this is something that has been discussed elsewhere from time to time and would be nice to do. I'd be game to move forward with that as part of this project.

In addition to that, we'll also want to parse these events into fields. Looking at the log format as-is we can accomplish that using a grok filter in logstash. There may be some patterns already existing for iptables format that we can use.

Its also worth pointing out that ulogd supports native json output[1] but only to a separate log file not syslog. Prometheus is was simpler to keep things raw and in (r)syslog which may be the case here as well. In relation to writing our own grok rules i suspect the mtail patterns will be a good starting point[2]

[1]https://phabricator.wikimedia.org/T116011#4927275
[2]https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/mtail/files/programs/ulogd.mtail

I'm for erring on the side of simplicity. Since these logs are useful on the command line of an individual host, on centrallog, and in Kibana, it makes sense to me to stream the ulogd syslogs formatted as shown in the description to logstash and parse them with a grok pattern.

sounds good

Change 505737 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] kafka shipper: move kafka rsyslog shipping to base profile

https://gerrit.wikimedia.org/r/505737

Change 505750 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] kafka shipper: add ulogd to kafka forwarding rules

https://gerrit.wikimedia.org/r/505750

Change 505783 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] ulogd logstash: Add rule to parse ulogd ouput to json

https://gerrit.wikimedia.org/r/505783

Change 505817 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] kafka: It was pointed out that kafak shipping may not work for all hosts

https://gerrit.wikimedia.org/r/505817

Change 505817 abandoned by Jbond:
kafka: It was pointed out that kafak shipping may not work for all hosts

Reason:
replaced with 505737

https://gerrit.wikimedia.org/r/505817

Change 506377 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] nflog: add logging prefix to firewall log entries

https://gerrit.wikimedia.org/r/506377

Change 506377 merged by Jbond:
[operations/puppet@production] nflog: add logging prefix to firewall log entries

https://gerrit.wikimedia.org/r/506377

Change 506400 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] logstash: add ulog parser to logstash

https://gerrit.wikimedia.org/r/506400

Change 505783 abandoned by Jbond:
ulogd logstash: Add rule to parse ulogd ouput to json

Reason:
Replaced for 506400

https://gerrit.wikimedia.org/r/505783

Mentioned in SAL (#wikimedia-operations) [2019-04-30T14:26:33Z] <jbond42> disable-puppet "T220987: global kafaka log shipping - staged rollout (jbond)"

Change 505737 merged by Jbond:
[operations/puppet@production] kafka shipper: move kafka rsyslog shipping to base profile

https://gerrit.wikimedia.org/r/505737

Mentioned in SAL (#wikimedia-operations) [2019-04-30T14:58:42Z] <jbond42> enable-puppet "T220987: global kafaka log shipping - staged rollout (jbond)"

Change 505750 merged by Jbond:
[operations/puppet@production] kafka shipper: add ulogd to kafka forwarding rules

https://gerrit.wikimedia.org/r/505750

Change 506400 merged by Jbond:
[operations/puppet@production] logstash: add ulog parser to logstash

https://gerrit.wikimedia.org/r/506400