Page MenuHomePhabricator

Ferm: send ferm/iptables/ulogd logs to Kafaka/logstash/elasticsearch
Closed, ResolvedPublic

Description

Update puppet config so that ulogd logs are sent to kafaka. below is an example log entry

Apr 15 14:29:44 actinium ulogd[20403]:  IN=eth0 OUT= MAC=aa:00:00:1c:4b:fe:84:18:88:0d:df:c1:08:00 SRC=192.210.171.38 DST=208.80.154.49 LEN=40 TOS=00 PREC=0x00 TTL=246 ID=50569 PROTO=TCP SPT=50556 DPT=445 SEQ=2399362565 ACK=0 WINDOW=1024 SYN URGP=0 MARK=0

The intention is that ulogd logs from all servers will be sent to kafaka as such it would seem to make senses to move ::profile::rsyslog::kafka_shipper to the ::standard class

Event Timeline

jbond triaged this task as Normal priority.Apr 15 2019, 2:32 PM
jbond created this task.
jbond updated the task description. (Show Details)

The intention is that ulogd logs from all servers will be sent to kafaka as such it would seem to make senses to move ::profile::rsyslog::kafka_shipper to the ::standard class

Indeed this is something that has been discussed elsewhere from time to time and would be nice to do. I'd be game to move forward with that as part of this project.

In addition to that, we'll also want to parse these events into fields. Looking at the log format as-is we can accomplish that using a grok filter in logstash. There may be some patterns already existing for iptables format that we can use.

herron moved this task from Backlog to Up next on the Wikimedia-Logstash board.
jbond added a comment.Apr 15 2019, 6:44 PM

Its also worth pointing out that ulogd supports native json output[1] but only to a separate log file not syslog. Prometheus is was simpler to keep things raw and in (r)syslog which may be the case here as well. In relation to writing our own grok rules i suspect the mtail patterns will be a good starting point[2]

[1]https://phabricator.wikimedia.org/T116011#4927275
[2]https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/mtail/files/programs/ulogd.mtail

I'm for erring on the side of simplicity. Since these logs are useful on the command line of an individual host, on centrallog, and in Kibana, it makes sense to me to stream the ulogd syslogs formatted as shown in the description to logstash and parse them with a grok pattern.

jbond added a comment.Apr 17 2019, 3:50 PM

sounds good

Change 505737 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] kafka shipper: move kafka rsyslog shipping to base profile

https://gerrit.wikimedia.org/r/505737

Change 505750 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] kafka shipper: add ulogd to kafka forwarding rules

https://gerrit.wikimedia.org/r/505750

Change 505783 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] ulogd logstash: Add rule to parse ulogd ouput to json

https://gerrit.wikimedia.org/r/505783

Change 505817 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] kafka: It was pointed out that kafak shipping may not work for all hosts

https://gerrit.wikimedia.org/r/505817

Change 505817 abandoned by Jbond:
kafka: It was pointed out that kafak shipping may not work for all hosts

Reason:
replaced with 505737

https://gerrit.wikimedia.org/r/505817

Change 506377 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] nflog: add logging prefix to firewall log entries

https://gerrit.wikimedia.org/r/506377

Change 506377 merged by Jbond:
[operations/puppet@production] nflog: add logging prefix to firewall log entries

https://gerrit.wikimedia.org/r/506377

Change 506400 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] logstash: add ulog parser to logstash

https://gerrit.wikimedia.org/r/506400

Change 505783 abandoned by Jbond:
ulogd logstash: Add rule to parse ulogd ouput to json

Reason:
Replaced for 506400

https://gerrit.wikimedia.org/r/505783

Mentioned in SAL (#wikimedia-operations) [2019-04-30T14:26:33Z] <jbond42> disable-puppet "T220987: global kafaka log shipping - staged rollout (jbond)"

Change 505737 merged by Jbond:
[operations/puppet@production] kafka shipper: move kafka rsyslog shipping to base profile

https://gerrit.wikimedia.org/r/505737

Mentioned in SAL (#wikimedia-operations) [2019-04-30T14:58:42Z] <jbond42> enable-puppet "T220987: global kafaka log shipping - staged rollout (jbond)"

Change 505750 merged by Jbond:
[operations/puppet@production] kafka shipper: add ulogd to kafka forwarding rules

https://gerrit.wikimedia.org/r/505750

Change 506400 merged by Jbond:
[operations/puppet@production] logstash: add ulog parser to logstash

https://gerrit.wikimedia.org/r/506400

jbond closed this task as Resolved.May 14 2019, 9:13 AM

logs are now been sent to kafka, however we still need to role the profile::firewall::logging module to all infrastructure which is been tracked in T116011