diffscan now scans all ports on all external networks. This produces a lot of logs which we dont really care about as such we should drop the traffic before it hits the logging rule.
We could resolve this by adding a rule to iptables simlar to the one used to filter out dhcp/bootp traffic. however it would be better if we didn;t have a specific rule just for the diffscanners. As such its worth first exploring if this additional data is actually causing any issues. looking at idp-test it seems a scan generates about 5MB (18587 lines i think each line represents one logstash event ) of logs for each host scan. and there are currently about 400 active hosts which translates to an additional 7.5M events/day.
If this is to much for logstash then it may be better to update the mtail rule to filter out diffscan instead of explicitly dropping it in iptables.
If the figure of 7.5M logs/day just from ulog dropping expected internal scans is accurate then it does make a difference (about +10% overall log daily volume).
To clarify: I suggested filtering out drop traffic from expected source IPs to get a higher SNR in ulog logs, what would be the value in having expected drop traffic indexed/stored?
What are the downsides to using iptables rules for this?
Also, what are thoughts about creating a generalized iptables "skiplog" chain, and adding rules as needed to direct traffic (e.g. diffscan source addresses) to this chain before it hits the log rule?
Downside of having a special iptables rule for diffscan's IP is that it will not be treated anymore as a "random external host" and could cause false positive/negative.
Plus having to keep track of that source IP somewhere.