We could resolve this by adding a rule to iptables simlar to the one used to filter out dhcp/bootp traffic. however it would be better if we didn;t have a specific rule just for the diffscanners. As such its worth first exploring if this additional data is actually causing any issues. looking at idp-test it seems a scan generates about 5MB (18587 lines i think each line represents one logstash event ) of logs for each host scan. and there are currently about 400 active hosts which translates to an additional 7.5M events/day.

If this is to much for logstash then it may be better to update the mtail rule to filter out diffscan instead of explicitly dropping it in iptables.

I spoke with @fgiunchedi on irc and he pointed out that mtail is not the right bit to filter things out and suggested we filter the entries out in rsyslog before they get sent to logstash

The servers with a public IP already have lots of noise from random bots/portscans (e.g. on bast3004 40kish log per day), so this doesn't make too much of a difference.

If the figure of 7.5M logs/day just from ulog dropping expected internal scans is accurate then it does make a difference (about +10% overall log daily volume).

To clarify: I suggested filtering out drop traffic from expected source IPs to get a higher SNR in ulog logs, what would be the value in having expected drop traffic indexed/stored?

Shameless plug T238414.

Also my vote goes to not have a specific iptables rule for diffscan.

What are the downsides to using iptables rules for this?

Also, what are thoughts about creating a generalized iptables "skiplog" chain, and adding rules as needed to direct traffic (e.g. diffscan source addresses) to this chain before it hits the log rule?

Downside of having a special iptables rule for diffscan's IP is that it will not be treated anymore as a "random external host" and could cause false positive/negative.
Plus having to keep track of that source IP somewhere.