Looks like a scraper

I've manually dropped the abuser: sudo nft insert rule inet base input ip saddr 93.123.109.83 drop it was a top hitter in the logs, but there might be more.

This is yet another reason to prioritize T286066: Put lists.wikimedia.org web interface behind LVS and T278495: Figure out plan for mailman IP situation

Puppet was disabled on lists1004 to prevent rolling back the nftables change.

I created and then merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148432

This added generic throttling (not based on allow/deny lists) on lists servers.

The default values apply, so port is 443 and 32 parallel connections are allowed or they get banned for 300 seconds.

This is now active on lists2001 (fail-over machine) but not yet on lists1004 (active machine) because puppet is disabled there still with @ABran-WMF 's message.

If there are no concerns we can enable puppet again and it will also be applied on lists1004.

Regarding additional throttling with actual lists of bad IPs/networks.. there are 2 more changes pending / to be discussed:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148826 by Arnaudb which makes throttling more generic and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148433/1 by myself which copied code over from Gerrit without making it generic.

In T394519#10846230, @Dzahn wrote:

I created and then merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148432

This added generic throttling (not based on allow/deny lists) on lists servers.

The default values apply, so port is 443 and 32 parallel connections are allowed or they get banned for 300 seconds.

Thanks for enabling throttling on the lists machine! The default policy is accept, so no throttling is taking place:

ip saddr @DENYLIST accept
ip6 saddr @DENYLIST_V6 accept

If there are no concerns we can enable puppet again and it will also be applied on lists1004.

Regarding additional throttling with actual lists of bad IPs/networks.. there are 2 more changes pending / to be discussed:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148826 by Arnaudb which makes throttling more generic and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148433/1 by myself which copied code over from Gerrit without making it generic.

We will enable generic throttling and add the single IP to the abusers list for lists host. Arnaud is preparing a patch for that.

puppet has been reenabled, the patch has been applied on lists1004, everything seems to be running OK

Change #1148433 abandoned by Dzahn:

[operations/puppet@production] lists: add parameter and code to block abusers using nftables

Reason:

another solution that uses the same list of abusers for multiple services by Arnaudb has been implemented.

https://gerrit.wikimedia.org/r/1148433