Disk full because of slapd logging:
-rw-r----- 1 root adm 6.9G Oct 3 07:16 syslog -rw-r----- 1 root adm 6.9G Oct 3 07:16 debug
Description
Description
Event Timeline
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2016-10-03T07:24:55Z] <volans> emptying /var/log/debug on dubnium because of disk full (the same data is on syslog) T147173
Comment Actions
Some space was freed but looks like it will not last for very long, the logs are quite flooded and looks like most of the requests comes from mx1001.
On exim logs on mx1001 are flooded by routing defer (-51): retry time not reached for emails to root@.
Looking at a couple of messages from the logs (mainlog and mainlog.1) they were deferred because of The user you are trying to contact is receiving mail too quickly and according to the help page linked in the message it happens when those limits are reached.
The exim queue has 20757 messages right now of which 20745 are logging the "routing defer" message.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2016-10-03T09:14:03Z] <akosiaris> T147173 clean exim queues on mx1001 from backscatter spam
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2016-10-03T09:32:35Z] <akosiaris> T147173 clean exim queues on mx1001 from backscatter spam. Seems to be originating from mx.{east,west}.cox.net, blocked them for now
Comment Actions
Setting to stalled for a while.. will have a look later in the day whether the backscatter continues
Comment Actions
Reopening, has happened again, the issue is once more due to mx1001's queues. Probably a backscatter spam attack again, investigating
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2016-10-10T08:12:41Z] <akosiaris> clear mx1001's queues from backscatter spam T147173