Page MenuHomePhabricator

dubnium disk full
Closed, ResolvedPublic


Disk full because of slapd logging:

-rw-r----- 1 root        adm    6.9G Oct  3 07:16 syslog
-rw-r----- 1 root        adm    6.9G Oct  3 07:16 debug

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2016-10-03T07:24:55Z] <volans> emptying /var/log/debug on dubnium because of disk full (the same data is on syslog) T147173

Some space was freed but looks like it will not last for very long, the logs are quite flooded and looks like most of the requests comes from mx1001.

On exim logs on mx1001 are flooded by routing defer (-51): retry time not reached for emails to root@.
Looking at a couple of messages from the logs (mainlog and mainlog.1) they were deferred because of The user you are trying to contact is receiving mail too quickly and according to the help page linked in the message it happens when those limits are reached.

The exim queue has 20757 messages right now of which 20745 are logging the "routing defer" message.

Volans removed a subscriber: SRE.

Mentioned in SAL (#wikimedia-operations) [2016-10-03T09:14:03Z] <akosiaris> T147173 clean exim queues on mx1001 from backscatter spam

Mentioned in SAL (#wikimedia-operations) [2016-10-03T09:32:35Z] <akosiaris> T147173 clean exim queues on mx1001 from backscatter spam. Seems to be originating from mx.{east,west}, blocked them for now

akosiaris changed the task status from Open to Stalled.Oct 3 2016, 9:40 AM
akosiaris triaged this task as Low priority.
akosiaris subscribed.

Setting to stalled for a while.. will have a look later in the day whether the backscatter continues

Backscatter has stopped, blocks removed for now. Resolving

Reopening, has happened again, the issue is once more due to mx1001's queues. Probably a backscatter spam attack again, investigating

Queues on mx1001 cleared to stop exim from querying dubnium too much

Mentioned in SAL (#wikimedia-operations) [2016-10-10T08:12:41Z] <akosiaris> clear mx1001's queues from backscatter spam T147173