Presently MXes have a 20G root filesystem only which is normally about half utilized.
Filesystem Size Used Avail Use% Mounted on /dev/vda1 19G 11G 7.2G 60% /
It was seen today that a significant spike in deferred mail caused the exim mainlog to become quite large (5+ GB) and nearly fill the disk.
Let's consider how best to add some space to these hosts, and also think about how to prevent large logs from filling the root filesystem which would interrupt mail flow.
We could add a second disk to the mx* VMs and move /var or to it, but this sounds rather something to factor it for the new VMs running the new setup? (The immediate log consumption from today's event can probably just be mitigated with a manual logrotate run?)
I rotated the log file and then compressed it on another host for this specific incident, but it was cumbersome. I think we should definitely embiggen the disks for the new Postfix based hosts. I am less sure if it is worth the effort for these hosts.
We've just had a repeat, and again exim mainlog is I think too big for logrotate to succeed with :-/
mvernon@mx1001:~$ ls -lsh /var/log/exim4/mainlog 3.9G -rw-r----- 1 Debian-exim adm 3.9G Apr 8 12:56 /var/log/exim4/mainlog mvernon@mx1001:~$ df -lh /var/log/exim4/ Filesystem Size Used Avail Use% Mounted on /dev/vda1 19G 15G 3.0G 84% /
Mentioned in SAL (#wikimedia-operations) [2022-11-18T18:21:40Z] <herron> removed older exim logs to free space T305567
Mentioned in SAL (#wikimedia-operations) [2022-12-07T23:24:22Z] <mutante> mx1001 about to run out of disk again - apt-get clean, gzip /var/log/exim4/mainlog.1 find -mtime +31 -delete in /var/log/exim4 - deleting old logs to prevent mail server running out of disk - it was alerting in Icinga but same as conf* - monitoring works, alerting does not T305567
I think the priority is surprisingly low for this being the main prod mail server and almost running out of disk multiple times.
@Dzahn I agree, sorry for letting this one slip out of my todo list, I'll take care of it tomorrow