Page MenuHomePhabricator

MX: increasing disk space
Open, MediumPublic

Description

Presently MXes have a 20G root filesystem only which is normally about half utilized.

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        19G   11G  7.2G  60% /

It was seen today that a significant spike in deferred mail caused the exim mainlog to become quite large (5+ GB) and nearly fill the disk.

Let's consider how best to add some space to these hosts, and also think about how to prevent large logs from filling the root filesystem which would interrupt mail flow.

Event Timeline

herron renamed this task from MX: increasing disks space to MX: increasing disk space.Apr 6 2022, 4:47 PM

We could add a second disk to the mx* VMs and move /var or to it, but this sounds rather something to factor it for the new VMs running the new setup? (The immediate log consumption from today's event can probably just be mitigated with a manual logrotate run?)

I rotated the log file and then compressed it on another host for this specific incident, but it was cumbersome. I think we should definitely embiggen the disks for the new Postfix based hosts. I am less sure if it is worth the effort for these hosts.

We've just had a repeat, and again exim mainlog is I think too big for logrotate to succeed with :-/

mvernon@mx1001:~$ ls -lsh /var/log/exim4/mainlog
3.9G -rw-r----- 1 Debian-exim adm 3.9G Apr  8 12:56 /var/log/exim4/mainlog
mvernon@mx1001:~$ df -lh /var/log/exim4/
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        19G   15G  3.0G  84% /

ok, thanks, I'll rotate it manually and plan on embiggening the existing
hosts.

jhathaway triaged this task as Medium priority.