Page MenuHomePhabricator

deployment-bastion.eqiad.wmflabs `/var/` keeps filling up
Closed, ResolvedPublic

Description

The /var (size: 1.9GB) partition on deployment-bastion fills to 95% or so pretty much daily.

The largest factor in this is log files:

thcipriani@deployment-bastion:log
❯ sudo du -chs /var/log/* | sort -rh | head
813M    total
226M    /var/log/account
45M     /var/log/atop.log.4
45M     /var/log/atop.log.1
44M     /var/log/atop.log.9
44M     /var/log/atop.log.11
43M     /var/log/atop.log.7
39M     /var/log/atop.log.2
38M     /var/log/atop.log.6
34M     /var/log/atop.log.8

Account is currently handled by /etc/cron.daily/acct which uses savelog, which doesn't gzip the most recent logfiles in case something is writing to it. Gziping that one log file (/var/log/account/pacct.0) is generally enough to keep deployment-bastion happy, but there are probably other things that can be done here.

Event Timeline

thcipriani raised the priority of this task from to Needs Triage.
thcipriani updated the task description. (Show Details)
thcipriani added subscribers: thcipriani, hashar, greg.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2015, 3:58 PM

Process accounting and /var being too small got filled as T91354: Process accounting routinely fill up /var on deployment-bastion already.

Yuvi provided new labs images that have a unified and larger root partition. So we will need to recreate deployment-bastion (apparently we can't resize).

thcipriani closed this task as Resolved.May 12 2015, 9:05 PM
thcipriani claimed this task.

Closing as duplicate of T91354