Page MenuHomePhabricator

atop (monitoring system) logs fill up instances /var/ partition
Closed, DeclinedPublic

Description

I noticed 'atop' generates log files in /var/log/ which is on a 2GB partition. atop has a logrotate rule of 15 days / uncompressed.

On deployment-bastion.eqiad.wmflabs that is 334MB currently, or a sixth of /var/ available disk space.

I guess we want to compress the old archives.


Version: unspecified
Severity: normal

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 3:33 AM
bzimport added a project: Cloud-VPS.
bzimport set Reference to bz69605.
bzimport added a subscriber: Unknown Object (MLST).

Example on deployment-bastion:

$ du --total -h /var/log/atop*
4.0K /var/log/atop
2.7M /var/log/atop.log
41M /var/log/atop.log.1
32M /var/log/atop.log.10
40M /var/log/atop.log.11
41M /var/log/atop.log.12
41M /var/log/atop.log.13
35M /var/log/atop.log.2
69M /var/log/atop.log.3
28M /var/log/atop.log.4
18M /var/log/atop.log.5
19M /var/log/atop.log.6
37M /var/log/atop.log.7
41M /var/log/atop.log.8
64M /var/log/atop.log.9
501M total

$df -h /var
Filesystem Size Used Avail Use% Mounted on
/dev/vda2 1.9G 1.5G 324M 83% /var

Change 176909 had a related patch set uploaded (by Yuvipanda):
base: Make atop log retention configurable

https://gerrit.wikimedia.org/r/176909

Patch-For-Review

Change 176909 abandoned by Yuvipanda:
base: Make atop log retention configurable

Reason:
This is too complicated. Newer labs instances should just have a unified / (coming sooooon)

https://gerrit.wikimedia.org/r/176909

The labs instance now have /var in the / partition that is 20GBytes. So that atop logs size is no more of an issue.