Page MenuHomePhabricator

Boron disk space alert
Closed, ResolvedPublic

Description

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=boron&service=Disk+space

DISK WARNING - free space: / 27666 MB (9% inode=80%):

Making this task a persistent comment to the Icinga alert so whatever is done to solve it would be easy to re-do

Event Timeline

ayounsi triaged this task as High priority.

Mentioned in SAL (#wikimedia-operations) [2019-11-07T21:28:43Z] <mutante> boron apt-get clean (saved 9G on /) (T237649)

home dirs over 1 Gigabyte:

2.1G	elukey
2.2G	ema
8.9G	filippo
1.1G	fsero
2.1G	jiji
7.1G	jmm
4.2G	oblivian
11G	otto
1.4G	vgutierrez

@Ottomata @fgiunchedi @Muehlenhoff Is there maybe something that is not needed anymore ^?

One big disk space hog is the fact that we don't expire old builds in /var/cache/pbuilder/result/*, there are builds which date back to 2016. There's certain value to keep the last, say six months of builds (e.g. for rollbacks), but anything else does not need to be retained. We could simply add a cron to purge old stuff, that should free ~ 40G or so.

MoritzMuehlenhoff claimed this task.

I removed a bunch of old data (e.g. trusty leftovers), we now have over 70G free diskspace again and I filed https://phabricator.wikimedia.org/T237713 for pruning old builds.