https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=boron&service=Disk+space
DISK WARNING - free space: / 27666 MB (9% inode=80%):
Making this task a persistent comment to the Icinga alert so whatever is done to solve it would be easy to re-do
Mentioned in SAL (#wikimedia-operations) [2019-11-07T21:28:43Z] <mutante> boron apt-get clean (saved 9G on /) (T237649)
home dirs over 1 Gigabyte:
2.1G elukey 2.2G ema 8.9G filippo 1.1G fsero 2.1G jiji 7.1G jmm 4.2G oblivian 11G otto 1.4G vgutierrez
@Ottomata @fgiunchedi @Muehlenhoff Is there maybe something that is not needed anymore ^?
One big disk space hog is the fact that we don't expire old builds in /var/cache/pbuilder/result/*, there are builds which date back to 2016. There's certain value to keep the last, say six months of builds (e.g. for rollbacks), but anything else does not need to be retained. We could simply add a cron to purge old stuff, that should free ~ 40G or so.
I removed a bunch of old data (e.g. trusty leftovers), we now have over 70G free diskspace again and I filed https://phabricator.wikimedia.org/T237713 for pruning old builds.