Page MenuHomePhabricator

Boron disk space alert
Closed, ResolvedPublic

Description

https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=boron&service=Disk+space

DISK WARNING - free space: / 27666 MB (9% inode=80%):

Making this task a persistent comment to the Icinga alert so whatever is done to solve it would be easy to re-do

Event Timeline

ayounsi triaged this task as High priority.Thu, Nov 7, 3:59 PM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Nov 7, 3:59 PM
jbond added a subscriber: jbond.
jbond moved this task from Unsorted 💣 to Friday tasks on the User-jbond board.Thu, Nov 7, 5:32 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-07T21:28:43Z] <mutante> boron apt-get clean (saved 9G on /) (T237649)

Dzahn added a comment.Thu, Nov 7, 9:30 PM

home dirs over 1 Gigabyte:

2.1G	elukey
2.2G	ema
8.9G	filippo
1.1G	fsero
2.1G	jiji
7.1G	jmm
4.2G	oblivian
11G	otto
1.4G	vgutierrez
Dzahn added subscribers: Ottomata, fgiunchedi.EditedThu, Nov 7, 9:31 PM

@Ottomata @fgiunchedi @Muehlenhoff Is there maybe something that is not needed anymore ^?

Removed a buncha stuff!

One big disk space hog is the fact that we don't expire old builds in /var/cache/pbuilder/result/*, there are builds which date back to 2016. There's certain value to keep the last, say six months of builds (e.g. for rollbacks), but anything else does not need to be retained. We could simply add a cron to purge old stuff, that should free ~ 40G or so.

MoritzMuehlenhoff closed this task as Resolved.Fri, Nov 8, 8:01 AM
MoritzMuehlenhoff claimed this task.

I removed a bunch of old data (e.g. trusty leftovers), we now have over 70G free diskspace again and I filed https://phabricator.wikimedia.org/T237713 for pruning old builds.

Removed a buncha stuff!

Ditto!