[22:35:00] <icinga-wm> PROBLEM - Disk space on tin is CRITICAL: DISK CRITICAL - free space: / 582 MB (1% inode=78%)
tin.eqiad.wmnet$ df -h / Filesystem Size Used Avail Use% Mounted on /dev/dm-0 37G 33G 2.1G 95% /
[22:35:00] <icinga-wm> PROBLEM - Disk space on tin is CRITICAL: DISK CRITICAL - free space: / 582 MB (1% inode=78%)
tin.eqiad.wmnet$ df -h / Filesystem Size Used Avail Use% Mounted on /dev/dm-0 37G 33G 2.1G 95% /
$ du -h /var/lib/l10nupdate/caches/ 1.5G /var/lib/l10nupdate/caches/cache-1.29.0-wmf.2 1.5G /var/lib/l10nupdate/caches/cache-1.29.0-wmf.3 1.5G /var/lib/l10nupdate/caches/cache-1.29.0-wmf.10 1.5G /var/lib/l10nupdate/caches/cache-1.29.0-wmf.9 1.5G /var/lib/l10nupdate/caches/cache-1.29.0-wmf.5 1.5G /var/lib/l10nupdate/caches/cache-1.28.0-wmf.23 1.5G /var/lib/l10nupdate/caches/cache-1.29.0-wmf.6 1.5G /var/lib/l10nupdate/caches/cache-1.29.0-wmf.1 1.5G /var/lib/l10nupdate/caches/cache-1.29.0-wmf.12 1.5G /var/lib/l10nupdate/caches/cache-1.28.0-wmf.22 1.5G /var/lib/l10nupdate/caches/cache-1.29.0-wmf.7 1.5G /var/lib/l10nupdate/caches/cache-1.29.0-wmf.11 1.5G /var/lib/l10nupdate/caches/cache-1.29.0-wmf.8 1.5G /var/lib/l10nupdate/caches/cache-1.29.0-wmf.4 20G /var/lib/l10nupdate/caches/
From IRC supposedly we had a cron job to garbage collect the old caches.
$ sudo -u l10nupdate -s crontab -l 0 2 * * * /usr/local/bin/l10nupdate-1 --verbose >> /var/log/l10nupdatelog/l10nupdate.log 2>&1
We also have full clones of all mediawiki repositories under /var/lib/l10nupdate/mediawiki which are rather dirty and could use remote prune / git gc :(
/ is no longer full, and there are other tickets to resolve the bigger problems. Resolving.