Page MenuHomePhabricator

tin.eqiad.wmnet / partition is full
Closed, ResolvedPublic

Description

[22:35:00] <icinga-wm> PROBLEM - Disk space on tin is CRITICAL: DISK CRITICAL - free space: / 582 MB (1% inode=78%)

tin.eqiad.wmnet$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/dm-0        37G   33G  2.1G  95% /

see also

Event Timeline

$ du -h /var/lib/l10nupdate/caches/
1.5G    /var/lib/l10nupdate/caches/cache-1.29.0-wmf.2
1.5G    /var/lib/l10nupdate/caches/cache-1.29.0-wmf.3
1.5G    /var/lib/l10nupdate/caches/cache-1.29.0-wmf.10
1.5G    /var/lib/l10nupdate/caches/cache-1.29.0-wmf.9
1.5G    /var/lib/l10nupdate/caches/cache-1.29.0-wmf.5
1.5G    /var/lib/l10nupdate/caches/cache-1.28.0-wmf.23
1.5G    /var/lib/l10nupdate/caches/cache-1.29.0-wmf.6
1.5G    /var/lib/l10nupdate/caches/cache-1.29.0-wmf.1
1.5G    /var/lib/l10nupdate/caches/cache-1.29.0-wmf.12
1.5G    /var/lib/l10nupdate/caches/cache-1.28.0-wmf.22
1.5G    /var/lib/l10nupdate/caches/cache-1.29.0-wmf.7
1.5G    /var/lib/l10nupdate/caches/cache-1.29.0-wmf.11
1.5G    /var/lib/l10nupdate/caches/cache-1.29.0-wmf.8
1.5G    /var/lib/l10nupdate/caches/cache-1.29.0-wmf.4
20G     /var/lib/l10nupdate/caches/

From IRC supposedly we had a cron job to garbage collect the old caches.

$ sudo -u l10nupdate -s crontab -l
0 2 * * * /usr/local/bin/l10nupdate-1 --verbose >> /var/log/l10nupdatelog/l10nupdate.log 2>&1

We also have full clones of all mediawiki repositories under /var/lib/l10nupdate/mediawiki which are rather dirty and could use remote prune / git gc :(

I ran "apt-get clean" on tin which freed another 2G or so

I killed the 1.28 l10nupdate cache folders, and the 1.29 ones < .10

Makes me think how mira is doing. (checked, it has plenty space)

Ottomata claimed this task.
Ottomata subscribed.

/ is no longer full, and there are other tickets to resolve the bigger problems. Resolving.